Multiple concurrent upload/download threads

Feature
,
#1

Is it currently possible to use multiple upload and/or download threads? The CLI help only lists a -threads option while backing up, and only for Backblaze.

If not currently supported, is this planned at all for other providers in the future? Transferring a single chunk at a time seems to be a limiting factor for faster connection speeds. Being able to make use of maybe 2-3 simultaneous transfer threads would speed things up greatly. I know I can increase chunk size, though from what I understand this will lower the effectiveness of deduplication.

#2

Currently only uploading to Backblaze allows multiple threads. We’ll add multi-threaded uploading to other storages if there is enough user demand. What is the storage backend you’re using?

#3

Google Drive.

I find using the Google Drive API in other products there is generally a 2s delay before a download begins - single threaded speeds are generally fine for individual large files in this case, but when there are a number of smaller files, such as Duplicacy’s chunks, multiple threads can make quite a difference - especially noticeable on a faster connection.

This is more of an issue with restores, as Google Drive’s response times are a lot lower for uploads, although backups would also benefit from multiple threads.

#4

2 posts were merged into an existing topic: Backblaze B2 optimum number of threads and chunk size?

#5

Just to clarify, parallel download/upload have been implemented for all storage backends since version 1.2 which was more than a year ago.

4 Likes
#6