Web Edition (0.2.1, Linux Arm32) Slow Speeds On Wasabi

Support
, ,
#1

I am using the web edition of Duplicacy with Linux Arm 0.2.1. My storage is set up as Wasabi and appears to work correctly. The Backup and Schedule also seem to be accurately configured in the web edition ui.

However, when backing up data, the transfer rate eventually drops from 10 MB/s to around 300-400 KB/s after about 2-3 hours.

This is on a residential line and my upload capacity is 10 MB/s. I have contacted the ISP and they claim to not be throttling this traffic; to my knowledge, Wasabi also does not place a throttle on uploads.

How can I get or keep Duplicacy transferring at 10MB/s and prevent it from dropping to 300-400 KB/s?

#2

What kind of files are you backing up (e.g., a lot of small files, VM images, etc)?

How many threads are you using for the backup? The default is 1.

#3

I am just backing up general files. They include Word documents, PowerPoint files, some music and video, and other misc. AV. There’s about 11,000 files, about 250 GB.

I have not changed or updated anything related to threads. So, it is just using defaults.
Also, Duplicacy data is housed on a spindle disk. IIRC, there were posts about when Duplicacy runs from a spindle disk to keep threads at 1. In any case, I have not updated or changed anything regarding threads, so it’s at default.

#4

I’m pretty sure any such post was in respect of backup storage located on HDD, not the repository itself. In your case, it’s Wasabi and you’ll very likely benefit from multiple threads.

In your backup schedule, click the dash (-) under options and put -threads 4 as a starting point.

2 Likes
#5

I think there’s a missunderstanding here (but please link those posts, so we can clear them):

  • duplicacy uses 1 single thread at all times to read from the source/repository disk
  • duplicacy can use multiple threads for uploading to the storage
    • if you are backing up to a “self hosted” storage which resides on a single HDD (or multiple HDD in raid 1 – mirror) then you should probably not use more than 2 threads since you may kill the performance on the storage (the destination)
    • if you are backing up to a cloud provider then multiple threads is the way to go – the more, the merrier – just pay attention when they throttle (rate limit) you so that you know what’s the maximum number of threads you should use.

Again: please tell me where you saw info about using a single thread for backup, coz the info there could be wrong and i’d like to fix it.

Self note: this response should be added to the wiki

4 Likes
#6

If this is the thread you’re referencing, then I think @Droolio is right. It’s about the backup storage being local HDDs in a RAID 5 over USB (!) – so not representative of Wasabi or other cloud storage providers.

#8

Thanks for comments. I’ll have to dig up the post that highlighted the storage on regular HDDs as best suited with not more than 1 thread. A quick search didn’t find it. For the meantime, I’ll add the “-threads 4” to the backup options. Hopefully that’ll help.

Edit: After 18 hours, and with 4 threads, backup speed has dropped from 10 MB/s to 3.2 MB/s. 3.2 MB/s is better than ~300 KB/s I was previously getting. Still, it’s not as fast as when it starts and I’m not sure why the rate drops over time.

Are there other means to keep upload speed from dropping ?

Edit2: Yes, the thread that @leerspace references is the one I was thinking about. There is a particular quote in there from gchen which says:

gchen Dec 25 9:30PM 2017
Yes, that makes total sense. For spinning disks we should always stick with -threads 1 .

This was where it was my understanding that I needed 1 thread. Prior commenters on this thread have clarified that, if you are uploading from a local HDD, threads do not need to be 1.

But I still continue to experience substantially decrease in upload speeds, to Wasabi, over time…

1 Like