Upload Speed Gets Progressively Worse

Support
#1

Hi there.
I love duplicacy but sadly have one issue, my duplicacy upload speed when using it gets worse as the upload progresses, or more accurately, the more files already backed up as this persists in further runs.

I am uploading to a GSuite account and using the following command: duplicacy backup -stats -vss -threads 8

The repository is about 4TB in size.

I noticed high memory usage, then I read somewhere about a month ago that the ‘DUPLICACY_ATTRIBUTE_THRESHOLD’ environment variable being set to ‘1’ helps, so I’ve done this and now only 8/16GB of RAM is used total on the machine (previously Duplicacy would use all available RAM) but see no improvement.

The speed I had at the start of the upload was maxing out the connection, but now I am getting is about 2 megabits per second towards the end of the upload, and stays that slow on subsequent runs even now when the initial upload has completed. This resulted in the initial upload taking a month or two to complete. A speedtest on the same computer returns a stable 35 Megabits upload speed.

I am running Windows Server 2019 with the latest updates installed, with duplicacy command line 2.7.1. I recently upgraded from 2.6.2 which had the same issue.

I feel like there’s an obvious mistake I’m making or a switch somewhere I need to toggle, any ideas what could be causing the speed issues?

#2

Can you add the -d option to see if you’re hitting the rate limit?

duplicacy -d backup -stats -vss -threads 8
#3

Just tried this while resuming an upload that disconnected last night, here’s the output:

Skipped chunk 8693aeb0f80540021d880c9e2a995ed276f9a666fe5cdfe5c47785d107e25edc in cache
Skipped chunk 18 size 2449286, 9.69MB/s 00:02:33 3.1%
Chunk 7a935c96df6e02cd426130f47d562cb81cf680145b878e5958a93fb0dac4ef7b already exists
Skipped chunk 7a935c96df6e02cd426130f47d562cb81cf680145b878e5958a93fb0dac4ef7b in cache
Skipped chunk 17 size 1751946, 10.02MB/s 00:02:28 3.2%
[1] HTTP status code 502; retrying after 1.10 seconds (backoff: 2, attempts: 1)
Chunk 81f229f41bcfea6b5b3b35e75b28b0063a2ff913799104134181e0d1f6c14265 has been uploaded
Uploaded chunk 22 size 4813839, 394KB/s 01:03:45 3.5%
Chunk b6fd7e8134c5c5e271f1ddfaf792352d65457272876508a6a0e77f468046bf57 has been uploaded
Uploaded chunk 16 size 5278232, 425KB/s 00:59:00 3.9%
Chunk 33285ff64c7986b33bc038665ee37d1e0ada0c8c816b3bad5dd88448a274a3b5 has been uploaded
Uploaded chunk 21 size 4964701, 452KB/s 00:55:15 4.2%
Chunk ce7c2983d8b93ad5f2f0ca1fdec1ca9fa24acb110ab66d833da0a6a79c0669bb has been uploaded
Uploaded chunk 23 size 1472943, 363KB/s 01:08:48 4.3%
Chunk e13ea220e92884277ddb3b0f49093ee4a039dee883fc9d5d73cbc5e86f2da70f has been uploaded
Uploaded chunk 25 size 1883133, 345KB/s 01:12:18 4.4%
Chunk d4cc9e77c9d5950d4026330de8061740998389b02019310ed5c2c5065f21a79d has been uploaded
Uploaded chunk 24 size 2025157, 348KB/s 01:11:35 4.5%
Chunk 366481adb111f59181feb5faed8cfb4c5f505ff1ae9378b916b8f01281d158a1 has been uploaded
Uploaded chunk 19 size 8497690, 339KB/s 01:13:06 5.0%
[3] HTTP status code 502; retrying after 2.99 seconds (backoff: 2, attempts: 1)
Chunk 7173b78bfd73e572a104075dd5e7204a082f9deb529a09d64b5ad0e69108c0f5 has been uploaded
Uploaded chunk 15 size 10541500, 315KB/s 01:17:57 5.7%
Chunk bc3ccb76c41ae02df3bcd30cb3e76e106d61473654f96859c8bd6b850b2a25b4 has been uploaded
Uploaded chunk 29 size 3517589, 278KB/s 01:28:03 5.9%
Chunk a21ac687abd613dead6c6ede9d878f7367b561903e4aba4c4127aa7cd61ee7ec has been uploaded
Uploaded chunk 20 size 12530828, 309KB/s 01:18:34 6.7%

The 502’s are unusual…
Is there another part to look at to identify if it’s rate limiting that I haven’t included?

#4

@gchen Is there any other information I should provide?

#5

I don’t know why Google returned 502 but that seems to be an temporary error. Do you still have this problem now? You can run the CLI command duplicacy benchmark to test the upload speeds.

#6

Still have the speed issue. Issue has been ongoing for a couple of months or so.

C:\Users\Administrator\AppData\Roaming\Duplicacy\dDriveMisc>duplicacy benchmark
Repository set to D:\
Storage set to gcd://duplicacy/WinServer
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 1.82s: 140.75M/s
Reading the random data from local disk
Read 256.00M bytes in 0.07s: 3554.21M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.81s: 141.51M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 2.60s: 98.31M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 2.58s: 99.07M/s
Generating 64 chunks
Uploaded 256.00M bytes in 1318.67s: 199K/s
Downloaded 256.00M bytes in 1498.18s: 175K/s
Deleted 64 temporary files from the storage

Did a speedtest on the same PC after the benchmark, 87 Megabits down, 36 Megabits up.

#7

Seems there’s something iffy with your connection to the internet. Doesn’t look to be an issue with Duplicacy, but just so you know, the benchmark command has two options - -upload-threads and -download-threads - that you might want to try. Your single thread speeds look terrible, though.

2 Likes
#8

duplicacy -d benchmark may reveal if Google Drive is rate limiting or returning errors.

#9

No rate limits or errors by the looks of it. Although I did another about half an hour later, and now I’m just confused. Outputs of both below. Any ideas? The internet connection is reliable and stable, as evidenced by speed tests always being around 90/35 Megabits with no packet loss, regardless of Duplicacy performance. The cabling is also only about a year old.

First run:

C:\Users\Administrator\AppData\Roaming\Duplicacy\dDriveMisc>duplicacy -d benchmark
Repository set to D:\
Storage set to gcd://duplicacy/WinServer
Reading the environment variable DUPLICACY_GCD_TOKEN
Reading gcd_token from keychain/keyring
Reading the environment variable DUPLICACY_GCD_TOKEN
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 1.79s: 142.67M/s
Reading the random data from local disk
Read 256.00M bytes in 0.07s: 3606.44M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.79s: 142.86M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 2.46s: 104.02M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 2.45s: 104.32M/s
Generating 64 chunks
Uploaded 256.00M bytes in 1327.39s: 197K/s
Downloaded 256.00M bytes in 1514.33s: 173K/s
Deleted 64 temporary files from the storage

Second run:

C:\Users\Administrator\AppData\Roaming\Duplicacy\dDriveMisc>duplicacy -d benchmark
Repository set to D:\
Storage set to gcd://duplicacy/WinServer
Reading the environment variable DUPLICACY_GCD_TOKEN
Reading gcd_token from keychain/keyring
Reading the environment variable DUPLICACY_GCD_TOKEN
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 1.84s: 138.84M/s
Reading the random data from local disk
Read 256.00M bytes in 0.07s: 3506.93M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.97s: 130.21M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 2.80s: 91.39M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 2.80s: 91.49M/s
Generating 64 chunks
Uploaded 256.00M bytes in 165.73s: 1.54M/s
Downloaded 256.00M bytes in 204.21s: 1.25M/s
Deleted 64 temporary files from the storage
#10

I just performed a single thread speed test on speedtest.net and while the speed was significantly less than multi-threaded, at 40/35 it still doesn’t explain Duplicacy being so much slower. Immediately after the speedtest, I ran a benchmark in Duplicacy and got ‘Uploaded 256.00M bytes in 962.14s: 272K/s’

I know it’s easy to pass this off by saying I have connection issues but it is only Duplicacy that has slow speeds, every other program performs without any issues, be that a web browser, a media server, P2P program or online games.

#11

Worth noting this was temporary and the speeds have returned to being extremely slow. Seems it ran properly as a once off.

#12

I don’t think this is necessarily a representative test. Speedtest.net tries to choose an optimal server for the speed test by default, sometimes even a server hosted by your ISP; but that tells you almost nothing about the connection to whatever Google server you’re talking to. All the speed test really tells you is maximum performance you can expect from your connection.

For what it’s worth, I could max out my 300mbps symmetric connection using Duplicacy with Google Drive back when I used Google Drive (as recently as a month ago). So I’d be surprised if the issue is caused by Duplicacy, rather than Duplicacy shedding light on an underlying issue.

If you wanted to try to do more troubleshooting to narrow down the cause, it might be valuable to also do a test with rclone and Google Drive. If rclone is also slow when uploading to Google Drive, that would suggest the issue is specific to Google Drive – whether it’s Google throttling, or some network-related issue between your computer and Google’s servers (e.g. insufficient bandwidth between your ISP and Google, throttling from your ISP possibly targeted at YouTube or some other Google service, packet loss, etc).

edit: if rclone is fast while Duplicacy is slow, there are some things that are also different between these two applications – like different application-specific quotas from Google – that could be difficult to rule out. But it would still be an interesting data point.

#13

Using rclone, I’ve never noticed any significant speed issues, but I’ve just given it a go and noticed both similarities and differences.

With default settings and a supplied Client ID and Secret, uploading was around 300 Kilobytes per second, and downloading hovers at around 1 Megabyte per second. Still awful speeds, but higher than duplicacy gets, especially for download. Tried creating and sharing a team drive and using it on rClone with another count and got similar speeds.

Throttling by the ISP is unlikely, the ISP I use is well regarded for being one of the few that doesn’t throttle, and is usually pretty good with this stuff. Got changed off a CG-NAT and had all standard 80, 443 etc port blocks removed within 5 mins.

Does Google implement any throttling or restrictions outside of the known caps on 750GB up and 10TB down in a 24 hour period?

Edit: Just to be certain, I ran the TorGuard VPN client while using rclone and duplicacy, both had the same speeds with it running as without it running. That should eliminate ISP throttling.

#14

Okay, this is getting weird now.

rclone gets similar speeds to Duplicacy on the same machine, but if I run rclone on a different machine on the same network, connected to the same Google account, rclone fully saturates the upload. I imagine Duplicacy would be the same on this machine.

Speedtests run on both PCs return similar results, and the speed test servers are run by different ISPs. Tried different speedtest sites and using the CLI version, but all get similar results. Downloading and uploading files over the network and internet work fine, as the ‘slow’ PC is used as a media server as well.

That means the slow connection is on one specific computer on the network and only affects Google drive. I suppose the next steps are to see if a VM running on the same PC suffers the same speed issues, or if changing networks makes a difference.

#15

I’m not sure if anyone besides Google knows under what conditions they apply throttling besides the daily caps and app-specific quotas. As of October this year, the limit for Duplicacy appears to be 2000 queries per 100 seconds (source). I’m not sure what the per-app quota is for Duplicacy users in total, if any.

Really weird. I’m not sure I have too much to contribute at this point without looking at packet captures or something potentially invasive like that. But sounds like you’re getting closer to figuring out this mystery. I’m not really sure off the top of my head what could account for this behavior.

It’d probably be worth double checking that Duplicacy also is faster on the different machine like you suspect, just to make sure that the rclone result was not a red-herring.

#16

I just set up a VM using the same network interface to see how fast it runs with rclone.

It uploaded a file at 300-600 Kilobytes per second, varying greatly. For some reason the speedtest were also slow for upload. I decided to upload the same file at the same time on the host, the machine with the ‘slow’ connection.

3.5 Megabytes per second. A normal upload speed. So I decided to re-run the duplicacy benchmark on the host. Stable 1.64 Megabytes per second. Less than half of rclone, but significantly better than normal. Download speed was only 1.2 though.

Tried a duplicacy benchmark on the other PC that has no issues and got 1.58 Megabytes per second upload. Looks like rclone may have been a red herring as you suggested, as while 1.6 Megabytes is better than earlier, it’s very slow, especially when the same PC on the same Gsuite gets 3.9 Megabytes per second upload via rclone.

I suppose once Arq 7 launches, I’ll give it a go and see if it suffers from the same speed issues.

Edit: Arq 6 has upload speed of 3-4 Megabytes per second on the same test file to the same GSuite Google Drive. I’m not going to actively look for a solution to this anymore, knowing that Arq 7 is coming out soon and will likely work just as well.

#17

If you could consistently get 1.6 M/s per thread to GSuite, you could theoretically get more than 200 mbps up with 16 upload threads. One thing I do remember about GSUite is that I had to use a lot of threads when compared to a non-consumer product like B2.

Edit: oops, missed your most recent edit.

#18

I decided as one last shot, to run the backup script again with the same command used in my initial post. duplicacy backup -stats -vss -threads 8

For some reason it now has consistent 4.25 Megabytes per second upload and has for about 12 hours.

I guess I’ll stop questioning the magic that is Gsuite and just hope it works reliably.