Restore Timeouts with Azure or Onedrive

Support
#1

I am hitting this same issue (using Duplicacy WebEdition 1.4.1).

@spruit / @jani - when doing “Restore” operation from Azure, what download speeds are you seeing?

  • I was seeing horrendously slow downloads (~1.6MBps) and then tried running the restore with “-threads 10” which increased the speed to (~3.5MBps) and after a few hours I started hitting connection reset issue :frowning:

2020-11-10 00:43:15.709 ERROR DOWNLOAD_CHUNK Failed to download the chunk dfd5ae9a1f414d2b5c352e6125c22a7e4c12b0c21b382ac9bfff324474b8949d: read tcp 192.168.86.45:62898->13.107.42.12:443: read: connection reset by peer

*Just to check how it fares, I tried using OneDrive as well from WebEdition. While I got upload speeds of ~17MBps when doing a “Copy” operation, and then to validate the backup I tried a “Restore” operation and I was seeing ~8 MBps and again after a few hours I hit the following error

The restore command encountered an error:
Failed to download the chunk ca49706b411489777a076c202afdc227bac26132a833e07291e351eb3589f524: failed to refresh the access token: 400 Unexpected response

My backup is ~515GB and based on the comments on forum, looks like many folks do backups regularly of this size - so I was hoping this is not too much to expect - not sure what am i doing wrong here?

So far I have been using local backups and Duplicacy has been great - but after seeing 5TB HDD end up with bad sectors, I was really hoping to backup to the cloud, but so far was only able to do a “Copy” and have NOT been able to do any cloud based “Restore” successfully yet :roll_eyes: - every time I am getting some error or other and the restore operation fails.

I am not sure how it is working for some folks and others are facing OneDrive timeouts

CC: @gchen, @TheBestPessimist, @towerbr

Restore Timeouts with Azure
#2

First I would suggest a thread number of 2 or 4. With 10 threads you only double the speed which means 10 is too many.

This read timeout error could be a result of throttling. If so, reducing the number of threads will help.

This looks like an authorization error. Please make sure that the token file is valid and you didn’t revoke the token while the program is running.

#3

While in WebUI I used 10 threads to see if the upload speed goes up, after reading @Droolio’s suggestion of "Might be counter-intuitive but have you tried specifying -threads 4 or something small at first?" in Restore Timeouts with Azure, when using CLI runs, I used “-threads 4” option - but still ran into the failures. I will try it out with 2 threads and see.

About auth issues - the tokens/keys surely were NOT revoked while running. I have been tinkering with various options to see what works and across the runs I am using the same access keys / token and the runs always starts fine (so tokens are fine) but eventually run into failures.

#4

@gchen - since I reduced the threads to 2, the downloads from Azure was able to complete successfully without hitting errors - but the download speeds are very slow (~7MBps) :frowning:

As per Azure documentation (I also confirmed it with a Azure Storage support engineer) that Azure should be able to easily support up to 60MBps unless there are other bottlenecks. Not sure why I am seeing such low speeds, even though my internet speed tests are consistently showing 120+ MBps.

#5

Can you run the CLI benchmark command with different threads numbers to determine the maximum speed?