Very slow webdav restore speeds - High latency connection

Support
#1

I’m in New Zealand and I’ve been backing up to Hetzner StorageBox in Germany. Backup speeds were fantastic, with 6-8 threads I was getting 20MB/s. Over 2TB was done within 48 hours.

But testing a restore now, with 12 threads, I’m only able to get 100-200KB/s restoring, this would take me over 200 days to restore my data. I’ve tried the restore with more threads, like 20 and even 50, and it helps a little at 1MB/s but still a very slow speed.

Is there anything that can be done? My latency to Hetzner is about 280ms.

Running in Docker on unRAID.

16GB DDR3
Intel i7 4770

Here’s a restore log with 50 threads for example:

2024-05-06 13:33:24.047 INFO REPOSITORY_SET Repository set to /data/appdata/test
2024-05-06 13:33:24.047 INFO STORAGE_SET Storage set to webdav://HIDDEN@HIDDEN.your-storagebox.de/duplicacy
2024-05-06 13:33:26.096 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-05-06 13:33:26.992 INFO RESTORE_INPLACE Forcing in-place mode with a non-default preference path
2024-05-06 13:33:26.992 INFO RESTORE_INDEXING Indexing /data/appdata/test
2024-05-06 13:33:26.992 INFO SNAPSHOT_FILTER Parsing filter file /cache/localhost/restore/.duplicacy/filters
2024-05-06 13:33:26.992 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2024-05-06 13:33:27.565 INFO RESTORE_START Restoring /data/appdata/test to revision 2
2024-05-06 13:33:35.583 INFO DOWNLOAD_PROGRESS Downloaded chunk 44 size 1122677, 122KB/s 03:45:24 0.0%
2024-05-06 13:33:45.640 INFO DOWNLOAD_PROGRESS Downloaded chunk 7 size 1255826, 122KB/s 03:44:26 0.1%
2024-05-06 13:33:47.807 INFO DOWNLOAD_PROGRESS Downloaded chunk 8 size 1200791, 166KB/s 02:44:44 0.2%
2024-05-06 13:33:48.162 INFO DOWNLOAD_PROGRESS Downloaded chunk 12 size 1098239, 208KB/s 02:11:58 0.2%
2024-05-06 13:33:48.508 INFO DOWNLOAD_PROGRESS Downloaded chunk 5 size 1235529, 262KB/s 01:44:19 0.3%
2024-05-06 13:33:49.938 INFO DOWNLOAD_PROGRESS Downloaded chunk 22 size 1219062, 303KB/s 01:30:21 0.4%
2024-05-06 13:33:53.825 INFO DOWNLOAD_PROGRESS Downloaded chunk 17 size 1572980, 315KB/s 01:26:49 0.5%
2024-05-06 13:33:54.095 INFO DOWNLOAD_PROGRESS Downloaded chunk 10 size 2253778, 382KB/s 01:11:26 0.6%
2024-05-06 13:33:54.100 INFO DOWNLOAD_PROGRESS Downloaded chunk 6 size 1506690, 435KB/s 01:02:44 0.7%
2024-05-06 13:33:54.194 INFO DOWNLOAD_PROGRESS Downloaded chunk 43 size 2774826, 532KB/s 00:51:14 0.9%
2024-05-06 13:33:55.158 INFO DOWNLOAD_PROGRESS Downloaded chunk 41 size 1293750, 557KB/s 00:48:52 0.9%
2024-05-06 13:33:56.492 INFO DOWNLOAD_PROGRESS Downloaded chunk 32 size 1502912, 587KB/s 00:46:18 1.0%
2024-05-06 13:33:58.769 INFO DOWNLOAD_PROGRESS Downloaded chunk 13 size 1949940, 610KB/s 00:44:31 1.1%
2024-05-06 13:34:01.632 INFO DOWNLOAD_PROGRESS Downloaded chunk 11 size 1854421, 609KB/s 00:44:31 1.2%
2024-05-06 13:34:03.808 INFO DOWNLOAD_PROGRESS Downloaded chunk 39 size 1919762, 627KB/s 00:43:12 1.4%
2024-05-06 13:34:05.808 INFO DOWNLOAD_PROGRESS Downloaded chunk 24 size 2366367, 654KB/s 00:41:21 1.5%
2024-05-06 13:34:06.656 INFO DOWNLOAD_PROGRESS Downloaded chunk 26 size 2284501, 694KB/s 00:38:57 1.6%
2024-05-06 13:34:18.289 INFO DOWNLOAD_PROGRESS Downloaded chunk 40 size 2694186, 584KB/s 00:46:10 1.8%
2024-05-06 13:34:20.884 INFO DOWNLOAD_PROGRESS Downloaded chunk 37 size 3252742, 621KB/s 00:43:20 2.0%
2024-05-06 13:34:21.088 INFO DOWNLOAD_PROGRESS Downloaded chunk 27 size 3187808, 667KB/s 00:40:18 2.2%
2024-05-06 13:34:21.157 INFO DOWNLOAD_PROGRESS Downloaded chunk 9 size 3073585, 721KB/s 00:37:11 2.4%
2024-05-06 13:34:21.917 INFO DOWNLOAD_PROGRESS Downloaded chunk 31 size 2860898, 772KB/s 00:34:41 2.5%
2024-05-06 13:34:25.188 INFO DOWNLOAD_PROGRESS Downloaded chunk 23 size 3500480, 778KB/s 00:34:21 2.7%
#2

Do you have access to other protocols? Can you try restoring via SFTP? To rule out web server issues from connectivity. WebDAV is not really design for this type workload.

Perhaps measuring speed of downloading a large file will be useful too — to distinguish throughput from latency. There can be there are some weird peering issues. I would look at MTR from both sides.

#3

Using SFTP gives me even slower speeds. The protocols I have are:
FTP
SFTP
WebDAV
Samba
SSH (rsync and BorgBackup)

Here is an MTR from my house to the server, I cannot do a test the other way as StorageBox does not provide an SSH shell.

image
image763×386 15.2 KB

#4

That’s… quite a journey!

Can you try SFTP uploading and downloading a single large file.

If that is also slow — not much you will be able to do.

#5

Yes, a very long trip indeed. I believe this is due to the submarine fibre cables being cut in the red sea.

I’ve tried uploading and downloading using SFTP and the speed is about 1MB/s each way, so it seems it’s a routing issue. Sure enough I tried using Cloudflare Warp VPN and I was able to get 30MB/s on a single-thread SFTP transfer.

Thanks for your help :slight_smile: