How increase WebDav timeouts

Flibble · 21 April 2024 13:45

Hello, I’m using Duplicacy to backup to remote Qnap NAS via WebDav protocol. I was fine for couple years, but recently I noticed that check and prune commands started to fail. Backup work fine, but listing chunks ends with timeouts or connection reset. I probably reached some Qnap performance limits (backup have 125 000 chunks)

My question:
Is there some way I can increase Duplicacy timeouts or slow down check command? Or something else I can do on Duplicacy side? Thanks

This is log I’m getting from check command:

Running check command from .duplicacy-web/repositories/localhost/all
Options: [-log -v check -storage abc -persist -id t_down]
2024-04-21 13:47:49.964 INFO STORAGE_SET Storage set to webdav://webdavtarget.com:5001/USBDisk1/Duplicacy/targetfolder
2024-04-21 13:47:53.120 TRACE WEBDAV_MKDIR Can't create directory snapshots: Moved permanently; error ignored
2024-04-21 13:47:53.151 TRACE WEBDAV_MKDIR Can't create directory chunks: Moved permanently; error ignored
2024-04-21 13:47:53.212 TRACE CONFIG_ITERATIONS Using 123 iterations for key derivation
2024-04-21 13:47:53.266 INFO SNAPSHOT_CHECK Listing all chunks
2024-04-21 13:47:53.266 TRACE LIST_FILES Listing chunks/
2024-04-21 13:47:53.390 TRACE LIST_FILES Listing chunks/37/
2024-04-21 13:47:53.520 TRACE LIST_FILES Listing chunks/ea/
2024-04-21 13:47:53.651 TRACE LIST_FILES Listing chunks/51/
2024-04-21 13:47:53.738 TRACE LIST_FILES Listing chunks/87/
2024-04-21 13:47:53.852 TRACE LIST_FILES Listing chunks/25/
2024-04-21 13:47:53.963 TRACE LIST_FILES Listing chunks/d8/
2024-04-21 13:47:54.086 TRACE LIST_FILES Listing chunks/c6/
2024-04-21 13:47:54.866 TRACE LIST_FILES Listing chunks/ab/
2024-04-21 13:48:25.758 TRACE LIST_FILES Listing chunks/a5/
2024-04-21 13:48:50.441 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/a5/' returned an error (Propfind "https://webdavtarget.com": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:49:12.229 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/a5/' returned an error (Propfind "https://webdavtarget.com": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:49:34.394 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/a5/' returned an error (Propfind "https://webdavtarget.com": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:49:57.433 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/a5/' returned an error (Propfind "https://webdavtarget.com": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:50:23.585 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/a5/' returned an error (Propfind "https://webdavtarget.com": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:50:49.316 TRACE LIST_FILES Listing chunks/7a/
2024-04-21 13:50:49.490 TRACE LIST_FILES Listing chunks/76/
2024-04-21 13:50:49.623 TRACE LIST_FILES Listing chunks/3c/
2024-04-21 13:50:49.677 TRACE LIST_FILES Listing chunks/3b/
2024-04-21 13:50:49.794 TRACE LIST_FILES Listing chunks/df/
2024-04-21 13:50:49.910 TRACE LIST_FILES Listing chunks/af/
2024-04-21 13:50:49.993 TRACE LIST_FILES Listing chunks/86/
2024-04-21 13:50:50.655 TRACE LIST_FILES Listing chunks/52/
2024-04-21 13:50:51.287 TRACE LIST_FILES Listing chunks/bc/
2024-04-21 13:51:43.297 TRACE LIST_FILES Listing chunks/b6/
2024-04-21 13:52:05.501 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/b6/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/b6/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:52:27.453 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/b6/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/b6/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:52:50.413 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/b6/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/b6/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:53:14.289 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/b6/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/b6/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:53:39.915 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/b6/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/b6/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:54:03.709 TRACE LIST_FILES Listing chunks/57/
2024-04-21 13:54:03.845 TRACE LIST_FILES Listing chunks/f8/
2024-04-21 13:54:03.945 TRACE LIST_FILES Listing chunks/b5/
2024-04-21 13:54:04.035 TRACE LIST_FILES Listing chunks/a6/
2024-04-21 13:54:04.145 TRACE LIST_FILES Listing chunks/7e/
2024-04-21 13:54:04.351 TRACE LIST_FILES Listing chunks/79/
2024-04-21 13:54:04.492 TRACE LIST_FILES Listing chunks/8e/
2024-04-21 13:54:05.326 TRACE LIST_FILES Listing chunks/fb/
2024-04-21 13:54:06.362 TRACE LIST_FILES Listing chunks/5b/
2024-04-21 13:54:07.005 TRACE LIST_FILES Listing chunks/9a/
2024-04-21 13:54:49.791 TRACE LIST_FILES Listing chunks/97/
2024-04-21 13:55:11.105 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/97/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/97/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:55:32.760 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/97/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/97/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:55:55.339 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/97/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/97/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:56:19.821 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/97/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/97/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:56:36.071 TRACE LIST_FILES Listing chunks/5a/
2024-04-21 13:56:36.205 TRACE LIST_FILES Listing chunks/da/
2024-04-21 13:56:36.317 TRACE LIST_FILES Listing chunks/b4/
2024-04-21 13:56:36.427 TRACE LIST_FILES Listing chunks/6e/
2024-04-21 13:56:36.554 TRACE LIST_FILES Listing chunks/6d/
2024-04-21 13:56:36.688 TRACE LIST_FILES Listing chunks/1f/
2024-04-21 13:56:36.800 TRACE LIST_FILES Listing chunks/8a/
2024-04-21 13:56:37.596 TRACE LIST_FILES Listing chunks/81/
2024-04-21 13:57:20.493 TRACE LIST_FILES Listing chunks/67/
2024-04-21 13:57:43.757 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:58:05.691 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:58:28.320 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:58:52.569 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:59:20.213 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 13:59:54.972 TRACE WEBDAV_ERROR URL request 'PROPFIND chunks/67/' returned an error (Propfind "https://webdavtarget.com/targetfolder/chunks/67/": dial tcp 123.123.123.123:5001: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.)
2024-04-21 14:00:26.270 TRACE LIST_FILES Listing chunks/27/
2024-04-21 14:00:26.332 TRACE LIST_FILES Listing chunks/22/
2024-04-21 14:00:26.445 TRACE LIST_FILES Listing chunks/0b/
2024-04-21 14:00:26.586 TRACE LIST_FILES Listing chunks/ba/
2024-04-21 14:00:26.681 TRACE LIST_FILES Listing chunks/ce/
2024-04-21 14:00:26.779 TRACE LIST_FILES Listing chunks/63/
2024-04-21 14:00:26.904 TRACE LIST_FILES Listing chunks/4d/
2024-04-21 14:00:27.054 TRACE LIST_FILES Listing chunks/b3/
2024-04-21 14:00:27.837 TRACE LIST_FILES Listing chunks/95/
2024-04-21 14:02:40.095 ERROR LIST_FILES Failed to list the directory chunks/95/: read tcpsourceIP:53474->123.123.123.123:5001: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Failed to list the directory chunks/95/: read tcpsourceIP:53474->123.123.123.123:5001: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

saspus · 21 April 2024 15:10

Switch to sftp

Flibble · 21 April 2024 15:37

Sadly QNAP offer SFTP only for admin user group which is not an option.

That’s why I’m looking for solution on Duplicacy side. My other option in this scenario in is to replace Duplicacy with something else.

saspus · 21 April 2024 15:58

I googled it a bit and it’s jaw dropping. Synology did similarly dumb stunts by the way, qnap are not alone in their incompetence.

I would drop qnap at this point and switch to a network storage solution made by people who know what they are doing. NAS that only allows SFTP for admin users is laughable garbage, sorry.

In the meantime, you can install another copy of OpenSSH on a different port and use it instead of qnap provided one.

Another alternative would be to configure VPN between your host and client (including site-to-site VPN, Tailscale and/or Zerotier) and use SMB endpoint (see Samba here: Storage Backends · gilbertchen/duplicacy Wiki · GitHub)

I assume your qnap is on the different network. If it is not, I.e. if it is in the same LAN — use smb endpoint directly.

WebDAV is not suitable for large number of files. It was designed for document exchange, not bulk storage. Attempting to fetch millions of files will be increasingly overwhelming your qnap as number of chunk files continue to grow.

Can you add more ram to your NAS? Filesystem caching may help reduce the timeouts. I would suggest at least 8 to 16 GB.

Flibble · 21 April 2024 17:06

Your right, Qnap is very poor device … politely speaking

The situation is that Qnap is in a remote location and I have no possibility to do anything with it. Only WebDAV is available. Just use it or not. But thank you very much for the advice

However, other findings:
It seems that the check behavior differs depending on the OS on which Duplicacy is running. I get timeouts on Duplicacy on Windows, but if I try the Docker version, the check runs fine there… Weird right?
The connection is the same, only the OS under Duplicacy is different. I made at least a dozen attempts, so I think I can rule out coincidence.

saspus · 21 April 2024 22:07

Does it not provide remote shell? You said ssh works for admin. Or you don’t have admin access to there?

It’s a bit weird for qnap to fear SSH access from users and at the same time expose web server to the world.

On the same windows? Docker on windows runs inside Linux virtual machine that runs on HyperV; maybe with all that overhead it might be going a bit slower, just enough to avoid overwhelming the server?

You can look at the logs and see how often are the requests sent between Linux and windows. In your logs duplicacy actually mostly just waits for the sever to return file list and metadata. It can’t really do anything if the server times out.

I’m not sure if it is possible to increase those timeouts, but even so, it would be just kicking the can down the road — with increased load it will start taking even longer, exceeding the new thresholds, or maybe even hitting internal limits and dropping connections.

Adding ram to cache metadata on the server will help, but you don’t have access to it, even if ram was upgradable.

I will venture to offtopic now again:
The question I would worry about now is — how much do you trust that remote barely alive qnap server you can’t control to keep your data healthy and available? if you need to restore your data urgently?

And if you can’t be sure it’s rock solid - maybe don’t use it, even if it is a secondary backup?

Flibble · 27 April 2024 09:32

Sadly no, only WebDav is available for remote backup, and for security reason we have WebDav available only for whitelisted IPs. Qnap is not trustworthy enough : )

I did some testing:
On same machine i have two VMs:

On Windows - original Duplicacy where i have timeout problem: Same situation - timeout after some check time
Ubuntu 24.04 with freshly installed Duplicacy web and added storage: the check goes through without a problem the first time without a single timeout

I originally thought that HW might have an influence on the problem (in an earlier test, the check passed without a problem in docker on a weaker Synology), but even when limiting CPU performance in Windows, the timeouts did not change. …So now I’m not sure about anything anymore
So far, it looks like a specific problem with Duplication on Windows when the target is a slow WebDav. Or something specifically on my side, but it’s practically pure Windows without changes.

Backups continue to work from the same Windows machine to the same Webdav through other applications (Duplication)
And from another Linux VM with the same slow webdav server as the target is also not a problem.

So the only thing I can say for sure is that this topic will probably not help anyone else