Error: too many open files

Christoph · 14 February 2020 07:33

It didn’t take so long until it happened again. Not exactly the same, but “too manu open files” is part of it:

2020-02-14 02:40:33.867 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot PC_C
2020-02-14 02:40:35.002 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:40:35.930 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:40:37.330 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:40:40.022 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:40:47.109 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:40:57.148 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:41:23.622 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:42:27.121 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/1' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/1: dial tcp: lookup webdav.pcloud.com on 127.0.0.53:53: dial udp 127.0.0.53:53: socket: too many open files)
2020-02-14 02:44:11.152 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot PC_C at revision 1: Maximum backoff reached
Failed to get the information on the snapshot PC_C at revision 1: Maximum backoff reached

This is a strange combination of another error I have been seeing and the one reported in the OP.

Adding to all the strangeness in the other error (that the DNS lookup suddenly fails after the remote host has been contacted only minutes earlier), this one is even stranger the problem with the lookup suddenly is too many open files instead of no such host.

@gchen, could you explain how duplicacy constructs these error messages? I’m guessing that it is somehow passing on error messages that it gets from somewhere else?

gchen · 17 February 2020 05:50

This explained why the CLI opened so many files – each udp socket is a file descriptor too, and the Go resolver used too many sockets since the DNS resolver was down. So if you can figure out why the DNS resolver was down you won’t see this “too may files” error.

Christoph · 17 February 2020 08:18

I love it when completely weird situations (see also this topic) suddenly start to make sense

Could this be the solution:

Christoph · 20 July 2020 11:46

Unfortunately, this “too many open files” error keeps plaguing me even though I have set up a new server with a different operating system (Debian/ OpenMediaVault):

2020-07-20 04:23:08.749 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot NAS
2020-07-20 04:29:39.534 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/NAS/349 to the snapshot cache: open /.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/NAS/349.iorybnta.tmp: too many open files
2020-07-20 04:29:39.678 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:29:40.232 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:29:41.243 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:29:43.476 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:29:50.732 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:30:03.246 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:30:35.225 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:31:16.510 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/350' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/350: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-20 04:33:11.195 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot NAS at revision 350: Maximum backoff reached
Failed to get the information on the snapshot NAS at revision 350: Maximum backoff reached

So I’m not sure anymore about that bug that I mentioned above. It is being discussed by the ubuntu people and although ubuntu is derived from Debian, I’m not sure I can assume that the same problem exists on my Debian server. And if it does, I’m not sure if I can just disable systemd-resolved as suggested here.

Any suggestions how I can finally get duplicacy web to work on my server?

gchen · 23 July 2020 03:52

2020-07-20 04:29:39.534 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/NAS/349 to the snapshot cache: open /.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/NAS/349.iorybnta.tmp: too many open files

This means it already ran out of file descriptors before the DNS resolver was trying to resolve webdav.pcloud.com. So it was not the DNS resolver’s fault – it just didn’t have a free file descriptor to open a socket.

What if you move the temporary directory from /.duplicacy-web/repositories/ to something under /tmp, like /tmp/duplicacy/repositories?

Christoph · 25 July 2020 00:31

Indeed, after changing to /tmp/, I seem to be no longer getting “too many open files”. How can changing the path have this effect?

But the job nevertheless fails. Now like this:

2020-07-25 01:57:23.583 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot SERVER
2020-07-25 02:01:31.686 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned status code 401
2020-07-25 02:01:33.314 INFO WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/204' returned status code 401
2020-07-25 02:02:12.717 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned status code 401
2020-07-25 02:02:14.662 INFO WEBDAV_RETRY URL request 'GET snapshots/SERVER/238' returned status code 401
2020-07-25 02:02:59.632 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned status code 401
2020-07-25 02:09:01.534 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ALPHA_C
2020-07-25 02:15:02.943 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot NAS
2020-07-25 02:18:22.649 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:18:23.196 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:18:24.260 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:18:27.188 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:18:31.328 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:18:43.355 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:19:08.521 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:19:47.403 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/159' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/159: dial tcp: lookup webdav.pcloud.com on 208.67.222.2:53: no such host)
2020-07-25 02:21:10.069 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot NAS at revision 159: Maximum backoff reached
Failed to get the information on the snapshot NAS at revision 159: Maximum backoff reached

What does this mean?

gchen · 26 July 2020 16:25

Your DNS resolver running on 208.67.222.2:53 is down.

Christoph · 26 July 2020 17:15

Not sure what’s going on there. 208.67.222.2 is from OpenDNS and it is my secondary DNS, the primary one being 1.1.1.1 (cloudflare). No idea why it’s not using cloudflare. But I guess, I should get rid of the openDNS server.

Edit:
Plus, what I understand even less: why does duplicacy need to look up webdav.pcloud.com mid way through the process, after it has been downloading data from that same url for an hour or so? Doesn’t it remember the IP?

saspus · 26 July 2020 20:29

By default some (most?) resolvers use all configured upsteam resolvers. If you use dnsmasq see the option --all-servers. You can configure it to use secondary resolvers only if the primary is down: see --strict-order for dnsmasq.

And on a tangential – if you use OpenDNS for family/malware filtering Cloudflare now also have family/malware filtered resolvers – 1.0.0. and 1.1.1. ending .2 and .3

Christoph · 26 July 2020 21:21

As I’m thinking about it, my new setup is actually a bit more complex. I’m using pihole on a local machine as my primary DNS and OpenDNS as secondary (for no particular reason). Pihole then is set up to use one or more of cloudflare’s DNS.

So I didn’t actually expect the secondary DNS to be used at all because the first one is backed by multiple others. So thanks for explaining that secondary doesn’t really mean secondary. I now realise that my redundancy setup is wrong and I need to get rid of the secondary DNS (especially when it’s failing me anyway).

But let me try and get this back on topic: I’m still surprised that a single failing DNS server can make duplicacy fail. Why is that?

Christoph · 29 July 2020 22:10

Okay, now that duplicacy is using a functioning DNS server again, we’re back to the too many open files error again:

2020-07-29 03](nxphone:2020-07-29 03):42:47.379 INFO WEBDAV_RETRY URL request 'GET snapshots/NAS/400' returned status code 401
2020-07-29 03:43:00.227 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned status code 401
2020-07-29 03:43:28.826 INFO WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/436' returned status code 401
2020-07-29 03:43:53.806 INFO WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/457' returned status code 401
2020-07-29 03:43:55.789 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned status code 401
2020-07-29 03:45:05.202 INFO WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned status code 401
2020-07-29 03:48:37.413 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot PC_C
2020-07-29 03:49:18.437 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/PC_C/281 to the snapshot cache: open /tmp/.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/PC_C/281.mmmxwsca.tmp: too many open files
2020-07-29 03:49:18.578 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:49:19.508 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:49:20.656 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:49:23.746 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:49:30.407 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:49:43.434 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:50:05.770 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:50:51.012 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/284' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/284: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-07-29 03:52:36.784 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot PC_C at revision 284: Maximum backoff reached
Failed to get the information on the snapshot PC_C at revision 284: Maximum backoff reached

There also seems to be an authentication issue which is probably pCloud’s fault (their webdav interface intermittently claims that I don’t have access to some of my files or directories but that usually goes away again) but it doesn’t seem to cause the job to fail.

Christoph · 3 August 2020 20:57

@gchen Errors just keep piling up. In addition to the usual “too many open files” error that eventually kills the whole job, there are now even EOF errors TLS handshake timeout errors, and connection reset by peer, I would assume that the latter are pclouds fault but the biggest problem is the too many open files error which, as far as I can see, has nothing to do with pcloud…

2020-08-03 14:55:54.584 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot SERVER
2020-08-03 15:00:36.427 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/302' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/302: EOF)
2020-08-03 15:01:01.940 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/309' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/309: EOF)
2020-08-03 15:02:00.243 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/351' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/351: EOF)
2020-08-03 15:02:28.192 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/362' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/362: EOF)
2020-08-03 15:03:36.858 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/418' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/418: EOF)
2020-08-03 15:05:17.215 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-08-03 15:05:33.472 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-08-03 15:07:06.861 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/558' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/558: net/http: TLS handshake timeout)
2020-08-03 15:07:09.039 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot SURFACE
2020-08-03 15:07:11.180 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ext_MEETGOV
2020-08-03 15:07:13.436 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot NAS_media
2020-08-03 15:07:14.661 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot PC_C
2020-08-03 15:10:51.497 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/368' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/368: EOF)
2020-08-03 15:12:40.752 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/399' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/399: EOF)
2020-08-03 15:13:32.064 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/402' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/402: EOF)
2020-08-03 15:13:55.211 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: read tcp 192.168.1.4:57474->74.120.8.227:443: read: connection reset by peer)
2020-08-03 15:15:29.698 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/415' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/415: EOF)
2020-08-03 15:16:46.433 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: EOF)
2020-08-03 15:17:32.275 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/448' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/448: net/http: TLS handshake timeout)
2020-08-03 15:21:04.631 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/510' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/510: EOF)
2020-08-03 15:21:17.415 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: EOF)
2020-08-03 15:21:51.264 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/519' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/519: EOF)
2020-08-03 15:22:28.778 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/523' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/523: EOF)
2020-08-03 15:22:59.490 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/525' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/525: EOF)
2020-08-03 15:23:24.157 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: EOF)
2020-08-03 15:23:52.493 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: EOF)
2020-08-03 15:24:03.587 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: net/http: TLS handshake timeout)
2020-08-03 15:24:15.640 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: net/http: TLS handshake timeout)
2020-08-03 15:24:28.674 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: net/http: TLS handshake timeout)
2020-08-03 15:25:01.676 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C: read tcp 192.168.1.4:58080->74.120.8.227:443: read: connection reset by peer)
2020-08-03 15:26:20.427 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/557' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/557: EOF)
2020-08-03 15:26:57.442 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/559' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/559: EOF)
2020-08-03 15:27:44.235 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/559' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/559: EOF)
2020-08-03 15:28:37.823 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/579' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/579: net/http: TLS handshake timeout)
2020-08-03 15:30:54.426 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/636' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/636: EOF)
2020-08-03 15:32:28.371 TRACE WEBDAV_RETRY URL request 'GET snapshots/PC_C/658' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/658: EOF)
2020-08-03 15:33:04.132 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_C/661' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_C/661: EOF)
2020-08-03 15:33:46.328 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot PC_deleted
2020-08-03 15:33:47.877 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ALPHA_D_COP
2020-08-03 15:33:58.043 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot NAS
2020-08-03 15:34:37.866 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/NAS/24 to the snapshot cache: open /tmp/.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/NAS/24.qhciziks.tmp: too many open files
2020-08-03 15:34:38.031 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:34:38.608 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:34:40.152 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:34:43.237 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:34:47.653 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:34:57.343 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:35:14.259 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:35:46.833 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/25' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/25: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-08-03 15:36:56.577 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot NAS at revision 25: Maximum backoff reached
Failed to get the information on the snapshot NAS at revision 25: Maximum backoff reached

Christoph · 3 September 2020 19:49

@gchen I’m still stuck with this issue. Not a single check command has been completed because of this. Allthough I am inclined to believe that pCloud’s webdav interface is not exactly making things better, It seems clear that the too many open files problem is a local problem so that even if pCloud is creating some errors, duplicacy is also somehow failing in handling these.

It also seems likely that this is a duplicacy problem, because I have been seeing this very same problem on two different machines, one Ubuntu 18.04 server and now on a freshly installed Debian 10 machine (OpenMediaVault). I have not modified the Debian OS (all my apps are running in various docker containers) so I don’t see how this is not a duplicacy bug or design fault…

2020-09-02 23:34:58.996 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/252' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/252: EOF)
2020-09-02 23:35:23.063 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/ALPHA_C' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C: EOF)
2020-09-02 23:35:49.455 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/ALPHA_C/272 to the snapshot cache: open /tmp/.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/ALPHA_C/272.pcwrzhfe.tmp: too many open files
2020-09-02 23:35:49.612 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:35:50.483 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:35:52.041 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:35:55.082 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:36:00.451 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:36:09.710 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:36:27.985 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:37:09.796 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/ALPHA_C/273' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/ALPHA_C/273: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: dial udp 1.1.1.1:53: socket: too many open files)
2020-09-02 23:39:14.283 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot ALPHA_C at revision 273: Maximum backoff reached
Failed to get the information on the snapshot ALPHA_C at revision 273: Maximum backoff reached

saspus · 3 September 2020 20:54

Not exactly helpful comment but this is exactly why Duplicacy should drop support for file sharing storage backends. They are not designed for bulk storage and will fail in weird in wonderful way — exemplified by this topic — and ultimately result is user frustration and extra futile support workload for Acrosync.

More organized thoughts on this here: (Newbie) Backup entire machine?

Christoph · 3 September 2020 23:41

I’m willing to admit that I wasted - I think it was 500 USD - when buying 4TB of lifetime storage from pCloud; I was not aware of the differences that you explain so well in the other post… But are you saying that only pCloud is to blame? I don’t understand the logs 100% but it seems clear to me that duplicacy is not handling whatever problems pCloud is having properly, no?

saspus · 3 September 2020 23:53

I did not watch the thread too carefully but 401(unauthorized) and TLS handshake timeout is fairly certainly either service issues or their anti-abuse measures. Either way not good for intended purpose.

Duplicacy should keep trying to back off and retry with increasing interval — and maybe it does fail to do so in some cases but it’s too late anyway. Service has already face planted.

The too many open files does look like client side issue — perhaps you are hitting limit on max open files? You can run lsof to see who has holding what handles and then act accordingly (murder the offending process or report Duplicacy bug or increase handles limit). Maybe indeed there is a bug handling failed connections and Duplicacy is not letting go the orphaned connections?

gchen · 4 September 2020 01:16

What is the TTL of webdav.pcloud.com on your machine? On mine it is 7200 seconds. If this TTL is too small there will be too many DNS requests.

$ dig +nocmd +noall +answer webdav.pcloud.com
webdav.pcloud.com.	7199	IN	A	74.120.8.227

I think you should keep running lsof once Duplicacy starts. Maybe write a script to save the output of lsof to a file every minute. You should be watching all file descriptors opened by the Duplicacy CLI, not the web GUI. If the list of file descriptors keeps grow, it may be fairly obvious what is leaking.

Christoph · 4 September 2020 14:10

Aaah, TTL! that is the answer to this question:

Anyway: a couple of hours ago, I was also getting 7200. But now (after having started by check command on the web-ui) I’m getting slightly less:

$ dig +nocmd +noall +answer webdav.pcloud.com
webdav.pcloud.com.      6188    IN      A       74.120.8.227

And now, I couple of minutes later, it’s less again:

$ dig +nocmd +noall +answer webdav.pcloud.com
webdav.pcloud.com.      5862    IN      A       74.120.8.227

I’m assuming that this is reflecting the countdown of those two hours that was triggered when duplicacy queried the DNS.

As suggested, I am also monitoring which files duplicacy has opened. I did the following:

$ ps aux | grep duplicacy
root       959  0.0  0.2 196800 21184 ?        Ssl  Aug25   3:05 /usr/bin/duplicacy-web
root     28590  5.3  0.5 126696 40192 ?        Sl   15:30   0:01 /.duplicacy-web/bin/duplicacy_linux_x64_2.6.1 -log -v check -storage pcloud -a -tabular
root     29241  0.0  0.0   6204   888 pts/4    S+   15:30   0:00 grep duplicacy

Then I used the pid of the CLI process to do

$ lsof -p 28590 -r300 >> /zfs/NAS/christoph/duplicacy_open_files.log &

(I thought logging every five minutes would be sufficient.)

So currently I’m getting:

COMMAND     PID USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
duplicacy 28590 root  cwd       DIR      0,50       60        16 /tmp/.duplicacy-web/repositories/localhost/all
duplicacy 28590 root  rtd       DIR       8,1     4096         2 /
duplicacy 28590 root  txt       REG       8,1 31740199  13369359 /.duplicacy-web/bin/duplicacy_linux_x64_2.6.1
duplicacy 28590 root    0r      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    1w     FIFO      0,13      0t0 104332902 pipe
duplicacy 28590 root    2w      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    3u     sock       0,9      0t0 104332951 protocol: TCP
duplicacy 28590 root    4u  a_inode      0,14        0     11185 [eventpoll]
duplicacy 28590 root    5u     sock       0,9      0t0 104333609 protocol: TCP
duplicacy 28590 root    6u     IPv4 104770091      0t0       TCP server.lan:42440->api13.pcloud.com:https (ESTABLISHED)

I’ll get back once the process has failed.

This is the kind of thing I am suspecting, but I’m unable to trace down that kind of stuff by myself. So let’s see what that log will reveal.

Christoph · 4 September 2020 21:06

So, here we go. It didn’t take very long until things started piling up. I’ll give you the last “ordinary” log and then the one five minutes later and another five minutes later (to stay within the max character limit per post I’m cutting out repetitive lines):

COMMAND     PID USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
duplicacy 28590 root  cwd       DIR      0,50       60        16 /tmp/.duplicacy-web/repositories/localhost/all
duplicacy 28590 root  rtd       DIR       8,1     4096         2 /
duplicacy 28590 root  txt       REG       8,1 31740199  13369359 /.duplicacy-web/bin/duplicacy_linux_x64_2.6.1
duplicacy 28590 root    0r      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    1w     FIFO      0,13      0t0 104332902 pipe
duplicacy 28590 root    2w      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    3u     sock       0,9      0t0 104332951 protocol: TCP
duplicacy 28590 root    4u  a_inode      0,14        0     11185 [eventpoll]
duplicacy 28590 root    5u     sock       0,9      0t0 104333609 protocol: TCP
duplicacy 28590 root    6u     IPv4 104856773      0t0       TCP server.lan:43182->api13.pcloud.com:https (ESTABLISHED)
=======
COMMAND     PID USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
duplicacy 28590 root  cwd       DIR      0,50       60        16 /tmp/.duplicacy-web/repositories/localhost/all
duplicacy 28590 root  rtd       DIR       8,1     4096         2 /
duplicacy 28590 root  txt       REG       8,1 31740199  13369359 /.duplicacy-web/bin/duplicacy_linux_x64_2.6.1
duplicacy 28590 root    0r      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    1w     FIFO      0,13      0t0 104332902 pipe
duplicacy 28590 root    2w      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    3u     sock       0,9      0t0 104332951 protocol: TCP
duplicacy 28590 root    4u  a_inode      0,14        0     11185 [eventpoll]
duplicacy 28590 root    5u     sock       0,9      0t0 104333609 protocol: TCP
duplicacy 28590 root    6u     IPv4 104880962      0t0       TCP server.lan:50474->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root    7u     IPv4 104880999      0t0       TCP server.lan:50526->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root    8u     IPv4 104881032      0t0       TCP server.lan:50532->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root    9u     IPv4 104881785      0t0       TCP server.lan:50550->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   10u     IPv4 104881082      0t0       TCP server.lan:50576->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   11u     IPv4 104881116      0t0       TCP server.lan:50634->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   12u     IPv4 104881146      0t0       TCP server.lan:50708->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   13u     IPv4 104883139      0t0       TCP server.lan:51406->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   14u     IPv4 104885905      0t0       TCP server.lan:51414->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   15u     IPv4 104883169      0t0       TCP server.lan:51450->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   16u     IPv4 104885917      0t0       TCP server.lan:51500->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   17u     IPv4 104885930      0t0       TCP server.lan:51540->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   18u     IPv4 104886288      0t0       TCP server.lan:51584->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   19u     IPv4 104885975      0t0       TCP server.lan:51588->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   20u     IPv4 104885987      0t0       TCP server.lan:51590->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   21u     IPv4 104885999      0t0       TCP server.lan:51618->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   22u     IPv4 104886322      0t0       TCP server.lan:51676->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   23u     IPv4 104886354      0t0       TCP server.lan:51698->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   24u     IPv4 104886450      0t0       TCP server.lan:51748->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   25u     IPv4 104886115      0t0       TCP server.lan:51776->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   26u     IPv4 104886505      0t0       TCP server.lan:51794->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   27u     IPv4 104886178      0t0       TCP server.lan:51864->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   28u     IPv4 104886515      0t0       TCP server.lan:51880->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   29u     IPv4 104886209      0t0       TCP server.lan:51930->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   30u     IPv4 104888482      0t0       TCP server.lan:51958->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   31u     IPv4 104888490      0t0       TCP server.lan:51962->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   32u     IPv4 104888495      0t0       TCP server.lan:51980->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   33u     IPv4 104888528      0t0       TCP server.lan:52042->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   34u     IPv4 104888543      0t0       TCP server.lan:52058->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   35u     IPv4 104889426      0t0       TCP server.lan:52072->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   36u     IPv4 104889429      0t0       TCP server.lan:52110->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   37u     IPv4 104888612      0t0       TCP server.lan:52132->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   38u     IPv4 104889468      0t0       TCP server.lan:52144->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   39u     IPv4 104888620      0t0       TCP server.lan:52146->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   40u     IPv4 104888637      0t0       TCP server.lan:52208->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   41u     IPv4 104889564      0t0       TCP server.lan:52240->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   42u     IPv4 104889570      0t0       TCP server.lan:52248->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   43u     IPv4 104888691      0t0       TCP server.lan:52296->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   44u     IPv4 104888693      0t0       TCP server.lan:52300->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   45u     IPv4 104889587      0t0       TCP server.lan:52326->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   46u     IPv4 104888708      0t0       TCP server.lan:52328->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   47u     IPv4 104888728      0t0       TCP server.lan:52378->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   48u     IPv4 104889017      0t0       TCP server.lan:52404->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   49u     IPv4 104890227      0t0       TCP server.lan:52454->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   50u     IPv4 104889104      0t0       TCP server.lan:52482->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   51u     IPv4 104889114      0t0       TCP server.lan:52502->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   52u     IPv4 104890250      0t0       TCP server.lan:52510->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   53u     IPv4 104889128      0t0       TCP server.lan:52556->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   54u     IPv4 104890305      0t0       TCP server.lan:52576->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   55u     IPv4 104889175      0t0       TCP server.lan:52594->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root   56u     IPv4 104890324      0t0       TCP server.lan:52646->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   57u     IPv4 104889207      0t0       TCP server.lan:52666->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root   58u     IPv4 104890336      0t0       TCP server.lan:52688->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root   59u     IPv4 104891458      0t0       TCP server.lan:52844->api13.pcloud.com:https (ESTABLISHED)
=======
COMMAND     PID USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
duplicacy 28590 root  cwd       DIR      0,50       60        16 /tmp/.duplicacy-web/repositories/localhost/all
duplicacy 28590 root  rtd       DIR       8,1     4096         2 /
duplicacy 28590 root  txt       REG       8,1 31740199  13369359 /.duplicacy-web/bin/duplicacy_linux_x64_2.6.1
duplicacy 28590 root    0r      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    1w     FIFO      0,13      0t0 104332902 pipe
duplicacy 28590 root    2w      CHR       1,3      0t0         6 /dev/null
duplicacy 28590 root    3u     sock       0,9      0t0 104332951 protocol: TCP
duplicacy 28590 root    4u  a_inode      0,14        0     11185 [eventpoll]
duplicacy 28590 root    5u     sock       0,9      0t0 104333609 protocol: TCP
duplicacy 28590 root    6u     sock       0,9      0t0 104880962 protocol: TCP
duplicacy 28590 root    7u     sock       0,9      0t0 104880999 protocol: TCP
duplicacy 28590 root    8u     sock       0,9      0t0 104881032 protocol: TCP
duplicacy 28590 root    9u     sock       0,9      0t0 104881785 protocol: TCP
duplicacy 28590 root   10u     sock       0,9      0t0 104881082 protocol: TCP
duplicacy 28590 root   11u     sock       0,9      0t0 104881116 protocol: TCP
duplicacy 28590 root   12u     sock       0,9      0t0 104881146 protocol: TCP
duplicacy 28590 root   13u     sock       0,9      0t0 104883139 protocol: TCP
duplicacy 28590 root   14u     sock       0,9      0t0 104885905 protocol: TCP
duplicacy 28590 root   15u     sock       0,9      0t0 104883169 protocol: TCP
duplicacy 28590 root   16u     sock       0,9      0t0 104885917 protocol: TCP
duplicacy 28590 root   17u     sock       0,9      0t0 104885930 protocol: TCP
duplicacy 28590 root   18u     sock       0,9      0t0 104886288 protocol: TCP
duplicacy 28590 root   19u     sock       0,9      0t0 104885975 protocol: TCP
duplicacy 28590 root   20u     sock       0,9      0t0 104885987 protocol: TCP
duplicacy 28590 root   21u     sock       0,9      0t0 104885999 protocol: TCP
duplicacy 28590 root   22u     sock       0,9      0t0 104886322 protocol: TCP
duplicacy 28590 root   23u     sock       0,9      0t0 104886354 protocol: TCP
duplicacy 28590 root   24u     sock       0,9      0t0 104886450 protocol: TCP
duplicacy 28590 root   25u     sock       0,9      0t0 104886115 protocol: TCP
duplicacy 28590 root   26u     sock       0,9      0t0 104886505 protocol: TCP
duplicacy 28590 root   27u     sock       0,9      0t0 104886178 protocol: TCP
duplicacy 28590 root   28u     sock       0,9      0t0 104886515 protocol: TCP
duplicacy 28590 root   29u     sock       0,9      0t0 104886209 protocol: TCP
duplicacy 28590 root   30u     sock       0,9      0t0 104888482 protocol: TCP
duplicacy 28590 root   31u     sock       0,9      0t0 104888490 protocol: TCP
duplicacy 28590 root   32u     sock       0,9      0t0 104888495 protocol: TCP
duplicacy 28590 root   33u     sock       0,9      0t0 104888528 protocol: TCP
duplicacy 28590 root   34u     sock       0,9      0t0 104888543 protocol: TCP
duplicacy 28590 root   35u     sock       0,9      0t0 104889426 protocol: TCP
duplicacy 28590 root   36u     sock       0,9      0t0 104889429 protocol: TCP
duplicacy 28590 root   37u     sock       0,9      0t0 104888612 protocol: TCP
duplicacy 28590 root   38u     sock       0,9      0t0 104889468 protocol: TCP
duplicacy 28590 root   39u     sock       0,9      0t0 104888620 protocol: TCP
.
.
[160 lines cut]
.
.
duplicacy 28590 root  200u     sock       0,9      0t0 104910169 protocol: TCP
duplicacy 28590 root  201u     sock       0,9      0t0 104910884 protocol: TCP
duplicacy 28590 root  202u     sock       0,9      0t0 104910197 protocol: TCP
duplicacy 28590 root  203u     sock       0,9      0t0 104910917 protocol: TCP
duplicacy 28590 root  204u     sock       0,9      0t0 104910939 protocol: TCP
duplicacy 28590 root  205u     sock       0,9      0t0 104910947 protocol: TCP
duplicacy 28590 root  206u     sock       0,9      0t0 104911036 protocol: TCP
duplicacy 28590 root  207u     sock       0,9      0t0 104911100 protocol: TCP
duplicacy 28590 root  208u     sock       0,9      0t0 104910633 protocol: TCP
duplicacy 28590 root  209u     sock       0,9      0t0 104911733 protocol: TCP
duplicacy 28590 root  210u     sock       0,9      0t0 104910656 protocol: TCP
duplicacy 28590 root  211u     sock       0,9      0t0 104910670 protocol: TCP
duplicacy 28590 root  212u     sock       0,9      0t0 104911782 protocol: TCP
duplicacy 28590 root  213u     sock       0,9      0t0 104910686 protocol: TCP
duplicacy 28590 root  214u     sock       0,9      0t0 104911792 protocol: TCP
duplicacy 28590 root  215u     sock       0,9      0t0 104911803 protocol: TCP
duplicacy 28590 root  216u     sock       0,9      0t0 104911878 protocol: TCP
duplicacy 28590 root  217u     sock       0,9      0t0 104911852 protocol: TCP
duplicacy 28590 root  218u     sock       0,9      0t0 104912913 protocol: TCP
duplicacy 28590 root  219u     IPv4 104912052      0t0       TCP server.lan:59000->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  220u     sock       0,9      0t0 104912060 protocol: TCP
duplicacy 28590 root  221u     sock       0,9      0t0 104912068 protocol: TCP
duplicacy 28590 root  222u     sock       0,9      0t0 104912071 protocol: TCP
duplicacy 28590 root  223u     sock       0,9      0t0 104913009 protocol: TCP
duplicacy 28590 root  224u     IPv4 104912126      0t0       TCP server.lan:59142->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  225u     sock       0,9      0t0 104913028 protocol: TCP
duplicacy 28590 root  226u     sock       0,9      0t0 104915012 protocol: TCP
duplicacy 28590 root  227u     IPv4 104915017      0t0       TCP server.lan:59180->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  228u     sock       0,9      0t0 104915033 protocol: TCP
duplicacy 28590 root  229u     sock       0,9      0t0 104913052 protocol: TCP
duplicacy 28590 root  230u     IPv4 104915061      0t0       TCP server.lan:59258->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  231u     sock       0,9      0t0 104913068 protocol: TCP
duplicacy 28590 root  232u     sock       0,9      0t0 104913161 protocol: TCP
duplicacy 28590 root  233u     IPv4 104915177      0t0       TCP server.lan:59340->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  234u     IPv4 104913181      0t0       TCP server.lan:59350->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  235u     IPv4 104913190      0t0       TCP server.lan:59384->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  236u     IPv4 104913199      0t0       TCP server.lan:59400->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  237u     sock       0,9      0t0 104915237 protocol: TCP
duplicacy 28590 root  238u     IPv4 104915246      0t0       TCP server.lan:59482->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  239u     sock       0,9      0t0 104915266 protocol: TCP
duplicacy 28590 root  240u     sock       0,9      0t0 104915277 protocol: TCP
duplicacy 28590 root  241u     sock       0,9      0t0 104915312 protocol: TCP
duplicacy 28590 root  242u     sock       0,9      0t0 104915323 protocol: TCP
duplicacy 28590 root  243u     IPv4 104915341      0t0       TCP server.lan:59570->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  244u     IPv4 104915363      0t0       TCP server.lan:59628->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  245u     sock       0,9      0t0 104915401 protocol: TCP
duplicacy 28590 root  246u     IPv4 104915423      0t0       TCP server.lan:59660->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  247u     IPv4 104915821      0t0       TCP server.lan:59674->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  248u     IPv4 104915832      0t0       TCP server.lan:59690->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  249u     sock       0,9      0t0 104916103 protocol: TCP
duplicacy 28590 root  250u     IPv4 104916112      0t0       TCP server.lan:59780->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  251u     sock       0,9      0t0 104916127 protocol: TCP
duplicacy 28590 root  252u     IPv4 104915883      0t0       TCP server.lan:59826->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  253u     IPv4 104915886      0t0       TCP server.lan:59840->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  254u     IPv4 104916175      0t0       TCP server.lan:59858->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  255u     IPv4 104915907      0t0       TCP server.lan:59878->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  256u     IPv4 104915932      0t0       TCP server.lan:59920->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  257u     IPv4 104915967      0t0       TCP server.lan:59950->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  258u     IPv4 104916278      0t0       TCP server.lan:59958->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  259u     IPv4 104916285      0t0       TCP server.lan:59984->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  260u     IPv4 104916293      0t0       TCP server.lan:60000->api13.pcloud.com:https (CLOSE_WAIT)
.
.
[50 lines cut]
.
.
duplicacy 28590 root  311u     IPv4 104919469      0t0       TCP server.lan:33076->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  312u     IPv4 104919476      0t0       TCP server.lan:33086->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  313u     IPv4 104921699      0t0       TCP server.lan:33124->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  314u     IPv4 104919518      0t0       TCP server.lan:33132->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  315u     IPv4 104919660      0t0       TCP server.lan:33280->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  316u     IPv4 104919667      0t0       TCP server.lan:33298->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  317u     IPv4 104921938      0t0       TCP server.lan:33338->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  318u     IPv4 104923150      0t0       TCP server.lan:33540->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  319u     IPv4 104922302      0t0       TCP server.lan:33590->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  320u     IPv4 104923194      0t0       TCP server.lan:33596->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  321u     IPv4 104923293      0t0       TCP server.lan:33688->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  322u     IPv4 104923319      0t0       TCP server.lan:33744->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  323u     IPv4 104923322      0t0       TCP server.lan:33748->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  324u     IPv4 104923354      0t0       TCP server.lan:33760->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  325u     IPv4 104922495      0t0       TCP server.lan:33778->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  326u     IPv4 104923401      0t0       TCP server.lan:33806->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  327u     IPv4 104922516      0t0       TCP server.lan:33832->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  328u     IPv4 104922530      0t0       TCP server.lan:33852->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  329u     IPv4 104923445      0t0       TCP server.lan:33908->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  330u     IPv4 104923497      0t0       TCP server.lan:33912->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  331u     IPv4 104922622      0t0       TCP server.lan:33914->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  332u     IPv4 104922632      0t0       TCP server.lan:33932->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  333u     IPv4 104923515      0t0       TCP server.lan:33966->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  334u     IPv4 104923521      0t0       TCP server.lan:33992->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  335u     IPv4 104923527      0t0       TCP server.lan:34002->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  336u     IPv4 104922744      0t0       TCP server.lan:34068->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  337u     IPv4 104923548      0t0       TCP server.lan:34078->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  338u     IPv4 104923554      0t0       TCP server.lan:34080->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  339u     IPv4 104922767      0t0       TCP server.lan:34092->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  340u     IPv4 104923579      0t0       TCP server.lan:34112->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  341u     IPv4 104922813      0t0       TCP server.lan:34142->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  342u     IPv4 104922822      0t0       TCP server.lan:34162->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  343u     IPv4 104922827      0t0       TCP server.lan:34216->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  344u     IPv4 104923708      0t0       TCP server.lan:34240->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  345u     IPv4 104922899      0t0       TCP server.lan:34242->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  346u     IPv4 104922917      0t0       TCP server.lan:34252->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  347u     IPv4 104924057      0t0       TCP server.lan:34262->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  348u     IPv4 104924216      0t0       TCP server.lan:34294->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  349u     IPv4 104924093      0t0       TCP server.lan:34322->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  350u     IPv4 104924099      0t0       TCP server.lan:34364->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root  351u     IPv4 104924249      0t0       TCP server.lan:34394->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  352u     IPv4 104924118      0t0       TCP server.lan:34404->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  353u     IPv4 104924262      0t0       TCP server.lan:34412->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root  354u     IPv4 104924129      0t0       TCP server.lan:34426->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  355u     IPv4 104924276      0t0       TCP server.lan:34444->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  356u     IPv4 104924323      0t0       TCP server.lan:34494->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  357u     IPv4 104925262      0t0       TCP server.lan:34540->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  358u     IPv4 104925325      0t0       TCP server.lan:34564->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root  359u     IPv4 104925330      0t0       TCP server.lan:34574->api13.pcloud.com:https (ESTABLISHED)
duplicacy 28590 root  360u     IPv4 104925338      0t0       TCP server.lan:34580->api13.pcloud.com:https (CLOSE_WAIT)
duplicacy 28590 root  361u     IPv4 104925343      0t0       TCP server.lan:34596->api13.pcloud.com:https (ESTABLISHED)

The number of open files then increases every five minutes. The last log before the command failed has over 1000 entries.

Does “CLOSE_WAIT” mean that @saspus was right about the orphaned connections?

gchen · 5 September 2020 04:08

This could have been fixed by this PR: Improve WebDAV performance by Jos635 · Pull Request #549 · gilbertchen/duplicacy · GitHub

I compiled a linux binary from this PR if you want to test it: https://acrosync.com/duplicacy/duplicacy_linux_x64_2.6.11