Webdav error: unexpected EOF reading trailer

Support
,
#1

I’m continuing the webdav discussion from Github here.

I am seeing the following issue:

PS C:\> duplicacy check -tabular
Storage set to webdav://user@name@webdav.pcloud.com/Backup/Duplicacy
Listing all chunks
Failed to list the directory chunks/da/: http: unexpected EOF reading trailer
PS C:\> duplicacy check -tabular
Storage set to webdav://user@name@webdav.pcloud.com/Backup/Duplicacy
Listing all chunks
Failed to list the directory chunks/90/: http: unexpected EOF reading trailer
PS C:\> duplicacy check -tabular
Storage set to webdav://user@name@webdav.pcloud.com/Backup/Duplicacy
Listing all chunks
Failed to list the directory chunks/bd/: http: unexpected EOF reading trailer
PS C:\> duplicacy check -tabular
Storage set to webdav://user@name@webdav.pcloud.com/Backup/Duplicacy
Listing all chunks
Failed to list the directory chunks/: unexpected EOF
PS C:\>

I would not be surprised if this is yet another failure of pcloud’s webdav interface, but perhaps there is something that can be improved in duplicacy’s side?

Lots of missing chunks with pCloud
#2

Today it seems to work, so it was probably pcloud’s problem (not the first time their webdav interface is throwing errors)

#3

The problem seems to be back:

$duplicacy check -tabular
Storage set to webdav://<myusername>@webdav.pcloud.com/Backup/Duplicacy
Listing all chunks
Failed to list the directory chunks/: http: unexpected EOF reading trailer
$

This is not limited to the check command, btw. Also my backups are currently failing.

@gchen, could you explain what the “unexpected EOF” message means? Does it mean that the server starts sending the file and then suddenly stops? Or Is it possible that the server doesn’t even start sending the file?

I’m asking because I need to be very exact with the pCloud support, otherwise they just keep coming up with all kinds of excuses. Is there a simple way I can provoke the above error without using duplicacy (i.e. using more generic webdav command?)

#4

“unexpected EOF” means the server unexpectedly closed the connection. “unexpected EOF reading trailer” means the server unexpectedly closed the connection when the client is expecting the http trailers. So maybe their servers don’t send the trailers in a proper way?

#5

Last night the scheduled backup worked (without me chaning anything). So I won’t bother discussing this with their support right now.

But since I think I mentioned elsewhere that pCloud is not officially supporting WebDAV (because that is what theuir support told me), I should mention that someone else from pCloud claims that they officially do support it here.

#6

So I finally had to take this up with pCloud support and they are asking me to try via another browser (duh) or if I’m not accessing via browser “tell us how you use the pCloud’s WebDAV exactly?”

I want to give them the answer that is most likely going to lead to a solution and that means, in particular, that I give them as few alternative sources for the error. (Unfortunately, this was previously one of their routine answers: e.g. “it’s the fault of the software you’re using”, or: “we don’t support that software” , which is of course nonsense when all I’m asking them is to support webdav.

Anyway, I’m thinking the best might be to describe how duplicacy is accessing the webdav storage at a low level. In this case the problem is occurring when trying to do a check, so I’ll stick to that.

@gchen could you tell me how exactly duplicacy is interacting with webdav?

Edit: To provide some context, here are the most recent failure logs:

Running check command from /srv/NAS/duplicacy/duplicacy-web/repositories/localhost/all
Options: [-log check -storage pCloud -a -tabular]
2020-02-02 02:00:01.957 INFO STORAGE_SET Storage set to webdav://myusername@webdav.pcloud.com/Backup/Duplicacy
2020-02-02 02:00:05.669 INFO SNAPSHOT_CHECK Listing all chunks
2020-02-02 04:01:46.900 ERROR LIST_FILES Failed to list the directory chunks/29/: http: unexpected EOF reading trailer
Failed to list the directory chunks/29/: http: unexpected EOF reading trailer

Running check command from /srv/NAS/duplicacy/duplicacy-web/repositories/localhost/all
Options: [-log check -storage pCloud -a -tabular]
2020-02-03 02:00:01.225 INFO STORAGE_SET Storage set to webdav://myusername@webdav.pcloud.com/Backup/Duplicacy
2020-02-03 02:00:04.249 INFO SNAPSHOT_CHECK Listing all chunks
2020-02-03 10:02:48.625 ERROR LIST_FILES Failed to list the directory chunks/6d/: http: unexpected EOF reading trailer
Failed to list the directory chunks/6d/: http: unexpected EOF reading trailer

Running check command from /srv/NAS/duplicacy/duplicacy-web/repositories/localhost/all
Options: [-log check -storage pCloud -a -tabular]
2020-02-04 18:58:31.933 INFO STORAGE_SET Storage set to webdav://myusername@webdav.pcloud.com/Backup/Duplicacy
2020-02-04 18:58:34.307 INFO SNAPSHOT_CHECK Listing all chunks
2020-02-04 19:02:10.573 ERROR LIST_FILES Failed to list the directory chunks/2c/: http: unexpected EOF reading trailer
Failed to list the directory chunks/2c/: http: unexpected EOF reading trailer

Running check command from /srv/NAS/duplicacy/duplicacy-web/repositories/localhost/all
Options: [-log check -storage pCloud -a -tabular]
2020-02-05 20:23:37.839 INFO STORAGE_SET Storage set to webdav://myusername@webdav.pcloud.com/Backup/Duplicacy
2020-02-05 20:23:40.315 INFO SNAPSHOT_CHECK Listing all chunks
2020-02-05 20:27:08.215 ERROR LIST_FILES Failed to list the directory chunks/2f/: http: unexpected EOF reading trailer
Failed to list the directory chunks/2f/: http: unexpected EOF reading trailer
#7

As expected, pCloud now claim that they can’t see any fault on their side and they are asking for a video that shows how the error occurs. Or some other evidence that they can work with to reproduce the error.

@gchen, how exactly is duplicacy interacting with webdav storage when the check command is listing chunks in a directory?

I am suspecting that the problem has something to one or more of the following

  • the large number of files being listed
  • resource problems on the client (the error only seem to occur on my Linux server, not on my desktop PC)
  • a bug in either duplicacy or Golang in how these requests are handled (When searching for “http: unexpected EOF reading trailer”, Google spits suspiciously many hits related to Golang…)
  • a weakness in pClouds webdav implementation (which probably only becomes a problem when combined with another weakness/bug in duplicacy/Golang)

@gchen: if you look at the following post on the duplicati forum, does it give you any ideas?

Or this discussion on Google Groups

Or this?

#8

Currently the WebDAV backend has a simple retry mechanism, but unfortunately this error doesn’t trigger it. So an easy solution is to retry on this error, so we don’t need to figure out whether it is a bug in pCloud’s servers or in Golang.

I’ll make this fix in a couple of days…

1 Like
#9

Excellent! I’m curious whether this simple fix will solve the problem.

#10

Just checking to see if there has been an update on this. I’m currently running into the exact same problem while running duplicacy check. I’m also using pcloud.

#11

Sorry I didn’t do it last time. I’ll try to get the fix in before the next CLI release.

1 Like
#12

Could you pls report here what version it is in, I am having same problems.
Thanks

#13

This has been fixed by Retry on "unexpected EOF" errors for the webdav backend. · gilbertchen/duplicacy@f2f07a1 · GitHub, and should be included in the next release (planned for this weekend).

1 Like
#14

I’m getting lots of unexpected EOF reading trailer with 2.6.2. Is this fix included in that release?

#15

Yes, 2.6.2 should include that fix.

#16

If the fix is indeed included in 2.6.2. it did not fix the issue. Here is the log from my most recent check (all lines not including “EOF” have been removed)

2020-09-10 14:02:30.532 TRACE WEBDAV_RETRY URL request 'GET config' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/config: EOF)
2020-09-10 14:02:56.644 TRACE WEBDAV_RETRY URL request 'GET config' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/config: EOF)
2020-09-10 14:03:21.684 TRACE WEBDAV_RETRY URL request 'GET config' returned an error (Get https://webdav.pcloud.com/Backup/Duplicacy/config: EOF)
2020-09-10 14:03:58.263 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/f5/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/f5/: EOF)
2020-09-10 14:05:02.527 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/69/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/69/: EOF)
2020-09-10 14:06:01.071 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:07:02.061 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:08:24.339 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:08:52.209 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:09:55.101 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:11:41.948 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:12:36.769 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/e7/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/e7/: EOF)
2020-09-10 14:13:11.390 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:13:55.784 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:15:17.347 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:16:20.258 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:16:43.138 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/0e/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/0e/: EOF)
2020-09-10 14:17:11.871 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:18:13.655 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:19:59.042 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:21:56.079 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:22:41.063 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:23:29.655 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:23:55.620 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:24:54.621 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/d6/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/d6/: EOF)
2020-09-10 14:25:32.187 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 14:25:57.860 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:26:24.557 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:27:23.019 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:27:45.767 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:29:13.438 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:29:38.416 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:30:22.292 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:31:13.514 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:32:38.218 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:33:50.655 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:36:27.567 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:36:48.669 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:39:45.293 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:40:06.970 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/d9/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/d9/: EOF)
2020-09-10 14:40:30.806 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:41:39.862 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:42:44.773 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/a4/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/a4/: EOF)
2020-09-10 14:43:58.269 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:45:19.598 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:46:54.586 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:47:25.498 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:47:53.017 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/05/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/05/: EOF)
2020-09-10 14:48:15.594 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:48:37.985 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/05/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/05/: EOF)
2020-09-10 14:49:08.951 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:49:33.623 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/05/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/05/: EOF)
2020-09-10 14:51:04.396 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:51:26.495 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:53:07.937 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:54:17.405 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:57:25.686 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 14:58:28.091 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 14:59:08.350 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 14:59:30.201 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:01:18.694 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:01:53.171 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:04:32.611 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 15:05:08.076 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:05:52.194 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:06:30.887 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:06:54.397 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 15:09:11.439 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:09:53.228 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:11:04.089 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:11:50.708 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:12:29.151 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:12:51.981 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:14:41.496 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:16:02.602 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:16:22.755 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/20/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/20/: EOF)
2020-09-10 15:18:03.286 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:19:37.733 WARN WEBDAV_RETRY Retrying on unexpected EOF
2020-09-10 15:20:00.735 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:20:36.406 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/50/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/50/: EOF)
2020-09-10 15:21:59.380 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:22:07.869 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:23:17.888 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:24:11.508 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:25:54.202 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:26:33.120 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:27:45.155 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-10 15:28:16.452 TRACE WEBDAV_RETRY URL request 'PROPFIND chunks/ee/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/chunks/ee/: EOF)
2020-09-10 15:30:03.119 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:30:10.837 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:30:42.540 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:30:54.741 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:32:35.742 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:33:02.933 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/363' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/363: EOF)
2020-09-10 15:33:40.349 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/386' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/386: EOF)
2020-09-10 15:35:09.143 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:36:09.370 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/473' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/473: EOF)
2020-09-10 15:36:10.859 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:36:54.614 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:36:57.833 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:37:24.582 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/522' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/522: EOF)
2020-09-10 15:37:36.904 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/PC_D' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D: EOF)
2020-09-10 15:38:04.585 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/540' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/540: EOF)
2020-09-10 15:38:46.141 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/PC_D/561' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/PC_D/561: EOF)
2020-09-10 15:40:06.868 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:41:13.986 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/44' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/44: EOF)
2020-09-10 15:42:02.866 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/78' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/78: EOF)
2020-09-10 15:42:48.590 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/97' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/97: EOF)
2020-09-10 15:43:31.901 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/101' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/101: EOF)
2020-09-10 15:47:39.714 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/289' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/289: EOF)
2020-09-10 15:48:11.357 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:48:41.931 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/306' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/306: EOF)
2020-09-10 15:49:20.875 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/314' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/314: EOF)
2020-09-10 15:49:54.681 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/314' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/314: EOF)
2020-09-10 15:50:27.291 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/SERVER/324' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER/324: EOF)
2020-09-10 15:50:53.686 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:51:33.454 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:55:11.183 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:56:13.875 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:56:16.513 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/SERVER' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/SERVER: EOF)
2020-09-10 15:56:51.717 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/: EOF)
2020-09-10 15:57:26.049 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS: EOF)
2020-09-10 15:57:57.256 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/40' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/40: EOF)
2020-09-10 16:00:08.800 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS: EOF)
2020-09-10 16:00:32.501 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS: EOF)
#17

The fix is to retry when the unexpected EOF reading trailer error occurs. So from the log it appears the fix was working. Did check abort with an error?

#18

Right. I somehow thought the aim was to get rid of the EOF errors, but, of course that is not something duplicacy can do.

Yes, but unrelated to the EOF issue.

2020-09-10 16:00:08.800 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS: EOF)
2020-09-10 16:00:30.769 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/121' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/121: net/http: TLS handshake timeout)
2020-09-10 16:00:32.501 TRACE WEBDAV_RETRY URL request 'MKCOL snapshots/NAS' returned an error (Mkcol https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS: EOF)
2020-09-10 16:00:49.653 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/143' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/143: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: no such host)
2020-09-10 16:00:50.646 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/143' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/143: dial tcp: lookup webdav.pcloud.com on 1.1.1.1:53: no such host)
2020-09-10 16:00:54.624 WARN DOWNLOAD_FILE_CACHE Failed to add the file snapshots/NAS/143 to the snapshot cache: open /tmp/.duplicacy-web/repositories/localhost/all/.duplicacy/cache/pcloud/snapshots/NAS/143.oncqsdvo.tmp: too many open files
2020-09-10 16:00:54.772 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:00:55.450 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:00:57.157 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:00:59.290 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:01:06.315 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:01:17.421 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:01:48.240 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:02:29.076 TRACE WEBDAV_RETRY URL request 'PROPFIND snapshots/NAS/144' returned an error (Propfind https://webdav.pcloud.com/Backup/Duplicacy/snapshots/NAS/144: dial tcp: lookup webdav.pcloud.com on 8.8.8.8:53: dial udp 8.8.8.8:53: socket: too many open files)
2020-09-10 16:03:47.734 ERROR SNAPSHOT_INFO Failed to get the information on the snapshot NAS at revision 144: Maximum backoff reached
Failed to get the information on the snapshot NAS at revision 144: Maximum backoff reached

Running the check from my Windows machine, it seems to have gone through. It’s a bit cryptic. The web-ui says the job failed but the log ends like this:

2020-09-11 01:10:45.954 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_D at revision 581 exist
2020-09-11 01:10:50.366 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_D at revision 582 exist
2020-09-11 01:10:57.030 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_D at revision 583 exist
2020-09-11 01:11:03.732 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_D at revision 584 exist
2020-09-11 01:11:07.689 INFO SNAPSHOT_VALIDATE Chunk 01d515ceeca2afb1c0190f4762beb9da03e40cf359c005e7f90d4ef4d6abd1ed is confirmed to exist
2020-09-11 01:11:07.784 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_D at revision 585 exist
2020-09-11 01:11:07.982 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ext_MEETG at revision 1 exist
2020-09-11 01:11:08.148 INFO SNAPSHOT_CHECK All chunks referenced by snapshot ext_MEETG at revision 2 exist
2020-09-11 01:11:08.245 INFO SNAPSHOT_CHECK All chunks referenced by snapshot NAS_vmimages at revision 1 exist
2020-09-11 01:11:08.257 INFO SNAPSHOT_CHECK All chunks referenced by snapshot PC_deleted at revision 1 exist
2020-09-11 01:11:08.257 ERROR SNAPSHOT_CHECK 9 chunks have a size of 0
9 chunks have a size of 0

I read this as the check command actually succeeding but it returns an error because nine chunks are corrupted. If this is correct, then I think this could be clarified in the web-ui. Perhaps instead of it just saying “Failed” it could say something like “Check completed successfully but errors were found”.

Error: too many open files
#19

Tried the check command on my windows PC (via the web-ui)

Running check command from C:\Users\Christoph/.duplicacy-web/repositories/localhost/all
Options: [-log -d check -storage pCloud -a -tabular]

and it failed several times. The logfile was rather unhelpful. Here are the last couple of lines (note that I ran this with global option -d but it seems to give me less rather than more details than with -v):

2020-09-14 19:12:15.035 TRACE LIST_FILES Listing chunks/b1/
2020-09-14 19:12:37.330 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:12:59.196 TRACE LIST_FILES Listing chunks/49/
2020-09-14 19:13:06.264 TRACE LIST_FILES Listing chunks/37/
2020-09-14 19:13:29.474 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:14:03.535 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:14:37.695 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:15:16.814 TRACE LIST_FILES Listing chunks/31/
2020-09-14 19:15:46.652 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:16:11.826 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:16:34.385 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-09-14 19:16:57.683 ERROR LIST_FILES Failed to list the directory chunks/31/: unexpected EOF
Failed to list the directory chunks/31/: unexpected EOF

So it retried three times to list the files in /chunks/31/ and then gave up?

I noticed that in other situation it already specifies that it it is not the command that failed but that the result of the command was “Missing chunks”. That’s good!

#20

It seems that after the recent fix, the check command is now working. Now, I’m having trouble with the prune command failing with unexpected EOF reading trailer. I have tried twice now and both times it failed. Could it be that the fix needs to be applied to the prune code too?

2020-10-21 02:01:39.315 INFO SNAPSHOT_DELETE Deleting snapshot ALPHA_C at revision 299
2020-10-21 02:04:31.257 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:07:03.562 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:08:07.800 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:09:36.368 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:12:06.366 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:14:33.294 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:17:38.821 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:22:27.578 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:37:56.102 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:41:24.115 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:45:51.481 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:46:16.618 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:47:04.865 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:47:42.321 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:48:04.849 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 02:48:35.644 ERROR LIST_FILES Failed to list the directory chunks/7b/: unexpected EOF
Failed to list the directory chunks/7b/: unexpected EOF

Edit: on third try, it failed again:

2020-10-21 10:23:19.642 INFO SNAPSHOT_DELETE Deleting snapshot SERVER at revision 488
2020-10-21 10:33:54.555 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:34:58.807 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:35:23.805 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:38:25.054 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:40:16.806 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:42:42.056 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:45:29.478 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:45:54.529 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:46:42.201 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:53:48.812 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:57:19.573 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:58:04.352 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 10:58:44.679 WARN WEBDAV_RETRY Retrying on http: unexpected EOF reading trailer
2020-10-21 11:03:33.926 INFO WEBDAV_RETRY URL request 'MOVE chunks/6b/498ed85825e11805b5c6746a67deda4077d4ce7a1e2e0c437f37ffcfda77fb' returned status code 403
2020-10-21 11:03:53.945 INFO WEBDAV_RETRY URL request 'MOVE chunks/6b/5c6786f5d120106a45d7729d54e706e421917b73fc7fe8bdde25f254051d92' returned status code 403
2020-10-21 11:14:40.911 INFO WEBDAV_RETRY URL request 'MOVE chunks/96/b564116efa0589f931be30fee814b1931055a9e12424862d80d789d0eae555' returned status code 500
2020-10-21 11:15:02.036 INFO WEBDAV_RETRY URL request 'MOVE chunks/96/b564116efa0589f931be30fee814b1931055a9e12424862d80d789d0eae555' returned status code 500
2020-10-21 11:22:44.634 INFO WEBDAV_RETRY URL request 'MOVE chunks/8e/d86ca280a80baa8ae84de42f108d95a591713af0779daa03ab6e39e78733d4' returned status code 500
2020-10-21 11:32:08.986 INFO WEBDAV_RETRY URL request 'MOVE chunks/ff/c14674687c04b2cf9213cea914875f5c1a8e6ae788c28154dd3bc3c97376c8' returned status code 500
2020-10-21 11:39:47.654 INFO WEBDAV_RETRY URL request 'MOVE chunks/31/5bd7699de73b82be1eaec5c13c65aa2ba7c1cf103038420a596ded99c80ed4' returned status code 500
2020-10-21 11:40:08.888 INFO WEBDAV_RETRY URL request 'MOVE chunks/31/5bd7699de73b82be1eaec5c13c65aa2ba7c1cf103038420a596ded99c80ed4' returned status code 500
2020-10-21 11:40:30.350 INFO WEBDAV_RETRY URL request 'MOVE chunks/31/5bd7699de73b82be1eaec5c13c65aa2ba7c1cf103038420a596ded99c80ed4' returned status code 500
2020-10-21 12:23:18.834 INFO WEBDAV_RETRY URL request 'MOVE chunks/73/950f4519ef6903f650deee76fac36df1019831b932852895a55cedbe0d2436' returned status code 500
2020-10-21 12:23:40.159 INFO WEBDAV_RETRY URL request 'MOVE chunks/73/950f4519ef6903f650deee76fac36df1019831b932852895a55cedbe0d2436' returned status code 500
2020-10-21 12:24:03.332 INFO WEBDAV_RETRY URL request 'MOVE chunks/73/4bb9f11e1b0a679727633ea5744b752661cb15fffb98ecf2cbc5f979ba2fd1' returned status code 500
2020-10-21 12:24:07.731 INFO WEBDAV_RETRY URL request 'MOVE chunks/73/4bb9f11e1b0a679727633ea5744b752661cb15fffb98ecf2cbc5f979ba2fd1' returned status code 500
2020-10-21 12:24:10.291 ERROR CHUNK_DELETE Failed to fossilize the chunk 734bb9f11e1b0a679727633ea5744b752661cb15fffb98ecf2cbc5f979ba2fd1: Path does not exist
Failed to fossilize the chunk 734bb9f11e1b0a679727633ea5744b752661cb15fffb98ecf2cbc5f979ba2fd1: Path does not exist