Hi,
I am copying 1 storage location from Wasabi US West to AP Northeast. The problem I have is that I am having the job fail multiple times, but logs always suggest some sort of issues copying from the source. Sometimes it appears to be an unexpected “unexpected EOF” and sometimes it is “net/http: TLS handshake timeout”. It always seems to be an issue with the source. If I restart the job, it all seems to continue ok and it keeps going until it fails again, and I need to restart it.
I few details of my setup :-
- Duplicacy Web GUI 1.5.0
- Running on a QNAP box.
- There is no sign of problem as recognised by Wasabi
- There is no sign of any problems with the internet connection my end, or in fact between my location and either of the Wasabi destinations which I have got very finely monitored which would show up any issue that is longer than a couple of seconds.
I can’t rule out that it is some issue on the Wasabi end that is very intermittent or transient. But the fact that I 1) Wasabi are not reporting any issues, and 2) that I can just restart, to me suggests that there might be improvements we can do with Duplicacy to improve its robustness so it can handle these sorts of transient issues??
Any insights to make this more reliable would be appreciated, as I have concerns this sort of issue would significantly impact my ability to do fast restores if I need them in the event of a failure.
See below for the log messages.
Thanks heaps.
Cheers
Matthew
2021-07-09 05:27:51.486 INFO COPY_PROGRESS Copied chunk 8a001c62ea9afc355d23ea1fb0b255727cbb9fa42ee990a5b6039bf990199bcf (8138/36420) 2.12MB/s 18:12:49 22.3%
2021-07-09 05:27:54.317 INFO COPY_PROGRESS Copied chunk 7829c13878a375794b33cb2bd88664202b5ddf3ed7b5be879465b275452985d8 (8156/36420) 2.12MB/s 18:09:53 22.4%
2021-07-09 05:27:54.941 INFO COPY_PROGRESS Copied chunk 5cb5b0499b499c3dfef69bbe88ab34168168607eb1ca65f4adc1774a233b7f54 (8157/36420) 2.12MB/s 18:09:45 22.4%
2021-07-09 05:27:56.395 WARN DOWNLOAD_RETRY Failed to download the chunk f69d097a6a0c21168e786842a5229e578346c8747f9131f671882febd6d6598b: unexpected EOF; retrying
2021-07-09 05:27:57.542 INFO COPY_PROGRESS Copied chunk 46f0cb99ce009b04fa2ef774ec4ad557947f602ebad6668ff349433ea1bbf15c (8158/36420) 2.12MB/s 18:09:43 22.4%
2021-07-09 05:27:57.743 INFO COPY_PROGRESS Copied chunk f1152301f3c4a99bb1d71b2ceb7f08c242030fff65e1c9da8972983e3fbd706e (8155/36420) 2.12MB/s 18:10:15 22.4%
2021-07-09 05:27:58.559 WARN DOWNLOAD_RETRY Failed to download the chunk f69d097a6a0c21168e786842a5229e578346c8747f9131f671882febd6d6598b: unexpected EOF; retrying
2021-07-09 05:27:59.719 WARN DOWNLOAD_RETRY Failed to download the chunk f69d097a6a0c21168e786842a5229e578346c8747f9131f671882febd6d6598b: unexpected EOF; retrying
2021-07-09 05:28:00.320 INFO COPY_PROGRESS Copied chunk d7e1dc4123a385a25f9657ad717aad15ce1022e3a7ff0f2edd572288ce4cd701 (8159/36420) 2.12MB/s 18:09:43 22.4%
2021-07-09 05:28:00.779 ERROR DOWNLOAD_CHUNK Failed to download the chunk f69d097a6a0c21168e786842a5229e578346c8747f9131f671882febd6d6598b: unexpected EOF
Failed to download the chunk f69d097a6a0c21168e786842a5229e578346c8747f9131f671882febd6d6598b: unexpected EOF
2021-07-09 06:20:37.473 INFO COPY_PROGRESS Copied chunk c6c725aec8064260945696ecdb04c94a74a9145ac5df664e4fd20aabc9e58094 (1225/28261) 2.21MB/s 16:13:53 4.3%
2021-07-09 06:20:45.163 INFO COPY_PROGRESS Copied chunk f7b25c0390a39014ec846188ae40778b1eca46cc211b99a5ddbc659109cf81d5 (1221/28261) 2.21MB/s 16:20:03 4.3%
2021-07-09 06:20:45.823 INFO COPY_PROGRESS Copied chunk 9b88aa39df2a010c8f3e4b3cf05125e1d02a9adba23c2bda2d7aff6bd98e572d (1220/28261) 2.22MB/s 16:21:08 4.3%
2021-07-09 06:21:11.612 ERROR DOWNLOAD_CHUNK Failed to download the chunk df53ee1baf09b9fb368ddeaf3dfcf02ca26546741b981e4ac5c4d25fad988c84: RequestError: send request failed
caused by: Get https://msduplicacy.s3.us-west-1.wasabisys.com/documents/chunks/df/53ee1baf09b9fb368ddeaf3dfcf02ca26546741b981e4ac5c4d25fad988c84: net/http: TLS handshake timeout
Failed to download the chunk df53ee1baf09b9fb368ddeaf3dfcf02ca26546741b981e4ac5c4d25fad988c84: RequestError: send request failed
2021-07-08 23:02:51.526 INFO COPY_PROGRESS Copied chunk 661005c6e86c18b5f2a548804bb7628bd4eee692d4aaa82bcd9b046503bba5ce (592/38167) 2.10MB/s 1 day 01:39:43 1.6%
2021-07-08 23:02:52.027 INFO COPY_PROGRESS Copied chunk 3fa77ad8eeb83dcfbd817467b6a144947a9868f90147a84bfccf19a929549d43 (596/38167) 2.11MB/s 1 day 01:29:45 1.6%
2021-07-08 23:03:21.936 ERROR DOWNLOAD_CHUNK Failed to download the chunk 33899d676228af5c438e7e802910f0426178ffb1b3084c62993b35d1c107015b: RequestError: send request failed
caused by: Get https://msduplicacy.s3.us-west-1.wasabisys.com/documents/chunks/33/899d676228af5c438e7e802910f0426178ffb1b3084c62993b35d1c107015b: net/http: TLS handshake timeout
Failed to download the chunk 33899d676228af5c438e7e802910f0426178ffb1b3084c62993b35d1c107015b: RequestError: send request failed
2021-07-08 22:22:03.381 INFO COPY_PROGRESS Copied chunk 4e72a421d06d14cef6e4bb8762814d160a4d57ead07c449a90c627e831c061f8 (1260/39443) 2.14MB/s 1 day 00:14:45 3.2%
2021-07-08 22:22:03.977 INFO COPY_PROGRESS Copied chunk 2440688da2606ffbce7f82165319986ce5984650fa9f2154de9d043096fd7abe (1275/39443) 2.14MB/s 23:57:22 3.2%
2021-07-08 22:22:05.718 WARN DOWNLOAD_RETRY Failed to download the chunk 66f3148a6308656359080270a8e7098da3f30975721fe8b8f1820dc7545a48c3: unexpected EOF; retrying
2021-07-08 22:22:12.000 INFO COPY_PROGRESS Copied chunk 086e7cf612cc44d65f778801197700d26b84f26f4fbd2fa68b6bf2a5c90e48a0 (1274/39443) 2.14MB/s 1 day 00:02:32 3.2%
2021-07-08 22:22:13.161 INFO COPY_PROGRESS Copied chunk 20b84f6e3ebf298ce4f492355e9339806e5e2af6ab8ed305844f7c6f26a609bd (1276/39443) 2.14MB/s 1 day 00:00:47 3.2%
2021-07-08 22:22:20.992 INFO COPY_PROGRESS Copied chunk 7dd3e084c60ec16f83f19ae89d7579af0c90f7266f212682aeb15ef04201957c (1273/39443) 2.14MB/s 1 day 00:08:12 3.2%
2021-07-08 22:22:21.107 WARN DOWNLOAD_RETRY Failed to download the chunk 66f3148a6308656359080270a8e7098da3f30975721fe8b8f1820dc7545a48c3: unexpected EOF; retrying
2021-07-08 22:22:26.601 WARN DOWNLOAD_RETRY Failed to download the chunk 66f3148a6308656359080270a8e7098da3f30975721fe8b8f1820dc7545a48c3: unexpected EOF; retrying
2021-07-08 22:22:29.952 ERROR DOWNLOAD_CHUNK Failed to download the chunk 66f3148a6308656359080270a8e7098da3f30975721fe8b8f1820dc7545a48c3: unexpected EOF
Failed to download the chunk 66f3148a6308656359080270a8e7098da3f30975721fe8b8f1820dc7545a48c3: unexpected EOF
2021-07-08 21:22:15.945 INFO COPY_PROGRESS Copied chunk a0b9938bf847d830ad35704f477cc49b04cf40e7ca72eb4c9b11130d2accb174 (865/40310) 2.02MB/s 1 day 02:03:06 2.1%
2021-07-08 21:22:16.200 INFO COPY_PROGRESS Copied chunk bd7c3f5a0a0c3ac6d1e7847858ef938d8535f90637ec37a814f698bb42175fe0 (866/40310) 2.02MB/s 1 day 02:01:27 2.1%
2021-07-08 21:22:16.565 INFO COPY_PROGRESS Copied chunk c84a50926fd75379ef8414d600733b1b09dc11366beb6c38ef2cd79f7e76a54a (858/40310) 2.02MB/s 1 day 02:16:36 2.1%
2021-07-08 21:22:16.906 INFO COPY_PROGRESS Copied chunk 6d3af526b4dc69cc1c32d32350baeca12ae13b769d747dfd3ee8e19304b503a7 (857/40310) 2.02MB/s 1 day 02:18:45 2.1%
2021-07-08 21:22:53.102 ERROR DOWNLOAD_CHUNK Failed to download the chunk 65b26b2193e55c28c43fe63031bba81a2cc352f522c35aa67e2865252737f2a8: RequestError: send request failed
caused by: Get https://msduplicacy.s3.us-west-1.wasabisys.com/documents/chunks/65/b26b2193e55c28c43fe63031bba81a2cc352f522c35aa67e2865252737f2a8: net/http: TLS handshake timeout
Failed to download the chunk 65b26b2193e55c28c43fe63031bba81a2cc352f522c35aa67e2865252737f2a8: RequestError: send request failed