Duplicacy not retrying when getting `Failed to download the chunk`

Support
#1

When running a duplicacy check, duplicacy doesn’t retry on any download errors (eg 4xx/5xx), even if they’re transient and will immediately exit on the first error it encounters.

Is there a way to make duplicacy retry automatically on these transient chunk download errors?

...
Verified chunk 6711c4559cf863f123b51ab5f8438b7a9405737b98a6ffc2b0c55bdd1c41ae39 (7856/1976783), 11.25MB/s 9 days 23:54:58 0.4%
Prefetching chunk ee3c1b65f32eba2d7db4dc9088630e44f5d9a5bbca2bd081be152e80a897fc79
Chunk ee3c1b65f32eba2d7db4dc9088630e44f5d9a5bbca2bd081be152e80a897fc79 has been downloaded
Failed to download the chunk 360d0f6deefc71fff0c9ba673c2e2b8fcbe43860e9a2d5c9bb3c3300e2baf4bc: googleapi: got HTTP response code 400 with body: {"error":{"errors":[{"domain":"global","reason":"failedPrecondition","message":"Bad Request"}],"code":400,"message":"Bad Request"}}
goroutine 147 [running]:
runtime/debug.Stack(0xc0000bc0c0, 0x0, 0xc052487700)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/gilbertchen/duplicacy/src.CatchLogException()
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:227 +0x88
panic(0xf49d60, 0xc00294f3e0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/gilbertchen/duplicacy/src.logf(0x2, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc052487cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:180 +0x8f9
github.com/gilbertchen/duplicacy/src.LOG_WERROR(0xc02ab4e300, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc052487cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:114 +0xe6
github.com/gilbertchen/duplicacy/src.(*ChunkDownloader).Download(0xc0004b8000, 0x2, 0x0, 0x221e, 0xc00ad91220, 0x20, 0x0, 0x1, 0x1b02c01)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:453 +0x400
github.com/gilbertchen/duplicacy/src.CreateChunkDownloader.func1(0xc0004b8000, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:87 +0xb7
created by github.com/gilbertchen/duplicacy/src.CreateChunkDownloader
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:82 +0x1e6
Added 7856 chunks to the list of verified chunks

I’ve also gotten errors like Failed to download the chunk 18027d9645f322f7a920cfddc51e23412edd5e1a3578c592b41d37aacddd503e: stream error: stream ID 11337; INTERNAL_ERROR, but was running without debugging.

#2

Which storage backend is this? The Google Drive backend can retry on this 400 failedPrecondition error but the Google Cloud Storage backend can’t.

#3

Sorry, forgot to mention that. This is using the Google Drive backend.

In all of the cases, I can download the chunks from the Google Drive WebUI without any issues.

#4 
Prefetching chunk a2a532b1e9fedc6e27b084d68783e9abe9f9dd16c95e8ebd545a118d4a593c28
Chunk 18e932dc80676f847c32f4cb5a8bde1b3527f87ef581a849040359cc520f3d42 has been downloaded
Chunk 0b1052d200d2a020270b736a6a7d65a5d782422b75b71b1025023e60531ed039 has been downloaded
Chunk f043d0a236c3706163b7d7d4921fc1ecafcea19178aeaaa22ebb2671267e545f has been downloaded
Chunk 421c98ba6e8ddb05a89d8d029bb4addd2796ad9e1606d3ee92880abb059ee34e has been downloaded
Chunk 3e264d8a539748e71bf925d7847a7408b13d914bb08594611fa639794f6c4e3d has been downloaded
Chunk ed4f993cca36a48866eb13eba23fee980abaa41c455e083a2b86fe89a03cd2b8 has been downloaded
Chunk a2a532b1e9fedc6e27b084d68783e9abe9f9dd16c95e8ebd545a118d4a593c28 has been downloaded
[4] HTTP status code 503; retrying after 3.20 seconds (backoff: 2, attempts: 1)
[4] HTTP status code 503; retrying after 3.75 seconds (backoff: 4, attempts: 2)
[4] HTTP status code 503; retrying after 14.33 seconds (backoff: 8, attempts: 3)
[4] HTTP status code 503; retrying after 12.13 seconds (backoff: 16, attempts: 4)
Failed to download the chunk ff3fe706421ac2971ea03d76fc4f20f46022f07dc2b585550374a913044c48f6: stream error: stream ID 15057; INTERNAL_ERROR
goroutine 149 [running]:
runtime/debug.Stack(0xc0000f6060, 0x0, 0xc001943700)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/gilbertchen/duplicacy/src.CatchLogException()
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:227 +0x88
panic(0xf49d60, 0xc00e1a1e60)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/gilbertchen/duplicacy/src.logf(0x2, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc001943cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:180 +0x8f9
github.com/gilbertchen/duplicacy/src.LOG_WERROR(0xc07433ad00, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc001943cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:114 +0xe6
github.com/gilbertchen/duplicacy/src.(*ChunkDownloader).Download(0xc0001242c0, 0x4, 0x0, 0xd13, 0xc0571b17a0, 0x20, 0x0, 0x1, 0x1)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:453 +0x400
github.com/gilbertchen/duplicacy/src.CreateChunkDownloader.func1(0xc0001242c0, 0x4)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:87 +0xb7
created by github.com/gilbertchen/duplicacy/src.CreateChunkDownloader
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:82 +0x1e6
Added 2499 chunks to the list of verified chunks

Is there any way to configure the amount of retries to work around Google Drive stability issues?

#5

Looks like disconnections are also not being retried.

Prefetching chunk 88f02c04d0e41b0c591166ecfe9267fb60cbdf20b194f3f1d2d9a98cfc83e5bf
Chunk c4c2f11a4cf744c6274ae7f6b3da2d572609efbab5e35ce0ff4090e4000f1c97 has been downloaded
Verified chunk c4c2f11a4cf744c6274ae7f6b3da2d572609efbab5e35ce0ff4090e4000f1c97 (1829/1871825), 2.87MB/s 36 days 13:34:37 0.1%
Prefetching chunk 33084ebd3f40a92409dbcd78ba4fbe7a0e21a85db2aa5af59fc4ac0514533bb9
Failed to download the chunk 88f02c04d0e41b0c591166ecfe9267fb60cbdf20b194f3f1d2d9a98cfc83e5bf: http2: server sent GOAWAY and closed the connection; LastStreamID=11011, ErrCode=NO_ERROR, debug="server_shutting_down"
goroutine 145 [running]:
runtime/debug.Stack(0xc0000bc0c0, 0x0, 0xc0001a7700)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x22
github.com/gilbertchen/duplicacy/src.CatchLogException()
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:227 +0x88
panic(0xf49d60, 0xc00b6ac3f0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/gilbertchen/duplicacy/src.logf(0x2, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc0001a7cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:180 +0x8f9
github.com/gilbertchen/duplicacy/src.LOG_WERROR(0xc024aa9800, 0x1027d12, 0xe, 0x1040875, 0x23, 0xc0001a7cf0, 0x2, 0x2)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_log.go:114 +0xe6
github.com/gilbertchen/duplicacy/src.(*ChunkDownloader).Download(0xc000756000, 0x0, 0x0, 0xa75, 0xc00eda6360, 0x20, 0x0, 0x1, 0x52454320444e4501)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:453 +0x400
github.com/gilbertchen/duplicacy/src.CreateChunkDownloader.func1(0xc000756000, 0x0)
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:87 +0xb7
created by github.com/gilbertchen/duplicacy/src.CreateChunkDownloader
        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:82 +0x1e6
Added 1829 chunks to the list of verified chunks