Duplicacy shall retry at least once on broken S3 connection

saspus · 7 June 2023 17:41

Please describe what you are doing to trigger the bug:
Backup to flaky S3 endpoint over flaky connection

Please describe what you expect to happen (but doesn’t):
I expect duplciacy to retry S3 request if it fails spuriously and continue backup

Please describe what actually happens (the wrong behaviour):
Duplicacy aborts backup with the message similar to:

Failed to upload the chunk 25f315347a7b6bf50f5fd363729945c4785717b77cf10d91003c51fd078a2687: RequestError: send request failed caused by: Put "https://duplicacy.gateway.storjshare.io/duplicacy/chunks/25/f315347a7b6bf50f5fd363729945c4785717b77cf10d91003c51fd078a2687": read tcp 10.0.17.48:31127->185.244.226.3:443: read: connection reset by peer

gchen · 9 June 2023 01:48

I think the AWS s3 client already has a built-in retry mechanism:

github.com

aws/aws-sdk-go/blob/a5bef1542952d979233bd92569aa09eeb7686c6a/aws/client/client.go#L66-L71

    
      
          	default:
          		maxRetries := aws.IntValue(cfg.MaxRetries)
          		if cfg.MaxRetries == nil || maxRetries == aws.UseServiceDefaultRetries {
          			maxRetries = DefaultRetryerMaxNumRetries
          		}
          		svc.Retryer = DefaultRetryer{NumMaxRetries: maxRetries}

Maybe the default 3 retries aren’t enough for Storj. Then we need to add a method to configure the number of retries.

saspus · 9 June 2023 02:01

If I remember correctly there were no other logs indicating that retries have happened, but I’ll double check later tonight and get back.

gchen · 9 June 2023 03:16

Retries happen inside the AWS client (github.com/aws/aws-sdk-go), so there won’t be any Duplicacy log showing that.

saspus · 9 June 2023 03:54

Oh yes, you are right. My bad.

If three times it failed it’s unlikely it will succeed the fourth.