Upload to STORJ S3 sometimes failed with error "You must provide the Content-Length HTTP header"

Bug
#1

Hi! I have a problem with making backup to STORJ storage, configured as S3.
Sometimes creating backup failed with error:

04:42:00 Uploaded chunk 5449 size 4632566, 10.60MB/s 00:06:45 86.1%
04:42:01 Uploaded chunk 5450 size 3492829, 10.60MB/s 00:06:45 86.1%
04:42:01 Uploaded chunk 5451 size 7229636, 10.60MB/s 00:06:44 86.1%
04:42:02 Uploaded chunk 5452 size 1114264, 10.60MB/s 00:06:44 86.1%
04:42:02 Uploaded chunk 5453 size 4016324, 10.60MB/s 00:06:43 86.2%
04:42:04 Uploaded chunk 5454 size 16777216, 10.60MB/s 00:06:42 86.2%
04:42:05 Uploaded chunk 5455 size 11464605, 10.60MB/s 00:06:41 86.2%
04:42:06 Uploaded chunk 5457 size 9344686, 10.60MB/s 00:06:40 86.3%
04:42:06 Failed to upload the chunk f7968537ee52075c7be82b3736578f469fcb5d76930b770f235aa5adca091247: You must provide the Content-Length HTTP header.
04:42:06 Error executing command: exit status 100

If I re-run backup job, it finished successfully.
I also tried to set lower chunk size (2M), but it doesn’t help.
Some time ago I used duplicacy on my Raspberry Pi to make backups of my home lab configs to this storage and never had problems with that. Now I moved to x86 mini PC and using Duplicacy to backup my Proxmox backups to cloud storage and sometimes this fail happens.

What can I do with that?

#2

Bogus error message returned due to intermittent s3 gateway issues? It’s not that duplicacy randomly decides not to provide the content length in the request :slight_smile: You can add -d flag for more verbose logging.

I would switch to native storj backend.

#3

Thanks for reply, I’ll try native
BTW, is there any retry mechanism to re-upload chunk if it is failed?

#4

Without looking in the code and judging by this duplicacy seems to rely on backend adaptation code to do the retries.

Whether S3 does not retry, or does not retry for this specific error code —I’m
not sure. I guess it would make sense to retry at least once even on failures that seemingly are client issues (such as bad api call), because those can be misreported. @gchen?