Initial backup error - 500 Internal Server Error

UX
#1

The initial backup is not able to terminate.

Here is a snipped of the output, with verbosity and debug enabled:

2016-08-13 09:40:02.836 DEBUG CHUNK_EXIST Skipped chunk adb30f485e6072fffeae32d4cecc5a7008cefcf0673642516f81cd3c45a0a4ed in the storage
2016-08-13 09:40:02.836 INFO UPLOAD_PROGRESS Skipped chunk 9351 size 63137, 194KB/s 16:20:18 6.0%
2016-08-13 09:40:03.323 DEBUG CHUNK_DUPLICATE Chunk 5d85c6e26bdd1cb3070a8cba8c4bf672769d5198e9ad5481376db012ab752d6d already exists
2016-08-13 09:40:03.323 DEBUG CHUNK_EXIST Skipped chunk 5d85c6e26bdd1cb3070a8cba8c4bf672769d5198e9ad5481376db012ab752d6d in the storage
2016-08-13 09:40:03.323 INFO UPLOAD_PROGRESS Skipped chunk 9352 size 137150, 194KB/s 16:20:22 6.0%
2016-08-13 09:40:03.951 DEBUG CHUNK_DUPLICATE Chunk ffee93fbfed12b244c2b7daaa45da0b2c45d5e12917fe76ca719e0dfea833139 already exists
2016-08-13 09:40:03.951 DEBUG CHUNK_EXIST Skipped chunk ffee93fbfed12b244c2b7daaa45da0b2c45d5e12917fe76ca719e0dfea833139 in the storage
2016-08-13 09:40:03.951 INFO UPLOAD_PROGRESS Skipped chunk 9353 size 28313, 194KB/s 16:20:20 6.0%
2016-08-13 09:40:09.293 ERROR UPLOAD_CHUNK Failed to find the path for the chunk c929d28983e4d11dde1116de9d3eb1167af961bd6db25252a9c2ab5c178dbb82: 500 Internal Server Error
{2 UPLOAD_CHUNK Failed to find the path for the chunk c929d28983e4d11dde1116de9d3eb1167af961bd6db25252a9c2ab5c178dbb82: 500 Internal Server Error}

#2

which storage are you using?

#3

Amazon S3

#4

I’m not sure of the exact cause – it could have been a transient network error; or the retry count (of 5) is not enough. However, I do notice that chunk sizes are small. Using a bigger average size (for example the default 4M) will definitely help and improve the upload speed significantly.

#5

I restarted the backup multiple times and the same issue occurs each time.

I’m doing a backup of a mail directory for multiple users. It’s around 40K files with an average size of 175KB. in this scenario, a chunk size of 4MB would be good?

#6

Duplicacy actually combines files first before splitting them into chunks, so it doesn’t matter how small the files are. By using a larger chunk size you will reduce the API request rate and that is why I think it would help.

#7

Thanks for the info about chunk size.

So I tried again by re-initializing the storage with the default chunk size. Although the backup went further, it still failing:

2016-08-15 06:00:00.444 INFO UPLOAD_PROGRESS Uploaded chunk 1511 size 2933937, 6.44MB/s 00:11:56 61.0%
2016-08-15 06:00:00.459 INFO PACK_END Packed var/vmail/.Sent/cur/1471033506.M213103P11619.mail-srv,S=500826,W=507363:2,S (500826)
2016-08-15 06:00:00.460 TRACE PACK_START Packing var/vmail/.Sent/cur/1471033509.M485495P11619.mail-srv,S=3023058,W=3062341:2,S
2016-08-15 06:00:07.594 ERROR UPLOAD_CHUNK Failed to upload the chunk d1e8c6b4aa9f16f6173413a0e079697f5ae9ba5bc2b48c3ed7f032b980ba25e1: We encountered an internal error. Please try again.
{2 UPLOAD_CHUNK Failed to upload the chunk d1e8c6b4aa9f16f6173413a0e079697f5ae9ba5bc2b48c3ed7f032b980ba25e1: We encountered an internal error. Please try again.}r

#8

After few tries, the backup finally terminated…

#9

I uploaded a new version with an improved retry mechanism for S3.

#10

Thank you, I will try it.