Hi,
Could you please explain how duplicacy handles interrupted upload to cloud storage? Are there any risks of data corruption / chunk missing in such situation?
Thanks.
For file-based storage or SFTP storage, a chunk is always uploaded to a temporary file – only after the upload is completed will the temporary file be renamed into the real destination file, so no data corruption can happen to these storages.
For http-based cloud storages, we don’t upload to temporary files first, to avoid the api call that renames the file. Instead, we make the assumption that the cloud server will not save the partial file if the upload is interrupted. Most of these cloud storage takes a content-length header when accepting a file upload, so it is reasonable to assume that this header will be honored even if they don’t handle an interrupted tcp connection properly. If you’re really concerned about this issue, you should run the check command with the -files option to perform a full verification after a backup is completed. Duplicacy guarantees file immutability by design, so once a backup passes the check it will never need to be checked again.
1 Like
Thank you for your reply and detailed answer.
After the interruption, I need to run duplicacy again to backup. So, how does duplicacy deal with resuming?
If that is the initial backup, Duplicacy will skip all chunks that have been uploaded but it still needs to scan all files. For a large repository it can take quite a while to get to where it was interrupted. I’m working on a new feature that will skip files that have already been uploaded in order to get to the resuming point almost instantly. This feature should be available by the end of this week.
If that is not the initial backup, Duplicacy will compare files against the last completed backup by timestamp and size, and skip files with the same timestamp and size. Generally this will be very fast so it will reach the resuming point fairly quickly.