Corrupted backup - guiding needed

Support
,
#1

Hi,

I’m testing the backup I created for the last month to decide whether I keep or not. The restore did not work . It’s currently running with the command “persist”
The backup is a disk on unraid stored on unraid on the same server on XFS drives. (Don’t worry, that data is also on the cloud and stored somewhere else remote)

2023-11-25 15:59:22.640 WARN DOWNLOAD_RETRY The chunk 5a7ff2ac099f6eec3d656c6e952680ec5302dcbafb288f7912b64a74bd5e1ba5 has a hash id of 5f0dcf8bcf64da49866b29c913439fa27e696331f18d56d450934c94e2a0e29d; retrying
2023-11-25 15:59:22.658 WARN DOWNLOAD_RETRY The chunk 5a7ff2ac099f6eec3d656c6e952680ec5302dcbafb288f7912b64a74bd5e1ba5 has a hash id of 5f0dcf8bcf64da49866b29c913439fa27e696331f18d56d450934c94e2a0e29d; retrying
2023-11-25 15:59:22.676 WARN DOWNLOAD_RETRY The chunk 5a7ff2ac099f6eec3d656c6e952680ec5302dcbafb288f7912b64a74bd5e1ba5 has a hash id of 5f0dcf8bcf64da49866b29c913439fa27e696331f18d56d450934c94e2a0e29d; retrying
2023-11-25 15:59:22.693 ERROR DOWNLOAD_CORRUPTED The chunk 5a7ff2ac099f6eec3d656c6e952680ec5302dcbafb288f7912b64a74bd5e1ba5 has a hash id of 5f0dcf8bcf64da49866b29c913439fa27e696331f18d56d450934c94e2a0e29d

1- I feel there should be a how to on how to handle corrupted chunks
2 - Those are the related post that I found. I commented those and left some question there. I would appreciate your inputs.

If I add the “-chunks” parameter when I run the check after every backup. It will check the chunks only once. So if it becomes corrupted, will I ever get a warning about it? I wouldn’t mind to have a scheduled check from duplicacy if that would validate each chunk each time.

The list of verified chunks will be saved to .duplicacy/cache/storage/verified_chunks and chunks in this file will not be verified again in subsequent runs. You can delete this file to force the check command to verify all chunks.

meaning this can’t be automated easily in duplicacy, am I right?

Thank you for your help and time.

#2

This error indicates that the chunk was already corrupted before it was uploaded to the storage server. Actually the chunk content was somehow modified after the chunk id was calculated but before the content was compressed and encrypted. This must be the case, because if the content were modified after the compression or encryption, the decompressor or the decryptor would have caught the corruption and reported the error.

Yes, you’ll need to manually delete the chunk. Then in the GUI create a new backup, using a different backup in but with the same source directory and the same storage. Run the new backup once. If the files do not change then you might be able to recover the deleted chunk. If that doesn’t work then you’ll have to delete the affected revision (by deleting the corresponding revision file under snapshots/backupid/n in the storage server where n is the revision number) and then run prune with options -exhaustive -exclusive to clean up unreferenced chunks (make sure no other backup jobs are running due to the -exclusive option).

Checking with -chunks after every backup is the only method to guard against this kind of error. There is no need to check a chunk more than once, unless the reliability of the storage server is in question.

#3

Hi @gchen
First, thank you very much for your time.

“-chunks” option

I enabled -chunks for each checks for each storage. What happens if it find an error? will it correct it?

corrupted chunk

I went ahead to delete the faulty chunk., maybe I’m dumb but I can’t find it. I’m able to easily find any chunks, but this one, I can’t find it. To be sure I tried a restore minutes ago, and still got the same log with the faulty chunk.
to be sure, I wasn’t checking in the wrong disk, I searched in the entire “mnt” folder

image
image863×56 11.9 KB

It returned no results. That chunk should be inside /mnt/disk4/. Is that possible that the chunk doesn’t exist?
I also copied the files inside the snapshot folder into a another folder and opened them in notepad++ and search for that chunk in all of the snapshot revision, but it returned no result. This is confusing me.

I’m currently doing another backup as you suggested of the same data in the same storage. I’ll see where this goes

If I were to delete a snapshot revision, how would I know which revision to erase? Only the revision I tried? Let say I was trying to restore revision50. Can the problematic chunk be referenced in the second snapshot (#19), if so how do I know which revision to delete considering I couldn’t find the chunk name in the snapshot files.
I have revision, 1, 19, 34, 40, 43, 46, 49, 50

EDIT: specified what file I was opening in notepad++

#4

@gchen

Creating a new backup in the same storage, result

I did create another backup in the same storage. Then I tried to restore the new backup and I did get the same error. I retried the original backup and it also failed with the same error. I would guess that my problem relies on the fact that I can’t find that corrupted chunk to delete it.

Cant find the corrupted chunk

I did the check with the -chunks option all my other backup. I did get another corrupted chunk from another backup

corrupted chunk from another backup

2023-12-08 19:16:49.516 WARN DOWNLOAD_RETRY The chunk 5542e3e27cdaabb02097eab297cb069d81f8b189c1035b2032c4418dd738948f has a hash id of 60264865c587dace448994a163d32425d36fa2fb5beea717e092beb26fbd5e94; retrying
2023-12-08 19:16:49.533 WARN DOWNLOAD_RETRY The chunk 5542e3e27cdaabb02097eab297cb069d81f8b189c1035b2032c4418dd738948f has a hash id of 60264865c587dace448994a163d32425d36fa2fb5beea717e092beb26fbd5e94; retrying
2023-12-08 19:16:49.551 WARN DOWNLOAD_RETRY The chunk 5542e3e27cdaabb02097eab297cb069d81f8b189c1035b2032c4418dd738948f has a hash id of 60264865c587dace448994a163d32425d36fa2fb5beea717e092beb26fbd5e94; retrying
2023-12-08 19:16:49.570 ERROR DOWNLOAD_CORRUPTED The chunk 5542e3e27cdaabb02097eab297cb069d81f8b189c1035b2032c4418dd738948f has a hash id of 60264865c587dace448994a163d32425d36fa2fb5beea717e092beb26fbd5e94

I also can’t find this 2nd corrupted chunk.

I would think that my problem is that I can’t find the chunk.

I went into the directory “chunks”, took note of one chunk filename randomly. Ran the same “find” command and it returned it’s path. So I think I’m using this command properly

image
image912×61 7.34 KB

⤷testing the find command with a chunk that I took randomly⤴

I’m puzzled about what to do. Why can’t I find the corrupted chunk that the log reports?

emailreport

Is there a way to get error only on the email report? the check -chunks create the longest email I have ever seen, but even then, it doesn’t contain the complete log (understandably). summary + errors as the body of the email and complete log as an attachment would be ideal.

#5

You shall delete the corrupted chunk and only then run another backup to the same storage, if the hope is to get the chunk recreated.

You can run check with -persist option to list all corrupted chunks.

Chunk file named abcdef… will be under the path chunks/ab/cdef…

#6

OMG! thank you.
I didn’t know that the first 2 char were the name of the folder where they we stored. So I only had to remove the first to characters to find the chunk! Ok, I just found it. Ideleted the erroenous chunk, created a new backup using the same storage. Will try once more when that new backup is finished running

Thanks

1 Like
#7

Hi, thank you both

I was able to correct the corrupted chunks. I deleted the corrupted chunks using check -chunks -persist
Then I deleted the chunks, the first 2 char are the name of the folder containing the chunk
Then I created a new backup of the same data as the original backup, in the same storage.
Then I validated that the chunks are all valid with a check -chunk -persis
Then I was able to restore. Success!

I still have 3 questions

  • How would I know what revision actually owns the corrupted chunk if I wanted to delete the snapshots that contains it?
  • If I run check with -chunks -persist, will it return a failure if it encouter an error even if it continues after that error? I’m wondering If I should leave that on or not
  • I now run check with -chunks everytime. If it encounter a corruption error, what happens? Do I have to do the same procedure again? Or is there a way to automate the repair?

Thank you for your time

#8

after deleing the bad chunks you can run check -persist -a, it shall report which revisions are bad. Then you can delete those revisions, and follow up with prune -exhaustive -a to delete any orphaned chunks.

Check -chunks will check each chunk only once. It keeps a list of chunks that had already been checked and won’t check those again. The primary purpose of this is to ensure that the chunk was delivered to storage successfully. It’s then up to the storage to keep it valid.