ERROR SNAPSHOT_CHUNK Hash has garbled characters and also message authentication failed

Support
, , ,
#1

There are some threads on recovering from bad chunks that I think I can follow to get back into a good state but I’m posting this more for the investigation and understanding of what happened because it seems really weird.

Yesterday while recovering from some filesystem trouble I was able to successfully run my Backup schedule (7 parallel backups, 1 serial backup, and then 1 serial check all local storage) and also my Copy schedule (copy local to B2 and then check both in parallel) and everything succeeded. Everything seemed to be in good shape but oernight when the jobs ran though I had some failures.

The copy errored with

2022-10-03 11:57:42.054 ERROR DOWNLOAD_DECRYPT Failed to decrypt the chunk eef32bb9cd06c71a57e1d3590979776bdaf44a558fc733c66d006432498f46f9: cipher: message authentication failed
Failed to decrypt the chunk eef32bb9cd06c71a57e1d3590979776bdaf44a558fc733c66d006432498f46f9: cipher: message authentication failed

The Backup schedule failed at the check with a really lengthy string with garbled characters:

2022-10-03 09:13:11.034 ERROR SNAPSHOT_CHUNK Failed to load chunks for snapshot unraid_paul_archive at revision 1059: The chunk hash e3dae49874de225a1692e207c6d301a1f5b463c1a2bc8da6d2d015a01ac472605e367ded223e5bf03dd603fef24c6c0f692bcee8b27d3123f28ca619acb6209f8251df088add22f470fd05c4c592e2bf31fe6d4eddad541f507e000fee8db74dcad581b6197dd90aac36db4205d05eb07db53c858d6a3ec29e9cecd901f6c797546ca33f22ad77cf12fc0a5f3155ad5dedca7892d7e424a5014f84d85cafe8a617354e46cd02397ce2b85374c6cf4eb.........{omitted for brevity}...........Ql>+Œh›k½Aû«Ùƒy!~Íà[DzL‰½©&Ãó±ê>ƒäøadóå¸ÑVÕ„™â[²ª������������������������������������������������������������������������������������������������ ����������������������÷>SIœJ›`^¦]±`ØôaÕXyä£;%½ï8à
bufio.Scanner: token too long

But the Check against the same local storage that ran after the Copy job failed succeeded with no problems. The Check that ran against B2 after the Copy job failed also had the same failure with garbled characters.

I’m not quite sure what’s going on, any thoughts?

#2

This error can be caused by a corrupted chunk file in the local storage.

This error means the metadata chunk stored in the local cache (~.duplicacy/cache) is corrupted. You can delete the cache and try again but I guess there is still some problem with the file system that you need to fix first.

#3

The Backup just ran on schedule and the local Check succeeded but then when the Copy ran on schedule it failed (haven’t fixed the chunk yet) but then the same exact local Check that just passed minutes earlier failed. Could this also be the result of a bad cache or is it indicitive of something else?

#4

Check without the --chunks option will only check if every chunk exists, so it won’t detect a corrupted chunk.

The reason why the second Check failed could be that the chunk downloaded by the first Check became corrupted. It could be something else, but I would run a disk check first.

#5

The same Check passed again without me changing anything which is weird. I’ll look into a disk/filesystem check. In the meantime I’ll follow the threads I linked to recover from the message authentication failed error. Thanks for explaining!

#6

If your underlying media is failing, it is entirely possible to have reads that sometimes work, and sometimes return garbled mess. Chasing those kinds of errors from the application side is not productive, you need to make sure you solve the underlying problem first (repair/replace, most likely).

1 Like
#7

Is there a best-practices way to check the underlying media?

#8

chkdsk X: /r if you’re on Windows. On Linux, you probably would want to run something like smartctl for testing, or oldschool badblocks.

#9

An extended smartctl was clean and I seem to be in a good state now. Thanks!

#10

All these tools don’t check for data integrity. So they may be all passing and data may still be corrupted. Just to keep in mind.

#11

Ah that’s a good point. Would a check with -chunks be the way to verify? I’m not sure if XFS has the right info to be able to check.

#12

Yes, check -chunks should verify chunks consistency. You might need to delete the .duplicacy/cache/storage/verified_chunks file, because by default duplicacy checks each chunk only once; that file contains list of chunks it has already checked and will therefore skip on subsequent checks

xfs does not keep data checksums and cannot detect or recover from bit-rot.

#13

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.