Chunks - Checksum Mismatch on Backup

Support
#1

Good morning! I’ve searched through several posts but havent found an exact match to my situation.

ISSUE:
Quarterly disk scrub found checksum mismatch with several chunks on the backup. The same chunks have shown up in logs during test run in November and now again this month. Check command shows all snapshots and revisions exist (~15TB). shown below:

2024-12-18 07:00:02.164 INFO SNAPSHOT_CHECK Listing all chunks
2024-12-18 07:01:46.367 INFO SNAPSHOT_CHECK 5 snapshots and 79 revisions
2024-12-18 07:01:46.486 INFO SNAPSHOT_CHECK Total chunk size is 14255G in 3003109 chunks
2024-12-18 07:01:47.720 INFO SNAPSHOT_CHECK All chunks referenced by snapshot personal at revis…

I have not ran the Prune command to cleanup any snapshots.

Check command finds all chunks although I’ve not tried to do a full restore and would believe several files would fail based on the checksum mismatch.

Is there a way to determine which backup is effected by the corrurpted files?
What is the proper method to remove the bad chunks and backup the missing data?
I havent ran check command with -files or -chunks <<< would this be my best option for finding the corrupted files? Before attempting and regretting a mistake, I thought I’d ask questions first.

SETUP:
Duplicacy (Web-UI) runninng on a Synololgy DS1819; backing up to a Synology DS1815 on local network

DSM LOG:

Warning System 12/17/24 5:07 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/fc/351ed75115572c57b9c215328c599c16a755d492eaffe450b102bdef72d29e].
Warning System 12/17/24 4:59 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/94/d27bee28af153f0f08336e86f64c2dcda0589acc44cddee9a5e4df86633c97].
Warning System 12/17/24 4:05 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/80/39f04b76a182af38fc6cfa7f3d1fb70c5f7846afd78d9ebf1268f58e967df8].
Warning System 12/17/24 3:38 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/47/d1e04e8973e22d6ca0b5d2020f11bfa6c66e7d6e6a1e769b458eb3336d70ce].
Warning System 12/17/24 3:34 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/1d/2da395baf7a0fc9d5641d5045c6c289301be5bcc5bb0ea826ff902211608e0].
Warning System 12/17/24 3:33 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/a9/a42822f836dd0cddf91692e7bf03aa583eb5c0cdecf5ffdf10f90f0061e039].
Warning System 12/17/24 3:31 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/5d/cd35d95dcd5aa6079987605f76ed7731aaad32e673236faf03963729b89079].
Warning System 12/17/24 3:22 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/81/7be8627450de2b66fa89e9a4309497c4cbbb3304b5de19249f18765c425dad].
Warning System 12/17/24 3:15 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/a1/7059369a58a8193255bfb2055f243c673453926008da9bedcdc7d6be3e3b03].
Warning System 12/17/24 3:01 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/12/a2e672bb2c1e166a7477595e77c9f578fdbdd76d86dd98189f3f8d7322f1f3].
Warning System 12/17/24 2:42 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/10/720dad9056dc2c3bc805a0798a44d4dfa5cb832cc2d582b6c2c4fd809b412c].
Warning System 12/17/24 2:42 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/90/c57ce46fb6b091da0707b365828035d7e3cf02409d7ec624a147b8c706087b].
Warning System 12/17/24 2:36 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/3a/4ce10b1a1a3507987b1b72f0d898d64f86b6c98559efccfb458bc9f73c3c9e].
Warning System 12/17/24 2:32 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/db/6bfdaa1321a670609ac8cf552bc908acd3bd123b6ec761bd1917d66df10cff].
Warning System 12/17/24 2:04 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/b6/088633d2da63120f586d446ee08124b757fe755c89e9a20a45e62ff36b258c].
Warning System 12/17/24 1:24 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/ab/e60a0b1c6a985083b45f2eb580edee619004d1c3a44237e4c455fb257862f2].
Warning System 12/17/24 1:14 SYSTEM Checksum mismatch on file [/volume1/backup/duplicacy/chunks/b0/67aa775c0d98c7f4ad3b7f3d06246882386ceb1e8924a104d2c63ca6cc62bf].

Thank you for the help and I’m happy to add more detail as necessary.

#2

The filesystem scrub should have successfully repaired the data using redundancy from the array and checksumming to determine which copy is correct. That message is purely informational. I would, however, consider replacing disks that exhibited data loss, and increasing frequency of scrub to monthly. I know, synology’s scrub degrades the performance too much – but I belive you can schedule it to run at night and pause during the day to minimize impact.

But yes, you can run check -chunks; but if you have ran that before – delete the “verified_chunks” file form duplicacy folder, otherwise it will skip chunks that were already checked previously.

#3

Thank you for the quick response and suggestion. Unfortunately, the Btrfs file system cannot repair damaged data. DSM recommends restoring the corrupted files from a backup… ha, thse files are my backup. None of my disks (6 drives, running RAID 5, 40TB) on the backup DS1815 are giving me errors through disk checks, defrag, etc. so I have no idea which drive could be causing the errors and all data is spanned across the drives.

Just to be clear, errors are from Synology DSM not through Duplicacy.

I’ll run check -chunks now and report what I find in the log.

Thanks again.

#4

This is not true. Since DSM6.1 Synology implemented data healing in their “BTRFs over md” concoction. All redundant configs support healing. There is no other reason in fact to tolerate slow and obnoxious BTRFs on Synology if not for that feature.

https://kb.synology.com/en-us/DSM/tutorial/How_to_enable_file_self_healing_on_DSM

From this white paper: https://global.download.synology.com/download/Document/Software/WhitePaper/Firmware/DSM/All/enu/Synology_Data_Protection_White_Paper.pdf

Silent data corruption detection and recovery (file self-healing)
In DSM 6.1 and above, Btrfs file system has the ability to not only detect the silent corruption on the user data but also to recover it. If checksum mismatch is detected, the file system layer instructs MD layer to read redundant copy (parity or mirror) of the damaged block. It will then automatically recover the damaged block with the good redundant copy. This file self-healing feature requires the RAID volumes to run RAID 1, 5, 6, 10, F1, or SHR.

Check the system log and smart reports on drives.

#5

If you find bad chunks you can recover some of them:

  • delete bad chunks from storage
  • create new snapshot id
  • make full backup into that new snapshot id
  • delete the new snapshot ID
  • run exhaustive prune
  • run check again. If you are lucky — many chunks would be recreated. Those that are still missing — run check, and delete all affected snapshots manually, followed by exhaustive prune.

And most importantly, if it turned out that you get a single chunk corrupted — defenestrate your Synology boxes; switch to something that actually works.

#6

I misspoke previously; you are correct. However, the error from DSM shows a checksum mismatch for several files as noted in my earlier post and no option to repair the data. Furhermore, DSM states I should restore the data from a backup (see image below). I’ve conducted tests on all the drives with no errors or issues with all drives. All are noted as healthy.

CleanShot 2024-12-19 at 09.32.46

I can backup the data again from the main server. I hoping to determine which backup contains the corrurpted files to speed up this process; Is there a way to determine which backup saved the corrupted chunks in a Duplicacy log?

I’m still in the process of running check -chunks -persist.

Synology may not work for all but has been fine for my needs; I dont like several of the DSM apps — hence the reason I purchased Duplicacy to fill this gap. Thank you again for the time and help.

#7

I think that message is bogus. Because otherwise that would mean the main reason they exist must be broken. To be fair, DSM is broken and very buggy — but not to this degree: their core technology team is very good. (Hopefully; haven’t dealt with Synology in years) The apps team is horrific shite. I feel the whole team is one overworked intern.

Wait for check -chunks … to finish. I’d expect all to pass.

But I can’t stress this enough: at this point you shall abandon Synology: either it lost data or it thinks it lost data. In either case you can’t trust it. We can have a separate discussion on how to pick storage appliance in a separate thread.

Until it unnecessarily causes you to waste time in panic mode with bogus messages.

Literally all DSM apps are broken if you try to deviate from one usecase they tested them with. All. DSM apps are there to sell the NAS to you. Once you buy NAS — you don’t use those apps past few months once you realize it’s a lipstick on a pig, but you already bought the NAS. So you continue using it as what you bought it for — a NAS, not a repacement for the whole saas industry (Dropbox/office/music/videos/etc). I see this story over and over and I have been a victim myself. I only wasted two year with their retched products thankfully. Fight the sunk cost fallacy and do what’s right sooner than later. You shall be able to trust your storage appliance to maintain integrity of your data no matter what. It’s literally it’s one job. If it does not, or makes you think it does not — I don’t know what’s worse — then the defenestration is the only reasonable solution. But I digress.

#8

Thank you for the help! I’m aware of Synology drawbacks (somewhat) just trying to live with what I’ve got… and this from Synology:

https://kb.synology.com/en-global/DSM/tutorial/Btrfs_checksum_mismatch

CleanShot 2024-12-19 at 12.01.45
CleanShot 2024-12-19 at 12.01.451119×680 62.2 KB

Very true - no panic yet. My core data is fine its my backup data that has the issue. So yes, fix it now before it a problem.

Still running check…

Regardless, my questions are about Duplicacy and commands to fix any damaged or missing files…

Questions:
-Could I delete the 16 damaged files, create a new snapshot id and complete a new backup? Only the missing files would be added to my previous backup as the other files would exist, if I understand correctly?
-Can I determine which backup the damaged files are from via a log in Duplicacy?

Thank you!

#9

There are circumstances when btrfs indeed cannot repair data – for example, if both copies of data got corrupted. Probability of this, bar complete drive failure, is miniscule. Another possibility is memory failure – and it is detecting bogus mismatches (or attempting to fix them, which is more scary). Just for a peace of mind I would run memory test (you can do it via synology utility from your PC).

Yes, the goal of this would be for missing chunks to get re-generated and uploaded to the storage, hoping that if data layout did not change much, exact same chunks will get created as the missing ones, thus restoring the integrity of the datastore.

Yes, once you delete damaged files run check -persist -a ,it will check all snapshots in all backup sets for the presence of chunks they reference, reporting if any are missing.

Then you can delete affected snapshot revision files and run duplicacy prune -a -exhaustive to delete chunks that no longer belong to any remaining snapshot