Check reports "Missing chunks" even though no pruning

Support
#1

Every day I do a backup and check:

image
image1954×301 28.5 KB

Every day the check fails with “Missing chunks”. I then delete from Backblaze the revision containing missing chunks. I backup again, and same thing, the check fails with “Missing chunks”.

The logs of the last failed check:

Running check command from /home/lcswillems/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage Backblaze -fossils -resurrect -persist -a -tabular]
2025-02-24 23:44:52.085 INFO STORAGE_SET Storage set to b2://lucas-willems-backup-duplicacy
2025-02-24 23:44:52.638 INFO BACKBLAZE_URL Download URL is: https://f004.backblazeb2.com
2025-02-24 23:44:54.333 INFO SNAPSHOT_CHECK Listing all chunks
2025-02-24 23:45:35.028 INFO SNAPSHOT_CHECK 1 snapshots and 28 revisions
2025-02-24 23:45:35.033 INFO SNAPSHOT_CHECK Total chunk size is 304,158M in 108257 chunks
2025-02-24 23:45:35.112 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 1 exist
2025-02-24 23:45:35.179 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 43 exist
2025-02-24 23:45:35.245 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 149 exist
2025-02-24 23:45:35.302 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 183 exist
2025-02-24 23:45:35.389 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 217 exist
2025-02-24 23:45:35.479 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 238 exist
2025-02-24 23:45:35.548 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 270 exist
2025-02-24 23:45:35.641 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 305 exist
2025-02-24 23:45:35.754 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 340 exist
2025-02-24 23:45:35.882 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 371 exist
2025-02-24 23:45:36.002 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 405 exist
2025-02-24 23:45:36.137 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 427 exist
2025-02-24 23:45:36.297 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 459 exist
2025-02-24 23:45:36.426 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 482 exist
2025-02-24 23:45:36.573 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 483 exist
2025-02-24 23:45:36.744 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 511 exist
2025-02-24 23:45:36.909 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 521 exist
2025-02-24 23:45:37.090 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 529 exist
2025-02-24 23:45:37.296 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 536 exist
2025-02-24 23:45:37.484 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 540 exist
2025-02-24 23:45:37.664 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 541 exist
2025-02-24 23:45:37.854 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 542 exist
2025-02-24 23:45:38.038 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 543 exist
2025-02-24 23:45:38.229 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 544 exist
2025-02-24 23:45:38.435 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 545 exist
2025-02-24 23:45:38.623 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 546 exist
2025-02-24 23:45:38.816 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 547 exist
2025-02-24 23:45:39.625 WARN SNAPSHOT_VALIDATE Chunk 598d78045b88289e8f4820a9a90c0eddd3e379c0bc6e3b582449e2da90302385 referenced by snapshot Data at revision 548 does not exist
2025-02-24 23:45:39.941 WARN SNAPSHOT_VALIDATE Chunk 47f78e9f0a7755cccc5faf6b3554550d2fffb1791801ffe9cff5d19c2fd9ab51 referenced by snapshot Data at revision 548 does not exist
2025-02-24 23:45:40.247 WARN SNAPSHOT_VALIDATE Chunk aff4510da0e84ab67a6e9a2dac32bc476733a9bd1338fe7788c05f304ce43c8a referenced by snapshot Data at revision 548 does not exist
2025-02-24 23:45:40.554 WARN SNAPSHOT_VALIDATE Chunk bb13dfadcbb96e41cc3dff94089001b616bbed4c55db407dfb9b288dbcd28f70 referenced by snapshot Data at revision 548 does not exist
2025-02-24 23:45:40.561 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 548 are missing
2025-02-24 23:45:40.561 ERROR SNAPSHOT_CHECK Some chunks referenced by some snapshots do not exist in the storage
Some chunks referenced by some snapshots do not exist in the storage

Any idea why I’m getting those errors?

#2

You can search logs for the missing chunk id to track when it was created, uploaded, and deleted.

I would also delete local cache and retry the check.

#3

The missing chunk is only referenced in check logs. There is no log saying when it was created.

I’ve deleted the local cache and started a check again but I get the exact same errors. The new logs:

Running check command from /home/lcswillems/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage Backblaze -fossils -resurrect -persist -a -tabular]
2025-03-04 21:56:07.761 INFO STORAGE_SET Storage set to b2://lucas-willems-backup-duplicacy
2025-03-04 21:56:08.334 INFO BACKBLAZE_URL Download URL is: https://f004.backblazeb2.com
2025-03-04 21:56:18.312 INFO SNAPSHOT_CHECK Listing all chunks
2025-03-04 21:58:15.050 INFO SNAPSHOT_CHECK 1 snapshots and 35 revisions
2025-03-04 21:58:15.058 INFO SNAPSHOT_CHECK Total chunk size is 307,836M in 109870 chunks
2025-03-04 21:58:16.572 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 1 exist
2025-03-04 21:58:27.345 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 43 exist
2025-03-04 21:58:51.535 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 149 exist
2025-03-04 21:59:04.771 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 183 exist
2025-03-04 21:59:07.077 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 217 exist
2025-03-04 21:59:20.426 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 238 exist
2025-03-04 21:59:24.133 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 270 exist
2025-03-04 21:59:47.799 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 305 exist
2025-03-04 22:01:54.982 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 340 exist
2025-03-04 22:02:31.376 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 371 exist
2025-03-04 22:02:51.046 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 405 exist
2025-03-04 22:02:54.602 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 427 exist
2025-03-04 22:03:10.452 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 459 exist
2025-03-04 22:04:14.124 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 482 exist
2025-03-04 22:04:27.028 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 483 exist
2025-03-04 22:05:04.548 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 511 exist
2025-03-04 22:06:03.114 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 521 exist
2025-03-04 22:06:31.126 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 529 exist
2025-03-04 22:07:44.903 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 536 exist
2025-03-04 22:08:08.837 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 540 exist
2025-03-04 22:08:25.001 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 541 exist
2025-03-04 22:08:45.380 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 542 exist
2025-03-04 22:09:46.470 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 543 exist
2025-03-04 22:10:26.356 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 544 exist
2025-03-04 22:10:27.896 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 545 exist
2025-03-04 22:11:13.861 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 546 exist
2025-03-04 22:12:02.155 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Data at revision 547 exist
2025-03-04 22:12:38.191 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 548 does not exist
2025-03-04 22:12:38.192 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 548 are missing
2025-03-04 22:12:41.555 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 549 does not exist
2025-03-04 22:12:41.570 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 549 are missing
2025-03-04 22:13:30.361 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 550 does not exist
2025-03-04 22:13:30.643 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 550 does not exist
2025-03-04 22:13:30.647 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 550 are missing
2025-03-04 22:13:34.412 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 551 does not exist
2025-03-04 22:13:34.693 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 551 does not exist
2025-03-04 22:13:34.694 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 551 are missing
2025-03-04 22:14:41.805 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 552 does not exist
2025-03-04 22:14:42.085 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 552 does not exist
2025-03-04 22:14:42.088 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 552 are missing
2025-03-04 22:14:57.443 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 553 does not exist
2025-03-04 22:14:57.720 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 553 does not exist
2025-03-04 22:14:57.722 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 553 are missing
2025-03-04 22:15:12.264 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 554 does not exist
2025-03-04 22:15:12.546 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 554 does not exist
2025-03-04 22:15:12.553 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 554 are missing
2025-03-04 22:15:26.581 WARN SNAPSHOT_VALIDATE Chunk 4195da9803fe55588c4f11d304fda5608d38c73a5108638c06ca7e95690c4e75 referenced by snapshot Data at revision 555 does not exist
2025-03-04 22:15:26.860 WARN SNAPSHOT_VALIDATE Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 referenced by snapshot Data at revision 555 does not exist
2025-03-04 22:15:26.866 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Data at revision 555 are missing
2025-03-04 22:15:26.866 ERROR SNAPSHOT_CHECK Some chunks referenced by some snapshots do not exist in the storage
Some chunks referenced by some snapshots do not exist in the storage
#4

When duplicacy reports chunk does not exist – can you manually confirm on back blaze that it actually is missing? with some other tools, like CybeDuck?

Are you running prune? Does prune succeed in deleting the revisions it says it’s deleting? I had issues with google drive where delete would succeed but the revision file would remain

#5

I can’t find it in Backblaze indeed. I’m looking at chunks/78/7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 and can’t find it.

I’m not running prune at all. I just do “backup” and then “check”.

Each time the “check” fails saying that a chunk is missing. So I delete the snapshot with the missing chunk. I start a new backup and the check fails again saying that a chunk is missing.

I’ve also tried to delete the cache but it changed nothing.

#6

Chunk 7837ffb837f9b297f1ef30f6675b6b01a18fbebd260c3e9e46fc26b719520658 will be stored as chunks/78/37ff... - first two digits are move to the directory pathname.

Could this be related to B2s Lifecycle rules?

#7

Maybe I misunderstood how it works. I’ve also checked the pattern you give:

chunks / 78 / 37ff...

And I could not find a chunk too. @saspus @gchen any idea? This is quite concerning.

Basically, I backup, do a check just after (no prune) and there is a chunk missing.

#8

Indeed quite troubling.

Is this the only duplicacy instance that touches b2 at the time? And the missing chunks are always in the latest, just uploaded snapshot?

What if you run check 10 min after backup — does it still find same missing chunks?

Enable logging for backup job (-d global flag), to see chunks that have been uploaded. Then if subsequent check fails to find chunk — try to find it in the backup log, to confirm that duplicacy actually uploaded it.

If it’s three it would be an egregious b2 bug. Or if check after 10 min success — it could be related to “eventual consistency” that most cloud storage feature; maybe that eventuality delay becomes huge on b2

#9

Also one trick you can try - if these are new backups - is do a fresh backup using a brand new ID. Duplicacy should skip most existing chunks and upload a new snapshot revision. Delete all other revisions and run a check.

If you’re clearing the cache, make sure to do it in the numbered repository directories (0, 1, 2 etc.) and all.

#10

Thank you for your help @saspus @Droolio !

So today I’ve done the following:

  • deleted the snapshots with missing chunks
  • cleared the cache
  • run a new backup
  • run a check

And it has successfully passed!

The previous time, I had just cleared the cache and run a check (without running a new backup). So maybe that’s the reason…

I’ll continue to monitor, and let you know in case a new error appear. Also, I keep in mind the option “-d” to log the chunks that are backed up.