Hi team,
Running GUI 1.8.3 which is backed by CLI 3.2.4.
I backup with a “create only” key to B2 most of the time, and occasionally load in a prune key for pruning operations.
I’ve attempted that today, first running a check on all snapshots across all backup machines, which came back successfully. Then, I ran a prune - this failed on “missing chunk”:
ERROR CHUNK_FIND Chunk 30ac4ad96fabedf04a43af2744474f6f715fd908137e52195d6c167eba316ebc does not exist in the storage
I verified this is not in storage. I checked logs for web GUI:
➜ logs grep 30ac4ad96fabedf04a43af2744474f6f715fd908137e52195d6c167eba316ebc ./*
./prune-log-20250924-100025:Marked fossil 30ac4ad96fabedf04a43af2744474f6f715fd908137e52195d6c167eba316ebc
I re-ran a check (duplicacy check -a via CLI), and now, plenty of chunks are not found - some marked as fossil and some just missing (many revisions snipped but you get the idea):
All chunks referenced by snapshot Iridium at revision 331 exist
All chunks referenced by snapshot Iridium at revision 332 exist
Chunk 634b8e212fb1d09c61223acd22e8f77ff653c44e77ead5db4c27f5d2e354ece5 is a fossil
Chunk 634b8e212fb1d09c61223acd22e8f77ff653c44e77ead5db4c27f5d2e354ece5 referenced by snapshot Iridium at revision 333 does not exist
Some chunks referenced by snapshot Iridium at revision 333 are missing
All chunks referenced by snapshot Iridium at revision 334 exist
All chunks referenced by snapshot Iridium at revision 335 exist
All chunks referenced by snapshot squidward at revision 293 exist
Chunk 9896dc8efe749512bb28d550b859cef223fec77e3c1b767d8f271027063532ac referenced by snapshot squidward at revision 294 does not exist
Some chunks referenced by snapshot squidward at revision 294 are missing
All chunks referenced by snapshot squidward at revision 295 exist
However the missing chunk that shows up in Web UI (30ac4ad96fabedf04a43af2744474f6f715fd908137e52195d6c167eba316ebc) doesn’t show in these logs anywhere?
It seems to me that the prune operation is not working properly somehow. The last time I had to do a prune, I encountered the same thing, and it was a painful process of manually deleting every snapshot ID from storage and rerunning exhaustive prunes after that. Is there any better method, and any indication why a prune would “partially work”?