Missing Chunks and failed prune

I have a failing prune job and a “chunks missing” check job. see screenshots. I have tried adding -fossils and a few other options I found in the forums and nothing seems to be working. The one that seems to be failing is a several TB backup that I really don’t want to rebuild. Is there anyway to resolve this without starting the backup over?

Thanks.


Prune first deletes chunks and then proceeds to delete snapshots. If it’s interrupted after chunks got deleted but before snapshots, those ghost snapshots that refer to missing chunks remain in the storage and will be failing checks.

The proposed solution was to implement two step snapshot deletion process but was never implemented.

Essentially this means your datastore is still consistent, no data loss occurred; the only broken parts are those that should have been deleted anyway, and it’s fairly easy to clean up.

  1. run check -all -persist
  2. From the log collect IDs of revisions that are reported bad and delete them manually from the storage. They will be under the snapshots folder
  3. Then to cleanup remaining orphaned chunks run prune -all -exhaustive.

And that’s it.

Thank you @saspus – but check -all -persist is erroring out with invalid options

Make sure you put it to check’s parameters, not global options. I don’t remember details, but I think there are two separate boxes when editing the options.

Try -a -persist. -all is a valid option for the CLI but it seems the web GUI failed to parse it correctly.

So missing chunks are all in snapshot 1 at revision 1. This is a bit unusual because prune never deletes the first revision. I think you can just manually remove the revision file snapshots/1/1 in the storage and then run the one-time prune job with the -exhaustive -exclusive options (while making sure no backup jobs are in progress). This is to clean up any unreferenced chunks caused by the manual deletion of the revision file.

There are command options and global options. I was putting them in the command options but no dice.

@gchen – are you telling me to go into the cloud storage and delete files? All folders and files are encrypted so I’m not sure if I’ll be able to even tell which folders are tied to what. I will take a look later this afternoon. FYI -a -persist -all failed in the same way with invalid options.

-a instead of -all, not in addition.

Yes, delete snapshot revision file on the storage. It’s just a file with the name equals to the revision. 1, in your case.

But it’s possible there are more ghost snapshots, so you might still need to do the whole thing described above.

Violla – that worked. Thanks folks!

Ran the following commands in this order and it was resolved.

check -a -persist
prune -threads 12 -keep 0:180 -keep 7:30 -keep 1:7 -a