What is a Ghost Snapshot? Should I start worrying?

Support
#1

Hi,

I noticed these messages in a prune log:
“Snapshot X revision Y should have been deleted already”
“The fossil collection file fossils/9 has been ignored due to ghost snapshots”.

What is a ghost snapshot?
Why should Snapshot X have been deleted already but isn’t?
Will this issue resolve itself eventually, or do I have to take action?

Running prune command from /Users/blahbla/...
Options: [-log prune -storage Backup -keep 0:360 -keep 30:180 -keep 7:90 -a]
2020-06-28 22:05:15.653 INFO STORAGE_SET Storage set to blah://blah/...
2020-06-28 22:05:16.462 INFO RETENTION_POLICY Keep no snapshots older than 360 days
2020-06-28 22:05:16.462 INFO RETENTION_POLICY Keep 1 snapshot every 30 day(s) if older than 180 day(s)
2020-06-28 22:05:16.462 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 90 day(s)
2020-06-28 22:05:23.049 INFO FOSSIL_COLLECT Fossil collection 10 found
2020-06-28 22:05:23.049 INFO SNAPSHOT_INACTIVE Ignore snapshot dev whose last revision was created 7 days ago
2020-06-28 22:05:23.049 INFO FOSSIL_POSTPONE Fossils from collection 10 can't be deleted because deletion criteria aren't met
2020-06-28 22:05:23.050 INFO FOSSIL_COLLECT Fossil collection 11 found
2020-06-28 22:05:23.050 INFO FOSSIL_POSTPONE Fossils from collection 11 can't be deleted because deletion criteria aren't met
2020-06-28 22:05:23.050 INFO FOSSIL_COLLECT Fossil collection 12 found
2020-06-28 22:05:23.050 INFO SNAPSHOT_INACTIVE Ignore snapshot dotfiles whose last revision was created 7 days ago
2020-06-28 22:05:23.050 INFO FOSSIL_POSTPONE Fossils from collection 12 can't be deleted because deletion criteria aren't met
2020-06-28 22:05:23.051 INFO FOSSIL_COLLECT Fossil collection 13 found
2020-06-28 22:05:23.051 INFO FOSSIL_POSTPONE Fossils from collection 13 can't be deleted because deletion criteria aren't met
2020-06-28 22:05:23.052 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Minecraft revision 3 should have been deleted already
2020-06-28 22:05:23.052 INFO FOSSIL_IGNORE The fossil collection file fossils/9 has been ignored due to ghost snapshots
2020-06-28 22:05:23.052 INFO SNAPSHOT_NONE No snapshot to delete
Three subsequent checks return three different missing chunks
#2 
2020-06-28 22:05:23.052 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Minecraft revision 3 should have been deleted already
2020-06-28 22:05:23.052 INFO FOSSIL_IGNORE The fossil collection file fossils/9 has been ignored due to ghost snapshots

This means revision 3 was deleted when creating the fossil collection file fossils/9 but somehow the corresponding file snapshots/Minecraft/3 still exists in the storage. This can be caused by an error in the deletion operation or the storage not working properly. You can find out what went wrong by looking at the log for the prune job that created fossils/9 (all prune logs can be found under .duplicacy/logs).

#3

Thank you @gchen for explaining the details.
When I search all log files for “fossils/9”, I get only one result, which is the output above.

A regexp for fossil.*9 returns this line from a prune log file:

2020-06-09 21:01:32.648 INFO FOSSIL_COLLECT Fossil collection 9 saved

No adjacent errors or warnings though.

Could this issue perhaps be caused by my attempt to repair a partly deleted storage (Switch to different storage backend)? The repair apparently succeeded as all backups are running fine again, but check and prune are now complaining.

Worth noting, this repair was done after the above prune from June 9th.

#4

The ghost snapshot could be a result of removing a revision and then recreating it again with the same revision number. If this is the case then it can also cause missing chunks due to a stale cache (the snapshot file in the local cache becomes different from the corresponding one in the storage). So you’ll need to clean the local cache by removing the entire .duplicacy/cache directory and try a check again.

#5

I am using the UI only. In ~/.duplicacy-web there is no cache folder. Where is the Web edition’s cache located?

#6

There will be several - one for each repository - under:

.duplicacy-web\repositories\localhost\*\.duplicacy\cache\

…where * is 0, 1, all etc…

#7

Thanks @Droolio for clarifying. Found them, moved them to the trash, ran the checks again, and I have no more missing chunks. So apparently the problem was a stale cache.

Thank you @gchen and @Droolio for your help!

Three subsequent checks return three different missing chunks