Missing Chunks, FOSSIL_GHOSTSNAPSHOT

Today, 2021-04-17, prune reported FOSSIL_GHOSTSNAPSHOT on 8 of 9 snapshots on B2 storage:

[~/.duplicacy-web/logs] # grep -e 'FOSSIL_GHOSTSNAPSHOT\|FOSSIL_IGNORE\|WARN\|revision 9[37] has been removed' `grep -l b2:// prune*.log`
prune-20210417-001525.log:2021-04-17 00:19:40.894 INFO SNAPSHOT_DELETE The snapshot Duplicacy-config at revision 93 has been removed
prune-20210417-002445.log:2021-04-17 00:25:08.891 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Duplicacy-config revision 93 should have been deleted already
prune-20210417-002445.log:2021-04-17 00:25:08.891 INFO FOSSIL_IGNORE The fossil collection file fossils/543 has been ignored due to ghost snapshots
prune-20210417-002445.log:2021-04-17 00:25:25.813 WARN DOWNLOAD_RESURRECT Fossil 5744a3f6248a28123d20bb80e644fb1f6e7c302a0f1a46297a9df354b9e5951d has been resurrected
prune-20210417-002445.log:2021-04-17 00:27:59.618 INFO SNAPSHOT_DELETE The snapshot Joint at revision 97 has been removed
prune-20210417-003220.log:2021-04-17 00:32:43.089 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Joint revision 97 should have been deleted already
prune-20210417-003220.log:2021-04-17 00:32:43.089 INFO FOSSIL_IGNORE The fossil collection file fossils/544 has been ignored due to ghost snapshots
prune-20210417-003220.log:2021-04-17 00:35:01.312 WARN DOWNLOAD_RESURRECT Fossil 636fca4f581ad36859e0105b3e5b7f24ee3b2717efa72d7a330a8e6a68e3c78a has been resurrected
prune-20210417-003220.log:2021-04-17 00:35:14.630 INFO SNAPSHOT_DELETE The snapshot Household at revision 97 has been removed
prune-20210417-003929.log:2021-04-17 00:39:52.135 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Household revision 97 should have been deleted already
prune-20210417-003929.log:2021-04-17 00:39:52.135 INFO FOSSIL_IGNORE The fossil collection file fossils/545 has been ignored due to ghost snapshots
prune-20210417-003929.log:2021-04-17 00:41:57.304 WARN DOWNLOAD_RESURRECT Fossil 8478ea4615dab363125726df0be255699baba40ab944f466a785f10688974711 has been resurrected
prune-20210417-003929.log:2021-04-17 00:42:27.316 INFO SNAPSHOT_DELETE The snapshot Carol at revision 97 has been removed
prune-20210417-004641.log:2021-04-17 00:47:03.172 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Carol revision 97 should have been deleted already
prune-20210417-004641.log:2021-04-17 00:47:03.172 INFO FOSSIL_IGNORE The fossil collection file fossils/546 has been ignored due to ghost snapshots
prune-20210417-004641.log:2021-04-17 00:49:27.087 WARN DOWNLOAD_RESURRECT Fossil 59e25592a2009fd8a2996584048b006cff29fb1be069e1ed05dfc33c3498c5b6 has been resurrected
prune-20210417-004641.log:2021-04-17 00:50:41.147 INFO SNAPSHOT_DELETE The snapshot Keith at revision 97 has been removed
prune-20210417-005507.log:2021-04-17 00:55:29.493 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Keith revision 97 should have been deleted already
prune-20210417-005507.log:2021-04-17 00:55:29.493 INFO FOSSIL_IGNORE The fossil collection file fossils/547 has been ignored due to ghost snapshots
prune-20210417-005507.log:2021-04-17 00:55:40.370 WARN DOWNLOAD_RESURRECT Fossil 192166355da76910e8b6ee882fc6934e9bd0db8f84a8bf1929c0b29fffa7c8f8 has been resurrected
prune-20210417-005507.log:2021-04-17 00:59:04.111 INFO SNAPSHOT_DELETE The snapshot Keith-Pictures at revision 97 has been removed
prune-20210417-010446.log:2021-04-17 01:05:17.457 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Keith-Pictures revision 97 should have been deleted already
prune-20210417-010446.log:2021-04-17 01:05:17.457 INFO FOSSIL_IGNORE The fossil collection file fossils/548 has been ignored due to ghost snapshots
prune-20210417-010446.log:2021-04-17 01:06:04.189 WARN DOWNLOAD_RESURRECT Fossil 6a04662ccf59826a78bdb02058d1db15acb5d55a65588258a8ca74f13c27b3df has been resurrected
prune-20210417-010446.log:2021-04-17 01:08:01.561 INFO SNAPSHOT_DELETE The snapshot Public at revision 97 has been removed
prune-20210417-011250.log:2021-04-17 01:13:15.194 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Public revision 97 should have been deleted already
prune-20210417-011250.log:2021-04-17 01:13:15.194 INFO FOSSIL_IGNORE The fossil collection file fossils/549 has been ignored due to ghost snapshots
prune-20210417-011250.log:2021-04-17 01:15:49.408 INFO SNAPSHOT_DELETE The snapshot Multimedia at revision 97 has been removed
prune-20210417-012000.log:2021-04-17 01:20:23.125 INFO FOSSIL_GHOSTSNAPSHOT Snapshot Multimedia revision 97 should have been deleted already
prune-20210417-012000.log:2021-04-17 01:20:23.125 INFO FOSSIL_IGNORE The fossil collection file fossils/550 has been ignored due to ghost snapshots

Subsequent check reported missing chunks:

[~/.duplicacy-web/logs] # grep "WARN SNAPSHOT_CHECK" check*.log
check-20210417-022447.log:2021-04-17 02:25:45.583 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Multimedia at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:25:52.008 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Carol at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:25:58.499 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Duplicacy-config at revision 93 are missing
check-20210417-022447.log:2021-04-17 02:26:25.792 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Keith-Pictures at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:28:54.471 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Joint at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:29:01.687 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Keith at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:29:16.229 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Public at revision 97 are missing
check-20210417-022447.log:2021-04-17 02:29:20.654 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Household at revision 97 are missing

Any idea of what could have caused this? How do I recover from these errors?

These revisions should have been deleted by last prune job but for some reason the deletion wasn’t successful. You can find the error in the log of that prune job.

To fix it you can just manually delete these revisions.

No previous error was reported.

I didn’t have the opportunity to manually delete the revisions before the next prune, which deleted them and logged a bunch of CHUNK_FOSSILIZE warnings, after which the check succeeded. Today, everything ran without errors/warnings.

Recently, I had tried running prune with -threads 4. Subsequently, a prune reported a CHUNK_DELETE warning. Since I’d not seen any prune warnings before, and since prune wasn’t running noticeably faster, I removed the -threads option. So I’m wondering if these problems are related to using multiple threads.