"Missing Chunks" reported following a purge

Support
#1

One of my computers is reporting “Missing Chunks” after each Check. I currently have the computer configured to complete a backup every 2 hours followed by a check.

Full error from logs:

Running check command from C:\ProgramData/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage OneDrive-Laptop -a -tabular]
2021-08-13 12:01:31.006 INFO STORAGE_SET Storage set to odb://Duplicacy/Jadog
2021-08-13 12:01:32.372 INFO SNAPSHOT_CHECK Listing all chunks
2021-08-13 12:04:05.694 INFO SNAPSHOT_CHECK 4 snapshots and 19 revisions
2021-08-13 12:04:05.694 INFO SNAPSHOT_CHECK Total chunk size is 41,815M in 10322 chunks
2021-08-13 12:04:05.727 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 1 exist
2021-08-13 12:04:05.776 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 23 exist
2021-08-13 12:04:05.815 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 25 exist
2021-08-13 12:04:05.849 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 29 exist
2021-08-13 12:04:05.871 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 35 exist
2021-08-13 12:04:05.894 INFO SNAPSHOT_CHECK All chunks referenced by snapshot SurfaceBook at revision 40 exist
2021-08-13 12:04:06.080 WARN SNAPSHOT_VALIDATE Chunk e2744bcc7c1ba12785044e27cb06edcfcefc26e50e7b8c78516cd0267f306785 referenced by snapshot SurfaceBook at revision 47 does not exist
2021-08-13 12:04:06.083 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot SurfaceBook at revision 47 are missing
2021-08-13 12:04:06.249 WARN SNAPSHOT_VALIDATE Chunk e2744bcc7c1ba12785044e27cb06edcfcefc26e50e7b8c78516cd0267f306785 referenced by snapshot SurfaceBook at revision 48 does not exist
2021-08-13 12:04:06.250 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot SurfaceBook at revision 48 are missing
2021-08-13 12:04:06.994 WARN SNAPSHOT_VALIDATE Chunk e2744bcc7c1ba12785044e27cb06edcfcefc26e50e7b8c78516cd0267f306785 referenced by snapshot SurfaceBook at revision 49 does not exist
2021-08-13 12:04:06.995 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot SurfaceBook at revision 49 are missing
2021-08-13 12:04:06.999 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC at revision 1 exist
2021-08-13 12:04:07.004 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC at revision 25 exist
2021-08-13 12:04:07.008 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC at revision 41 exist
2021-08-13 12:04:07.013 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC at revision 65 exist
2021-08-13 12:04:07.018 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC at revision 67 exist
2021-08-13 12:04:07.041 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC_Desktop at revision 1 exist
2021-08-13 12:04:07.061 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC_Desktop at revision 28 exist
2021-08-13 12:04:07.080 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC_Desktop at revision 49 exist
2021-08-13 12:04:07.120 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC_Desktop at revision 80 exist
2021-08-13 12:04:07.145 INFO SNAPSHOT_CHECK All chunks referenced by snapshot HomePC_Desktop at revision 82 exist
2021-08-13 12:04:07.145 ERROR SNAPSHOT_CHECK Some chunks referenced by some snapshots do not exist in the storage
Some chunks referenced by some snapshots do not exist in the storage

I installed Duplicacy first on my server where I configured the “Purge -all” command to take place weekly. From reading the documentation, it appears to be best to have this purge run on only a single computer to purge for all backups on my network. Is it possible that I’m encountering that message because the purge doesn’t run from this computer?

How do I fix and prevent from happening again?

#2

Likely what happened here is one of prunes did not compete or otherwise failed to delete snapshots after deleting chunks leaving ghost snapshots.

Delete those snapshots manually and clear local cache.

I had similar thing happen recently described here: -exhaustive implies -exclusive? Corrupted snapshots, can't delete with "No snapshot to delete"

@gchen, prune shall first delete snapshot and then its chunks. If prune is interrupted or fails mid way it’s much better to leave orphan chunks in the datastore than to have ghost snapshots scare users that datastore is corrupted.

In other words, on each step the datastore must be consistent and any process must be safely interruptible at any point, without breaking consistency. Reversing order of deleting snapshots vs chunks in prune will make prune consistent and eliminate the need for this almost monthly topic…

3 Likes
Duplicacy can't back up empty files?
Cleanup/maintenance after initial ingestion to B2
Snapshot file format
#3

Ok, thanks. How do I clear the local cache and at which computer do I perform that at? I presume that would be done at the machine the logs show have the missing chunks.

#4

Yes; if you go to Settings in web GUI the contents of the folder pointed to by Temporary Directory can be nuked.

Alternatively, you can run check with arguments -fossils -resurrect to resurrect the deleted chunks to heal snapshots abandoned by failed prune; and also replace -a with -id <snapshot-id> in the prune command. This seems (because we don’t yet know the root cause here) to have fixed pruning for me by avoiding what looks like a corner case in the prune command where it fails to delete snapshot files after deleting chunks.

Either way, deleting cache should not hurt.

#5

There is only one chunk e274... missing. Can you check the chunks directory to see if it is there (should be under the e2 subdirectory with file name starting with 74...)?

#6

Also please check if you have more than one e2 (duplicate) directory (which can happen under some circumstances)

#7

Ok, I checked the chunks directory and see a SINGLE e2 directory under the path:

C:\ProgramData.duplicacy-web\repositories\localhost\all\duplicacy\cache\OneDrive-Laptop\chunks

Within this folder there is a file named ceb03bb2dc3098fdcfc1a751a1387852717ca94fbdd0d02160f2fe7b17134d.

I also have a folder at C:\ProgramData.duplicacy-web\repositories\localhost\0\duplicacy\cache\OneDrive-Laptop\chunks\e2

This path does NOT have any files in this folder. The difference between the two above paths are the first is “ALL” and the second is “0”.

So what does this mean I should do next?

#8

Ok, so should I just follow the original instruction here by deleting the snapshots using the prune command?

Prune -log -ID SurfaceBook -Storage OneDrive-Laptop - r r48-r51

Then I presume I just delete the entire cache folder found at:

C:\ProgramData.duplicacy-web\repositories\localhost\0\duplicacy\cache
C:\ProgramData.duplicacy-web\repositories\localhost\ALL\duplicacy\cache\