Missing Chunks on check

h11 · 30 October 2023 03:07

Good day,

I am using Duplicacy (web GUI) on unraid. Everything seems to be fine with actual backup but check reports missing chunks.

Honestly, I don’t really care why. I just want check to run so that it can update the stats.

I have gone through the forum on missing chunks and deleted cache. Run various forms of prune and still check reports missing chunks and does not update the dashboard.

I will copy the log from the last check

Running check command from /cache/localhost/all
Options: [-log check -storage k-serv1-bkup -a -tabular]
2023-10-30 04:59:45.521 INFO STORAGE_SET Storage set to /mnt/remotes/k-serv1_k-serv4-bkup
2023-10-30 04:59:45.534 INFO SNAPSHOT_CHECK Listing all chunks
2023-10-30 05:00:26.427 INFO SNAPSHOT_CHECK 5 snapshots and 91 revisions
2023-10-30 05:00:26.429 INFO SNAPSHOT_CHECK Total chunk size is 440,629M in 95437 chunks
2023-10-30 05:00:26.430 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 1 exist
2023-10-30 05:00:26.431 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 2 exist
2023-10-30 05:00:26.433 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 3 exist
2023-10-30 05:00:26.434 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 4 exist
2023-10-30 05:00:26.435 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 5 exist
2023-10-30 05:00:26.436 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 6 exist
2023-10-30 05:00:26.446 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 7 exist
2023-10-30 05:00:26.449 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 8 exist
2023-10-30 05:00:26.454 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 9 exist
2023-10-30 05:00:26.458 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 10 exist
2023-10-30 05:00:26.461 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 11 exist
2023-10-30 05:00:26.465 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 12 exist
2023-10-30 05:00:26.468 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 13 exist
2023-10-30 05:00:26.472 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 14 exist
2023-10-30 05:00:26.475 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 15 exist
2023-10-30 05:00:26.478 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 16 exist
2023-10-30 05:00:26.482 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 17 exist
2023-10-30 05:00:26.485 INFO SNAPSHOT_CHECK All chunks referenced by snapshot Bianca at revision 18 exist
2023-10-30 05:00:26.678 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 1 does not exist
2023-10-30 05:00:26.718 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 1 are missing
2023-10-30 05:00:26.928 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 2 does not exist
2023-10-30 05:00:26.947 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 2 are missing
2023-10-30 05:00:27.162 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 3 does not exist
2023-10-30 05:00:27.186 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 3 are missing
2023-10-30 05:00:27.366 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 4 does not exist
2023-10-30 05:00:27.405 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 4 are missing
2023-10-30 05:00:27.605 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 5 does not exist
2023-10-30 05:00:27.616 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 5 are missing
2023-10-30 05:00:27.826 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 6 does not exist
2023-10-30 05:00:27.831 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 6 are missing
2023-10-30 05:00:28.019 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 7 does not exist
2023-10-30 05:00:28.042 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 7 are missing
2023-10-30 05:00:28.245 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 8 does not exist
2023-10-30 05:00:28.250 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 8 are missing
2023-10-30 05:00:28.442 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 9 does not exist
2023-10-30 05:00:28.456 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 9 are missing
2023-10-30 05:00:28.657 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 10 does not exist
2023-10-30 05:00:28.661 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 10 are missing
2023-10-30 05:00:28.865 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 11 does not exist
2023-10-30 05:00:28.882 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 11 are missing
2023-10-30 05:00:29.072 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 12 does not exist
2023-10-30 05:00:29.091 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 12 are missing
2023-10-30 05:00:29.281 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 13 does not exist
2023-10-30 05:00:29.301 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 13 are missing
2023-10-30 05:00:29.501 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 14 does not exist
2023-10-30 05:00:29.519 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 14 are missing
2023-10-30 05:00:29.711 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 15 does not exist
2023-10-30 05:00:29.728 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 15 are missing
2023-10-30 05:00:29.960 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 16 does not exist
2023-10-30 05:00:29.962 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 16 are missing
2023-10-30 05:00:30.148 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 17 does not exist
2023-10-30 05:00:30.182 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 17 are missing
2023-10-30 05:00:30.368 WARN SNAPSHOT_VALIDATE Chunk de33c943cde7475e6a5a4de5ae6c3b7db3dbcb7953daf731fcfe160327222810 referenced by snapshot HRK at revision 18 does not exist
2023-10-30 05:00:30.400 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot HRK at revision 18 are missing
2023-10-30 05:00:30.426 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 1 exist
2023-10-30 05:00:30.450 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 2 exist
2023-10-30 05:00:30.474 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 4 exist
2023-10-30 05:00:30.500 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 5 exist
2023-10-30 05:00:30.527 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 6 exist
2023-10-30 05:00:30.553 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 7 exist
2023-10-30 05:00:30.578 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 8 exist
2023-10-30 05:00:30.606 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 9 exist
2023-10-30 05:00:30.633 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 10 exist
2023-10-30 05:00:30.659 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 11 exist
2023-10-30 05:00:30.684 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 12 exist
2023-10-30 05:00:30.712 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 13 exist
2023-10-30 05:00:30.738 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 14 exist
2023-10-30 05:00:30.764 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 15 exist
2023-10-30 05:00:30.796 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 16 exist
2023-10-30 05:00:30.822 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 17 exist
2023-10-30 05:00:30.847 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 18 exist
2023-10-30 05:00:30.878 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 19 exist
2023-10-30 05:00:30.909 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-ADMIN at revision 20 exist
2023-10-30 05:00:30.951 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 1 exist
2023-10-30 05:00:30.999 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 3 exist
2023-10-30 05:00:31.043 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 4 exist
2023-10-30 05:00:31.085 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 5 exist
2023-10-30 05:00:31.126 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 6 exist
2023-10-30 05:00:31.167 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 7 exist
2023-10-30 05:00:31.214 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 8 exist
2023-10-30 05:00:31.254 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 9 exist
2023-10-30 05:00:31.299 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 10 exist
2023-10-30 05:00:31.340 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 11 exist
2023-10-30 05:00:31.383 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 12 exist
2023-10-30 05:00:31.423 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 13 exist
2023-10-30 05:00:31.464 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 14 exist
2023-10-30 05:00:31.514 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 15 exist
2023-10-30 05:00:31.557 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 16 exist
2023-10-30 05:00:31.599 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 17 exist
2023-10-30 05:00:31.640 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 18 exist
2023-10-30 05:00:31.688 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-DEBT at revision 19 exist
2023-10-30 05:00:31.699 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 1 exist
2023-10-30 05:00:31.710 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 2 exist
2023-10-30 05:00:31.722 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 3 exist
2023-10-30 05:00:31.732 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 4 exist
2023-10-30 05:00:31.742 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 5 exist
2023-10-30 05:00:31.752 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 6 exist
2023-10-30 05:00:31.763 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 7 exist
2023-10-30 05:00:31.773 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 8 exist
2023-10-30 05:00:31.786 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 9 exist
2023-10-30 05:00:31.796 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 10 exist
2023-10-30 05:00:31.806 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 11 exist
2023-10-30 05:00:31.816 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 12 exist
2023-10-30 05:00:31.826 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 13 exist
2023-10-30 05:00:31.836 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 14 exist
2023-10-30 05:00:31.846 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 15 exist
2023-10-30 05:00:31.856 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 16 exist
2023-10-30 05:00:31.865 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 17 exist
2023-10-30 05:00:31.875 INFO SNAPSHOT_CHECK All chunks referenced by snapshot K-SERV2-A at revision 18 exist
2023-10-30 05:00:31.875 ERROR SNAPSHOT_CHECK Some chunks referenced by some snapshots do not exist in the storage

Some chunks referenced by some snapshots do not exist in the storage

saspus · 30 October 2023 03:12

This was probably due to interrupted prune some ghost snapshots remained in the storage, because this has never been implemented.

To fix:

run check with -persist flag. This will continue past errors. Collect names of all bad snapshots.
Go to the storage, snapshots/snapshot-id/ and delete the bad snapshots.
Run prune -exhaustive -all to clean out remaining orphaned chunks from the storage.

h11 · 30 October 2023 03:35

Ok I ran the check with persist and deleted the bad snapshots. Was all of them for that snapshot ID.

Now when running prune I get this error

Running prune command from /cache/localhost/all
Options: [-log prune -storage k-serv1-bkup -exhaustive -all]
2023-10-30 05:28:54.779 INFO STORAGE_SET Storage set to /mnt/remotes/k-serv1_k-serv4-bkup
2023-10-30 05:32:39.500 ERROR CHUNK_DELETE Failed to fossilize the chunk de010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa: rename /mnt/remotes/k-serv1_k-serv4-bkup/chunks/de/010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa /mnt/remotes/k-serv1_k-serv4-bkup/chunks/de/010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa.fsl: no such file or directory
Failed to fossilize the chunk de010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa: rename /mnt/remotes/k-serv1_k-serv4-bkup/chunks/de/010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa /mnt/remotes/k-serv1_k-serv4-bkup/chunks/de/010a138ec42a8165dbe074f3198566fdc19de64317f68e020ed0c4def448aa.fsl: no such file or directory

h11 · 30 October 2023 04:03

Ok so it turns out that running this prune now causes the unraid server where the storage is to crash. So I guess I will first have to figure that out.

Droolio · 31 October 2023 14:22

Looks like you have no good snapshots for the HRK snapshot ID. However, you should run check -fossils to make sure that’s the case.

Using -persist doesn’t help here because you didn’t run into any errors that stopped check from completing, and you weren’t doing a -chunks or -files integrity check that would be relevant to that sort of error.

If indeed it’s the case HRK doesn’t have a single good revision, you can repair the backup storage to a working state. First, you must manually remove the HRK revisions (either the contents of snapshots/HRK or the whole directory).

Make sure you don’t run a prune at this point because you can then take advantage of the fact chunks still exist on the storage. When you run your HRK backup, it’ll create a new revision 1 and skip uploading chunks.

Do a fresh check and, when all looks good, run a prune -exhaustive.

h11 · 1 November 2023 02:40

Thanks,

I just decided to make a whole new storage folder on the remote server and start a whole new backup. Still running at this time since it is a couple of gigs.

But thanks I will definitely come back to this when I run into this issue again.