There is also a case with B2 that the chunk exists, but there are multiple versions of it and the latest one is zero size. I did not find instructions what to do in this case.
SOLVED:
The solution seems to be to log in B2, locate the “missing” chunks and delete the zero-sized versions. I have no idea why those have been created in the first place, but I suspect an interrupted backup. I am not just sure does this method guarantee that the chunk content is valid anymore…
I have been using Duplicacy for a couple years to back up several repositories to the same storage. All the repositories are on my computer and nobody else backs up to the storage. A couple days ago, I started getting an error when I run the check command (normally I don’t include -fossils but you’ll see why I’m including it in this case):
$ duplicacy check -a -fossils
Repository set to /Users/me
Storage set to b2://bucket-name
download URL is: https://f002.backblazeb2.com
Listing all chunks
17 snapshots and 1252 revisions
Total chunk size is 477,541M in 121572 chunks
All chunks referenced by snapshot usr-local at revision 1 exist
All chunks referenced by snapshot usr-local at revision 32 exist
...
All chunks referenced by snapshot Documents-other at revision 158 exist
All chunks referenced by snapshot Documents-other at revision 194 exist
Chunk aafaf71f51fa153647ad4266668c63c808439e3162b8a1d4888a93201549f425 can't be found
I checked the storage; the chunk is not in the “aa” directory of the “chunks” directory of the storage. Grepping for the chunk id in all the repositories’ log directories, I find:
Marked fossil aafaf71f51fa153647ad4266668c63c808439e3162b8a1d4888a93201549f425
The explanation on this page says
This is because another ongoing backup that was seen by the prune command may reference any of these chunks. To be safe, the prune command will turn them into fossils, which can be either permanently removed if no such backup exists, or turned back into normal chunks otherwise.
However I don’t see a corresponding log entry saying the chunk was permanently removed. (In contrast, the logs mention other chunks that have been permanently removed.) So I have two issues:
If the logs say the chunk was marked as a fossil, but they don’t say it has been removed, shouldn’t it still exist?
How can I determine which revision the missing chunk belongs to, so I can delete the snapshot as described above? The error message, as I have shown, does not give me a revision number.
I’m running CLI version 2.6.1 (ACEF01) on Mac OS 10.14.6 Mojave. Thanks in advance for your attention.
What is your B2 lifecycle setting? See Should I disable Backblaze B2 Cloud Lifecycle Settings?
If it is not set to keep all versions fossils may be deleted automatically by B2.
Thank you. My B2 bucket lifecycle setting was not keep all versions, and I corrected that. Just so I understand how this is relevant to fossils: Duplicacy marks a chunk as a fossil by moving/renaming it, but B2 doesn’t support move/rename, so a workaround is used that may lead to errors if B2 is not told to keep all versions – is that correct?
My other question is, what’s the easiest way to tell which revision(s) have missing chunks so I can delete them? The check command isn’t telling me.
Apparently a lot of chunks are missing from my backups (found out the hard way). I don’t understand why, the backups were running smoothly (finished without errors) and the prune command should remove entire revisions not single chunks from revisions (I can restore backups, but some chunks are missing).
When running duplicacy check -fossils -resurrect it stops at the first missing chunk, -persist makes it to skip errors.
@gchen Is there any command to prune all revisions which contain missing chunks? Half-working backups aren’t worth much, so I’d much rather have them removed to get an overview if there’s any fully working backup.
wow this sounds like a nice feature request.
I would not do that. Such an option would make users tend to remove problem revisions too quickly – in some cases missing chunks are fixable (such as those caused by a stale local cache).
Then combine this option with dropping the local cache before?
My problem was that half of my revisions had missing chunks, I deleted the local cache before, I had the ressurect option → I had to go through all revisions one by one and delete them manually, which was a lot of work. It also made me lose confidence in duplicacy, because this shouldn’t happen to such an extend‽ It’s not happening in newer revisions, so I hope it’s a bug in the past that got fixed.
Only the prune command can delete a chunk and deletions are always logged in the prune logs. So if Duplicacy removes a chunk by mistake you can always track it down. I would suggest running a check job right after every backup to detect missing chunks early.
Is it also possible that an upload of a chunk failed and thus the chunk is missing?
I have exactly the same problem. How did you solve it?
I hope this helps somebody but my backup kept on experiencing missing chunks and after some head scratching, I learned that the A/V tool considered some chunks to be malicious and QUARANTINE the files. Once I realized that, I recovered the chunks from the quarantine and set the exceptions to the storage folders.