Missing chunk, can't figure out why

[saspus]

I have scheduled check followed by prune on B2.

Check complained about missing chunks, which prune subsequently tried to fossilize but that chunk was apparently already a fossil. There is no other mention of that chunk anywhere. What could have happened?

alex@Tuchka:/var/services/homes/duplicacy-web/.duplicacy-web/logs$ grep -C 2 -R "cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae" .
./check-20200506-033001.log-2020-05-06 03:32:51.778 INFO SNAPSHOT_CHECK All chunks referenced by snapshot tuchka at revision 747 exist
./check-20200506-033001.log-2020-05-06 03:32:53.783 INFO SNAPSHOT_CHECK All chunks referenced by snapshot tuchka at revision 774 exist
./check-20200506-033001.log:2020-05-06 03:32:57.397 WARN SNAPSHOT_VALIDATE Chunk cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae referenced by snapshot tuchka at revision 776 does not exist
./check-20200506-033001.log-2020-05-06 03:32:57.620 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot tuchka at revision 776 are missing
./check-20200506-033001.log-2020-05-06 03:32:59.570 INFO SNAPSHOT_CHECK All chunks referenced by snapshot tuchka at revision 798 exist
--
./prune-20200506-034655.log-2020-05-06 03:48:49.054 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 891
./prune-20200506-034655.log-2020-05-06 03:48:50.480 INFO SNAPSHOT_DELETE Deleting snapshot tuchka at revision 892
./prune-20200506-034655.log:2020-05-06 03:59:54.253 WARN CHUNK_FOSSILIZE Chunk cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae is already a fossil
./prune-20200506-034655.log-2020-05-06 04:00:14.924 INFO FOSSIL_COLLECT Fossil collection 13 saved
./prune-20200506-034655.log-2020-05-06 04:00:15.081 INFO SNAPSHOT_DELETE The snapshot tuchka at revision 776 has been removed

The graph for some reason went into a bad state too:

[gchen]

There are another set of prune logs under /var/services/homes/duplicacy-web/.duplicacy-web/repositories/localhost/all/.duplicacy/logs, which should give more information on that chunk.

[saspus]

Hmm. I’m now completely confused

alex@Tuchka:/var/services/homes/duplicacy-web/.duplicacy-web/repositories/localhost/all/.duplicacy/logs$ grep -R "cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae" .
./prune-log-20200503-042409:Marked fossil cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae
./prune-log-20200506-034657:Marked fossil cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae
./prune-log-20200506-142217:Deleted fossil cba6d3dbb864ec31270dcfb57025ec426dc414877c2fdcc7890410b43b5bbeae (collection 13)

So, 05/06 it was marked fossil and then shortly after deleted, while it it definitely was used in one of the existing snapshots. Why could that be allowed to happen?

[gchen]

This is my interpretation: prune-log-20200503-042409 found this chunk was unreferenced after deleting some old revisions. At that time revision 776 was perhaps still in progress so prune-log-20200503-042409 didn’t know that this chunk was needed by revision 776.

But prune-log-20200503-042409 only marked this chunk as a fossil so it still existed in the storage. check-20200506-033001 saw revision 776 but it didn’t look for the fossil so it complained about the missing chunk (I believe at this time if you run the check command with -resurrect it would have recovered the chunk). After that prune-20200506-034655 came in and this time revision 776 was to be deleted and that chunk was again marked as a fossil.

So all is good. Since revision 776 has been deleted you don’t really need to worry about this chunk. But I think you can still recover this chunk if you want: being a fossil it is just hidden by b2_hide_file, so if you create a new dummy file with the same name on the b2 website then you should be able to see previous versions.

[saspus]

Sounds plausible. That backup dataset however does not change much – maybe that was a contributing factor.

I wish there was a way to not report harmless failures like that in the UI – seeing “check failed” is a bit unnerving.

Would that be remedied by running prune before check? It feels that its better to first check and then prune, no?

[gchen]

Add the -fossils option to the check job and this kind of errors won’t happen again. Maybe -fossils should be the default?

[saspus]

It might make sense to do that by default.

My understanding is that since fossilization is an inherent part of designs the state where a chunk refers to a fossil is not an exception or error but an expected behavior.

Therefore the datastore is not in a bad state — and hence the check should succeed.

I’m not 100% sure about this but it seems this way.

[Droolio]

I’ve always wondered why -fossils is an option in the first place - why wouldn’t it be a good idea to always check for fossils if its non-fossil counterpart couldn’t be found?

Likewise, where would it be a bad idea to always run check with -resurrect too? Should it be avoided when running other jobs, and that’s why it’s an option?

[09b074d1d348bf2d688a]

I saw this same scenario and was quite worried when I saw the “chunk does not exist” message until I came across this point.

I’m going to be using the --fossils option from now on, but I would vote for it being the default as that would reduce warning messages that don’t require action.