"Failed to load chunks for snapshot X at revision Y: unexpected end of JSON input" for copied storage

Support
#1

I performed a copy from disk storage to b2 which I believe completed successfully, but now that I am attempting to check (or add a revision) to any of the IDs that were copied I always get a ‘chunk missing’ error and a json error

example output for a check

Chunk 7e9c0dcf00e74c89983eb7cd66c90d5977ae0c811b328f846c7a0ab092b4e2a3 can't be found
Failed to load chunks for snapshot t450 at revision 1: unexpected end of JSON input

this is a fatal error and cannot by bypassed with -persist.

I recall seeing there was a message about an “old format” for the snapshot when doing the copy.

I can confirm chunk 7e9c0d… is not present (checked 7e directory and looked for 9c0d in b2).

Is this missing block the cause of the JSON error or is there something in the (encrypted) snapshot directory which is in a bad state?

#2

found this description of how some of the chunks are metadata chunks and if they are missing, will cause the check to fail with json error as the metadata chunk contains necessary json to continue. Prune aborts with "chunk can't be found" error/warning - #18 by Droolio

I checked every id I copy-ed and every one ended up with this error. I should add a check action to my procedure before deleting the older storage…

#3

You can recover from this situation by deleting (or, to be safer, renaming) the /snapshots directory on B2 and retry the copy job. It’ll skip all chunks that have successfully been copied, then populate a new /snapshots directory. Then you should run a check again.

#4

alas, I needed to move it for a reason and cleared up the space first. I should have known better.

#5

I found a (file) copy of the original repo buried somewhere and was able to validate it with check with a report that “all chunks exist” and am running a verification with --chunks now.

While the verification runs, I performed another copy after clearing the snapshots and ran into the same “chunk can’t be found” error upon check after an apparently successful copy. It may be notable that the missing chunk is not present in the --source chunks directory. Is it possible there is an issue with the chunks which represent the snapshot metadata during a copy?

#6

I see the entry in the log from the copy for the claimed missing chunk:

I did a copy with --threads, so I believe there are a few that are interleaved, so pulled out the relevant logs

Copying snapshot backup8 at revision 1
Chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc has been loaded from the snapshot cache
...
Copying chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc to 7368e7f2795cb35900bae6b4514b44c14cb9b347456009e7326c5c8220588d17
...
Chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc has been downloaded

but no other entry mentions 7368e7f2795cb35900bae6b4514b44c14cb9b347456009e7326c5c8220588d17 and it is not present in b2 (assuming I understand how they are stored in prefix directories) even though it was clearly attempting to upload something…

I would have expected an http entry uploading it or an “already exists”

I ran with -v -d so not sure what additional visibility I can get beyond network monitoring. I don’t have B2 access logs available in my account.

#7

Keep in mind, that if you didn’t initialise the destination repo with --bit-identical, the source chunk filenames won’t match that of the destination.

#8

thanks, that tracks with the “copy X to Y” logs, but the problem I am seeing is that the “to” chunk never appears to make it to the destination storage during copy.

#9

I think this may has something to do with the local cache. Clear the local cache (rm -rf .duplicacy/cache) before the copy job. You can run a small test to verify that everything works.