"Failed to load chunks for snapshot X at revision Y: unexpected end of JSON input" for copied storage

Support
#1

I performed a copy from disk storage to b2 which I believe completed successfully, but now that I am attempting to check (or add a revision) to any of the IDs that were copied I always get a ‘chunk missing’ error and a json error

example output for a check

Chunk 7e9c0dcf00e74c89983eb7cd66c90d5977ae0c811b328f846c7a0ab092b4e2a3 can't be found
Failed to load chunks for snapshot t450 at revision 1: unexpected end of JSON input

this is a fatal error and cannot by bypassed with -persist.

I recall seeing there was a message about an “old format” for the snapshot when doing the copy.

I can confirm chunk 7e9c0d… is not present (checked 7e directory and looked for 9c0d in b2).

Is this missing block the cause of the JSON error or is there something in the (encrypted) snapshot directory which is in a bad state?

#2

found this description of how some of the chunks are metadata chunks and if they are missing, will cause the check to fail with json error as the metadata chunk contains necessary json to continue. Prune aborts with "chunk can't be found" error/warning - #18 by Droolio

I checked every id I copy-ed and every one ended up with this error. I should add a check action to my procedure before deleting the older storage…

#3

You can recover from this situation by deleting (or, to be safer, renaming) the /snapshots directory on B2 and retry the copy job. It’ll skip all chunks that have successfully been copied, then populate a new /snapshots directory. Then you should run a check again.

#4

alas, I needed to move it for a reason and cleared up the space first. I should have known better.

#5

I found a (file) copy of the original repo buried somewhere and was able to validate it with check with a report that “all chunks exist” and am running a verification with --chunks now.

While the verification runs, I performed another copy after clearing the snapshots and ran into the same “chunk can’t be found” error upon check after an apparently successful copy. It may be notable that the missing chunk is not present in the --source chunks directory. Is it possible there is an issue with the chunks which represent the snapshot metadata during a copy?

#6

I see the entry in the log from the copy for the claimed missing chunk:

I did a copy with --threads, so I believe there are a few that are interleaved, so pulled out the relevant logs

Copying snapshot backup8 at revision 1
Chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc has been loaded from the snapshot cache
...
Copying chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc to 7368e7f2795cb35900bae6b4514b44c14cb9b347456009e7326c5c8220588d17
...
Chunk 415a5e6c800415be24673970c265317caa06ead8b152e0aefb79635043a7a2bc has been downloaded

but no other entry mentions 7368e7f2795cb35900bae6b4514b44c14cb9b347456009e7326c5c8220588d17 and it is not present in b2 (assuming I understand how they are stored in prefix directories) even though it was clearly attempting to upload something…

I would have expected an http entry uploading it or an “already exists”

I ran with -v -d so not sure what additional visibility I can get beyond network monitoring. I don’t have B2 access logs available in my account.

#7

Keep in mind, that if you didn’t initialise the destination repo with --bit-identical, the source chunk filenames won’t match that of the destination.

#8

thanks, that tracks with the “copy X to Y” logs, but the problem I am seeing is that the “to” chunk never appears to make it to the destination storage during copy.

#9

I think this may has something to do with the local cache. Clear the local cache (rm -rf .duplicacy/cache) before the copy job. You can run a small test to verify that everything works.

#10

@gchen doesn’t look like the cache clearing resolved the isssue

I tried clearing .cache in the directory where I am managing the copy and deleting the snapshot files in the destination.

copy action started up transferring quite a few additional chunks, but still getting the same fail to load chunk error on subsequent check from another directory where I cleared the snapshot cache as well.

check error

rm -rf .duplicacy/cache/default/snapshots/t450
~/duplicacy/duplicacy_main check -chunks -threads 12 -persist -id t450
Listing all chunks
1 snapshots and 12 revisions
Total chunk size is 9021G in 2123226 chunks
Chunk 7e9c0dcf00e74c89983eb7cd66c90d5977ae0c811b328f846c7a0ab092b4e2a3 can't be found
Failed to load chunks for snapshot t450 at revision 1: unexpected end of JSON input

I can see in the log from the copy

Listing revisions for snapshot t450s
Copying snapshot t450 at revision 1
Chunk b3cde9d81c4b62a8c07a6004bb1b6a3b6f4042c37d6c63c86fe8eb414b1b26f9 has been loaded from the snapshot cache
...
Copying chunk b3cde9d81c4b62a8c07a6004bb1b6a3b6f4042c37d6c63c86fe8eb414b1b26f9 to 7e9c0dcf00e74c89983eb7cd66c90d5977ae0c811b328f846c7a0ab092b4e2a3
...
Chunk b3cde9d81c4b62a8c07a6004bb1b6a3b6f4042c37d6c63c86fe8eb414b1b26f9 has been downloaded
...

neither chunk mentioned is in the destination (I made some assumptions about the structure of chunks directories)

➜  tmpdup b2 ls b2://achasen/chunks/7e/ | grep 9c0dc
➜  tmpdup b2 ls b2://achasen/chunks/b3/ | grep cde9d81
#11

You’re likely hitting a bug which can only manifest if the source storage was created by a CLI version older than 1.2.0. Can you check the compression level of the source and destination storages by running:

duplicacy -d list
duplicacy -d list -storage b2

If the compression level of the source storage is 0-9 while that of the b2 storage is 100, then 2 storages are not copy-compatible. The only way to make them copy-compatible is to use the -bit-identical option when creating the b2 storage:

duplicacy add -copy default -bit-identical ...
#12

Seems the compression mismatch appears to be the culprit. On disk shows -1 compression and b2 shows 100 compression.

Does the b2 storage need to be re-created/initialized or can I reuse an existing b2 storage but include the -bit-identical when adding to the backup directory? It doesn’t appear to affect the entry for the storage in .duplicacy/preferences

Possibly notable, the Chunk Seed and Hash Key are identical.

➜  tmpdup /home/achasen/go/src/github.com/gilbertchen/duplicacy/duplicacy_main -d list                    
Storage set to b2://achasen
Download URL is: https://f005.backblazeb2.com
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
...
File chunks are encrypted
Metadata chunks are encrypted

➜  tmpdup /home/achasen/go/src/github.com/gilbertchen/duplicacy/duplicacy_main -d list -storage dup20b8dup
Storage set to /run/media/achasen/Duplicacy/backup8/duplicacy/repo
Compression level: -1
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
...
File chunks are encrypted
Metadata chunks are encrypted
#13

The b2 storage has to be recreated. Yes, the Chunk Seed and Hash Key are identical but unfortunately because of the different compression levels, a new hash algorithm is used so chunk hashes become different.

#14

understood. Any chance someone has come up with a “transfer”/“merge” function for storage? I have manually checked out each snapshot and re-committed to a new storage, but seems like there should be an easier method that I just don’t know about yet.