Prune aborts with "chunk can't be found" error/warning

Support
#1

Hello,
I am having an issue with the prune command. I run the following command:

duplicacy prune -a -storage default -keep 0:14 -keep 1:3 -exclusive

but I receive following error;

Chunk 55381f9bf500ed80a204abd202f52ad3250b115e38af34a36371ab92726cf97c can’t be found

and the prune just aborts and does not complete. All the snapshots which should be deleted are still on the storage location. I check by doing a List.

I looked in the docs, but I could not find any information about what I need to do to solve it. Please advise.

Thanks

#2

Prune log files are kept under .duplicacy/logs. You may be able to find out when this chunk was deleted from these logs files.

Of course first double check the chunk doesn’t exist in the storage, otherwise it is a different issue.

To correct the state of the storage, you need to find out which snapshots reference this missing chunk by running the check command, and then manually remove these snapshot files from the storage.

After that, run duplicacy prune -exclusive -exhaustive to clean up the storage. Remember that no other backup can be running when you run this command (due to the use of -exclusive). This missing chunk is very likely to be the result of running the prune command with the -exclusive while another backup was running.

#3

Ok… Thanks for the instructions.

I found the log file where it was deleted. It does not tell me much except that it was deleted under -exclusive mode as you guessed.

I checked the storage and the missing chunk does not exist. good here.

I then ran the Check command and found out that the first few (i.e. the oldest snapshots) were referencing the missing chunks. So I went ahead and deleted them manually from the storage location, as you explained.

I now have snapshots from 400 to 511

I ran the Check command on these and received response that all chunks for all snapshots 400-511 exist;

C:\Duplicacy\Vaults>duplicacy check -r 400-511
Storage set to C:\BDup
Listing all chunks
All chunks referenced by snapshot SP4-Vaults at revision 400 exist
All chunks referenced by snapshot SP4-Vaults at revision 401 exist
…
…
…
All chunks referenced by snapshot SP4-Vaults at revision 509 exist
All chunks referenced by snapshot SP4-Vaults at revision 510 exist
All chunks referenced by snapshot SP4-Vaults at revision 511 exist

I then ran the Prune command with -exclusive and -exhaustive and received an error that another chunk can’t be found;

C:\Duplicacy\Vaults>duplicacy prune -r 400-511 -exclusive -exhaustive
Storage set to C:\BDup
Deleting snapshot SP4-Vaults at revision 400
Deleting snapshot SP4-Vaults at revision 401
…
…
…
Deleting snapshot SP4-Vaults at revision 509
Deleting snapshot SP4-Vaults at revision 510
Deleting snapshot SP4-Vaults at revision 511
Chunk e067d67852ad05910a06320d4912eaa314293239d71a2f0f5ff3660adf5e6f99 can’t be found

Now this is a different chunk from above.
The Prune command aborts again and the snapshots are not deleted.
I check the logs and the chunk e067d67…6f99 has also been deleted in the log under -exclusive mode and also this chunk does not exist in the storage location.

But now I am baffled what to do.
Check command says all chunks exist while Prune aborts because of missing chunks.

Please advise what to do now. Thanks
PS: sorry if it hard to read, but I can’t seem to get the formatting right to make it easier to read.

#4

There was a bug in the local cache cleaning code that happens if you attempt to delete the latest snapshot with the -exclusive option. This bug had been reported and fixed: https://github.com/gilbertchen/duplicacy/issues/182

You’ll need to build from the latest source to have the fix. But since the bug occurred when Duplicacy tried to clean the local cache as the last step of the prune command, you can also just nuke the entire local cache under .duplicacy/cache and everything should be working fine.

#5

Thanks for the response. I have no idea how to build stuff so I will need to wait for that.

But I did what you suggested. I deleted the local cache under .duplicacy/cache. However I still get the same error and the snapshots folder and snapshots 400 to 511 get rewritten to the cache folder.

C:\Duplicacy\Vaults>duplicacy prune -r 400-511 -exclusive -exhaustive
Storage set to C:\BDup
Chunk e067d67852ad05910a06320d4912eaa314293239d71a2f0f5ff3660adf5e6f99 can’t be found

There are multiple repositories backing up to the same storage location if that would have any influence (which at the time of the prune command, none were performing any backup though). For info, storage location is local. System is Win 10 Pro.

Still no luck. cheers.

#6

Run duplicacy check -a to see which snapshots reference this e067 chunk. It could be a snapshot created from a different repository.

You’ll then need to manually remove any snapshot files under the snapshots directory from the storage that reference this chunk (and all other missing chunks), until duplicacy check -a doesn’t report an error. And run this command to clean up the storage afterwards:

duplicacy -exclusive -exhaustive
#7

Ok thanks. I finally could get it to work and prune the storage. Needed to check all repositories for missing chunks and deleted the individual snapshots as you said.

For some storage locations I have 10-12 repositories backing up to it. This could a lot of work if this happens again and I think it would if I am not careful. I realise now that I need to be more careful using the -exclusive option in Prune.

Is there any way you could build a comprehensive build function into your software to make it automatic. Something to consider, as a feature request.

Thanks again for the instructions.

#8

So, very old topic and I face the same problem today.

However, I don’t have any snapshots but only revisions. Not sure if this makes a difference.

I think having to fix such a situation manually is really PITA. Is there any script or whatever way to do this fix automatically? Or at least in a guided way until all inconsistencies are solved?

#9

This is what I get:

[/share/qnap-backup] # duplicacy prune -r 574-880 -threads 20
Storage set to b2://duplicacy-qnap-backup
Download URL is: https://f001.backblazeb2.com
Deleting snapshot qnap-backup at revision 574
Chunk 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584 can't be found
[/share/qnap-backup] # cd .duplicacy/
[/share/qnap-backup/.duplicacy] # rg 25468d659
logs/prune-log-20231217-161647
1460162:Marked fossil 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584
[/share/qnap-backup/.duplicacy] # duplicacy prune -delete-only
Storage set to b2://duplicacy-qnap-backup
Download URL is: https://f001.backblazeb2.com
[/share/qnap-backup/.duplicacy] # rg 25468d659
logs/prune-log-20231217-161647
1460162:Marked fossil 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584

Why is the chunk still marked as fossil, even I deleted it explicitly? Why do I get the error when deleting the revision range 574-880, even the chunk is marked fossil. How to fix this?

#10

This doesn’t help as well:

[/share/qnap-backup/.duplicacy] # duplicacy prune -exclusive -exhaustive
Storage set to b2://duplicacy-qnap-backup
Download URL is: https://f001.backblazeb2.com
Chunk 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584 can't be found
#11
  1. Run duplicacy check -a -persist
  2. Collect revision numbers that are broken
  3. Go to the storage and physically delete those revisions from snapshots/…/
  4. Run exhaustive prune to get rid of remaining orphaned chunks.

The problem here is that duplicacy prune first deletes chunks and then snapshot revisions. If it is interrupted, those now broken ghost snapshot revisions remain in the storage.

The solution would be for duplicacy to implement two-step revision deletion, but that has never been implemented.

#12

Thanks, will do even the duplicacy check -a -persist command runs pretty long if you have a 20GB bucket of backup data…

#13

I’m still struggling:

# duplicacy check -a -persist -threads 20
Storage set to b2://duplicacy-qnap-backup
Download URL is: https://f001.backblazeb2.com
Listing all chunks
1 snapshots and 59 revisions
Total chunk size is 17925G in 3717135 chunks
Chunk 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584 can't be found
Chunk 2498f2f860506ee3fa3f87a8d8e89ae0aed04cc85145d607e4b7447b57ed98c0 can't be found
Chunk f3f7f155af2fdb44d389c8146ad40487e8829ee3b6f8aa5f3340f08092e43326 can't be found
Chunk e0ee80229f425a4df887c0210736a21174f16d648e1579dc7633884e257e8197 can't be found
Chunk af7cd915a888cdd4269b89546805645b3ee00bcc891aba0efee5000bd62c03a7 can't be found
Chunk b21dc8f205ab03366e21e387e533a3aedd2907dc137ba3526746ff57bf9b1461 can't be found
Chunk 611e8305211f9d6efa2b9e78fbec0df472cf3631da561e70cbab84b79deca293 can't be found
Chunk fa82629975bf802440a386dfe89d4823c1a1c8008cb6f669a199147c0cdb3257 can't be found
Chunk ce9e2483845d52513c8cd596eaf735c5e9ecf7195b27b771a76abe6fbcce4db4 can't be found
Chunk 02ef21fc3785d7350e6da9fbe1f41a00198b34cf846f3c1a4d35dbd2471ad849 can't be found
Chunk 7c2c8a458bf6fe0b68809146b11ba0b24bf1efb0461ec9ad550935da8d4081c6 can't be found
Chunk 0dc31aaa95f93a373637b411adbacdcf93f08eb2d41b00b16e96793d704311db can't be found
Chunk 5359dd5e06603978060424e9c4d5fdf7431925442dc3db209d9ac58f425b4801 can't be found
Chunk 41861accffa979342a154001f8016b26aecab349c05290f5f407d3bbc9e1d526 can't be found
Failed to load chunks for snapshot qnap-backup at revision 574: unexpected end of JSON input

As suggested I deleted revision 574 from the backup storage, and from the local cache folder. Then:

[/share/qnap-backup] # duplicacy prune -exclusive -exhaustive -threads 20
Storage set to b2://duplicacy-qnap-backup
Download URL is: https://f001.backblazeb2.com
Chunk 25468d659e0a4a5b5fb2e90aa7e248f9c5da3f65c3ac118a7eb695f81f020584 can't be found

But I still get the error. I wiped the complete cache folder but still get the same error.

I really don’t understand where this chunk reference is still coming from.

#14

This looks like a corrupted chunk. Duplicacy could not proceed, in spite of -persist flag.

Yes, clear the cache folder, in the hopes that the original chunk on the target is intact. Or since you have deleted the revision this particular chunk won’t be a problem anymore.

Now run check with persists again, to identify the rest of bad snapshots.

#15

Ok, and when I know the revisions I want to prune anyway, can I directly delete them all on the backup storage to avoid iterating step-by-step?

Then wiping the cache and running one final check and prune exhaustive?

#16

Yep. Hence the -persist flag, for duplicacy to proceed after finding issues and ultimately produce list of all affected revisions.

There is no need for final check - but I guess it won’t hurt as as assurance

#17

Ok this works now. Thanks a lot.

1 Like
#18

Note, -persist doesn’t always work as advertised.

It can skip snapshots where there are 100% missing metadata chunks (or the first metadata chunk is missing), but if the snapshot revision (which was supposed to be deleted but got left behind) references at least one present chunk, Duplicacy will error out when finding the next metadata chunk which doesn’t exist.

i.e. When the metadata is chained across several chunks and the last ones are missing, that’s when it chokes and even -persist won’t help. Because it’s hitting unexpected data mid-stream. Ideally, this needs to be fixed.

(In the end, I fixed my own issue by rclone mounting the remote storage and comparing the directory structure with my local storage using Beyond Compare - then manually deleted the orphaned snapshot revsions; there were quote a lot of them.)

1 Like
Check is *very* unhappy after adjusting prune, all revisions now have chunks that don't exist
Cipher: message authentication failed