Just wondering what I should do about this error from two storages on two separate computers
2020-03-08 08:35:11.457 FATAL DOWNLOAD_CHUNK Chunk 12d74b249747e65f71769ef9243ccd302e389d693ed89a08af0c3bb4ea9ebfbb can't be found
Chunk 12d74b249747e65f71769ef9243ccd302e389d693ed89a08af0c3bb4ea9ebfbb can't be found
I found some info in the Guide on thisā¦ I will try to figure it out.
OK so I worked through the guide on this.
The prune logs do not indicate ANY chunks EVER being removed from what I can tell
The actual missing chunk file(s) do not appear anywhere on the storage from what I can tell
So who knowās what happened.
I have two questions I guess:
1). Want to get it fixedā¦ the guide says delete snapshot files until the missing chunk message goes awayā¦ do you mean delete ārevisionsā? I only have ONE snapshot ā¦ but a bunch of revisions in that snapshot.
2). I understand I can run a check after a backup to make sure this doesnt happen againā¦ BUTā¦ I dont see a way to trap or catch this via the GUIā¦ are we talking about making a custom script to catch this condition and notify the user?
Thanks for your patience and help!!
Did two computers complain about the same chunk?
Yes, delete ārevisionsā.
Are you still running the old GUI? This is now supported in the new web GUI.
I am sending backing up to two different computers ā¦ each one complains about a different chunk.
I am using the web GUI so will look to see about adding a check operation after the backup. If the check failsā¦ will I be notified somehow or I need to check the logs daily?
OK I will delete revisions until the check runs clear then re-backup. Correct?
Thanks!!
Before deleting revisions, I would run check with the -fossils option just in caseā¦
And maybe delete the local cache directory. I always though the check operation would tell you what revisions the chunk was missing from, so the fact your logs donāt say anything, might suggest these are metadata chunks? In which case, your cache (metadata) might be old?
-fossils did not identify the chunksā¦ it seems they are missing for real. 
To delete snapshots do I do that from a command line or can it be done from the gui?
Actually- I donāt really understand why I just canāt simply tell Duplicacy to fix it upon the next uploadā¦ is there not a way to do this?
Thanks!!!
OK so I am using prune to try to delete a bunch of old revisions and hopefully get to a point where I dont need the missing chunk.
I added. -r 1-60 to the prune arguments in the gui and ran it.
I get this:
Running prune command from /Users/kevinvannorsdel/.duplicacy-web/repositories/localhost/all
Options: [-log prune -storage storage_timberwolf -r 1-60]
2020-03-09 20:48:45.800 INFO STORAGE_SET Storage set to sftp://backups@blahblah:22//share/homes/backups/KevinV
2020-03-09 20:48:46.397 INFO SNAPSHOT_NONE No snapshot to delete
What does it mean āno snapshot to deleteā? I am trying to delete a REVISION right?
Thanks again for any help.
Itās still not clear if the missing chunks in your case, are from historic revisions or is missing from the last backup. Your logs should say what revisions the missing chunk pertains to. So hopefully you only have to delete revisions that are affected.
Also, if itās not missing from the last backup, you only need to delete the revisions theyāre missing for. If itās missing for the last backup, you need to āfixā the missing chunk or delete enough revisions until you have an intact most-recent snapshot as your last revision.
I wouldnāt use prune to delete these revisions as it may abort when it discovers a chunk is missing, and cause more chunks to be renamed to fossils and not properly collected.
To delete the revisions, you simply delete the numbered files under the storage in the \snapshots\<repository_id> folderā¦
As far as fixing it upon the next upload, itās not quite as simple as that. To enable incremental backup, Duplicacy assumes that all the chunks referenced in the last revision actually exist. It doesnāt check if they do, so will skip them regardless.
But you can often force these missing chunks to be re-uploaded (if itās still part of you existing repository and not old data thatās long been deleted). One way to do this is to create a new temporary backup ID - pointing at the same repository location. Running such a backup will check which chunks exist on the storage and then rescan the whole repository and re-upload everything, skipping chunks that donāt need to be uploaded, uploading only the missing chunk(s). This should fix things but thereās a small chance the chunk doesnāt get recreated from the current state of the repository.
Oh and if you delete any revisions, keep a copy or put them in the recycle bin. If youāre able to restore the missing chunk(s), those revisions will be good again. Put them back and your storage should be good again.
Thanks much!
One question if I mayā¦ what logs do you think will tell me what revisions need the missing chunk?
I am not seeing that info in the check logā¦ is there a different log I should look at?
Thanks again.
Yes, it should be in the log for the check job. On the Dashboard at the bottom under Activities, hover your mouse over the red line - a ācheckā should show up which you can click and that particular log opens.
For reference, an example line should look like (scroll right):
2019-04-14 04:21:42.950 WARN SNAPSHOT_VALIDATE Chunk 605dcc6c289af06ebe14e8978028b7eea03b5e1fcc8332198c3950c0ff122221 referenced by snapshot Redacted at revision 231 does not exist
Edit: Oh and you may have several of those lines, followed by a summary for each revision:
2019-04-14 04:21:44.980 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot Redacted at revision 231 are missing
wow I dont see anything like thatā¦ perhaps I need to upgrade? Here is all I see:
Running check command from /Users/kevinvdel/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage storage_timberwolf -a -tabular]
2020-03-09 00:09:39.707 INFO STORAGE_SET Storage set to sftp://backups@my_machine//share/homes/backups/KevinV
2020-03-09 00:09:40.085 INFO SNAPSHOT_CHECK Listing all chunks
2020-03-09 00:10:20.963 INFO SNAPSHOT_CHECK 1 snapshots and 63 revisions
2020-03-09 00:10:20.965 INFO SNAPSHOT_CHECK Total chunk size is 311,203M in 79497 chunks
2020-03-09 00:10:20.969 FATAL DOWNLOAD_CHUNK Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd can't be found
Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd can't be found
Do you think it is a version thing?
It shouldnāt be - Duplicacy Web should automatically download the latest CLI engine, although I think it only does that on startup. Double-check that your Web client is up-to-date, though.
Maybe the chunk in question is a metadata chunk, and therefore may not highlight which snapshot revision itās failing to load the data for?
Otherwise, you should see a line for each revision like so:
2020-03-12 08:46:00.545 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dolores-c_users_droolio at revision 1 exist
2020-03-12 08:46:00.556 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dolores-c_users_droolio at revision 34 exist
2020-03-12 08:46:00.571 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dolores-c_users_droolio at revision 67 exist
2020-03-12 08:46:00.588 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dolores-c_users_droolio at revision 240 exist
...
2020-03-12 08:46:09.532 INFO SNAPSHOT_CHECK All chunks referenced by snapshot dolores-c_users_droolio at revision 5750 exist
etc.
Reviving thisā¦ I backed off all the way to revision 1 and a check ran cleanā¦ then less than a few days later I am getting this again from check.
I am getting fairly desperate hereā¦ any other ideas?
Running check command from /Users/kevinvsdel/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage storage_timberwolf -a -tabular]
2020-03-29 05:57:46.411 INFO STORAGE_SET Storage set to sftp://backups@kv.dns.org:2222//share/homes/backups/KevinV
2020-03-29 05:57:46.928 INFO SNAPSHOT_CHECK Listing all chunks
2020-03-29 08:32:46.975 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2020-03-29 08:32:55.394 INFO SNAPSHOT_CHECK 1 snapshots and 12 revisions
2020-03-29 08:32:55.395 INFO SNAPSHOT_CHECK Total chunk size is 323,570M in 83835 chunks
2020-03-29 08:32:55.400 FATAL DOWNLOAD_CHUNK Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd canāt be found
Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd canāt be found
Does this chunk exist in the storage?
If not, run the check command on one revision at a time (duplicacy check -r n) to see which revision this chunk is in. And the go through all the check logs (~/.duplicacy-web/check*.log) to find out since when the chunk became missing, and also all prune logs (~/.duplicacy-web/prune*.log) to see if any prune command remove this chunk.
OK so this is weirdā¦ thanks @gchen for your info.
I did this on the command line:
duplicacy -log init -e kevin_salvage1 sftp://backups@kv.dns.org:2222//share/hoxdrmes/dataps/KevinV
duplicacy -log check
And got this:
2020-03-30 18:57:06.098 INFO SNAPSHOT_CHECK Listing all chunks
2020-03-30 18:57:12.875 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 1 exist
2020-03-30 18:57:13.158 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 2 exist
2020-03-30 18:57:13.430 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 3 exist
2020-03-30 18:57:13.702 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 4 exist
2020-03-30 18:57:13.995 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 5 exist
2020-03-30 18:57:14.279 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 6 exist
2020-03-30 18:57:14.551 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 7 exist
2020-03-30 18:57:14.823 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 8 exist
2020-03-30 18:57:15.098 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 9 exist
2020-03-30 18:57:15.375 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 10 exist
2020-03-30 18:57:15.716 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 11 exist
2020-03-30 18:57:16.031 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 12 exist
2020-03-30 18:57:16.314 INFO SNAPSHOT_CHECK All chunks referenced by snapshot kevin_salvage1 at revision 13 exist
So it looks like everything is good from this point of view
But if I look at the GUI logs I see this:
and if I click on āMissing Chunksā I see these messages
Running check command from /Users/kevinvsdel/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage storage_timberwolf -a -tabular]
2020-03-29 05:57:46.411 INFO STORAGE_SET Storage set to sftp://backups@kv.ds.org:2222//share/hXXXXX/ups/KevinV
2020-03-29 05:57:46.928 INFO SNAPSHOT_CHECK Listing all chunks
2020-03-29 08:32:46.975 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2020-03-29 08:32:55.394 INFO SNAPSHOT_CHECK 1 snapshots and 12 revisions
2020-03-29 08:32:55.395 INFO SNAPSHOT_CHECK Total chunk size is 323,570M in 83835 chunks
2020-03-29 08:32:55.400 FATAL DOWNLOAD_CHUNK Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd canāt be found
Chunk db537514f0d6d66e4662622438b843cbab8f042f7a4ecaef0e6de78f3d7628cd canāt be found
Can you pls advise what I should try next?
Is it bad that the Check is the first thing in my schedule?
I suspect that youāre running an early version of Duplicacy CLI (and the old GUI) where there are 2 nesting levels for chunks, whereas recent versions of Duplicacy all uses a nesting level of 1. If the chunk db5375... is stored as chunks/db/53/75... then the nesting level is 2.
If that is the case then the fix is to write a script that move all chunks up one level.
OK can you clarify? The screen shot in my post above I think shows that I am using the web guiā¦
It is possibly that when I ran command lineā¦ I used an older version of the CLIā¦ but that said all chunks were present.
I am awful confused.
All chunks are there, but they are under subdirectories that are 2 levels down, while the web GUI expects them to be under subdirectories that are only 1 level down.
Can you first check the chunks directory on the storage to confirm that this is the case?
