Fix missing chunks

Sometimes, when you run the check command, it may complain about missing chunks:

$ duplicacy check
Storage set to sftp://gchen@192.168.1.125/AcrosyncTest/teststorage
Listing all chunks
Chunk 02c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5 referenced by snapshot test at revision 1 does not exist
Some chunks referenced by snapshot test at revision 1 are missing

All other commands can also report the same missing chunk messages. If that happens, it is recommended to run the check command instead as it can identify all missing chunks at once for a given snapshot, without any side effects.

Clear the cache

One common cause of missing chunks is a stale cache. This can happen for example when a revision is manually removed from the storage and then a new backup is uploaded with the same revision number. The revision file stored in the cache is still the old one and thus may reference some chunks that have already been deleted.

For this reason it is now recommended that the first thing to do when you see the above error is to completely remove the cache directory. The cache is usually located at the directory .duplicacy/cache under the current repository. For the location of the cache in the web GUI, please refer to Cache usage details.

Check the storage if the missing chunk actually exists on the storage


:exclamation: Note: The folder of the chunk are the first two characters of the chunk name, for example, chunk

02c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5

is in the folder 02

and it’s the file c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5


If the same command with a clean cache still produces the same error, the next step is to check by hand if those chunks actually exist on the storage. Some cloud storage services (such as OneDrive and Hubic) have a bug that prevents the complete chunk list to be returned. In other cases, a chunk may be stored in a wrong folder. For instance, the expected path for the chunk 02c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5 may be chunks\02\c2\25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5, but if it were stored as chunks\02\c225aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5, Duplicacy would have difficulty locating it.

Check if the chunk was not deleted by prune

If a chunk reported as missing in fact does not exist in the storage, then you may need to find out why it is missing.

The prune command is the only command that can delete chunks, and by default Duplicacy always produces a prune log and saved it under the .duplicacy/logs folder.

Here is a sample prune log:

$cat .duplicacy/logs/prune-log-20180124-205159
Deleted chunk 2302e87bf0a8c863112bbdcd4d7e94e8a12a9939defaa8a3f30423c791119d4c (exclusive mode)
Deleted chunk 7aa4f3192ecbf5a67f52a2e791cfac445116658ec1e3bd00f8ee35dda6964fb3 (exclusive mode)
Deleted chunk 02c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5 (exclusive mode)
Deleted chunk dbbd5c008e107703e59d8f6633d89f9a55075fa6695c113a2f191dd6cddacb53 (exclusive mode)
Deleted chunk 611c478edcc4201f8b48e206391e9929359e71eb31691afc23fb059418d53fb5 (exclusive mode)
Deleted chunk 297dcc3d83dc05b8e697535306a3af847435874cbe7d5a6b5e6918811d418649 (exclusive mode)
Deleted cached snapshot test at revision 1

This log indicates that these chunks were removed when the prune command was invoked with the -exclusive option, because these chunks are only referenced by the snapshot to be deleted, and the -exclusive assumes there weren’t any other ongoing backups.

This is an excerpt from another prune log:

Marked fossil 909a14a87d185b11ec933dba7069fc2b3744288bb169929a3fc096879348b4fc
Marked fossil 0e92f9aa69cc98cd3228fcfaea480585fe1ab64b098b86438a02f7a3c78e797a
Marked fossil 3ab0be596614dd39bcacc2279d49b6fc1e0095c71b594c509a7b5d504d6d111e
Marked fossil a8a1377cab0dd7f25cac4ac3fb451b9948f129904588d9f9b67bead7c878b7d0

These chunks weren’t immediately removed but rather marked as fossils. This is because another ongoing backup that was seen by the prune command may reference any of these chunks. To be safe, the prune command will turn them into fossils, which can be either permanently removed if no such backup exists, or turned back into normal chunks otherwise. Please refer to Lock free deduplication algorithm for a detailed explanation of this technique.

If you can find the missing chunk in any of these prune logs (on all the computers which backup and prune to this storage!), then it is clear that the prune command removed it in the exclusive mode or marked it as a fossil (which may be removed at a later time). If you think the prune command mistakenly removed or marked the chunk due to a bug, post a bug report in the forum with relevant logs attached.

:exclamation: Exceptional cases

Please be aware there are some corner cases when a fossil still needed may be mistakenly deleted.

Backups lasting longer than 7 days

If there is a repository doing a backup which takes more than 7 days and the backup started before the chunk was marked as fossil, then the prune command will think that that particular repository becomes inactive and will be excluded from the criteria for determining safe fossils to be deleted.

Initial backups

The other case happens when an initial backup from a newly recreated repository that also started before the chunk was marked as fossil. Since the prune command doesn’t know the existence of such a repository at the fossil deletion time, it may think the fossil isn’t needed any more by any snapshot and thus delete it permanently.

-exclusive mode

If you see from the log that a missing chunk was deleted in exclusive mode, then it means that the prune command was incorrectly invoked with the -exclusive option, while there was still a backup in progress from a different computer to the same storage.

Fixing a missing chunk

In all these cases, a check command after the backup finishes will immediately reveal the missing chunk.

What if the missing chunk can’t be found in any of these prune logs? We may not be able to track down who the culprit was. It could be a bug in Duplicacy, or a bug in the cloud storage service, or it could be a user error. If you do not want to see this happen again, you may need to run a check command after every backup or before every prune.

Is it possible to recover a missing chunk? Maybe, if the backup where the missing chunk comes from was done recently and the files in that backup haven’t changed since the backup. In this case, you can modify the .duplicacy/preferences file to assign to the repository a new id that hasn’t been used by any repositories connecting to the same storage, and then run a new backup. This backup will be an initial backup because of the new repository id and therefore attempt to upload all chunks that do not exist in the storage. If you are lucky, this procedure will be able to produce an identical copy of the missing chunk.

If you are uninterested in figuring out why the chunk went missing and just want to fix the issue, you can keep removing by hand the affected snapshot files under the snapshots folder in the storage, until the check -a command passes without reporting missing chunks. At this time, you should be able to run new backups. However, there will likely be many unreferenced chunks in the storage. To fix this, run prune -exhaustive and all unreferenced chunks will be identified and marked as fossils for removal by a subsequent prune command. Or if you’re very sure that no other backups are running, prune -exhaustive -exclusive can remove these unreferenced chunks immediately.

Can I just rm the invalid chunk from the storage, and the next backup will re-upload it (if necessary)?

Simply deleting the bad chunks won’t get Duplicacy to re-upload them, so this is not a solution.

4 Likes
How to fix Missing Chunk?
Prune operation returned an error, FATAL: chunk xxxx can't be found
Missing chunks in initial backup
Lots of missing chunks with pCloud
Backups have stopped, no emails reporting it and missing chunks
So one corrupted chunk can destroy all my backups?
Can't recover from Chunk not found error during prune
FATAL DOWNLOAD_CHUNK Chunk (w/ Wasabi)
Chunks have size of zero
Chunks have size of 0
Three subsequent checks return three different missing chunks
Auto-fix Missing Chunks
Configuring multiple storage backends in Web GUI
How can I tell which snapshot references a missing chunk?
Failure Validating Backups
SFTP issues with Synology
Restore in GUI doesn't expand
Serious problem with missing chunks
FATAL Chunk can't be found
How to fix Missing Chunk?
Duplicacy User Guide
Deleted chunk during backup
If you're running prune and backup in parallel, please upgrade to 2.1.1
Missing chunks (ZFS volume)
Prune does not delete snapshots
Missing chunk in a storage: can I fix it using a second storage?
Missing Chunks, Manually Locating Chunks Show Duplicate Chunk Files?
Missing chunk and .tmp files
Missing chunks after running prune
`duplicacy check -all -fossils -resurrect -persist` not skipping errors
Error download_chunk
My check fails with "Missing Chunks"
My check fails with "Missing Chunks"
My check fails with "Missing Chunks"
My check fails with "Missing Chunks"
Prune -all not working after changing odb-token
ERROR Chunk has a size of 0
Chunk xxx can't be found

4 posts were split to a new topic: Missing chunks (ZFS volume)

@gchen: hasn’t this been fixed when you added the support for multiple nested levels for the chunks folder?

From what i remember (and i have a bad memory) duplicacy now searches for some versions by default (even if that file which contains the nested levels doesn’t exist).

@gchen is there a need for the repo to be re -created? (i just want to delete the “re” if it’s ok)

As per the response in FATAL DOWNLOAD_CHUNK Chunk (w/ Wasabi) it should be pointed out that if someone deletes the snapshots they should also delete the .duplicacy/cache to make sure it works.

2 Likes

I’ve updated the guide. Thanks for pointing it out.

3 Likes

Can this please be automatic? i.e. if I run duplicacy -prune -r 1000-1003 I’d expect duplicacy to manage cache accordingly and keep it up to date. Or simply nuke it for me.

I knew about this and yet wasted few minutes today with this issue again… I would not expect users to go and read the documentation; they will panic and go create new support topic…

4 Likes

In the current implement the prune command does delete the copy from the cache when deleting a snapshot file, but it only does it in the cache under the current repository. It can’t do it for other repositories on the same computer or a different one.

Oh, you are absolutely right. Now thinking about it that’s exactly what happened. Maybe Duplicacy should annotate the storage with which client last performed prune and clients would distrust cache if it wasn’t them?

I think the solution is to compare the timestamp of the cached copy with that of the file in the storage. However, due to an oversight in the design, the backend API doesn’t return the modification times when listing files in the storage (although most storages should support it).

Where do I find that file in a duplicacy-web install (on linux)?

Those preferences files are auto-generated in the web GUI so it is not recommended to modify them. If you want to change the repository id (which is called a backup id in the web GUI), just create a new backup with a new backup id.

1 Like

Is there a way to duplicate and modify an existing backup? In order to follow the above instructions I obviously also need to use the same filters…

You can edit ~/.duplicacy-web/duplicacy.json directly – find the backup in computers -> repositories and then change the id.

Changing that doesn’t change the backup ID in the UI. Will it still work?

Forgot to mention that you’ll need to restart the web GUI for the changes in duplicacy.json to take effect. Better yet, edit duplicacy.json while the web GUI is not running otherwise your changes may be overwritten.

1 Like

Can I restart the web-ui while a backup job is running?

Here is my reply from the other thread earlier today:

The CLI can be terminated any time and it shouldn’t leave any half-uploaded files on the cloud storage server, if the server behaves properly, because the content length is always set and the server should never store an incomplete chunk file shorter than the content length. OneDrive for Business is an exception but we’ve fixed that in the latest CLI release by using a different upload API.

For non-cloud storages like sftp and local disk, the CLI uploads to a temporary file first and then rename the temporary file once the upload completes. Aborting should cause any partial upload.

So are you saying that restarting the web-ui will stopp the cli but ir doesn’t matter?

BTW: you can quote text from other topics/threads. That will create links between those topics.

So I just waited for the backup to finish and then edited duplicacy.json, then restarted the web-ui. The new backup-ID showed up in the ui, and the backup went through without problems. But I don’t think it worked as intended because it uploaded tons of files that were supposed to be excluded (and which were excluded before renaming the ID). Might it be that renaming the repo in the .json file results in duplicacy displaying filters in the web-ui but not actually applying them?