Fixed Missing Chunk? Please Check My Work

[Wesley]

While recently running a check operation, I received an error that a chunk was missing. I believe I’ve fixed the problem, but I’m hoping someone can review what I did just to assure me I handled it correctly.

In case any of it becomes relevant, here’s how I have my operations scheduled. I’ve got several drives all backing up to a local drive pool, then that gets copied to Backblaze. On Sundays, I run a prune, followed by a check, on both backups.

The problem occurred last Sunday during the check of the local copy, when I received the error that a particular chunk was missing:

Running check command from C:\Users\username/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage Local -a -tabular]
2024-04-21 23:00:01.511 INFO STORAGE_SET Storage set to Z:/
2024-04-21 23:00:01.519 INFO SNAPSHOT_CHECK Listing all chunks
2024-04-21 23:00:17.575 INFO SNAPSHOT_CHECK 6 snapshots and 800 revisions
2024-04-21 23:00:17.602 INFO SNAPSHOT_CHECK Total chunk size is 9084G in 1906697 chunks
2024-04-21 23:00:19.363 WARN SNAPSHOT_VALIDATE Chunk d863f9746fd9d987a24ee665ab0be129953f39f814c928ec4974b2368cfa4e2b referenced by snapshot MALCOLM-Video at revision 636 does not exist
2024-04-21 23:00:19.382 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video at revision 636 are missing
2024-04-21 23:00:20.642 WARN SNAPSHOT_VALIDATE Chunk d863f9746fd9d987a24ee665ab0be129953f39f814c928ec4974b2368cfa4e2b referenced by snapshot MALCOLM-Video at revision 1469 does not exist
2024-04-21 23:00:20.954 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video at revision 1469 are missing
2024-04-21 23:00:22.345 WARN SNAPSHOT_VALIDATE Chunk d863f9746fd9d987a24ee665ab0be129953f39f814c928ec4974b2368cfa4e2b referenced by snapshot MALCOLM-Video at revision 2310 does not exist
2024-04-21 23:00:22.622 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video at revision 2310 are missing
2024-04-21 23:00:23.866 WARN SNAPSHOT_VALIDATE Chunk d863f9746fd9d987a24ee665ab0be129953f39f814c928ec4974b2368cfa4e2b referenced by snapshot MALCOLM-Video at revision 3150 does not exist
2024-04-21 23:00:24.209 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video at revision 3150 are missing
2024-04-21 23:00:25.762 WARN SNAPSHOT_VALIDATE Chunk d863f9746fd9d987a24ee665ab0be129953f39f814c928ec4974b2368cfa4e2b referenced by snapshot MALCOLM-Video at revision 3990 does not exist
2024-04-21 23:00:25.889 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video at revision 3990 are missing
... a lot more references to the same chunk missing from many revisions ...
2024-04-21 23:05:27.564 WARN SNAPSHOT_CHECK Some chunks referenced by snapshot MALCOLM-Video-01 at revision 2089 are missing
2024-04-21 23:05:27.670 INFO SNAPSHOT_CHECK All chunks referenced by snapshot MALCOLM-Video-02 at revision 1 exist
2024-04-21 23:05:27.748 INFO SNAPSHOT_CHECK All chunks referenced by snapshot MALCOLM-Video-02 at revision 924 exist
.. a lot of chunks that do exist ...
2024-04-21 23:12:19.263 INFO SNAPSHOT_CHECK All chunks referenced by snapshot MALCOLM-User at revision 17277 exist
2024-04-21 23:12:19.263 ERROR SNAPSHOT_CHECK Some chunks referenced by some snapshots do not exist in the storage

I referenced the following posts to fix the problem:

Fix missing chunks

Sometimes, when you run the check command, it may complain about missing chunks: $ duplicacy check Storage set to sftp://gchen@192.168.1.125/AcrosyncTest/teststorage Listing all chunks Chunk 02c25aea4621acdd4c8751d5ab7ff438fb47308ce8738f030b7db0741c37ecb5 referenced by snapshot test at revision 1 does not exist Some chunks referenced by snapshot test at revision 1 are missing All other commands can also report the same missing chunk messages. If that happens, it is recommended to run the che…

How to fix Missing Chunk?

How is an initial backup different from running one with the -hash option in this scenario? Will Duplicacy look at hashes in the previous snapshots and assume that if they’re listed in the backup then the chunk must exist – and so skip uploading it?

Here’s what I did:

1. I searched all directories for the chunk. It did not exist.

2. I searched for references to the chunk in all log files under \Users\username\.duplicacy-web\logs to see if it may have been deleted. Nothing showed up.

3. I deleted all “cache” directories under \Users\username\.duplicacy-web. I then reran the check of the local copy. Same errors.

4. I replaced the backup ID “MALCOLM-Video-01” with a new backup ID, “MALCOLM-Video-01-fix”.

5. I ran the new backup, which took about 8 hours. The missing chunk was recreated, along with a few others, apparently.

Running backup command from C:\Users\username/.duplicacy-web/repositories/localhost/1 to back up E:/
Options: [-log backup -storage Local -stats]
2024-04-22 11:18:00.046 INFO REPOSITORY_SET Repository set to E:/
2024-04-22 11:18:00.046 INFO STORAGE_SET Storage set to Z:/
2024-04-22 11:18:00.055 INFO BACKUP_START No previous backup found
2024-04-22 11:18:00.055 INFO BACKUP_INDEXING Indexing E:\
2024-04-22 11:18:00.055 INFO SNAPSHOT_FILTER Parsing filter file \\?\C:\Users\username\.duplicacy-web\repositories\localhost\1\.duplicacy\filters
2024-04-22 11:18:00.055 INFO SNAPSHOT_FILTER Loaded 3 include/exclude pattern(s)
2024-04-22 19:14:18.471 INFO UPLOAD_FILE Uploaded ...
... a whole lot of files uploaded ...
2024-04-22 19:14:19.548 INFO BACKUP_END Backup for E:\ at revision 1 completed
2024-04-22 19:14:19.548 INFO BACKUP_STATS Files: 22405 total, 3550G bytes; 22403 new, 3550G bytes
2024-04-22 19:14:19.548 INFO BACKUP_STATS File chunks: 739924 total, 3550G bytes; 7 new, 58,267K bytes, 49,940K bytes uploaded
2024-04-22 19:14:19.585 INFO BACKUP_STATS Metadata chunks: 42 total, 57,871K bytes; 7 new, 11,321K bytes, 7,845K bytes uploaded
2024-04-22 19:14:19.585 INFO BACKUP_STATS All chunks: 739966 total, 3550G bytes; 14 new, 69,589K bytes, 57,786K bytes uploaded
2024-04-22 19:14:19.585 INFO BACKUP_STATS Total running time: 07:56:19

6. I replaced the backup ID “MALCOLM-Video-01-fix” with the original “MALCOLM-Video-01”.

7. I deleted the “snapshots” folder for “MALCOLM-Video-01-fix”.

8. I again deleted all “cache” directories under \Users\username\.duplicacy-web.

9. I reran the check of the local copy. No more errors.

Running check command from C:\Users\username/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage Local -a -tabular]
2024-04-22 20:29:27.247 INFO STORAGE_SET Storage set to Z:/
2024-04-22 20:29:27.258 INFO SNAPSHOT_CHECK Listing all chunks
2024-04-22 20:29:51.352 INFO SNAPSHOT_CHECK 6 snapshots and 821 revisions
2024-04-22 20:29:51.380 INFO SNAPSHOT_CHECK Total chunk size is 9084G in 1906864 chunks
2024-04-22 20:29:55.313 INFO SNAPSHOT_CHECK All chunks referenced by snapshot MALCOLM-Video-01 at revision 1 exist
2024-04-22 20:29:57.314 INFO SNAPSHOT_CHECK All chunks referenced by snapshot MALCOLM-Video-01 at revision 60 exist
... a lot more existing chunks ...
2024-04-22 21:07:12.770 INFO SNAPSHOT_CHECK 
          snap |   rev |                          | files | bytes | chunks | bytes | uniq |    bytes |    new |    bytes |
... file details ...

Questions:

• The chunk that was created with the same filename is guaranteed to contain the same data as the missing one, correct? (I’m curious to know how that works.)
• Should the additional chunks that were created be of any concern? (No file changes should have occurred on that drive since the last backup. It’s basically a video archive.)
• What should be my next steps? Have I forgotten anything? (I’ve kept all schedules turned off since the original error.)

Thanks for your help!

[saspus]

The missing chunks are usually result of interrupted prune: it deleted chunks but failed to delete the snapshots that referred to those chunks. Then on the next check those snapshots are obviously missing chunks. The solution (two-step snapshot deletion process) was proposed, but not implemented.

You can confirm by reviewing prune log, but I don’t think it would be worth your time.

There is no point in fixing those chunks; those revisions were supposed to be deleted anyway.

To find affected revisions I would run check -persist, note the numbers, and delete them from under the snapshots folder on the target.

Then run prune -a -exhaustive to take care of any remaining orphaned chunks.

@Wesley

The chunk that was created with the same filename is guaranteed to contain the same data as the missing one, correct? (I’m curious to know how that works.)

Yes. It’s called CAS – Content Addressable Storage. The bit of data is uniquely identified by its name (which is a cryptographically strong hash).

@Wesley

Should the additional chunks that were created be of any concern? (No file changes should have occurred on that drive since the last backup. It’s basically a video archive.)

Those additional chunks are likely metadata chunks. The snapshot manifest can be huge, so it itself is split into chunks and stored In the same storage.

@Wesley

What should be my next steps? Have I forgotten anything? (I’ve kept all schedules turned off since the original error.)

I would do the above. I would not turn off any schedules, duplicacy is fully concurrent (as long as you don’t use -exclusive flag).

Another reason for missing chunk is storage failure. It’s very unlikely that backblaze lost a file, unless someone explicitly deleted it. (why would that be the case?). Your arrays too, if properly setup, should not lose data. interrupted prune is far more likely culprit.

But if the data is indeed lost – then what you did is correct.

[Wesley]

Thanks for the reply.

I had been having trouble with my drive bay recently, so I couldn’t be sure if it was an interrupted prune or corrupted data. (I had reports of bad sectors on one of the drives, though those now appear to have been false. Just communication problems with the bay.)

Everything seems to be error-free with Duplicacy at this point, but I went ahead and ran check -persist on both the local and Backblaze copies and received no errors.

Then, I ran my prune operations on both with -exhaustive added received the following for the local copy:

2024-04-28 12:19:58.205 WARN CHUNK_UNKNOWN_FILE File de/_ED6A7B83-7AAA-4935-A929-EBF3EDAB3657.copytemp is not a chunk
2024-04-28 12:20:05.850 WARN CHUNK_UNKNOWN_FILE File cc/_2CCA6778-F5FD-4244-9D44-BB37D6494520.copytemp is not a chunk
2024-04-28 12:20:14.377 WARN CHUNK_UNKNOWN_FILE File ba/_C91CE70F-C9FE-4E78-8A23-41B584A67FD1.copytemp is not a chunk

Any idea what these are? I assume I can just delete them?

[saspus]

you can delete those files. They are likely artifacts of failures to save the chunks.

If you suspect the storage may not guarantee integrity (i.e. if bad block can damage data) and want to ensure that there is no data corruption – run check -chunks -all. This will check consistency of every chunk. If you have already ran it in the past, you would want to delete the verified_chunks (or similarly named) file in the local .duplicacy directory, to force it to re-check chunks that could have been already validated in the past.

[Droolio]

Since you have *.copytemp files, my guess is you use DrivePool under the hood? Perhaps you were running a balance at the time of the check (or it was aborted prior and they left behind). See here:

Is it safe to remove *.copytemp files?

[Wesley]

Good catch. Yeah, I’m running DrivePool. It’s no surprise it happened, because the Sabrent drive bay I’m using keeps losing its connection, which is how all this started.

And it’s started doing it again while I’m trying to run check -chunks -all. I’ve ordered a Mediasonic bay to replace it, so I’ll have to continue this adventure once I get that swapped over.