Check -chunks command doesn't seem to obey -id and -r arguments

Support
#1

Hi,

I’ve been trying to deal with corrupted chunks due to the sftp bug and have noticed that duplicacy check -chunks with either the -id or -r flags still seem to download and check the entire storage repository based on the number of chunks it reports. For example, duplicacy list -tabular shows that one particular snapshot ID has 18,963 total chunks using up 73gb of space, but duplicacy check -id <snap> -chunks -stats reports that it’s verifying 81,000 chunks, which would be about the total size of all snapshots in the storage. The same thing happens when I try to check a specific revision with -r.

Is this a bug or is check -chunks not supposed to work with these flags? I have quite a lot of data in B2 that I also want to verify but it’s going to be expensive if I have to download the entire bucket to verify only one of the snapshot IDs.

Thanks!

#2

Use -files -id <id> -r <n> instead of -chunks.

IMO a more efficient method would be to do an actual restore of your last revision(s), because if your internet connection is interrupted, you won’t have to start again…

Even more efficient (in terms of de-duplication) would be to do a copy to temporary, local, storage. That act alone verifies all downloaded chunks, and then you could do a final restore or check -files. And you can pick individual revisions.

#3

Thanks for the suggestion, but wow, is duplicacy really that brittle? One transient network error being able to silently render all of your backups useless and require a full restore from remote storage to fix is pretty…concerning, unless I’m misunderstanding something about the process.

#4

I wouldn’t say it’s brittle, nor that a failed backup will ‘render all of your backups useless’. That’s just not so. It’s not supposed to happen. However, if you’re concerned about B2 costs, you have to take extra precautions to minimise bandwidth usage - a check currently doesn’t keep track of progress if aborted. I guess, in future, it could. But there are better options - e.g. copy.

Otherwise, I don’t know of any issues with B2s backend… but this is why backups should be tested regularly.

I’ve only ever witnessed issues with sftp - particularly when the storage space runs out - and so, in that regard, there seems to be a small bug that would result in a bad snapshot being uploaded.

With resilient storage, Duplicacy’s chunk format is still pretty robust in that you have options to fix such situations. The storage format and the engine are actually pretty fantastic and you never really should need to start from scratch. Although, currently, I’d like to to see Duplicacy add more diagnostics (such as skipping bad chunks during restores and checks), so that bad backups can be cleaned up more efficiently.

#5

Otherwise, I don’t know of any issues with B2s backend… but this is why backups should be tested regularly.

This is the part I’m struggling with, because based on your explanation, I don’t see a bandwidth efficient option to test regularly (one that doesn’t involve downloading the entire storage with -chunks, or inefficiently downloading chunks multiple times with -files). Am I missing something?

#6

I consider it the opposite way: if a chunk is corrupted (due to problems with your storage or network) you will lose only the file associated with that chunk. It’s a more robust approach than that of backup software that is based on complete images.

However, I understand that the above argument applies only to discrete files, and does not apply to large single files, such as databases and VMs.

Periodic restore is a good practice regardless of the software / storage you use:

https://www.isaca.org/resources/isaca-journal/past-issues/2012/database-backup-and-recovery-best-practices

I periodically restore random files as a sample of the backup consistency. I never had any problems with :d: + B2 set.

You can restore as @droolio suggested or make a cloud-to-cloud copy with two providers.

1 Like
#7

Can you clarify the exact manual process for fixing this problem if I were to do full restore? I am not sure I understand the steps required to repair the backups after doing this.

I am also curious if there is a technical reason that -chunks can’t just verify specific snapshots or revisions, but files can. I am also curious if there’s a technical reason it has to abort when -chunks finds one bad chunk, requiring you to start the whole process over again to find multiple corrupted chunks.

Thanks!

#8

Take a look:

The answer to this question:

I’ll leave to @gchen :wink:

#9

I understand the process behind finding which files can’t be restored due to corrupt chunks, but I’m not 100% clear on how to fix the backup once you figure out what files are affected.

#10

While there are several ways you can verify, it’s unavoidable that you have to download chunks. The potential issue with check (-files or -chunks) is that you may have to start from scratch if you lose connection, which would be wasteful considering your remote storage charges for egress.

A restore will allow you to resume, and be selective about which revisions you want to test.

But if you have a lot of de-duplication, a more efficient approach would be copy-ing down the storage to a local disk (if you have the spare space). You can resume and it only needs to download each chunk once - whereas a restore may download the same chunk multiple times.

#11

You haven’t come across any corrupted chunks on your B2, though? And a regular check succeeded…

#12

duplicacy check -chunks -stats -threads 20 just completed on my B2 storage…took several days and cost a few bucks but didn’t find any corrupted chunks, so that’s a relief. I’m guessing based on the number of chunks it verified all snapshots in that storage and not just the snapshot ID I ran it from?

#13

check -stats implies -all, and -all “check snapshots with any id”. So yes, you checked all revisions of all snapshots.

That is also why, when you try to specify -id or -r, those are ignored. You need to remove the -stats argument to be able to use those.

1 Like
#14

Ah, I just saw that in the help after scrolling to the right. Thanks @GuillaumeBoudreau!

#15

I am late to the party but will just leave this here:

  • there is no way to avoid doing periodic restores with any backup solution (or equivalent to check -files with Duplicacy) if you want to maintain trust in availability of your data and since Duplicacy does not have sever side component this requires egress of all data you want to check.

  • the good news is that with B2 you can do this for free through bandwidth alliance — backblaze provides free egress from B2 to cloudflare and cloudflare provides personal plan at $0/month.

1 Like
#16

-chunks was a more recent addition and honestly I didn’t think too much about all its use cases. Before I thought a single corrupt chunk is bad enough that we would be more worried about how it happens than if there are more. This is not the case any more unfortunately with the sftp bug.

I’ll add more options to -chunks in the next CLI release.

2 Likes
#17

Thanks a lot @gchen, I really appreciate it.

#18

Ok, final question…what is the process for repairing the backup after you’ve identified which files are affected by the corrupted chunks? I’ve read the other thread and I didn’t really see a resolution other than to keep excluding those files from the restore.

Thanks again.

#19

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.