How to debug the check cli command with Wasabi

Support
#1

Hi everyone, I have a Wasabi Bucket with this stat : 690 144 M 143 442 chunks.

1 snapshots and 2 revisions

I upload at around 25 MB/s and I download at around 90 MB/s.

I have been running that command for 3 days in a row without any success :

duplicacy_win_x64_2.7.2.exe check -files -chunks -tabular -stats - threads -4

The only output that I have is this :
All files in snapshot Wasabi at revision 1 have been successfully

The command didn’t finish yet and I don’t understand why. Why this is so long to complete that task ?

Is there a way to verbose what is going on or a log to check what the command check is doing ?

Thanks :stuck_out_tongue:

#2

You can add -d to the command.

However why are you adding -chunks -files? This results in download of your entire datastore. Wasabi guarantees data consistency, there is no need to do that. It is however arguably useful to check that all chunks are present and Duplicacy data structures such as snapshots are consistent. Simple check would suffice. I would however add -fossils flag to the check command. And not run it if prune is in progress because you will get false positives.

#3

Hi, so you recommend this basically :slight_smile:

duplicacy_win_x64_2.7.2.exe check -fossils -tabular -stats -threads 4

#4

I would go a step further and do just this: duplicacy_win_x64_2.7.2.exe check -fossils, just to keep it simple. But generally, yes.

To elaborate, this will not validate content of the files, and relies on you trusting wasabi or whatever other backend you will be using to not corrupt files. There have been issues in the past with various backends (OneDrive that would keep partial file during failed upload leaving corrupted chunk, Backblaze returning bad data via API, etc, those issue were backend’s fault and fixed since. you can search this forum for the reference). There might be similar intermittent issues with other backends in the future and ultimately the only reliable test of whether you can restore your files is to actually try to restore the files (which -chunks -files comes very close to emulating).

But running that kind of tests periodically will not be feasible or productive (as this will at least result in full download of the whole dataset), and the only reason for doing it anyway would be if the storage provider cannot be trusted. Why use it then? Switching to another would be saner choice.

The thing is one should draw a line somewhere – trust the RAM works, trust CPU is computing correctly, TCP does not corrupt packets, encryption is not broken, and storage companies know what they are doing.

I would also trust duplicacy to work properly, but just checking for presence of chunks is cheap so might as well do it. Personally for the many years of [ab]using Duplicacy all the issues the check command encountered for me were self-inflicted or false positives (such as when prune deletes chunks but then fails to delete the snapshot file – that snapshot file will not pass check next time obviously. This is why IMO duplicacy should first delete snapshot file and only then delete chunks. Maybe even do this a two step process – fossilize the snapshot, and then every prune delete chunks from fossilized snapshots, and only once all are gone – delete the fossil. But that’s different conversation).

An alternative solution would be to never prune and never check: if you don’t let duplicacy delete anything – nothing can get corrupted. Storage is cheap… but then again, this is all covered by simple and cheap check that only requires download of a small amount of metadata.

#5

I’ve been following the topic, very interesting!

My 2 cents: I have 1.5TB on Google Drive (3 accounts), I work with them mounted with Rclone + encryption (Cryptomator has some problems with portable executables), and I backup them with Duplicacy to Wasabi (I’ve already used Wasabi years ago and had some problems, I went to B2 and now I’m giving Wasabi another try)).

This setup works perfectly…

P.S.: I have a small stack of hard drives here, does anyone want? :laughing:

#6

Meh, I don’t care what anybody says; if you don’t test a full restore of at least your more recent snapshots, semi-regularly (at least once, then maybe once or twice a year), you don’t really have a good backup strategy. This is a solid principle of backups in general, and goes beyond the integrity of your media, the code-worthiness of the software, or about the data itself.

You’re not just testing backups, you’re testing a disaster recovery scenario.

Did you do everything right when setting up the backup jobs? Did you forget to include or accidentally filter something? Did something change?

Where are you keeping your encryption password? Hopefully not in a password database that’s stored only in the backups.

How quickly can you recover with a cloud-only restore; let’s say your local backups and your mobile phone (often used for MFA et al) are out of the picture - do you know the steps, say, to authenticate with your cloud provider. It’s not only about the data. Test, test again, and test often. :slight_smile:

1 Like
#7

I agree, restore must be tested, ideally from scratch on a test VM (for the reasons you listed, including keys); annually or semi-annually is the right frequency for that, I did not argue with that. Not tested == broken.

What I did argue against was using --chunks --files with routine check (which often is scheduled after prune; there is no other reason to run it – backup workflow only adds files, never deletes) or running any check at all: the full check is is one inch away from actually restoring files so why not do that instead and remove the last leap of faith that the backup software can check itself?