Thank you, I will use dd rescue to make an img file and then work from that. However, what if the chunks cant be recovered? I would have a backup that i cant restore, as some previous versions earlier are showing up fine.
What’s stopping me from copying from another backup i had made onto another hard drive onto the new toshiba drive, rather than trying to rely on the failing drive? Could i not merge 2 different backups of the same source onto the new toshiba drive?
It depends. Did you create both storages as --bit-identical? If not, then you definitely cannot mix and match chunks from different storages. If your storages are bit identical, then you might be able to repair missing chunks. But there is no guarantee that you would have the same chunks on both storages, unless you simply cloned these. Still, there is a decent chance that you may find your missing chunks in this case.
If you still have missing chunks in some snapshots, then you should be able to recover snapshots that do not have missing chunks. If you have some missing chunks in a snapshot but these chunks are not metadata chunks, you should be able to do a partial recovery (basically, all the files that do not reference these chunks). If missing chunks are metadata chunks this would be a total loss of a snapshot.
Am I right that, bit identical means backup onto 1 hard drive, then copy the backup onto another? so you end up with identical backups
No, this is a special flag on storage creation, meaning that certain internal keys/seeds are shared. In particular, this means that chunk file names would be the same for the same chunks.
oh! dont think that had been used. is there a way to add this so that i can be used or does it need to have been enabled from the start?
Needs to have been done from the start. But your other backup drive (WD)… was that storage initialised with at least the copy-compatible option (again, from the start)? Coz then you might be able to repair the missing/bad chunks - even if they’re not -bit-identical.
In this case, you could temporarily rename your /snapshots directory out of the way and do a Duplicacy copy job from good to bad (after having cloned as much of the bad to a good drive with ddrescue). Then put back the /snapshots directory and see what’s corrupt with a check -chunks.
In future, create your storages to be copy-compatible with each other. Consider also creating single, local, drives to use Erasure Coding. Then you’ll have a much better chance of backup storage repair. Run checks and occasional restores often, use third party tools (e.g. Stablebit Scanner) to detect for drive failure earlier.
Again, I’m not even sure. I set duplicacy using hot io container for unraid following the ibracorp guide.
I guess i could reinialise the Toshiba drive or simply copy from the wd drive to the new Toshiba drive?
Why were these options not enabled by default, nor can I re enable them later down the line. What’s the point in having previous versions backups if 1 file is not available then the whole backup is useless? Seams very vulnerable to me?
It’s ok as the seagate drive I had stop using as it got full.
Too late. I had made the storage ages ago. Could I say recreate the storage on the Toshiba and copy across from the wd drive? This is so messed up.
The option is available (in the Web UI at least)… when you initialise a storage - you have an option to make it copy-compatible.
You have to go through the extra steps to make it copy-compatible, because it has to read the configuration and encryption keys of the source storage, which you actively have to choose. It can’t be magically done for you, that’s just the way it is.
Yes you can create a second backup storage, make it copy-compatible and enable Erasure Coding, then copy between them. Later, you could recreate the first storage with Erasure Coding in the same process. Cloud-based storages wouldn’t really benefit much from Erasure Coding though.
aye!
Thank you!
This seems too good to be true, so just to be clear,
Create a new storage on the toshiba which currently has partially copied files from the failing seagate drive as copy compatible and the other option erasure coding? which not 100% clear what that does. i asume make it easer to restore even if files needed are missing.
then just copy the files from the wd drive to the toshiba in windows explorer and thats it? would it be best to have automatic copying so when i back up, it backs up onto both drives at the same time?
Personally, in your situation I’d leave the Erasure Coding til later and concentrate on recovering your data first. (Erasure coding adds parity to chunks and isn’t a 100% guarantee, but it has a chance of recovering data on single drives with occasional bad sectors - but this is for future prevention, not now)
While Duplicacy copy can help you repair bad/missing chunks between storages that are copy-compatible (but not -bit-identical), I don’t believe it has a -persist flag so it may stop when encountering errors. You may not be able to recover all that’s possible to recover.
So if you want to use Windows Explorer you’ll need the storages to be bit-identical and hence use the add command from the CLI, with -copy -bit-identical flags. However, you can completely skip all this and just copy the config file (at the root of the storage) and, so long as it isn’t corrupted, bam you have an empty, copy- and bit-identical storage.
(You could also copy the /chunks and /snapshots directories too but you don’t know their current state…)
One of the advantages of a copy-compatible storage is that chunks are deterministic - the same chunk data (mostly) is produced again, from the same source data. You can therefore pre-seed your new storage with fresh backups from your primary data to dummy snapshot IDs. (Either use dummy backup IDs or delete the /snapshot directory after so you’re left with a bunch of good /chunks.)
A Duplicacy copy from the bad storage to this new storage would skip the chunks already present, and depending on the amount of bad chunks, you may be able to recover historic snapshots with this approach alone.
If Duplicacy copy encounters a corrupted chunk, you should be able to manually copy it over with Windows Explorer, as chunks IDs should have the same name with -bit-identical storages. Then do another copy, or simply use Windows Explorer copy making sure not to overwrite existing chunks.
Again, it really depends on the level of corruption, but the pre-seeding strategy can potentially recreate good chunks where no good chunk existed before. Hence highest chance of recovery IMO.
Right thank you for the help. so could i delete the storages for the toshiba and wd drive. Rename the config file of each storage with a bk extension.
Then, create a new storage for the toshiba drive with erasure and copy compatible with the wd drive.
once done, create a 2nd new storage on the wd drive again with erasure encoding and copy compatible with the toshiba drive?
Again, this needs to be a simple way to fix. Some parts are little jarginated, but reading through erasure i assume means that if essentially if parts of the data files is not recoverable, then the data can be recreated and thus recoverable?
Hmm not sure what you’re trying to do here. The ‘config’ file is specific to the way the chunks and snapshots are encrypted within - you can’t substitute it with another ‘config’, and nor should you mingle chunks from two different storages.
Again, I’d not mess with Erasure Coding till after you’ve recovered your existing storage. Unless you want to start from scratch. (Which you may have to do if your storages aren’t copy-compatible anyway.) To check if they’re copy-compatible, you could try copying a snapshot from one to the other - you’ll get an error if they’re not.
Well, it sounds like i don’t have much of a choice anyhow. as long but i don’t lose the previous revision backups I have made.
Ultimately, I should have done copy compatible so that when I backup, it backs up onto 2 drives at the same time. Which i guess is i should have enabled the copy compatible option. Again, don’t want to lose the backups and previous versions i have already made.
Fair enough, then so long as you have enough disk space, create a separate directory for your new storages and don’t fiddle around with the config file / snapshot dir structure.
ok, but would i not be reset back to revision 1 again. what I want to do is continue from where i left off. on the toshiba drive. so could i just move everything into a new folder? what happens to the previous chunks and backup? i dont want to start back at revision 1 again. im currently on something like revision 30 for the wd drive.
I could set up a copy option to go from the wd to Toshiba in schedule options. also to add is the configs on the toshiba drive and wd are exactly the same.
(Soz for the delay.)
Does the revision number matter? Why not start with different IDs?
Let’s say your last revision for a particular snapshot is 1234 - does that revision, from your source storage, have any corrupted chunks? If not, yea you could probably start from there and keep the existing chunks and revisions.
If it does have corruption, your next incremental 1235 revision may also reference those same missing/corrupt chunks, and it won’t tell you they’re corrupted until you do a check -chunks.
Either way, I’d suggest to start with a storage that is free from missing/bad chunks by making sure to manually delete any bad snapshot revisions or chunks, run -exclusive to clean up, and that check -chunks returns with no errors before resuming backups to that storage - wherever it’s copied from.
Is this an option in the gui? Don’t remember seeing this.
Not really. While you can add most flags to most job types, that one is involved in the add command - which is all handled for you when you add (or init) a storage. Only the equivalent copy compatible (add -copy) option can be done in the Web UI.
However, you can drop to a command line and create the storage there and add the storage in the GUI after. e.g.:
cd <config>\.duplicacy-web\repositories\localhost\all
..\..\..\bin\duplicacy_win_x64_3.1.0.exe -v add -encrypt -erasure-coding 5:2 -copy source-storage-name-as-per-gui -bit-identical new-storage-name-throwaway duplicacyweb "F:/newstoragepath"
Your Web UI config location (.duplicacy-web) would be in C:\Users\<user> or C:\ProgramData - depending on how you installed it.
Then add the new F:/newstoragepath storage to the GUI as normal.