Too late. I had made the storage ages ago. Could I say recreate the storage on the Toshiba and copy across from the wd drive? This is so messed up.
The option is available (in the Web UI at least)… when you initialise a storage - you have an option to make it copy-compatible.
You have to go through the extra steps to make it copy-compatible, because it has to read the configuration and encryption keys of the source storage, which you actively have to choose. It can’t be magically done for you, that’s just the way it is.
Yes you can create a second backup storage, make it copy-compatible and enable Erasure Coding, then copy between them. Later, you could recreate the first storage with Erasure Coding in the same process. Cloud-based storages wouldn’t really benefit much from Erasure Coding though.
aye!
Thank you!
This seems too good to be true, so just to be clear,
Create a new storage on the toshiba which currently has partially copied files from the failing seagate drive as copy compatible and the other option erasure coding? which not 100% clear what that does. i asume make it easer to restore even if files needed are missing.
then just copy the files from the wd drive to the toshiba in windows explorer and thats it? would it be best to have automatic copying so when i back up, it backs up onto both drives at the same time?
Personally, in your situation I’d leave the Erasure Coding til later and concentrate on recovering your data first. (Erasure coding adds parity to chunks and isn’t a 100% guarantee, but it has a chance of recovering data on single drives with occasional bad sectors - but this is for future prevention, not now)
While Duplicacy copy can help you repair bad/missing chunks between storages that are copy-compatible (but not -bit-identical), I don’t believe it has a -persist flag so it may stop when encountering errors. You may not be able to recover all that’s possible to recover.
So if you want to use Windows Explorer you’ll need the storages to be bit-identical and hence use the add command from the CLI, with -copy -bit-identical flags. However, you can completely skip all this and just copy the config file (at the root of the storage) and, so long as it isn’t corrupted, bam you have an empty, copy- and bit-identical storage.
(You could also copy the /chunks and /snapshots directories too but you don’t know their current state…)
One of the advantages of a copy-compatible storage is that chunks are deterministic - the same chunk data (mostly) is produced again, from the same source data. You can therefore pre-seed your new storage with fresh backups from your primary data to dummy snapshot IDs. (Either use dummy backup IDs or delete the /snapshot directory after so you’re left with a bunch of good /chunks.)
A Duplicacy copy from the bad storage to this new storage would skip the chunks already present, and depending on the amount of bad chunks, you may be able to recover historic snapshots with this approach alone.
If Duplicacy copy encounters a corrupted chunk, you should be able to manually copy it over with Windows Explorer, as chunks IDs should have the same name with -bit-identical storages. Then do another copy, or simply use Windows Explorer copy making sure not to overwrite existing chunks.
Again, it really depends on the level of corruption, but the pre-seeding strategy can potentially recreate good chunks where no good chunk existed before. Hence highest chance of recovery IMO.
Right thank you for the help. so could i delete the storages for the toshiba and wd drive. Rename the config file of each storage with a bk extension.
Then, create a new storage for the toshiba drive with erasure and copy compatible with the wd drive.
once done, create a 2nd new storage on the wd drive again with erasure encoding and copy compatible with the toshiba drive?
Again, this needs to be a simple way to fix. Some parts are little jarginated, but reading through erasure i assume means that if essentially if parts of the data files is not recoverable, then the data can be recreated and thus recoverable?
Hmm not sure what you’re trying to do here. The ‘config’ file is specific to the way the chunks and snapshots are encrypted within - you can’t substitute it with another ‘config’, and nor should you mingle chunks from two different storages.
Again, I’d not mess with Erasure Coding till after you’ve recovered your existing storage. Unless you want to start from scratch. (Which you may have to do if your storages aren’t copy-compatible anyway.) To check if they’re copy-compatible, you could try copying a snapshot from one to the other - you’ll get an error if they’re not.
Well, it sounds like i don’t have much of a choice anyhow. as long but i don’t lose the previous revision backups I have made.
Ultimately, I should have done copy compatible so that when I backup, it backs up onto 2 drives at the same time. Which i guess is i should have enabled the copy compatible option. Again, don’t want to lose the backups and previous versions i have already made.
Fair enough, then so long as you have enough disk space, create a separate directory for your new storages and don’t fiddle around with the config file / snapshot dir structure.
ok, but would i not be reset back to revision 1 again. what I want to do is continue from where i left off. on the toshiba drive. so could i just move everything into a new folder? what happens to the previous chunks and backup? i dont want to start back at revision 1 again. im currently on something like revision 30 for the wd drive.
I could set up a copy option to go from the wd to Toshiba in schedule options. also to add is the configs on the toshiba drive and wd are exactly the same.
(Soz for the delay.)
Does the revision number matter? Why not start with different IDs?
Let’s say your last revision for a particular snapshot is 1234 - does that revision, from your source storage, have any corrupted chunks? If not, yea you could probably start from there and keep the existing chunks and revisions.
If it does have corruption, your next incremental 1235 revision may also reference those same missing/corrupt chunks, and it won’t tell you they’re corrupted until you do a check -chunks.
Either way, I’d suggest to start with a storage that is free from missing/bad chunks by making sure to manually delete any bad snapshot revisions or chunks, run -exclusive to clean up, and that check -chunks returns with no errors before resuming backups to that storage - wherever it’s copied from.
Is this an option in the gui? Don’t remember seeing this.
Not really. While you can add most flags to most job types, that one is involved in the add command - which is all handled for you when you add (or init) a storage. Only the equivalent copy compatible (add -copy) option can be done in the Web UI.
However, you can drop to a command line and create the storage there and add the storage in the GUI after. e.g.:
cd <config>\.duplicacy-web\repositories\localhost\all
..\..\..\bin\duplicacy_win_x64_3.1.0.exe -v add -encrypt -erasure-coding 5:2 -copy source-storage-name-as-per-gui -bit-identical new-storage-name-throwaway duplicacyweb "F:/newstoragepath"
Your Web UI config location (.duplicacy-web) would be in C:\Users\<user> or C:\ProgramData - depending on how you installed it.
Then add the new F:/newstoragepath storage to the GUI as normal.
Ok, so this is a command line thing.
In hindsight, I could have saved meself days of figuring this out simply by creating a temp folder, then restoring each revision from the WD to each respected sub folder, then edit the docker unraid template to point to each respected sub folder to be backed up and then just move the chunks and snapshots folder into a sub folder on the wd drive create a new storage on the Toshiba drive. Then make a new storage on the wd drive as copy compatible.
Ok, this very alien to me. it seams the docker version was having issues restoring. im trying to run the commands you have given, but i have zero clue of cli or anything. i am on windows. plan is that if i could set up on windows via cli then i could just migrate to the docker unraid. I tried running this
…\bin\duplicacy_win_x64_3.1.0.exe -v add -erasure-coding 5:2 -copy Hoard-Duplicacy-Backup -bit-identical Toshiba duplicacyweb “D:”
but i end up with Unrecognizable storage URL: D:"
the web ul config is in my users folder at C:\Users\USERNAME.duplicacy-web\repositories\localhost\all
Due to the amount of time this has been going on. I have created a 2nd storage with erasure and made it copy compatible with my original wd drive. I can schedule a copy from the wd drive to the toshiba drive. It should be set as bit identical anyhow if there is a copy option
The storage URL is the last parameter in your command and most likely the error is due to a missing slash after D:. However, I’d advise you create a directory on your D: drive instead of putting everything in the root - i.e. "D:\Duplicacy_Backup".
It won’t have been set as bit-identical if you only used -copy - you have to use both flags.
ok I have saved the duplicacy_win_x64_3.1.0.exe in C:\Users\USERNAME\.duplicacy-web\repositories\localhost\all and opened a cmd prompt and ran what you have said with this output.
duplicacy_win_x64_3.1.0.exe -v add -erasure-coding 5:2 -copy WD -bit-identical Toshiba duplicacyweb “D:\Duplicacy_Backup”
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
Hash key: 6475706c6963616379
ID key: 6475706c6963616379
Data shards: 5, parity shards: 2
C:\Users\USERNAME.duplicacy-web\repositories\localhost\all will be backed up to D:\Duplicacy_Backup with id duplicacyweb
Dono if thats right?
Now what?
You didn’t need to put the exe in there as the leading ..\..\..\ would’ve pointed to the existing executable in .duplicacy-web\bin but, no matter, you managed to create a new storage. 
Now you just add the storage in the Web UI - point it to D:\Duplicacy_Backup, give it a new name and unlock it with the password you supplied.
ah ok. I think I am there. I had managed to create new storage in the subfolder name you said, and i am now backing up to hard drives at the same time with erasure and bit identical (at least thats what I hope, as i did include that in the command.
Sounds good
Incidentally, it just occurred to me that adding erasure coding to a -bit-identical storage - while it seems to be possible - might contradict the purpose of having a bit-identical storage.
The original idea was to support making a copy of a storage with third-party tools such as rsync, and so chunks could be individually swapped out (to repair missing/bad chunks as well).
Hopefully, chunks from two bit-identical storages can be interchanged when one has erasure coding and the other does not. @gchen can you confirm this is safe?
Oh! I hope bit identical becomes an option in the web gui asap. It would be useful if theres some way to “convert” to bit identical and have erasure encoding. which i guess erasure ensures that if a chunk is missing, like i had then there is a chance to recover “something”.
Only thing is if its possible to “resume” a backup i cant leave the pc on for 15 hours with the hard drive racket going on.