Help with a Duplicacy backup strategy

[Macrodata]

I am trying to find a backup strategy for my NAS where I can protect two systems from each other in case one is attacked or affected by ransomware.

I have an Unraid server and a Windows gaming computer. I want to maintain server-backups on the Windows-computer in case the server loses data.

My theory so far is to make Ⓐ (the live Unraid server) do the backups with Duplicacy and let my gaming computer Ⓑ copy the backups from the server.

If Ⓑ is hacked the attacker will not be able to read the data on Ⓐ. The backup is encrypted and the attacker will need the encryption key from Ⓐ to decrypt. The attacker might destroy data on Ⓑ, but the server data is backed up on Ⓐ.

On the other hand, if Ⓐ is hacked the attacker will not be able to read the contents of Ⓑ because the server does not have any access at all. Likewise the attacker might destroy data on Ⓐ, but the data is backed up on Ⓑ.

This feels like a solid plan, but here is the thing I am struggling with…

How should I handle incremental backups? I mean sure, Duplicacy will take care of that on Ⓐ, but I don’t want to transfer the entire backup every night to Ⓑ. It would be better to just transfer the increment. But how do you do that?

Should I use rclone to incrementally sync Duplicacy’s storage folder
and copy each new chunk files in /chunks/ and add a new /snapshots/... file.

or…

should I perhaps run Duplicacy on both computers with the copycommand.

[fronesis]

I confess that the logic of your overall strategy may be a bit beyond me, but I’m fairly confident in saying the following: If you want to create a duplicacy encrypted, incremental backup, and then you want to have a second copy of that backup somewhere else, you shouldn’t use rclone to copy the second storage destination, you should use duplicacy copy.

It would be different if you wanted to take one source data (say, a set of files on your desktop computer) and, first, rclone copy all or part of that set of files to remote A and then use duplicacy to make backups to remote B. But if you’ve already made a duplicacy backup to A, you don’t want to rclone from A to B. You want to duplicacy copy from A to B. I’m not even completely clear if the rclone copy version would be readable or usable to restore.

[Macrodata]

No I think you get the gist. You are spot on my question.

I forgot to mention that I am using the web GUI version of Duplicacy. I am not sure if copy is available in it.

Can I ask why you think it is a better idea to use Duplicacy over rclone?

[fronesis]

SEE @saspus below – I stand corrected!

original post follows, now superseded…

@Macrodata

Can I ask why you think it is a better idea to use Duplicacy over rclone?

If the goal is to have two encrypted duplicacy backups in two different places, either of which you could recover from if the other went down, then you need to be able to restore directly from either.

Duplicacy copy is designed to do this: to make sure that the backup on storage B is exactly like the backup on storage A, and that you could init a new repo anywhere, connect to the one storage still alive, and restore.

Here’s what I don’t know, but suspect: if you rclone copy your duplicacy files to a second location (B), and then lose access to the first storage location (A) – I don’t think rclone will preserve the data on the new storage in the exact same way. I don’t know that you could connect to that directory on storage B and have duplicacy work.

Someone smarter than me will, I hope, correct me if I’m wrong, but I don’t think this is what rclone copy is designed to do. If I take my files directly, not my duplicacy chunks, and rclone copy them somewhere else, then I can always go get them there – they are just my files. But when you rclone copy hashed duplicacy chunks to a totally different storage – I think there’s a good chance that data will be inaccessible. This is why when you init a new storage in duplicacy, you need to set the -copy flag and the -bit-identical flag. Rclone has no way to do that for you; it wasn’t built to deal with encrypted duplicacy chunks; it was built to deal with files.

[saspus]

Duplicacy chunks are just files, and encryption keys are in the config file, so if the goal is to create a full copy of a duplicacy storage you can do it with rclone, rsync, cp, etc. it’s literally nothing more than a bunch of files.

Best Practice: copy from local server to remote

In that case you can just use rsync/rclone to copy the entire storage directory to your remote server. The storage directory is designed to be location-independent, so you can freely move it from one place to another. There used to be a difference in the nesting levels between local and cloud storages, but with version 2.0.10 all storages have a nesting level of 1 so it is fully compatible with rsync/rclone.

What you cannot do is selectively copy chunks between two independently initialized but unrelated storages and expect it to work: (different encryption keys would be a dealbreaker, different chunking would lead to massive overheard)

When you init storage with -bit-identical flag — the encryption keys, chunking, and compression (?) parameters are copied. So with either storage duplicacy will generate same, but-identical chunks, and those can be copied between storages.

With -copy-compatible flags chunking parameters are the same but encryption can be different. You cannot copy chunks directly of course, but duplicacy copy command decrypts and re-encrypts data in chunks to make it possible.

If repositories were created without the copy-compatible flags, they will have different chunking parameters and while copy may still work, duplicacy disallows it.

How to make an existing backup copy compatible?

Maybe we should remove the constraint of being copy-compatible for the copy command to work. Technically, it is completely fine when you inject into an existing storage some chunks generated with different size and chunkseed parameters. As long as you can decrypt these chunks and the referencing snapshot files, these foreign backups should be self-contained and readable. The only issue is that you would lose the benefit of deduplication. New backups, even if all files remain the same, will c…

The common approach for having two storages is to backup to one and then copy to another.

The ransomware aspect can be easily dealt with by configuring target as WORM storage, or giving duplicacy credentials that only allow to write but not delete or modify. It can be done easily with TrueNAS and Samba, with B2 Backup Immutability - Object Lock support? - #22 by tallgrass, and with some dance using other devices, until duplicacy supports object lock.

@Macrodata

server-backups on the Windows-computer in case the server loses data.

I would not trust windows to save the day. If unraid and zfs failed, widows would have been long gone. Instead, backup important bits to the cloud or replicate to another zfs server (zfs send /zfs receive).

[Macrodata]

I am not sure if I completely understood everything you said there, but thanks for taking time to explain.

If I understand you correctly, because of the file structure of Duplicacy chunks, using rclone to make a copy of a backup shouldn’t be a problem. I just have to make sure I also grab a copy of the config file so I can decrypt the files later?

When you say “selectively copy chunks” does this mean I should not “delta copy” (copy only changed chunks)? Obviously what I am trying to avoid is copying the entire backup if only certain chunks have changed.

[saspus]

@Macrodata

If I understand you correctly, because of the file structure of Duplicacy chunks, using rclone to make a copy of a backup shouldn’t be a problem. I just have to make sure I also grab a copy of the config file so I can decrypt the files later?

Right, if the goal is to create a 1:1 copy of duplicacy datastore elsewhere, you can use rclone/rsync/resilio sync/etc, to keep remote copy up to date, as new chunks are added. Duplciacy does not care where the datastore is located, and does not rely any other metadata other than file names. So you can backup to e.g. your NAS, then copy everything to Amazon S3 and have duplicacy restore from them (as an example, it’s probably not a best way to restore full backup)

@Macrodata

When you say “selectively copy chunks” does this mean I should not “delta copy” (copy only changed chunks)?

You can delta-copy, and tools like rsync will do that automatically. The end result is that you have a copy elsewhere. And duplicacy does not care how was that copy created.

What I meant is that duplicacy has its own internal copy mechanism, that is more fine-grained compared to copying entire datastore. For example, you can copy just specific “snapshot IDs”, or even speicifif revisions.

Another advantage of using duplicacy copy over external copy program is that duplicacy copy will decrypt and verify every chunk as part of repackaging data for the target storage. This extra step can be very valuable if your source storage does not provide data integrity guarantees, for example, a local hard drive with filesystem without self-healing support.

But if that is not needed – full copy with rclone/rsync/robocopy/teracopy/whathaveyou is just fine.

[Macrodata]

Thanks for staying with me. Much appreciated.

@saspus

Another advantage of using duplicacy copy over external copy program is that duplicacy copy will decrypt and verify every chunk as part of repackaging data for the target storage. This extra step can be very valuable if your source storage does not provide data integrity guarantees, for example, a local hard drive with filesystem without self-healing support.

That sounds like an advantage. But security-wise is it a good idea? I assume both computers will need to have the decryption key saved? If computer (B) is attacked the attacker can decrypt the copied data from (A), right?

[saspus]

Do you mean the scenario where you backup from machine A to storage B, and then machine B copies to storage C? Then yes, to use duplicacy copy machine B would need to have a key, and that need is avoided by the use of rclone and copying/syncing the whole thing.

On the other hand, if machine A and B are local, you can use machine A to duplicacy copy between storage B and storage C. And since machine A has raw data anyway, no extra security exposure occurs. Neither B or C need to have a key.