I have Duplicacy installed on my local Unraid machine. I use Unraid for several things:
Local Backup storage for all the computers in my home, using hourly/daily Arq and Windows File History configurations (let’s call that repo Backups).
The second is for all my media at home (XX terabytes, let’s call that Media).
Lastly is the general settings and files for the Unraid install itself plus some miscellaneous stuff (let’s call those Other)
I only use Backblaze B2 as a disaster recovery option, for the case when my physical Unraid machine would be completely hosed and I’d need to rebuild (it has dual parity so is locally protected from drive issues). Keeping B2 cost down is definitely a goal here.
I really just want to keep the latest version of Backups and Other backed up on B2, let’s say once a week. I don’t want to keep Media on B2. I don’t need versioning.
I have Duplicacy working but am confused with pruning, copying, and the difference between snapshots and chunks. I’m impressed with how configurable it is, but I’m finding it’s duplicating (ha!) some of the functionality I’ve already solved (via Arq).
What would be the main settings for me to keep the slimmest B2 footprint, only keeping one version per week, of two Backup Directories? Should I be using Copy, or Backup? Really appreciate any help.
My backup needs are very similar to yours, including having a Backups, Media, and Other group. The most versioning I need is to be able to restore a file from the most recent daily backup. For me, any data that requires more than a daily backup is also covered by Syncthing (e.g., personal wiki, SSH keys, etc.).
It’s a lot like Legos…
Chunks are Lego bricks. A Snapshot is a boxed set of Lego bricks with instructions (one or more revisions) for assembling one or more models (files).
Although you can keep the original boxes to separately store the instructions and bricks for each set, combining all of the bricks and discarding any extra duplicates can save a lot of space in your storage bin (deduplication).
From that storage bin (i.e., DAS, NAS, cloud storage), you only need to be able to reassemble any given model (file) per its instructions (revision in a snapshot) with the available collection of bricks (chunks).
At some point you might decide that you no longer need a particular Lego model so you discard (prune) the instructions (revision in a snapshot) from your storage bin, which then might leave you with bricks (chunks) that aren’t used by any remaining model (file). Those bricks (chunks) that are left over (fossils) can either be permanently disposed of (e.g., duplicacy prune -exclusive -exhaustive), or left in the storage bin for another time when a model (file in a revision) might need to replace a damaged or lost brick (chunk).
If you run Duplicacy only on your Unraid NAS, one possible backup scenario might be:
cd /Backups ; duplicacy init Backups b2://bucketname
cd /Other ; duplicacy init Other b2://bucketname
cd /Backups ; duplicacy backup
cd /Other ; duplicacy backup
cd /Other ; duplicacy prune -all -exclusive -exhaustive -keep 0:7
Repeat steps 3-5 once each Sunday night.
Steps 1 and 2 initialize two source directories (repositories) named Backups and Other, assigning the snapshot IDs “Backups” and “Other”, respectively.
Combining Backups and Other snapshots into the same B2 bucket simplifies backup management and makes better use of deduplication.
The retention policy -keep 0:7 means “delete any revisions older than 7 days”. Because the backup is run only once every Sunday night, you end up with only a single revision of each snapshot at any given time. However, it also means that the backup could be up to 7 days old, i.e., you could lose up to 7 days worth of data if your B2 bucket is your last resort.
For your current setup, duplicacy copy would be useful for replicating a snapshot between your Unraid NAS and B2. The two snapshots would contain the same files but could have different security settings, e.g., unencrypted on your Unraid NAS but encrypted on B2 (family members could then restore files without having the password and/or encryption key). You could also replace Arq with Duplicacy on all the computers in your home, backup to the Unraid NAS, then duplicacy copy the latest snapshot revision(s) from the Unraid NAS to B2 once a week.
You didn’t mention how you currently back up to B2 (Arq?)…
I don’t think it makes a lot of sense to use Duplicacy to back up Arq’s backup files because Arq already does deduplication and compression. As you already noted, Arq and Duplicacy share a lot of basic features, so what is it about Duplicacy that’s making you contemplate using it instead of only Arq?
Personally, I would use either Arq or Duplicacy, but not both. Arq is more desktop-focused and only supports Windows and macOS so it was a nonstarter for me. At a bare minimum I must have Linux and Windows plus a CLI interface. (A GUI is nice, but I can live without it.)
FWIW I’m using the GUI in a Docker container, so whatever limitations that comes with. I’m able to follow along with the command line settings you mentioned however.
There are two different backup processes I have implemented. The first is local backups for all my machines at home. I use Arq for this because it’s easy to install/update, works for multiple platforms, and is user-friendly enough for my family to manage and troubleshoot. The Arq backups use my LAN to upload to the aforementioned Backups share in my local Unraid server. They are, like you mentioned, pretty efficient and I have plenty of space so have no problem letting them run often, and I have the server heavily monitored and protected with dual parity.
(I realize Arq can also write to B2. I’ve considered this, but I don’t need that level of granularity and cost, nor do I want that much WAN traffic.)
The second process is to backup my Unraid server (minus media), including the Backups share. This is for redundant, off-site emergency use only, for if my server itself was damaged or stolen. I really just want an easy to way to get my data back to whatever “new” replacement server I could put together. I’m totally file with losing up to 7 days of info if it means amost things can be restored and the system settings of Unraid plus all the Dockers I run can be set-up again easily. Not sure if this matters, but if I were to have the Unraid lost, I’d be more concerned with getting it back online and then resuming local backups, than the other way around.
I guess what I’m realizing is I just need a snapshot of my Unraid server stored in B2, updated every week. I’ll play around with the settings you recommending (particularly the keep and prune settings which were the most confusing).
I think for what you’re looking to accomplish, there’s no disadvantage to using the GUI (with or without Docker) instead of the CLI. The example CLI commands I provided just saved me a lot of typing compared to trying to describe the steps for the GUI.
Using Arq would require accessing the Backups network share on your Unraid server from a Windows/macOS machine in order for Arq to backup the contents to B2. Because of that, running Duplicacy on your Unraid server would actually be more efficient.
If your Arq backups are already encrypted, there might not be much additional benefit to adding another layer of encryption in Duplicacy (leave the password box empty when setting up the storage to disable encryption). Adding more encryption slows down the backup process and complicates recovery (yet one more password/key to manage).
In Duplicacy web edition, I’d start with the following configuration:
Add a B2 bucket under the Storage section.
Add a backup under the Backup section. Directory field is the directory used by Arq on your Unraid server, Storage field is the storage name assigned to the B2 bucket under the Storage section, and Backup ID field is the label for the snapshot (e.g., “Arq” would be a good one).
Add a schedule under the Schedule section. Set it to run only once a day and only on one day of the week (e.g., 12:00am every Sunday). Then add two jobs: a backup followed by a prune (leave the Parallel checkbox disabled).
For the prune job, edit the Options column, changing it to -all -exclusive -exhaustive -keep 0:7.
The single daily run per week combined with the pruning, you’ll end up with only one snapshot containing one revision on B2 (the revision number will increment by 1 every week).
Since Arq already de-duplicates and compresses its data chunks, it’s likely not worth tweaking the chunk size in Duplicacy because there’s going to be very little additional deduplication.
Once you’ve got one backup running successfully, then it’ll be a breeze to add a second backup to cover settings from your Unraid server. The second backup doesn’t need to do any pruning since the first backup already takes care of it.
To mirror Backups to B2, perhaps tools like rclone fit better. Since it’s from arqbackup, there’s no need to waste power and time for versioning, chucking, encryption etc. A simple rclone sync with a weekly cron job should do it.
As for Other, it doesn’t sound like a lot of files. Any settings with duplicacy should be good
