I have read a number of topics and posts on this topic (e.g. Move duplicacy folder/Use symlink repository, Filters/Include exclude patterns, Improving the instructions for include/exclude patterns (filters)) and I think I understand, but I want to double check.
I have a directory at /backup
that I intend to be my symlink repository. My first attempt at setting this up looked like this (please don’t make fun of my default rpi user haha):
/backup
├── boot
│ ├── cmdline.txt -> /boot/cmdline.txt
│ └── config.txt -> /boot/config.txt
├── etc -> /etc
├── home
│ └── pi -> /home/pi
└── var
├── backups
│ ├── group.bak -> /var/backups/group.bak
│ ├── gshadow.bak -> /var/backups/gshadow.bak
│ ├── passwd.bak -> /var/backups/passwd.bak
│ └── shadow.bak -> /var/backups/shadow.bak
├── log -> /var/log
└── www -> /var/www
There are only a couple of specific files I want out of /boot
, so I created a directory at /backup/boot
and symlinked those two files in there. All of /etc
can come (this is obviously not the greatest plan, but I didn’t feel like sifting through everything while I’m evaluating Duplicacy). There is only one specific user home directory that I want to back up, so once again I created /backup/home
and symlinked /home/pi
in there. The /var
situation is even more complicated, because there’s a ton of stuff in there that I know I don’t care about, but there is also a pretty scattered collection of stuff that I do want. So, I ended up with the symlink structure you see in the tree above.
However, as I’m sure you all well know, this kind of mutli-level symlink structure does not work with Duplicacy. When I run a duplicacy backup
job on this repository, the etc
files are the only files that get successfully captured in a revision. From the documentation on symbolic links:
I’m curious about that "by default"
, but for now I’m just taking it as gospel that there is no way to get Duplicacy to follow symlinks that exist at any depth of the directory structure other than the root level of the repository. So, with that in mind, I recreated my /backup
directory as follows:
/backup
├── boot -> /boot
├── etc -> /etc
├── home -> /home
└── var -> /var
Obviously Duplicacy will follow these root-level symbolic links. However, now I need to create a complex .duplicacy/filters
file in order to curate my repository such that it matches my original plan. I haven’t tested it yet (because, frankly, I’m finding this a little frustrating), but I believe this set of filters should work, maybe?
+boot/cmdline.txt
+boot/config.txt
+boot/
+etc/*
+etc/
+home/pi/*
+home/pi/
+home/
+var/backups/group.bak
+var/backups/gshadow.bak
+var/backups/passwd.bak
+var/backups/shadow.bak
+var/backups/
+var/log/*
+var/log/
+var/www/*
+var/www/
+var/
-*
However! Even if those filters do work, I’m not sure I would be particularly thrilled with this setup. I was very optimistic about the original — but doomed to fail — symlink repository structure, particularly due to the ease with which I could survey exactly which files were going to be backed up (i.e. simply by browsing the directory structure of /backup
). Under the new “correct” symlink structure, one must consult the .duplicacy/filters
file and reconcile those filter rules against the repository directory structure with meticulous care in order to ascertain the true nature of the files to be backed up. Am I missing something, or is this pretty much the shape of it? Is there some secret method for coercing Duplicacy into following symbolic links that are encountered deeper into the repository than the root? I can’t find any mention in the documentation if so. Or, are filters like this the only option?
Edit: I should add that in all other aspects of my evaluation, Duplicacy is proving to be a fantastic and delightful tool, so I don’t mean to sound negative on it overall. Duplicacy has successfully backed up multiple terabytes of my most sensitive and irreplaceable data (family photos, mostly) to B2, and the data deduplication approach has cut the storage cost to a fraction of what it would be if I simply copied my data up to a B2 bucket via rsync or similar. Very impressive!