Recommendations about Duplicacy on Unraid and usage of docker

Support
#1

Hey all,

I’m on Unraid using Duplicacy as my backup solution, it’s installed as a docker container from the Unraid’s app store. I’m creating this topic to discuss a sensible way to setup Duplicacy on Unraid so that it can manage the docker service (and others services if needed), although with least privileges as possible.

The problem in using Duplicacy containerized is trying to manage others docker containers (as already discussed superficially in the forum), and mapping in the docker socket doesn’t seem a sensible approach to me. I have gone back and forth around this subject multiples times, trying to figure out what would be the “recommended” way, by this I mean: the one that better integrates with Unraid’s tools/plugins (e.g., Appdata Backup, User Scripts) and with the least effort/complexity.

Today, Duplicacy is set up as:

  • Every hour updates 3 backups (nextcloud/, appdata/, media/) to local storage.
    • appdata/ comes from the archives (tar.gz) generated by Appdata Backup (it handles the container stop/start). Some parallel question here: as the file is compressed, it can impact Duplicacy backup/restore process? At least, I think it will impact the deduplication heuristics. yes, I shouldn’t compress backup source files
    • nextcloud/ represents the user’s files. They way Duplicacy is set, it makes the backups without managing the service life cycle (this is what raised my eyebrows mainly).
  • Every 4 hours, copy from local storage to remote (S3 bucket).
  • Daily, makes check and prune on local storage.
  • Daily, makes check and prune on remote storage.

(this schedule seems good? feedback appreciated)

I was happy with this setup for months, until I notice that the backup can be corrupted (I think) by some other process writing to the file being backup-ed (e.g., Nextcloud user’s files as commented previously).

My plan is:

  • Migrate Duplicacy from docker to standalone. I think I just need to download the linux binary and copy the existent config files to some place, correct? Probably, create a service to start it at login yada yada…
    • My idea is to keep it simple here, but if needed I can discard the GUI and use the CLI directly.
  • Manage Nextcloud service backup appropriately, using stopping/starting hooks of the containers to signalize it (probably via Appdata Backup). I pretend to create some script to properly dump the DB (as I’m not doing it now) and selectively save the important files from its config.
  • Some strategy to restore a random subset of files to validate that everything is working.
  • Except the above points, the Duplicacy’s schedule backup remains the same.

My local copy is in the same array of the original files, it’s ok to me streamline the backup process and persist it directly on the cloud?

Thanks in advance.

I tried to elaborate on my backup plan to enrich the context and help the discussion. Also, I think it can be useful as a reference to others Unraid users that can be stumbling on the same doubts as me.

#2

A lot to break down here in this thread.

  1. I am unsure why you are mentioning the docker socket, it shouldn’t be mounted to the container at all. Duplicacy doesn’t use docker sockets for anything as far as I am aware. If you are having problems with file paths not allowing read/write, then you need to place a path on the container such as /mnt/user/ to map everything. Note that it’s very important on unraid that you DO NOT MOUNT or map disk paths as this bypasses the fuse file storage layer.

  2. Nextcloud files corrupting while being backed up is a first for me, I also have nextcloud on unraid and this container backs up fine. I typically just make backups frequently and restores work fine. What I would suggest is perhaps a scripted shutdown of the container that coincides with your backup schedule. This stops nextcloud (which is the only thing that would be writing to nexcloud) from writing to itself.

  3. I personally back up once a day at 8am, and to multiple storage targets for my own reasons. 4 hours is fine. I would do a check and prune daily as well, that’s also fine.

  4. I dont see why you would migrate from docker to standalone, that seems unsafe for urnaid, and the container works fine.
    the GUI is one of the strengths of this tool, so I wouldn’t go to CLI without good reason.

  5. You should not “persist” any files on the cloud as they exist on the Unraid box. These backups must be encrypted with a password, as you are backing up your containers which themselves contain very sensitive passwords, keys, encryption tokens, certificates etc. Not to mention your own files. Always encrypt files going off your system to somewhere else.

Here is an image of my unraid config, this is bog standard and has served me for about 5 years now, no worries at all:

image
image814×653 46.8 KB

Some notes:

  • Privilaged is ONLY ON during a restore, just because I dont want to bother with errors/failures from setting permissions on files. I turn it off during normal operation.
  • /mnt/user is passed through as a full read / write operation.
  • The container sits on its own custom network isolated away from any other containers. I have made a custom docker network (docker network create yournamehere) that means only the host and the container can access itself.
  • I use tailscale sidecars to securely access the container, I do not map any ports and never will.
#3

Thanks for the reply @zackey_tnt,

I can manage another docker containers using the socket, see: https://stackoverflow.com/a/33183227. But, as pointed in the stack overflow, I don’t want to compromise security, and I believe this usage goes contrary to what docker preconize (and myself).

Yeah, combining schedules to stop the container with the backup process seems feasible, but has some downsides:

  • I will need to remember the different places where the schedules are configured, like User Scripts for the starting/stopping of the container and Duplicacy GUI for the backup process. If some container’s property changes, I will have to adjust it manually. This looks brittle to me.
  • Adding more containers to this setup, it can become very complex.
  • No unified view of the process.

If I run Duplicacy binary directly, these points aren’t a problem anymore.

I’m approaching this with the idea of “better be safe than sorry”. I know that the Nextcloud AllinOne container stops the services containers when the Borg backup tool (also a container) is running. I will admit that I’m lacking on testing by myself the recovery process and validate if my current setup breaks. But it just looks to me that a database corruption can occur with my current setup, or store it out of sync with the user’s data, and don’t want to worry about that.

Thanks to bring your experience on this.

Yes, I agree with “4”.

Yeah, for sure. I was wondering if I can make it “append” only or some functionality to protect me of ransomware.

Thanks for the notes. About this one:

It’s because without it Duplicacy fails to create files where root is the user during a restore? I noticed that I changed my UID/GID to 0:0.