Am I doing this right?

xhaloz · 22 January 2019 17:54

I am using Duplicacy to backup my unRAID box. Previously I was using Borg + RClone but because Duplicacy runs the backup and pushes it to Google I’m looking to switch.

With Borg I would just include the directories I wanted to backup. I understand with Duplicacy, I need to cd into the directory first and run the backup command. My question is, if I have 40 parent directories and I only want to backup let’s say 10, do I need to run the backup command on each directory? For example

cd /mnt/user/Folder1
duplicacy -v backup -threads 2

cd /mntuser/Folder 2
duplicacy -v backup -threads 2

Also do I need to init on each new directory? Or will an init of the root directory /mnt/ work fine for the above example? Can they all go to the same snapshot ID?

Finally…last question, I see the .duplicacy file in each directory I “init” with. I used the -pref-dir command but it still places a .duplicacy file in each directory that I use init with. I don’t like having misc files lingering around my file system. Is there anyway to not have that happen or is it a requirement?

TheBestPessimist · 22 January 2019 18:35

i think you want to use duplicacy in symlink mode: Move duplicacy folder/Use symlink repository.

Basically duplicacy follows first level symlinks in your repository as if they are normal folders.

towerbr · 22 January 2019 18:49

Several strategies can be used:

Set up a repository for each folder and run a backup job for each of them, as you described;
Configure a unique repository in the “root” folder (/mnt/user/ in your case) and select subfolders to be backed up with filters and include patterns;
Use symbolic links, as suggested by @TheBestPessimist;

To choose the best strategy must be considered aspects such as:

number and size of files and folders;
need (or not) of different backup configurations (size of chunks fixed or variable, same or diferentt storages, etc.);
need (or not) of different schedules;
Frequency of source files updating.

I particularly use the first option because it allows me a more granular control over what will be backed up and when it will be backed up. But that depends on each use case.

Take a look at the -repository option at the end of this page:

xhaloz · 22 January 2019 19:33

First off I want to say thanks!!

I managed to get this working with

duplicacy init -e -repository /mnt/user/Downloads/Manual/Test/ -pref-dir /usr/local/duplicacy/TDS TDS gcd://Backup/TDS

Before I read your message, I would CD into the directory and then initiate the backup command. How do I do this now that I am not changing directories?

Edit: Nevermind, I just type duplicacy backup and it runs. I am going to try this on multiple directories

xhaloz · 22 January 2019 19:44

Ok I am really confused now.

I ran the following two commands

duplicacy init -e -repository /mnt/user/Downloads/Manual/Test2/ -pref-dir /usr/local/duplicacy/TDS2 TDS2 gcd://Backup/TDS2

duplicacy init -e -repository /mnt/user/Downloads/Manual/Test/ -pref-dir /usr/local/duplicacy/TDS1 TDS1 gcd://Backup/TDS1

So now I have 2 directories that I want to backup “Test” and “Test2”. I cannot have them in the same -pref-dir? Also can I place them in the same location gcd://Backup/TDS or does that need to be unique too?

Finally, how do I initiate the backup for each of these repo’s without CD-ing into the directory? Thanks for your help, where do I donate for beer or coffee?

xhaloz · 22 January 2019 19:54

Ah yes, this seems to be working. I really wish I could just specify the folders I wanted to backup without going through all this symlink information. Borg does a great job at this with a bash script. Is there a way to bash script this so I can just deploy the script to lets say 20 identical machines without having to symlink stuff?

TheBestPessimist · 22 January 2019 20:30

But using symlink is really simple: save all the ln -sf /source/folder /destination/folder/link in a duplicacy-setup-script.sh and just run it on all the machines.

Afterwards you just init a single repository on each of those machines, in the folder where you symlinked everything.

gchen · 23 January 2019 01:04

If you use the -repository option then -pref-dir isn’t needed and you can avoid symlinks by initializing a pseudo repository for each directory to be backed up:

cd /mnt/user/Duplicacy/repository1
duplicacy -init -repository /mnt/user/Downloads/Manual/Test1 repository_id storage_url

cd /mnt/user/Duplicacy/repository2
duplicacy -init -repository /mnt/user/Downloads/Manual/Test2 repository_id storage_url

Here /mnt/user/Duplicacy/repository* are the pseudo repositories and the real directories to be backed are /mnt/user/Downloads/Manual/Test1*. This strategy also has the advantage that all .duplicacy directories will be created under /mnt/user/Duplicacy.

towerbr · 23 January 2019 11:12

Exactly, that is the strategy I use, with all Duplicacy settings centralized in a folder, outside the repositories.

xhaloz · 24 January 2019 06:03

This is perfect!!! Yeah I really like your software, just tricky setting up on unRAID and the .duplicacy files were driving my OCD off the wall. Thank you for taking the time to reply!

Christoph · 24 January 2019 21:31

A post was split to a new topic: How to initiate a backup?

Christoph · 24 January 2019 21:32

For anyone: Feel free to use the button on the posts that you found useful.

For the OP of any support topic: you can mark the post that solved your issue by ticking the box under the post. That of course may include your own post

xhaloz · 30 January 2019 16:30

Hey gchen, that -repository option is giving me “incorrect usage” Is this available on the CLI version for linux_64 2.1.0 ?

TheBestPessimist · 30 January 2019 17:05

Please upgrade to 2.1.2. Version 2.1.0 is way too old!

xhaloz · 30 January 2019 17:24

Awesome, thanks! Did not see that before.

xhaloz · 1 February 2019 15:57

Hello @TheBestPessimist

Per @gchen 's solution for -repository. Can I back multiple directories to the same repository?

IE:

duplicacy -init -repository /mnt/user/Downloads/Manual/Test1 REPOID1 STORAGE1
duplicacy -init -repository /mnt/user/Downloads/Manual/Test2 REPOID1 STORAGE1

If not, what would be the best way to put everything in the same destination folder under the same repo?

Thanks again, I have my script generated and running. I am just getting into the intricate portions.

TheBestPessimist · 1 February 2019 17:34

Just to be clear here:

repository means the folder you are backing up right now (the one which has .duplicacy folder)
repository id is the name of a repository (should be unique for each of your folders)
the storage is the destination of your backup

You can backup multiple repositories to the same storage as long as they have different repository ids and this is indicated as you get the benefit of deduplication.

So for your example:

repository: /mnt/user/Downloads/Manual/Test1 and /mnt/user/Downloads/Manual/Test2
repository id: REPOID1
storage: STORAGE1

The problem with your example is that for 2 different folders to backup (which represent 2 different repositories) you have the same repository id.

What you want is actually to create 2 repositories:

duplicacy -init -repository /mnt/user/Downloads/Manual/Test1 REPOID1 STORAGE1
duplicacy -init -repository /mnt/user/Downloads/Manual/Test2 REPOID2 STORAGE1
                                                             ^^^^^^^

And make your script cd into both of those and back them up.

A note regarding this: i believe @gchen’s example has a single mistake (that you followed hence the confusion) because he uses the same repository id for both repositories, which is wrong because those repositories are different (there are 2 different folders):

gchen:

cd /mnt/user/Duplicacy-backups/repository1 
duplicacy -init -repository /mnt/user/Downloads/Manual/Test1 repository_id1 storage_url 

cd /mnt/user/Duplicacy-backups/repository2 
duplicacy -init -repository /mnt/user/Downloads/Manual/Test2 repository_id2 storage_url
                                                             ^^^^^^^^^^^^^^

@gchen: is your example above wrong because of the same repository id? Can i fix the post if that’s the case?

I still believe (and this is how i do all my backups using duplicacy-utils) that using a single repository where i symlink all the folders that i need backed up is good enough.

xhaloz · 1 February 2019 23:58

Awesome,

I know I asked a million questions here and I just want to say thank you very much. I have done a complete write up for the unRAID community and I hope more people get on board with this software. The only other software I have seen like this is Duplicati. The problem is Duplicati has so many issues with the database stuff, the slow restoring/browsing etc. Completely insane. My borg backups take about 4 hours each week to check/upload. I ran Duplicacy and it took 1:10 seconds after the initial run.

With that being said, I believe your symlink solution is absolutely the best solution now that I have analyzed all possible options.

Will Duplicacy CLI remain free forever? I am willing to completely make the switch at this point but am not interested in a monthly fee as I already pay that to my cloud storage provider.

TheBestPessimist · 2 February 2019 06:20

Please see Duplicacy Pricing Information .

gchen · 3 February 2019 02:29

Yes, the CLI version will remain free for personal use and it wouldn’t make sense to change this.