Scripted Duplicacy CLI on Linux - care to share your scripts?

Derek_v2 · 29 May 2023 10:34

Hiya, i’ve got 3 licenses for Windows, and i’ll soon buy another for my Linux unRAID NAS. I currently run Borg on my Linux desktop, but i have decided to switch to Duplicacy. I was immensely proud of myself for creating my first (and only) BASH scripts to backup, check and prune my Borg repositories. And i also created my only systemd service and timer to automate the backup to happen at a certain time, or if that time was missed then the next boot. It even pops up a desktop notification - green/orange/red depending on how the backup went (and that notification even stays put if red!). I want to retain the only bit of ‘programming’ i’ve done since some very simple DOS batch files in the 90s. I’m not a programmer, and that was an immense effort (yeah, i’m proud of it.)

I’m struggling to find more complete guides for Duplicacy, or other people’s examples of their implementation for Linux. I know there’s some instructions on the forum, like here and here. But they’re more “here’s some available options” rather than complete examples. I feel like i’m flying blind

Are there people here with experience running on Linux that could share their setup and/or scripts? I find it easier to see what other people have done, understand where i do, or ask questions where i don’t. And then research further options for my preferences and make adjustments.

I hope there’s people who are happy to share what they’ve done for their systems.

tchoover · 29 May 2023 12:59

All machines backup to the synology by default. Additionally, the synology performs a ‘-copy’ to an external drive on my desktop computer (bethel) and to the cloud (Backblaze B2). I hope this is helpful.

#!/usr/bin/env bash

# set -e: exit script immediately upon error
# set -u: treat unset variables as an error
# set -o pipefail: cause a pipeline to fail, if any command within it fails
set -o pipefail

DUPLICACY=duplicacy

backupRepository()
{
    if [[ -d "$1" ]]; then
        cd "$1" || exit 1
    else
        echo ""
        echo "### $1 not mounted"
        return
    fi
    chmod -R g=,o= .duplicacy

    DUPLICACY_LOGS=.duplicacy/logs
    [[ -d $DUPLICACY_LOGS ]] || mkdir -p $DUPLICACY_LOGS

    echo ""
    echo "### backupRepository $1 ###"

    echo "# Backup filters, known_hosts & preferences..."
    BACKUP_DIR="${HOME}/.duplicacy-backup${1}"
    [[ -d ${BACKUP_DIR} ]] || mkdir -p ${BACKUP_DIR}
    cp -a .duplicacy/{filters,known_hosts,preferences} ${BACKUP_DIR}
    chmod -R g=,o= "${HOME}/.duplicacy-backup"
    echo "# Done"

    echo "# Delete old logs..."
    find ./$DUPLICACY_LOGS -name "*.log" -type f -mtime +14 -delete
    echo "# Done"

    echo "# Start Backup..."
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    if [[ $2 == "hash" ]]; then
        $DUPLICACY -log backup -stats -threads 4 -hash | tee "$DUPLICACY_LOGS/$DATETIME-backup.log"
    else
        $DUPLICACY -log backup -stats -threads 4       | tee "$DUPLICACY_LOGS/$DATETIME-backup.log"
    fi
    echo "# Done"

    if [[ $2 == "check" ]]; then
        echo ""
        echo "### Check Backups... ###"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        # $DUPLICACY -log check -all | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        $DUPLICACY -log check -all -storage synology -tabular -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        echo "# Done"
    fi
}

if   [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /etc "$1"
        $DUPLICACY -log check -all -storage synology -tabular -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        echo "# Done"
    fi
}

if   [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /etc "$1"
elif [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" == "pvhost[0-9]*" ]]; then
    DUPLICACY="/usr/local/sbin/duplicacy"
    backupRepository /etc
    backupRepository /root
    backupRepository /usr/local
    backupRepository /home/tom "$1"
elif [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /etc
    backupRepository /root
    backupRepository /usr/local "$1"
elif [[ "$(whoami)" == "tom" ]] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /Users/tom/Dropbox/tc hash
    backupRepository /Volumes/USBAC hash
    backupRepository /Users/tom "$1"
elif [[ "$(whoami)" == "tom" ]] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /home/tom "$1"
fi

if [[ "$(hostname -s)" == "synology" ]]; then
    DUPLICACY="/var/services/homes/tom/bin/duplicacy"
    backupRepository /volume1/archive
    backupRepository /volume1/audio
    backupRepository /volume1/homes
    backupRepository /volume1/music
    backupRepository /volume1/photo
    backupRepository /volume1/video
    backupRepository /volume1/zz_backups

    echo ""
    echo "### Copy to bethel... ###"
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    $DUPLICACY -log copy -from synology -to bethel -threads 40 | grep -v "INFO SNAPSHOT_EXIST Snapshot .* already exists at the destination storage" | tee "$DUPLICACY_LOGS/$DATETIME-copy-bethel.log"
    echo "# Done"

    echo ""
    echo "### Copy to Backblaze... ###"
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    $DUPLICACY -log copy -from synology -to b2     -threads 4  | grep -v "INFO SNAPSHOT_EXIST Snapshot .* already exists at the destination storage" | tee "$DUPLICACY_LOGS/$DATETIME-copy-b2.log"
    echo "# Done"

    if [[ $1 == "check" ]]; then
        cd /volume1/archive || exit
        echo ""
        echo "### Check Backups... ###"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        $DUPLICACY -log check -all -storage synology -tabular -chunks -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        $DUPLICACY -log check -all -storage bethel                                         | tee "$DUPLICACY_LOGS/$DATETIME-check-bethel.log"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        $DUPLICACY -log check -all -storage b2                                             | tee "$DUPLICACY_LOGS/$DATETIME-check-b2.log"
        echo "# Done"
    fi
fi

Derek_v2 · 1 June 2023 06:25

Whoa

Thanks. That will take me some time to get my head around!

How does Duplicacy apply the source path and filters? I can see reference, but i’m afraid i don’t understand how it puts the info together.

    echo ""
    echo "### backupRepository $1 ###"

    echo "# Backup filters, known_hosts & preferences..."
    BACKUP_DIR="${HOME}/.duplicacy-backup${1}"
    [[ -d ${BACKUP_DIR} ]] || mkdir -p ${BACKUP_DIR}
    cp -a .duplicacy/{filters,known_hosts,preferences} ${BACKUP_DIR}
    chmod -R g=,o= "${HOME}/.duplicacy-backup"
    echo "# Done"

and

$DUPLICACY -log backup -stats -threads 4 -hash | tee "$DUPLICACY_LOGS/$DATETIME-backup.log"

tchoover · 1 June 2023 12:49

backupRepository() is the function that does all of the work, and uses the appropriate filter/preference file found in the .duplicacy subdirectory for each repository. The last half of the script is where you designate the repositories to be backed up. For example:

elif [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /etc
    backupRepository /root
    backupRepository /usr/local "$1"

If the script is run as ‘root’ [see the $(whoami) above] on my laptop ‘theophilus’, ‘/etc’, ‘/root’, and ‘/usr/local’ are each backed up by calling the backupRepository() function. Before running the script the first time, one must initialize each repository by something similar to the following (as ‘root’):

cd /etc             && duplicacy init -e -storage-name synology $(hostname -s)-etc--synology       sftp://tom@synology//zz_duplicacy-backups
cd /root            && duplicacy init -e -storage-name synology $(hostname -s)-root--synology      sftp://tom@synology//zz_duplicacy-backups
cd /usr/local       && duplicacy init -e -storage-name synology $(hostname -s)-usr_local--synology sftp://tom@synology//zz_duplicacy-backups

Another example:

elif [[ "$(whoami)" == "tom" ]] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /home/tom "$1"

If the script is run as ‘tom’ [see the $(whoami) above] on my laptop ‘theophilus’, only ‘/home/tom’ is backed up by calling the backupRepository() function. Before running the script the first time, one must initialize the repository by something similar to the following (as ‘tom’):

cd /home/tom && duplicacy init -e -storage-name synology $(hostname -s)-tom--synology       sftp://tom@SYNOLOGY//zz_duplicacy-backups

Although the script can be executed manually, it is typically added to your crontab (or setup with a systemd timer).

The section of the function you referenced above:

    echo "# Backup filters, known_hosts & preferences..."
    BACKUP_DIR="${HOME}/.duplicacy-backup${1}"
    [[ -d ${BACKUP_DIR} ]] || mkdir -p ${BACKUP_DIR}
    cp -a .duplicacy/{filters,known_hosts,preferences} ${BACKUP_DIR}
    chmod -R g=,o= "${HOME}/.duplicacy-backup"
    echo "# Done"

has nothing to do with the actual duplicacy backup of the repository (you could delete that section, and the backup would still be accomplished). I recently added it, as duplicacy doesn’t backup the ‘.duplicacy’ directory itself. As I wanted to keep a copy my .duplicacy/filters, .duplicacy/known_hosts & .duplicacy/preferences files, they are copied to ${HOME}/.duplicacy-backup for preservation during the backup process for each repository. I hope that made sense–if not, please let me know.

saspus · 1 June 2023 17:04

Few suggestions here (feel free to ignore) :

there is a spurious } in the middle of the script.
each use of ${BACKUP_DIR} shall be quoted; as does every value in the comparisons (e.g. $2)
I would never delete old logs; if you want, archive them: if something goes wrong they are useful to track down what has happened to a specific chunk
Superuser is not guaranteed to be called “root”. I’d check with [ "$(id -u)" -eq 0 ] instead
instead of backing up audio, homes, misic, …, separately I would symlink them into a single folder and backup that folder instead.
instead of backing up individual files from .duplicacy/ (such as filter) I woudl keep them in the location that is being backed up, and symlink to the right place.
There is no need to check if the directory exists before calling mkdir -p
Copying to an HDD with 40 threads is likely much slower than with 1: disks have huge seek latency, but good sequential performance
Copying to back blaze on the other hand will benefit from multithreading.
The whole thing feels like three different scripts for three different hosts smushed together – why not have a separate, host specific (or even host agnostic) script?

tchoover · 1 June 2023 17:59

Great suggestions–Thanks! I’ll incorporate them into my script.

re #5: I never considered using symlinks, as I would have assumed only the symlinks themselves would be backed up. After some searching, I now see that duplicacy follows symlinks in the root of the repository, so that will allow me to simplify things.

re #10: my ‘bin’ git repo lives on my desktop/laptops, so I make and checkin all changes to the script on these machines. The duplicacy-backup script is then simply scp’d to my proxmox nodes ( pvhost{1,2,3} ) and my synology. I previously kept the scripts separate on various machines, but found I was spending too much time editing on multiple machines when I wanted to make a change.

tchoover · 1 June 2023 18:28

tchoover:

if   [[ "$(whoami)" == "root" ]] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /etc "$1"
        $DUPLICACY -log check -all -storage synology -tabular -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        echo "# Done"
    fi
}

re #1: The above section shouldn’t have been included (it’s not in the script on my machine–somehow I messed up during the copy/paste and ‘cleanup’ before posting the message).

tchoover · 1 June 2023 20:03

#2: added quotes and braces throughout

#3: old logs are only deleted if DELETE_LOGS is set

#4: changed method of identifying superuser

#7: removed directory check before ‘mkdir -p’

#8: removed ‘-threads 40’

Will work on #5 & #6.

Updated script follows (most recent version can always be found here):

#!/usr/bin/env bash

# set -e: exit script immediately upon error
# set -u: treat unset variables as an error
# set -o pipefail: cause a pipeline to fail, if any command within it fails
set -o pipefail

# uncomment the following line to delete logs older than 14 days (you may change '14' to whatever number of days you wish to keep)
#DELETE_LOGS=14

DUPLICACY=duplicacy

backupRepository()
{
    if [[ -d "${1}" ]]; then
        cd "${1}" || exit 1
    else
        echo ""
        echo "### ${1} not mounted"
        return
    fi
    chmod -R g=,o= .duplicacy

    DUPLICACY_LOGS=.duplicacy/logs
    mkdir -p "${DUPLICACY_LOGS}"

    echo ""
    echo "### backupRepository ${1} ###"

    echo "# Backup filters, known_hosts & preferences..."
    BACKUP_DIR="${HOME}/.duplicacy-backup${1}"
    mkdir -p "${BACKUP_DIR}"
    cp -a .duplicacy/{filters,known_hosts,preferences} "${BACKUP_DIR}"
    chmod -R g=,o= "${HOME}/.duplicacy-backup"
    echo "# Done"

    if [ "${DELETE_LOGS}" ]; then
        echo "# Delete logs older than ${DELETE_LOGS} days old..."
        find "./${DUPLICACY_LOGS}" -name "*.log" -type f -mtime "+${DELETE_LOGS}" -delete
        echo "# Done"
    fi

    echo "# Start Backup..."
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    if [[ "${2}" == "hash" ]]; then
        "${DUPLICACY}" -log backup -stats -threads 4 -hash | tee "$DUPLICACY_LOGS/$DATETIME-backup.log"
    else
        "${DUPLICACY}" -log backup -stats -threads 4       | tee "$DUPLICACY_LOGS/$DATETIME-backup.log"
    fi
    echo "# Done"

    if [[ "${2}" == "check" ]]; then
        echo ""
        echo "### Check Backups... ###"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        "${DUPLICACY}" -log check -all -storage synology -tabular -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        echo "# Done"
    fi
}

if   [ "$(id -u)" -eq 0 ] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /etc "${1}"
elif [ "$(id -u)" -eq 0 ] && [[ "$(hostname -s)" == "pvhost[0-9]*" ]]; then
    DUPLICACY="/usr/local/sbin/duplicacy"
    backupRepository /etc
    backupRepository /root
    backupRepository /usr/local
    backupRepository /home/tom "${1}"
elif [ "$(id -u)" -eq 0 ] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /etc
    backupRepository /root
    backupRepository /usr/local "${1}"
elif [[ "$(whoami)" == "tom" ]] && [[ "$(hostname -s)" =~ (ariel|bethel) ]]; then
    backupRepository /Users/tom/Dropbox/tc hash
    backupRepository /Volumes/USBAC hash
    backupRepository /Users/tom "${1}"
elif [[ "$(whoami)" == "tom" ]] && [[ "$(hostname -s)" == "theophilus" ]]; then
    backupRepository /home/tom "${1}"
fi

if [[ "$(hostname -s)" == "synology" ]]; then
    DUPLICACY="/var/services/homes/tom/bin/duplicacy"
    backupRepository /volume1/archive
    backupRepository /volume1/audio
    backupRepository /volume1/homes
    backupRepository /volume1/music
    backupRepository /volume1/photo
    # backupRepository /volume1/reference
    backupRepository /volume1/video
    backupRepository /volume1/zz_backups

    echo ""
    echo "### Copy to bethel... ###"
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    "${DUPLICACY}" -log copy -from synology -to bethel         | grep -v "INFO SNAPSHOT_EXIST Snapshot .* already exists at the destination storage" | tee "$DUPLICACY_LOGS/$DATETIME-copy-bethel.log"
    echo "# Done"

    echo ""
    echo "### Copy to Backblaze... ###"
    DATETIME=$(date "+%Y%m%d-%H%M%S")
    "${DUPLICACY}" -log copy -from synology -to b2 -threads 4  | grep -v "INFO SNAPSHOT_EXIST Snapshot .* already exists at the destination storage" | tee "$DUPLICACY_LOGS/$DATETIME-copy-b2.log"
    echo "# Done"

    if [[ "${1}" == "check" ]]; then
        cd /volume1/archive || exit
        echo ""
        echo "### Check Backups... ###"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        "${DUPLICACY}" -log check -all -storage synology -tabular -chunks -threads 40 -persist | tee "$DUPLICACY_LOGS/$DATETIME-check.log"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        "${DUPLICACY}" -log check -all -storage bethel                                         | tee "$DUPLICACY_LOGS/$DATETIME-check-bethel.log"
        DATETIME=$(date "+%Y%m%d-%H%M%S")
        "${DUPLICACY}" -log check -all -storage b2                                             | tee "$DUPLICACY_LOGS/$DATETIME-check-b2.log"
        echo "# Done"
    fi
fi

Derek_v2 · 2 June 2023 05:17

Borg doesn’t operate with a preferences file, it’s all in the command line/script, so this is all very weird to me . I will install and experiment on a VM to try to get my head around things. I’ve been spoiled by the web version to date

This isn’t strictly related to the title of this thread, and i’m happy to open another thread if people suggest it. I have a question about the installation directory, preferences location and cache location (anything else?). I’ve read some forum posts about moving these things, using a switch on install and even creating a symbolic link to the binary. Also several different places from which to run it: /opt or /usr/bin for example. I don’t understand the why.

My intention is to run as me (the user, not root), backup everything (with selected exclusions) in /home (i’m the only user - i don’t want to use it for whole of system backup). I’m perfectly happy with a /home/.duplicacy dir especially if that’s where my preferences are stored and therefor backed up. How to put it there and what reasons might there be to not have it there? As i’m not running Duplicacy in root, i presume i’d need to have the caching in /home too? If so, how to get it there? How big does it get? And i presume i need to (or should do) exclude it from the backups?

tchoover · 2 June 2023 16:20

You are correct in that your preferences are (by default) stored in the .duplicacy directory at the root of what what you desire to backup (i.e. /home/.duplicacy in your scenario); however, the .duplicacy directory is automatically excluded from your backup. Hence, the purpose of the section in my script that copies .duplicacy/preferences to another directory to serve as a “backup” for preferences (.duplicacy/filters & .duplicacy/known_hosts are also copied in my script). This isn’t necessarily a requirement, nor the best way to back them up, it’s just the way I wrote my backup script.

The .duplicacy directory is automatically created when you run the duplicacy init command. For example:

cd /home && duplicacy init -e -storage-name <STORAGE_NAME_HERE> <SNAPSHOT_ID_HERE> <STORAGE_URL_HERE>

On my system, the above command equates to:

cd /home && duplicacy init -e -storage-name synology $(hostname -s)-home--synology sftp://tom@SYNOLOGY//zz_duplicacy-backups

As the cache is also contained within the .duplicacy directory, one does not need to exlude it from the backups, as this directory is automatically excluded.

Derek_v2 · 4 June 2023 09:04

Ah yes. Thank you. I think i did read that someone in a post but it completely escaped my mind. I have had a play on a VM. A combination of your help, reading various posts and experimentation, i think i’ve got a handle of the basics now.

Can the .duplicacy directory, minus the cache be inserted into the backup with filters? I tried to get it to work, but something is wrong on my VM, and even when i have no filter file, directories (that aren’t empty) are not included in the output when i run Duplicacy with | tee backup.log. If i change the filters a bazillion times, can Duplicacy get in a twist about things?

Apart from the .duplicacy directory and the storage location, Is there anything else i need to delete to completely obliterate the OSes knowledge of Duplicacy?

tchoover · 4 June 2023 13:04

My understanding is “no”, hence my copying them into another directory for backup. Or, you can go with @saspus’ suggestion above to put them into another directory (which would be backed up) and then symlinking into the .duplicacy directory.

Here’s a way to verify what your filters will include/exclude, without actually running a backup (of course, change ‘volume1/archive’ as appropriate on your system):

cd /volume1/archive && duplicacy -d -log backup -enum-only -stats | tee .duplicacy/check-filters.log

Then, to see what will be exluded:

cd /volume1/archive && grep PATTERN_EXCLUDE .duplicacy/check-filters.log | less

To see what will be included:

cd /volume1/archive && grep PATTERN_INCLUDE .duplicacy/check-filters.log | less

Derek_v2 · 7 June 2023 09:11

Amazing tip! Thank you so much! Not sure exactly what happened before, but i started clean and it looks good done manually. Next step is scripting something up

I feel like i’m asking too many questions that aren’t tightly related to the original title. I think i have one more…

Recommended commands? I’ve seen people use -chunks, -threads, -fossils… in my Windows setups, i use -chunks for the check operation, but that’s it. I didn’t change the -threads or add anything else.

Happy to start a new thread somewhere.

tchoover · 7 June 2023 12:10

I agree–that’s probably best for another thread.