Script for configuring duplicacy CLI for Unattended (launchd) Throttled (cpulimit) All Users backup on macOS, without disabling SIP

Goal

Provide a starting configuration for unattended duplicacy backup on macOS for all users with throttling. Filter list provided is not exhaustive but strike a good balance between amount of maintenance needed vs amount of random crap being backed up.

What we want

  • Backup to run at the start of every hour
  • To remote SFTP server with key based authentication
  • Using no more than 40% of a single core when using AC power or 10% when on battery.
  • Backup All Users

What we expect

  • Storage to be already initialized and sftp access to the server is working with the specially created key pair. If not – create one (man ssh-keygen) and add to the target (man ssh-copy-id). Sample preferences file provided
  • cpulimit to be installed to /usr/local/bin/cpulimit. if not – brew install cpulimit
  • duplicacy CLI to be located at or symlinked to /usr/local/bin/duplicacy.

Limitations

Configuration files

Launchd daemon

To backup all users this needs to be a daemon, not an agent, so the launchd plist (man launchd.plist) must be placed to /Library/LaunchDaemons/. Call it something appropriate, like com.saspus.duplicacy.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>StandardOutPath</key>
    <string>/Library/Logs/duplicacy_out.log</string>

    <key>StandardErrorPath</key>
    <string>/Library/Logs/duplicacy_err.log</string>

    <key>Label</key>
    <string>com.saspus.duplicacy</string>

    <key>RunAtLoad</key>
    <false/>

    <key>WorkingDirectory</key>
    <string>/Users</string>

    <key>Program</key>
    <string>/Users/.duplicacy/throttler.sh</string>

    <key>StartCalendarInterval</key>
    <dict>
        <key>Minute</key>
        <integer>0</integer>
    </dict>
</dict>
</plist>

Throttler

We want the throttler to exit and kill duplicacy when the daemon is being stopped. Hence the dance with signals.

Add the following executable file /Users/.duplicacy/throttler.sh

#!/bin/bash

CPU_LIMIT_CORE_AC=40
CPU_LIMIT_CORE_BATTERY=10

case "$(pmset -g batt | grep 'Now drawing from')" in
*Battery*) CPU_LIMIT_CORE=${CPU_LIMIT_CORE_BATTERY} ;;
*)         CPU_LIMIT_CORE=${CPU_LIMIT_CORE_AC} ;;
esac

function terminator() {
  kill -TERM "${duplicacy}" 2>/dev/null
  kill -TERM "${throttler}" 2>/dev/null
}

trap terminator SIGHUP SIGINT SIGQUIT SIGTERM EXIT

/usr/local/bin/duplicacy backup & duplicacy=$!

/usr/local/bin/cpulimit --limit=${CPU_LIMIT_CORE} --include-children --pid=${duplicacy} & throttler=$!

wait ${throttler}

Duplicacy preferences file

Should be located at /Users/.duplicacy/preferences and point to the initialize repo. There is no reason for me to show this other than to illustrate where does ssh key file must go

[
    {
        "name": "target-name",
        "id": "mymac-all-users",
        "repository": "",
        "storage": "sftp://duplicacy@server//Backups/duplicacy",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
           "password": "super-crazy-diceware-password",
           "ssh_key_file":"/Users/.duplicacy/rsa_duplicacy"
        },
        "filters": ""
    }
]

Filters

This is a starting point to build your filter list and most of the stuff here is supposed to be excluded by time machine anyway. Once this is done some of the rules can be removed

To find out what folders and files are safe to exclude you can run find /Users -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -exec ls -dp {} \;. Most of the files there are small and it would be too much work to add exclusions for them manually. I’ve added the biggest offenders only

File /Users/.duplicacy/filters

# dot folders in the homes, e.g. `/Users/greg/.rbenv`
e:^[^/]*/\.(rbenv|vscode|gem|Trash)/.*

# Other misc folders in the homes, e.g. `/Users/greg/Movies`
e:^[^/]*/(Movies|Virtual Machines)/.*

# Random crap; monitor the progress for a first backup and see if anything extra is picked up that is large; you can add it there.
e:^[^/]*/Library/(Containers/com\.docker\.docker|Metadata/CoreSpotlight|Application Support/(VMware Fusion/Virtual Machines|Firefox/Profiles|Steam/Steam.AppBundle)|com\.apple\.icloud\.searchpartyd)/.*

# These files should be skipped, they don't make sense to backup and often locked
e:.*\.sock$

# Mail envelope index
e:^[^/]*/Library/Mail/V[0-9]*/MailData/Envelope Index(-wal|-shm)?$

# Caches
# exclude any cache files/directories with cache in the name (case insensitive)
e:(?i).*cache.*

# Pieces of photos library with com.apple.metadata:com_apple_backup_excludeItem set. Did not bother to regex it because it will likely go away soon once support for time machine exclusion is added to duplicacy
-*/Pictures/Photos Library.photoslibrary/database/
-*/Pictures/Photos Library.photoslibrary/resources/derivatives/thumbs/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/data/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/metadata/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/caches/
-*/Pictures/Photos Library.photoslibrary/resources/streams/
-*/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/
-*/Pictures/Photos Library.photoslibrary/external/

Permissions

Make the /Users/.duplicacy readable by root.

Updater

#!/bin/bash

DUPLICACY_CONFIG_DIR=/Users/.duplicacy

for cmd in wget jq curl
do
   if ! command -v $cmd > /dev/null ; then  echo "$cmd is missing"; exit 1; fi
done

if [[ $(id -u) != 0 ]]; then
    sudo -p 'Restarting as root, password: ' bash $0 "$@"
    exit $?
fi

AVAILABLE_STABLE_VERSION=$(curl -s 'https://duplicacy.com/latest_cli_version' |jq -r '.stable' 2>/dev/null)

LOCAL_EXECUTABLE_NAME="${DUPLICACY_CONFIG_DIR}/duplicacy_osx_x64_${AVAILABLE_STABLE_VERSION}"

if [ -f "${LOCAL_EXECUTABLE_NAME}" ]
then
   echo "Version ${AVAILABLE_STABLE_VERSION} is up to date"
else
    DOWNLOAD_URL="https://github.com/gilbertchen/duplicacy/releases/download/v${AVAILABLE_STABLE_VERSION}/duplicacy_osx_x64_${AVAILABLE_STABLE_VERSION}"
    if wget -O "${LOCAL_EXECUTABLE_NAME}" "${DOWNLOAD_URL}" ; then
       chmod +x "${LOCAL_EXECUTABLE_NAME}"
       rm -f /usr/local/bin/duplicacy
       ln -s "${LOCAL_EXECUTABLE_NAME}" /usr/local/bin/duplicacy
       echo "Updated to ${AVAILABLE_STABLE_VERSION}"
    else
        echo "Could not download ${DOWNLOAD_URL}"
        rm -f "${LOCAL_EXECUTABLE_NAME}"
    fi
fi

Loading the daemon

To load run launchctl load -w /Library/LaunchDaemons/com.saspus.duplicacy.plist
To start immediately run launchctl start com.saspus.duplicacy

Unloading the daemon

To stop the schedule and running instate: launchctl unload /Library/LaunchDaemons/com.saspus.duplicacy.plist.
To stop running instance – launchctl stop com.saspus.duplicacy

Troubleshooting

  • Make sure the plist and throttler do not have any extended attributes set, plist is owned by root:wheel and permissions are 644. Editing the file in misconfigured GUI editor may result in com.apple.quarantine extended attribute added that will prevent the script from running with amusing error message. Read more here. Clear all extended attributes with sudo xattr -c /Library/LaunchDaemons/com.saspus.duplicacy.plist
  • in a separate terminal window run sudo tail -F /var/log/system.log and start the daemon. See what happens.
4 Likes

Automating this, relaxing the requirement to disable sip, and providing update mechanism. The script is hosted here:

Features:

  • Allows to backup all users but without the need to disable System Integrity Protection: This is achieved by wrapping the script that runs duplicacy backup into app bundle (with platypus) and asking the user to add that bundle into Full Disk Access exceptions in macOS Privacy settings.
  • Prevents bursty CPU usage: this is done by running duplicacy backup under cpulimit. Depending on whether Mac is running on battery or wall power different limits are applied.
  • Scheduling is done via Launchd
  • Script downloads specified version of duplicacy or can use existing local binary. The version can be specified by version number or by release channel – e.g. Latest or Stable. To update – re-run the script.

Prerequisites:

The duplicacy repository must be already initialized in /Library/Duplicacy and point to backup /Users (or any other folder for that matter). In other words, cd /Library/Duplicacy && sudo duplicacy backup shall work.

Also a few utilities are needed depending on the installation mode; the scrip will prompt to install them; home-brew would be the easiest way to accomplish that (one or more of the following might be needed: wget, cpulimit, curl, jq, platypus.

By default the backup is configured to run hourly and prune weekly with -keep 31:360 -keep 7:90 -keep 1:14 -all policy. This can be modified in the beginning of the script.

4 Likes

Absolutely love this !! great work, really appreciate it !

1 Like