Goal
Provide a starting configuration for unattended duplicacy backup on macOS for all users with throttling. Filter list provided is not exhaustive but strike a good balance between amount of maintenance needed vs amount of random crap being backed up.
What we want
- Backup to run at the start of every hour
- To remote SFTP server with key based authentication
- Using no more than 40% of a single core when using AC power or 10% when on battery.
- Backup All Users
What we expect
- Storage to be already initialized and sftp access to the server is working with the specially created key pair. If not – create one (
man ssh-keygen) and add to the target (man ssh-copy-id). Sample preferences file provided -
cpulimitto be installed to/usr/local/bin/cpulimit. if not –brew install cpulimit -
duplicacyCLI to be located at or symlinked to/usr/local/bin/duplicacy.
Limitations
-
SIP needs to be disabled. (Boot to recovery partition and do
csrutil disable). This is needed until duplicacy can be added to Full Disk Access
Configuration files
Launchd daemon
To backup all users this needs to be a daemon, not an agent, so the launchd plist (man launchd.plist) must be placed to /Library/LaunchDaemons/. Call it something appropriate, like com.saspus.duplicacy.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>StandardOutPath</key>
<string>/Library/Logs/duplicacy_out.log</string>
<key>StandardErrorPath</key>
<string>/Library/Logs/duplicacy_err.log</string>
<key>Label</key>
<string>com.saspus.duplicacy</string>
<key>RunAtLoad</key>
<false/>
<key>WorkingDirectory</key>
<string>/Users</string>
<key>Program</key>
<string>/Users/.duplicacy/throttler.sh</string>
<key>StartCalendarInterval</key>
<dict>
<key>Minute</key>
<integer>0</integer>
</dict>
</dict>
</plist>
Throttler
We want the throttler to exit and kill duplicacy when the daemon is being stopped. Hence the dance with signals.
Add the following executable file /Users/.duplicacy/throttler.sh
#!/bin/bash
CPU_LIMIT_CORE_AC=40
CPU_LIMIT_CORE_BATTERY=10
case "$(pmset -g batt | grep 'Now drawing from')" in
*Battery*) CPU_LIMIT_CORE=${CPU_LIMIT_CORE_BATTERY} ;;
*) CPU_LIMIT_CORE=${CPU_LIMIT_CORE_AC} ;;
esac
function terminator() {
kill -TERM "${duplicacy}" 2>/dev/null
kill -TERM "${throttler}" 2>/dev/null
}
trap terminator SIGHUP SIGINT SIGQUIT SIGTERM EXIT
/usr/local/bin/duplicacy backup & duplicacy=$!
/usr/local/bin/cpulimit --limit=${CPU_LIMIT_CORE} --include-children --pid=${duplicacy} & throttler=$!
wait ${throttler}
Duplicacy preferences file
Should be located at /Users/.duplicacy/preferences and point to the initialize repo. There is no reason for me to show this other than to illustrate where does ssh key file must go
[
{
"name": "target-name",
"id": "mymac-all-users",
"repository": "",
"storage": "sftp://duplicacy@server//Backups/duplicacy",
"encrypted": true,
"no_backup": false,
"no_restore": false,
"no_save_password": false,
"nobackup_file": "",
"keys": {
"password": "super-crazy-diceware-password",
"ssh_key_file":"/Users/.duplicacy/rsa_duplicacy"
},
"filters": ""
}
]
Filters
This is a starting point to build your filter list and most of the stuff here is supposed to be excluded by time machine anyway. Once this is done some of the rules can be removed
To find out what folders and files are safe to exclude you can run find /Users -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -exec ls -dp {} \;. Most of the files there are small and it would be too much work to add exclusions for them manually. I’ve added the biggest offenders only
File /Users/.duplicacy/filters
# dot folders in the homes, e.g. `/Users/greg/.rbenv`
e:^[^/]*/\.(rbenv|vscode|gem|Trash)/.*
# Other misc folders in the homes, e.g. `/Users/greg/Movies`
e:^[^/]*/(Movies|Virtual Machines)/.*
# Random crap; monitor the progress for a first backup and see if anything extra is picked up that is large; you can add it there.
e:^[^/]*/Library/(Containers/com\.docker\.docker|Metadata/CoreSpotlight|Application Support/(VMware Fusion/Virtual Machines|Firefox/Profiles|Steam/Steam.AppBundle)|com\.apple\.icloud\.searchpartyd)/.*
# These files should be skipped, they don't make sense to backup and often locked
e:.*\.sock$
# Mail envelope index
e:^[^/]*/Library/Mail/V[0-9]*/MailData/Envelope Index(-wal|-shm)?$
# Caches
# exclude any cache files/directories with cache in the name (case insensitive)
e:(?i).*cache.*
# Pieces of photos library with com.apple.metadata:com_apple_backup_excludeItem set. Did not bother to regex it because it will likely go away soon once support for time machine exclusion is added to duplicacy
-*/Pictures/Photos Library.photoslibrary/database/
-*/Pictures/Photos Library.photoslibrary/resources/derivatives/thumbs/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/data/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/metadata/
-*/Pictures/Photos Library.photoslibrary/resources/cloudsharing/caches/
-*/Pictures/Photos Library.photoslibrary/resources/streams/
-*/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/
-*/Pictures/Photos Library.photoslibrary/external/
Permissions
Make the /Users/.duplicacy readable by root.
Updater
#!/bin/bash
DUPLICACY_CONFIG_DIR=/Users/.duplicacy
for cmd in wget jq curl
do
if ! command -v $cmd > /dev/null ; then echo "$cmd is missing"; exit 1; fi
done
if [[ $(id -u) != 0 ]]; then
sudo -p 'Restarting as root, password: ' bash $0 "$@"
exit $?
fi
AVAILABLE_STABLE_VERSION=$(curl -s 'https://duplicacy.com/latest_cli_version' |jq -r '.stable' 2>/dev/null)
LOCAL_EXECUTABLE_NAME="${DUPLICACY_CONFIG_DIR}/duplicacy_osx_x64_${AVAILABLE_STABLE_VERSION}"
if [ -f "${LOCAL_EXECUTABLE_NAME}" ]
then
echo "Version ${AVAILABLE_STABLE_VERSION} is up to date"
else
DOWNLOAD_URL="https://github.com/gilbertchen/duplicacy/releases/download/v${AVAILABLE_STABLE_VERSION}/duplicacy_osx_x64_${AVAILABLE_STABLE_VERSION}"
if wget -O "${LOCAL_EXECUTABLE_NAME}" "${DOWNLOAD_URL}" ; then
chmod +x "${LOCAL_EXECUTABLE_NAME}"
rm -f /usr/local/bin/duplicacy
ln -s "${LOCAL_EXECUTABLE_NAME}" /usr/local/bin/duplicacy
echo "Updated to ${AVAILABLE_STABLE_VERSION}"
else
echo "Could not download ${DOWNLOAD_URL}"
rm -f "${LOCAL_EXECUTABLE_NAME}"
fi
fi
Loading the daemon
To load run launchctl load -w /Library/LaunchDaemons/com.saspus.duplicacy.plist
To start immediately run launchctl start com.saspus.duplicacy
Unloading the daemon
To stop the schedule and running instate: launchctl unload /Library/LaunchDaemons/com.saspus.duplicacy.plist.
To stop running instance – launchctl stop com.saspus.duplicacy
Troubleshooting
- Make sure the plist and throttler do not have any extended attributes set, plist is owned by
root:wheeland permissions are 644. Editing the file in misconfigured GUI editor may result incom.apple.quarantineextended attribute added that will prevent the script from running with amusing error message. Read more here. Clear all extended attributes withsudo xattr -c /Library/LaunchDaemons/com.saspus.duplicacy.plist - in a separate terminal window run
sudo tail -F /var/log/system.logand start the daemon. See what happens.