Help with weekly backup script optimization plus prune command

Support
#1

Hello all, I am fairly new to Duplicacy though I’ve got a script written that I run manually usually monthly. I want to get this set up weekly though I’ve had the tab for the prune command open for probably a year and never got around to figuring it out till now.

Basically what I am trying to do is keep 1 backup per week, and keep the past 4 weekly backups deleting anything older. Theoretically then, I can restore anything I accidentally deleted with not a lot of storage overhead plus grab the most current backup at any point if disaster struck.

My backup is three repositories, one for each drive I backup. I backup from the root of each drive. Each drive is backed up to an external hard drive and then synced to 2 cloud accounts. However two local drives go to the same backup drive. C/D get backed up to Y, E gets backed up to Z.

My first question, is there any optimization I can do to the backup/copying process besides what I already have written?
cd C:
duplicacy_win_x64_2.1.2.exe backup -storage local -vss
duplicacy_win_x64_2.1.2.exe copy -from local -to remote1 -threads 20
duplicacy_win_x64_2.1.2.exe copy -from local -to remote2 -threads 20
Which is repeated for the 2 other drives, D/E.

Finally how do I write the prune command effectively? I am not sure, should I only prune the local storage after it’s backed up, and then copy that to the remote storages? I don’t think that removes it, though, so I’m a bit confused. Do I need 3 lines after the backup/copy as in:
duplicacy_win_x64_2.1.2.exe prune -storage local -keep 0:31 -keep 7:1
duplicacy_win_x64_2.1.2.exe prune -storage remote1 -keep 0:31 -keep 7:1
duplicacy_win_x64_2.1.2.exe prune -storage remote2 -keep 0:31 -keep 7:1
Again repeated for drives D/E?
Then however the prune command documentation said to use -all if multiple repositories back up to the same storage. However I don’t think I did that, simply because mapping one drive to one folder is simpler to manage. I’m not looking for storage efficiency. I initialized each drive root (C/D/E) into a separate folder with that drive letter on the backup drives. So, I don’t think I need to use -all?

I am sorry for these noob questions but guidance would be really appreciated.
Thank you!

#2

Are you saying you have three local storages for each drive C/D/E that you backup to?

Sounds like you only have the one - ‘local’. In that case, yes use -all in your prune’s because all three repositories exist in each backup storage. However, your description of Y/Z leads me to believe you do indeed have separate storages, even on Y (for C/D)?

Either way, using -all does no harm if each storage has just one repository.

But if you are using three local storages, are you then copying them to a one storage for each remote? Or are you copying them to a seperate storage for each repository for each remote? :slight_smile: If the former, you’ll have multiple repositories in each of the two remotes. That’s why you may have to use -all and, as I said, it does no harm if there’s just one repo in each. You should use -all. The only time you don’t want to use -all is if you want different retention periods for different repositories, or if you don’t want to prune certain repos. So use -all to be sure.

Also yes, prune the [local and remote] storages with the same rules at the same time, otherwise you’ll end up in a situation where you might delete different revision IDs on different storages and end up re-copying snapshots to fill the holes. Doing it to all three one after the other on the same day - like you’re doing - is crucial.

Keeping no snapshots after such a short period seems a bit risky to me, but otherwise your process looks fine to me.

#3

Hi Droolio, thanks for the info.

First off I will consider keeping additional snapshots if this is a good idea, mostly I’m trying to get a good balance between snapshots and keeping extra data - does monthly snapshots, past a month out until a year back sound fine?

Regarding the storages here is how they were initialized,
cd C:
duplicacy_win_x64_2.1.2.exe init -e -storage-name local drive_C Y:\drive_C
duplicacy_win_x64_2.1.2.exe add -e -copy local remote1 drive_C gcd://drive_C
duplicacy_win_x64_2.1.2.exe add -e -copy local remote2 drive_C gcd://drive_C
cd D:
duplicacy_win_x64_2.1.2.exe init -e -storage-name local drive_D Y:\drive_D
duplicacy_win_x64_2.1.2.exe add -e -copy local remote1 drive_D gcd://drive_D
duplicacy_win_x64_2.1.2.exe add -e -copy local remote2 drive_D gcd://drive_D
cd E:
duplicacy_win_x64_2.1.2.exe init -e -storage-name local drive_E Z:\drive_E
duplicacy_win_x64_2.1.2.exe add -e -copy local remote1 drive_E gcd://drive_E
duplicacy_win_x64_2.1.2.exe add -e -copy local remote2 drive_E gcd://drive_E

This creates the following mappings,
C (internal SSD 512GB), backed up to Y:\drive_C
D (internal HDD 2TB), backed up to Y:\drive_D
E (external HDD 3TB), backed up to Z:\drive_E
(both Y and Z are external 3TB)
And then all the backups are copied to the two remote storage locations which when viewed, have all the folders drive_C, drive_D, drive_E located in the root directory.

As far as I understand it, each repository (in my case, the root of each drive, C/D/E) has a storage called local, but each repository’s local and remotes don’t affect the other - such that deduplication can be performed on a per drive basis but not cross referencing other drives.

#4

Yea so it looks like you have storages for each repository - even on your remotes. In fact, you technically have 3 local storages and 6 remote storages…

Personally, this seems a bit messy when backing up from one client, but if you don’t expect to see much de-duplication then fair dos.

Monthly for a year definitely sounds like a worthwhile improvement imho. When you run a check -all -stats -tabular you’ll probably notice those extra snapshots don’t take up that much space due to de-duplication. And you can always change your mind if they do.

1 Like
#5

Thank you for the check command, I’ll look into that for the revisions now and see what’s up.