So, I am a bit confused and hope to get some insight on how prune works in this case,
I have a few folders on my PC mounted via CIFS shares (from my NAS) being backed up using duplicacy to my Onedrive. Each of those shares have a separate backup ID. Now some of those are archive folders, which I rarely access but want to keep forever. Suppose I stop daily backups of those shares, will prune delete those folders eventually? I won’t then have any copy of the folder right?
I am using the duplicacy-gui with default prune options,
-delete snapshots older than 1800 days
-keep 1 snapshot every 7 days if older than 30 days
-keep 1 snapshot every 1 days if older than 7 days
Should I leave the first one (1800 days) blank in my case?
As far as I’m aware, so long as you don’t use -exclusive flag, the very last backup revision won’t get deleted, but older revisions will as per prune retention period.
IMO, Duplicacy really needs to support multiple -id's in a single prune command so that a retention policy can run (with -collect-only) for a bunch of repositories and a final -delete-only takes care of the second step.
That, and/or an individual repository could be set never to be pruned - i.e. -no-prune=true a la -no-backup.
I’ve been running Duplicacy (successfully and relatively happily) for over a year, have several “archive” snapshots containing 10 years of data migrated from CrashPlan, and am still uncertain of how prune works.
For example, if you have active snapshots a, b, and c and archive snapshots a-archive, b-archive, and c-archive, you can avoid pruning the -archive snapshots by individually pruning all the active snapshots, which is what I’ve been doing, e.g:
duplicacy prune -id a -keep 0:1800 -keep 7:30 -keep 1:7
duplicacy prune -id b -keep 0:1800 -keep 7:30 -keep 1:7
duplicacy prune -id c -keep 30:365 -keep 7:90 -keep 1:7
This also allows different snapshots to have different retention policies.
But if all active snapshots have the same retention policy, I’m wondering if, after not backing up to the -archive snapshots for more than 7 days, the following command could be used:
because the documentation states that Duplicacy by default will ignore repositories that have no new backup in the past 7 days ….
The documentation seems to use the terms snapshot ID, backup ID, and repository interchangeably – are they the same? I ask because archive snapshots are not associated with repositories.
The prune documentation goes on to state … and you can also use the -ignore option to skip certain repositories when deciding the deletion criteria, which seems to refer to fossil deletion. Isn’t it true that fossils are orphans, and are not associated with any revision in any snapshot/repository? If so, there are no snapshots/repositories to be ignored, so what does -ignore actually do? I’d assumed that -ignore would affect fossil collection, not deletion.
Yes, I could figure this out with -dry-run experiments, but I’ve grown weary of that.
Could be wrong, but I believe the 7 day / -ignore thing is about how it deals with the deletion of fossilised chunks between the two step process.
A protection in case a long backup is still in progress (although I dunno why anyone would want to use the -ignore flag specifically; maybe if a backup is expected to last more than 7 days?), and where such chunks might have subsequently been ‘collected’, because they’re as yet unreferenced.
In all my experiences running prune, snapshots outside of global retention period will get removed regardless of if they’re ignored or not, so this only seems to matter for the last (or incomplete) backup.
As mentioned before, the most recent backup never gets removed unless running with -exclusive (I don’t have any ‘archive’ snapshots, although through laziness I do have old backups where I just let it prune everything, and only one revision remains). Thus from my understanding the 7 day / -ignore protection is about making sure a running backup gracefully becomes the ‘last’ backup.
Maybe gchen or someone can correct or explain it better?
BTW, I believe this would be more efficient, although I don’t have mixed retentions to test if this would save much time (nice if it would):
Thank you all for responding. I will try the Prune command with various options and see which one would suit my needs. I am not yet sure if I understand the retention policy completely but will read some docs again to get a hang of the Prune command!
I tried -collect-only on one of my jobs that copies a 2.8TB storage then does a prune on each of 13 snapshot IDs. It doesn’t appear to be faster. Without -collect-only, the first prune deletes all the fossil collections saved by previous prunes, and each new prune saves a consecutively-numbered fossil collection.
First prune without -collect-only:
prune-20220208-112953.log:2022-02-08 11:31:17.433 INFO FOSSIL_COLLECT Fossil collection 57 found
prune-20220208-112953.log:2022-02-08 11:31:17.433 INFO FOSSIL_DELETABLE Fossils from collection 57 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:31:26.985 INFO FOSSIL_COLLECT Fossil collection 58 found
prune-20220208-112953.log:2022-02-08 11:31:26.985 INFO FOSSIL_DELETABLE Fossils from collection 58 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:31:57.874 INFO FOSSIL_COLLECT Fossil collection 59 found
prune-20220208-112953.log:2022-02-08 11:31:57.874 INFO FOSSIL_DELETABLE Fossils from collection 59 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:32:04.323 INFO FOSSIL_COLLECT Fossil collection 60 found
prune-20220208-112953.log:2022-02-08 11:32:04.323 INFO FOSSIL_DELETABLE Fossils from collection 60 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:32:23.056 INFO FOSSIL_COLLECT Fossil collection 61 found
prune-20220208-112953.log:2022-02-08 11:32:23.056 INFO FOSSIL_DELETABLE Fossils from collection 61 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:33:39.664 INFO FOSSIL_COLLECT Fossil collection 62 found
prune-20220208-112953.log:2022-02-08 11:33:39.664 INFO FOSSIL_DELETABLE Fossils from collection 62 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:34:04.196 INFO FOSSIL_COLLECT Fossil collection 63 found
prune-20220208-112953.log:2022-02-08 11:34:04.196 INFO FOSSIL_DELETABLE Fossils from collection 63 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:34:10.085 INFO FOSSIL_COLLECT Fossil collection 64 found
prune-20220208-112953.log:2022-02-08 11:34:10.086 INFO FOSSIL_DELETABLE Fossils from collection 64 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:34:13.753 INFO FOSSIL_COLLECT Fossil collection 65 found
prune-20220208-112953.log:2022-02-08 11:34:13.753 INFO FOSSIL_DELETABLE Fossils from collection 65 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:34:18.390 INFO FOSSIL_COLLECT Fossil collection 66 found
prune-20220208-112953.log:2022-02-08 11:34:18.390 INFO FOSSIL_DELETABLE Fossils from collection 66 is eligible for deletion
prune-20220208-112953.log:2022-02-08 11:40:53.368 INFO FOSSIL_COLLECT Fossil collection 67 saved
But deletions of all these newly-created fossil collections in later prunes are quickly skipped, saving negligible time. E.g. here’s the fourth prune without -collect-only:
prune-20220208-115422.log:2022-02-08 11:55:42.053 INFO FOSSIL_COLLECT Fossil collection 67 found
prune-20220208-115422.log:2022-02-08 11:55:42.053 INFO FOSSIL_POSTPONE Fossils from collection 67 can't be deleted because deletion criteria aren't met
prune-20220208-115422.log:2022-02-08 11:55:42.054 INFO FOSSIL_COLLECT Fossil collection 68 found
prune-20220208-115422.log:2022-02-08 11:55:42.054 INFO FOSSIL_POSTPONE Fossils from collection 68 can't be deleted because deletion criteria aren't met
prune-20220208-115422.log:2022-02-08 11:55:42.054 INFO FOSSIL_COLLECT Fossil collection 69 found
prune-20220208-115422.log:2022-02-08 11:55:42.054 INFO FOSSIL_POSTPONE Fossils from collection 69 can't be deleted because deletion criteria aren't met
prune-20220208-115422.log:2022-02-08 12:01:11.826 INFO FOSSIL_COLLECT Fossil collection 70 saved
With the -collect-only option on all but the last prune, there are fewer fossil-related log messages, and deletion of fossils from previous prunes is done by the last prune:
prune-20220225-125316.log:2022-02-25 13:00:54.284 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-130054.log:2022-02-25 13:07:12.622 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-130713.log:2022-02-25 13:13:14.791 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-131315.log:2022-02-25 13:19:29.565 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-131930.log:2022-02-25 13:26:05.538 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-132607.log:2022-02-25 13:32:59.650 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-133300.log:2022-02-25 13:39:13.229 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-133914.log:2022-02-25 13:45:36.636 INFO FOSSIL_COLLECT Fossil collection 1 saved
prune-20220225-134846.log:2022-02-25 13:54:23.102 INFO FOSSIL_NONE No fossil collection has been created since deleted snapshots did not reference any unique chunks
prune-20220225-135423.log:2022-02-25 13:55:23.187 INFO FOSSIL_COLLECT Fossil collection 1 found
prune-20220225-135423.log:2022-02-25 13:55:23.187 INFO FOSSIL_POSTPONE Fossils from collection 1 can't be deleted because deletion criteria aren't met
prune-20220225-135423.log:2022-02-25 13:55:23.201 INFO FOSSIL_COLLECT Fossil collection 67 found
prune-20220225-135423.log:2022-02-25 13:55:23.201 INFO FOSSIL_DELETABLE Fossils from collection 67 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:55:33.757 INFO FOSSIL_COLLECT Fossil collection 68 found
prune-20220225-135423.log:2022-02-25 13:55:33.757 INFO FOSSIL_DELETABLE Fossils from collection 68 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:56:03.020 INFO FOSSIL_COLLECT Fossil collection 69 found
prune-20220225-135423.log:2022-02-25 13:56:03.020 INFO FOSSIL_DELETABLE Fossils from collection 69 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:56:08.904 INFO FOSSIL_COLLECT Fossil collection 70 found
prune-20220225-135423.log:2022-02-25 13:56:08.905 INFO FOSSIL_DELETABLE Fossils from collection 70 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:56:23.525 INFO FOSSIL_COLLECT Fossil collection 71 found
prune-20220225-135423.log:2022-02-25 13:56:23.526 INFO FOSSIL_DELETABLE Fossils from collection 71 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:57:27.170 INFO FOSSIL_COLLECT Fossil collection 72 found
prune-20220225-135423.log:2022-02-25 13:57:27.170 INFO FOSSIL_DELETABLE Fossils from collection 72 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:58:42.896 INFO FOSSIL_COLLECT Fossil collection 73 found
prune-20220225-135423.log:2022-02-25 13:58:42.896 INFO FOSSIL_DELETABLE Fossils from collection 73 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:58:59.507 INFO FOSSIL_COLLECT Fossil collection 74 found
prune-20220225-135423.log:2022-02-25 13:58:59.507 INFO FOSSIL_DELETABLE Fossils from collection 74 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:59:06.006 INFO FOSSIL_COLLECT Fossil collection 75 found
prune-20220225-135423.log:2022-02-25 13:59:06.006 INFO FOSSIL_DELETABLE Fossils from collection 75 is eligible for deletion
prune-20220225-135423.log:2022-02-25 13:59:11.256 INFO FOSSIL_COLLECT Fossil collection 76 found
prune-20220225-135423.log:2022-02-25 13:59:11.257 INFO FOSSIL_DELETABLE Fossils from collection 76 is eligible for deletion
prune-20220225-135423.log:2022-02-25 14:27:44.568 INFO FOSSIL_COLLECT Fossil collection 77 saved
BUT, I’m wondering if fossil collection is working, because every prune with the -collect-only option saves to collection 1, so is collection 1 being overwritten?
This is looking like a bug – prune -exhaustive -exclusive -dry-run shows only 34 chunks in collection 1, but 1,725 FOSSIL_UNREFERENCED. I presume I should remove these, yes?
