Trouble Pruning Multiple Snapshot IDs - "deletion criteria aren't met"

[CodingSquirrel]

I have multiple different repositories that backup to the same storage. Some of these repositories have different needs as far as retention, so instead of using prune -all I have a series of prune commands that specify snapshot id and use -collect-only and then one final call with just -delete-only. From what I understand this should work for my needs. I run this from one repository directory. The problem I’m having is after running these I get:

Fossils from collection 1 can’t be deleted because deletion criteria aren’t met

From looking at the documentation I expected this after the first run, but I’m still getting this same line a week later even though I have run backups since. I noticed that every prune run reports Fossil collection 1 saved. I also inspected the fossils directory at .duplicacy/cache/local/fossils/ and see that there is only one file (named 1) that only contains information from the last prune command. Nothing from the other runs for other snapshot ids.

Example of my bash script:

#!/bin/bash

cd /root/duplicacy/data || exit 1

# Data - 1:day after 1 week, 1:week after 2 weeks, 1:month after 2 months, Nothing after 1 year
echo "Pruning data at $(date)"
duplicacy prune -id data -collect-only -keep 0:365 -keep 30:60 -keep 7:14 -keep 1:7
# Storage - 1:day after 1 week, 1:week after 1 month, 1:month after 6 months, 1:year after 3 years
echo "Pruning storage at $(date)"
duplicacy prune -id storage -collect-only -keep 365:1095 -keep 30:180 -keep 7:30 -keep 1:7
# Storage_temp - 1:day after 1 week, 1:week after 2 weeks, Nothing after 3 months
echo "Pruning storage_temp at $(date)"
duplicacy prune -id storage_temp -collect-only -keep 0:90 -keep 7:14 -keep 1:7

# Final deletion
echo "Running prune delete at $(date)"
duplicacy prune -delete-only
echo "Finished at $(date)"

Output from duplicacy -log -v prune -delete-only:

2024-10-19 11:50:31.757 INFO REPOSITORY_SET Repository set to /srv/data
2024-10-19 11:50:31.758 INFO STORAGE_SET Storage set to /srv/backup/backup/duplicacy
2024-10-19 11:50:31.759 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2024-10-19 11:50:31.832 TRACE SNAPSHOT_LIST_IDS Listing all snapshot ids
2024-10-19 11:50:31.833 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot 2
2024-10-19 11:50:31.833 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot data
2024-10-19 11:50:32.313 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot storage
2024-10-19 11:50:32.418 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot storage_temp
2024-10-19 11:50:32.466 INFO FOSSIL_COLLECT Fossil collection 1 found
2024-10-19 11:50:32.466 TRACE SNAPSHOT_NO_NEW No new snapshot from data since the fossil collection step
2024-10-19 11:50:32.466 TRACE SNAPSHOT_NO_NEW No new snapshot from storage_temp since the fossil collection step
2024-10-19 11:50:32.466 TRACE SNAPSHOT_NO_NEW No new snapshot from storage since the fossil collection step
2024-10-19 11:50:32.466 INFO FOSSIL_POSTPONE Fossils from collection 1 can’t be deleted because deletion criteria aren’t met

duplicacy -help shows version:

VERSION:
3.2.3 (254953)

Is there something I’m doing wrong? I’m guessing I’ll at the very least have to run -exhaustive to fix the current state, but will it just do the same thing again? Any help is appreciated.

[saspus]

[Solved] Copy from local to cloud, prune inconsistent

It seems it works as designed. You can read more here duplicacy/duplicacy_paper.pdf at master · gilbertchen/duplicacy · GitHub but the gist of it is that duplicacy is fully concurrent, it has to support weird scenarios where five machines run backup to a specific storage and seven other machines try to prune it at the same time. There is safety built in preventing immediate deletion of chunks — that fossilization step is part of that mechanism. For example, you don’t want prune to nuke the chun…

[CodingSquirrel]

It’s giving this error even after I ran backups. I first ran prune for each snapshot id last week, then I ran again this week and it still didn’t remove anything and gave that “error”. It also doesn’t appear to be making more than one fossil collection. I only see a single fossil collection in the cache directory, and the delete step doesn’t appear to see any others either. I would expect each of the prune runs I did on each snapshot to have its own fossil collection, or all be listed in the one that’s there.

[saspus]

Not sure. Should not your second command have -a or -id?

Any reason you are separating collection and deletion in the first place?

[CodingSquirrel]

Are you referring to an id for the -delete-only command? I thought that didn’t need an id because it’s just deleting the fossils already collected.

As for why, in another thread I had found this advice for what seemed to be a similar setup and it made sense to me. Instead of running both collection and deletion and all the checks that probably go along with it over and over, you collect all first then delete all at once.

Pruning retention policies for multiple repositories

Definately agree that it’d be nice if multiple -ids could be specified in a single run… However, in order to prune a storage, all snapshots have to be considered each run - as far as I understand - it has to compile a full list of chunks in memory (taken from all the snapshots) across the storage in order to determine what it can safely delete - even if you only run it on one id. Though it’s not really listing all chunks on the disk (unless you specify -exhaustive). Additionally, if you’re app…

[CodingSquirrel]

I found this bug thread that kind of sounds like the issue I’m having, but is from a couple years ago. Is it possible this wasn’t fixed?

Prune -id -collect-only overwrites Fossil collection 1

Please describe what you are doing to trigger the bug: duplicacy prune -id a -keep 0:1800 -keep 7:30 -keep 1:7 -collect-only duplicacy prune -id b -keep 0:1800 -keep 7:30 -keep 1:7 -collect-only duplicacy prune -id c -keep 30:365 -keep 7:90 -keep 1:7 Please describe what you expect to happen (but doesn’t): Expect prune to run without creating unreferenced fossils. Please describe what actually happens (the wrong behaviour): First prune command saves Fossil collection 1; second prune command…

[saspus]

Maybe there are additional snapshot ids in that datastore that shared chunks with those being pruned?

@CodingSquirrel

Instead of running both collection and deletion and all the checks that probably go along with it over and over, you collect all first then delete all at once

But you are running it on a local storage — after first pass all metadata is already in filesystem cache, I doubt you will see measurable difference.

@CodingSquirrel

I found this bug thread that kind of sounds like the issue I’m having, but is from a couple years ago. Is it possible this wasn’t fixed

Not impossible — especially since very few people use it this way. I would ask in that thread.

[CodingSquirrel]

Thanks, I’ll give it a try without separating the steps and see if the issue persists or not, especially if the time difference wouldn’t be meaningful in my case. I’ll update with results.

I also followed up on the other thread, so hopefully that gives an answer on if that is still a lingering bug.

[CodingSquirrel]

Removing the -collect-only flags from the individual prune commands, and the following -delete-only call seems to have worked. New, separate, fossils are being created by the different runs instead of overwriting.

And duplicacy prune -exhaustive picked up all the unreferenced chunks from after my other attempts that didn’t properly generate fossils. Thankfully that didn’t take too terribly long, at about an hour. I reran my backups so they each had a fresh snapshot revision and then reran prune. That cleared up about 8TB of space and took about 12 hours to run.

Thankfully now with a working prune setup my security footage won’t accumulate in backups for a year.