Prune options for handling of deleted files

Feature
#1

This is somehow related with (part of) this other issue, but I’ll open a new thread to keep things focused.

I’m trying to figure out Duplicacy’s versioning/retention settings (and their limitations). So it is clear that I can set all kinds of sophisticated configurations such as

duplicacy prune -keep 0:360 -keep 30:180 -keep 7:30 -keep 1:7

which will 1 snapshot per day for snapshots older than 7 days, 1 snapshot every 7 days for snapshots older than 30 days, etc

So the retention works at the snapshot level. One important setting for me is, however, to never remove any deleted files (in other words: there should always be at least one version of every file ever backed up). This is not possible with duplicacy, right?

Could it be implemented at all?

Remove specific files from the storage
#2

This will be very hard to implement. Within a snapshot, a file is normally packed in chunks with other others, so it is difficult to keep certain files while removing others.

Equivalent "keep" rule to Duplicati's smart backup retention?
#3

But it could be done?

How about something like this:

Before deleting a fossil, duplicacy checks whether the filename to which the fossil belonged still exists in any snapshot, if it does, it deletes it, if not, it gathers all the other chunks that belong to that file and adds them to a kind of virtual snapshot in which it references all the deleted files. Perhaps this could be snapshot 0 in each snapshot ID.

Understanding pruning
#4

Before deleting a fossil, duplicacy checks whether the filename to which the fossil belonged still exists in any snapshot, if it does, it deletes it, if not, it gathers all the other chunks that belong to that file and adds them to a kind of virtual snapshot in which it references all the deleted files. Perhaps this could be snapshot 0 in each snapshot ID.

That is possible but I feel this makes the prune command much more complicated. In addition it violates two design principles: 1) a prune command now creates new chunks (the chunks that make of the file list in snapshot 0) and 2) files are no longer immutable (the snapshot file for snapshot 0 needs to be overwritten).

#5

a prune command now creates new chunks (the chunks that make of the file list in snapshot 0)

Well, not really. It just doesn’t delete chunks it would otherwise have deleted. Already now, prune manipulates chunks by renaming them. This would be no different. Instead of deleting the fossil, it would rename it back (just like it does now when it realizes that it is still needed).

The difference is rather that prune would write into a snapshot file, which, I think it currently doesn’t do. So if that is what bugs you (that’s how I understand your second point), I can understand your concern. It would be nicer to keep the principle: prune does not modify snapshots (except for deleting them, of course). But I would argue that this principle would still be maintained for all snapshots, except snapshot 0. And, importantly, snapshot zero is modified only by prune.

1 Like