Backup Immutability - Object Lock support?

I saw an announcement today that Backblaze implemented Data Immutability with Veeam supporting it right away… Digging deeper - looks like B2’s S3 API now support some Object Lock API:

I don’t think it is possible to use this as is with Duplicacy - it modifies some existing files after each backup - so it is not possible to protect entire bucket…
But this is a great feature and it would be very good to have this implemented in the code, if possible…

The fact that it is possible to delete/corrupt existing backup always tuned me off from usual back ends.
So far I know only two solutions which can help - one is iDrive - they can restore deleted files on request and rsync.net - they have immutable snapshots.

But it would be great if Duplicacy could create immutable backups directly - it would be then possible to use either immutable attribute in Linux or manually set object lock in B2…

I think this was discussed before, but I can’t find it for some reason.

1 Like

Excellent point. My understanding was that Duplicacy never modified any files on the target except config file; snapshots and chunks are only being created, deleted (during prune) and renamed (during backup and prune); but never modified.

I’m pretty sure it’s already possible to configure the bucket permissions to only allow those operations and disallow modify which will effectively keep all chunks and snapshots immutable.

I’m not sure how important is config file and how often does it change. Maybe I’m misremembering this.

If you keep permission to delete files, you will not have any protection - when ransomware strikes, people who launch attack do appropriate work to find all remote backups and delete them. I speak from experience in recovering friend’s business from such attack… Attackers discovered backup software and deleted all files. Luckily, backup vendor was able to restore from backup.

If object is locked, delete will fail - In this case Duplicacy would need to account for this during prune operation and only delete files when lock attribute expires. This may become very complex, unfortunately.

I played around with locking destination folder with immutable attribute on Linux - but it broke the backup:
I wanted to set things up so the client only does backup and prune is done on sever after temporary resetting immutable attribute for the duration of the operation.

Then I discovered rsync.net - the price is not too bad and free 7 daily immutable snapshots are a great option.

I also emailed Backblaze about recovering files in case of rogue action - and was told that no such option exists. Looks like they finally moving in the right direction with object lock support…

1 Like

For this to work in duplicacy, basically you would have 3 issues to deal with:

#1
Any duplicacy config files that get changed during every backup. This could be fixed by using a new config file copy every time, and deleting the old config files after the lock expires.

For example, instead of using config.data (or whatever), you would use
config.data.00001
then
config.data.00002

It should always copy the highest numbered file to a new one. The old ones can be deleted out as their locks expire.

#2
Pruning / deleting files from the backup. Just don’t set any pruning for less than however long the lock period is. (Or don’t worry about this issue at all, and silently ignore lock errors when deleting, knowing that eventually one day, the delete will work. Every prune will try to delete these, and eventually it will work.)

#3
Keep old files locked somehow. Since duplicacy reuses existing chunks forever, you don’t know how long to lock them for. So, after every prune, it would need to relock any files that are not locked (but ONLY files that are not locked). This would be tricky. Some files would be expire the lock for some time before the next prune to relock them and would be vulnerable during that time. There would instead need to be something indicate that a file needs to be allowed to expire the lock, and otherwise continually renew the lock.

Interestingly, B2 also has “Legal Holds” now in addition to the new locks. I’m not sure how the legal holds work within b2, and they don’t seem to be documented anywhere yet, but this could potentially be used as a different kind of locking mechanism we could use.

Interesting note on Legal Holds… I might send their tech support a question on this.

On the other points - yes, it won’t be simple to implement, but should be possible.
I wonder if prune process can extend lock every run if it knows that chunk is needed and then mark somewhere chunks which can be deleted on the next run, when lock expires.
This will require regular prune run with the period shorter than the lock…

I just realized that the config file (issue #1) wouldn’t be an issue at all . Just don’t lock that file.

So, really, the only issue is #3. Relocking (extending the lock) on all files after each prune, and also like you said, noting the files in another config file that need to be deleted on the next run.

That could be transaction costly through (on b2), having to access through every single chunk, every single week (or however often), and then updating the lock on them.

Or, just thinking here, what if instead of all of that, duplicacy use the b2 “hide” instead of “delete” command. B2 has a feature to automatically delete hidden files after xx days. (Or is hide already being used elsewhere in the duplicacy logic). Remove delete from your key (and any other unneeded functions).

If hide isn’t being used already by the logic, that would be a very simple code change.

Then, the only new feature that would need to be implemented would be a “fix after messing up” if a virus went in and “deleted” everything (and thus made it all hidden) … we would need some way to know what to unhide, programitically. Make some immutable file that is created after each run with a list of what chunks exist. Then, just run that file against an unhide script to “fix” or reverse the virus damage.