Duplicacy not backing up new revision of encrypted container with small changes

Support
#1

I have a 2GB encrypted container that has been backed up multiple times to the remote storage using duplicacy web edition. After any significant changes to the contents of the container, I dismount the container, copy it to another folder, and back up that other folder.

This time I ran duplicacy again with the only changes to the copied encrypted container having a few new log and cache files.

However the backup log shows no new chunks, neither file nor metadata chunks.

The modified date and the file size of the encrypted container stays the same all the time, as per the security settings of the encrypted container. But the date created will have changed since I’m making a new copy of the updated container.

I have never encountered this scenario in any of the previous backup runs. So I’m wondering which timestamp is duplicacy using to determine whether or not to back up the file if I don’t use the hash option?

#2

This is the problem. Duplicacy checks date and size.

You can relax those settings. Those are security theater, don’t improve “security” but can and do break things, like your duplicacy use-case demonstrates. Usually heuristic “same size + same modification date == no changes” is pretty robust, unless some app decides that it knows better and breaks assumptions.

As your testing confirmed it’s using modified date. It would not make sense to use created date – files can and do change since being created.

Separate discussion is on why are you backing up encrypted container? Backup contents of the container instead, while it’s mounted.

#3

As your testing confirmed it’s using modified date. It would not make sense to use created date – files can and do change since being created.>

Why was it that all the previous backups before today worked fine even though the date modified and the file size had not changed?

Separate discussion is on why are you backing up encrypted container? Backup contents of the container instead, while it’s mounted.>

If you back up the contents of a container instead, it defeats the purpose of an encrypted container. If you had to restore the contents of the encrypted container, those files would no longer be encrypted and unreadable, relative to the local system, which may not be the same secure system that you use to back it up.

#4

I don’t know. They should not have. Maybe something around changed, that caused modification date to get updated. You can check what modification dates were of the container in the version history.

How so? Encrypted container protects against local shenanigans. Your backup is also encrypted. Or do you trust your container encryption more than duplciacy’s? Both use standard libraries.

In the same vein if you mount your encrypted container your files are accessible plaintext. I don’t see any difference.

But let’s assume you still want to use container, for “plausible deniability”, (some believe it matters), or whatever.

Then you should not use duplicacy, or any other versioning tool to back it up. Instead, copy containers as is into remote storage. Because if your container is properly designed, small change in the content shall spread out over large are in the container, thus defeating any deduplication duplicacy (or any other backup tool) attempts to do.

So you are just increasing complexity, losing performance, and incurring costs for absolutely no reason. Moreover, to restore, you now have to egress full container worth of data from the remote, as opposed to mounting it remotely.

Side comment – if you backup binary blobs like virtual disks, including encrypted containers, use fixed-sized chunking, if you are not already. You will get much better performance.

#5

In the same vein if you mount your encrypted container your files are accessible plaintext. I don’t see any difference.>

There is a big difference. As I said, the system used to restore the backup may not be the same one that used to upload the container. Why would you think that it’s fine to back up the contents of the container from a secure computer, and then restore the contents of the container unencrypted to a different unsecure computer?

So you are just increasing complexity, losing performance, and incurring costs for absolutely no reason. Moreover, to restore, you now have to egress full container worth of data from the remote, as opposed to mounting it remotely.>

The contents of the container only require infrequent backup.

The only situation when the backup needs to be restored is if the computer is lost. If all the data is lost, then the entire content needs to be restored anyway, so egress doesn’t matter whether it’s a container or individual files.

There is not much more complexity other than 1 minute per month, and there is no additional cost.

#6

This makes no logical sense. Presumably, you have the encryption keys to both the container and the Dupliacy backup storage - so you get to choose how it’s restored, whether it’s individual files or back into another encrypted container. If you’re sharing the backup storage with other systems, there’s also RSA encryption.

You’re just losing de-duplication efficiency when dealing with containers versus file-based backup. It’s the same whether it’s a monolith zip file, disk image, or encrypted container. You’ll get far better results with files and you can still stay safe because Duplicacy’s encryption is just as robust as VeraCrypt or whatever, because only you have the keys, and you only have to keep the container unlocked long enough to take the backup.

Anyway, as saspus points out, fixed-sized chunking is better if you wanna carry on down this route, but also… you can run backup -hash to bypass the modified check.

#7

This makes no logical sense

It makes complete sense. As I keep saying, the computer used to restore the repository may not be the same computer used to make back up and could be unsecure.

If the computer used to restore the repository does not belong to you but is the only one available to use, you may not have a choice of installing encryption programs to secure the local storage.

You lose everything in your house in a disaster. You have a USB stick and your credentials. You need to find internet connection and a secure computer, but you may not find them both together at the same time. If you find the internet connection first but not a secure computer, you download the encrypted container to the USB stick so that the data remains secure until you can find your own secure computer.

I already know that deduplication does not work with the encrypted container. But it still transfers less than 5% of the entire container each time a back up is run. And the encrypted container is less than 1% of all my other backups, and changes and backups for the container are only occasional.

Anyway I set the containers date modified timestamp to be updated whenever the contents changed, so now duplicacy will work every time the backup is run.

#8

In the same vein, you can always make a local copy of your remote backup - either (without RSA) the whole encrypted backup, or partially (with RSA) - say, just the most recent backup, with Duplicacy doing the copying - and be 100% safe, until such time you restore it to a “secure computer”.

I mean, do it the way you wanna, but I’m telling you the better way is with RSA encryption. Whether you do it on an unsecured computer or not, makes no difference compared with the scenario you outline.

Don’t forget to used fixed-size chunks. It makes a huge difference.