Need help understanding encryption and S3

Support
#1

I’m considering switching my remote backups to S3 from a syncing service, as using a file syncing service seems to be the less efficient solution. However, I am having some trouble understanding how encryption works when using a S3 service, and would appreciate some help wrapping my head around it. Here is what I have done and (think) I understand thus far:

I set up a test bucked on Wasabi, created a duplicacy storage (via web ui), and ran a backup. When looking at the files on Wasabi, it seems clear the chunks are encrypted. So far so good, but I need help understanding how these files were encrypted.

When you set up a local storage, you are prompted for a password which will form the basis of the encryption (to my understanding), but there is no such option when using S3. My first guess is that the secret key is used to encrypt files when sending them to S3. However, given the secret key was generated by the S3 provider, does that mean they could decrypt my files? Or does duplicacy incorporate some other component?

At first I thought some element of the config files may be used, but was able to send a backup on one machine, and restore it on another (which presumably has somewhat different config file contents). I used the same master password for both machines though, so maybe that is the other element?

If someone could help fill in the gaps/correct my misunderstandings here it would be helpful.

For context, I have been using Duplicati to send files via rsync to my local server & pcloud, and am looking for other solutions. Any input would be appreciated. Thank you!

#2

Regardless of the target (S3, local folder, SFTP, etc) you can pass -e flag to init command and provide your own encryption key. That key will be used to wrap the generated keys stored in the config file on the target, that are used to actually encrypt the chunks. If you don’t provide the -e key the keys in the config file are not encrypted and anyone can restore.

There is another option: RSA encryption, where you can provide public key to backup and would need private key to restore.

more details on the encryption implementation in duplicacy is here: https://github.com/gilbertchen/duplicacy/blob/master/duplicacy_paper.pdf

#3

Thanks for the information. If I am understanding correctly, the file names I am seeing on Wasabi are coming from the chunk hash and the ID Key, but the files are not actually secure because I did to pass the -e flag when I created the storage in the web UI. It would not be possible to encrypt this storage after the fact.

Is there a way to pass the -e flag while using the web UI? When setting up S3, I was given an input screen for the S3 details, and then a second screen asking only for the storage ID. It seems odd that local storage will prompt for a password, while others will not give the opportunity. (I am assuming here that the password asked for when creating a local storage becomes the value for -e).

If there is no way to pass -e in the web UI, can I create the storage on the CLI, then interact with it from there on the webui? I am trying to maintain parity in usage between my unraid server (where using the CLI would be easier) and my windows desk top (where I’m not sure how to interact with the CLI).

EDIT: It seems I was wrong about the CLI being easier on unraid, as while accessing the console, there is no duplicacy, only duplicacy_web which does not seem to work in the same way.

#4

Hmmm. This is what I see on the second page after specifying S3 keys and endpoints:

image
image1214×1274 68.1 KB

Which version of WebUI are you using?

#5

That would explain my confusion! Here is what it looks like to me on both windows and unraid.

Untitled-2
Untitled-2651×1520 107 KB

On the settings page, it says I am using command line version version 3.1.0, but I can’t find anything specific for the webUI. I downloaded it from the website/unraid app store just this morning, so I would assume they are the most recent versions.

EDIT: I will add I do see a second screen like yours when creating a local storage.

#6

Got it! Because I created this specific bucket before, it was not providing all the same options! For posterity, you will only see the screen with password/RSA options when you create the storage for the bucket for the first time. And if I understand things right, because duplicacy remembers storages by name, even if you delete thes torage, then create it again, if will not provide this initial screen a second time unless you destroy whatever logs it keeps.

Thank you for the help saspus.

Addendum: This behavior seems to differ when creating local storage, as if I try to create a local storage for the same location twice, I always get the second page with credential options.

#7

See on the second screenshot “The storage has already been initialized. Enter a unique name for the
storage.”

That storage already contains initialized duplicacy datastore, so it simply adopts it and offers you to give it a name how it should be referred to. Since you cannot change encryption parameters after the initial initialization – they are not offered.

If you want to start from scratch – initialize a new storage, and provide all the encryption keys etc – you should specify a different “folder” or bucket, that does not already contain duplicacy data.

#8

Yes!!!

Hmmm this does not look right. Could be a bug? Let me try…

#9

This is what I see first time:

image
image1206×1282 67.7 KB

And this is second time.

image
image1216×1262 39 KB

#10

Yes, it shall be fresh then.

The version is printed at the bottom of the page on the Dashboard tab:

image
image1012×338 10.3 KB

#11

You’re right again. I didn’t go down a level when I tried the second time. Local does behave consistently with other options.

I also now understand the “logs” I was assuming existed before is actually the “config” file that is created in the storage location. Going from there, it makes sense how duplicacy is able to restore to multiple machines, as it reads these configs on the storage side, and deals with decryption using the info in that file and the password you would then provide.

Thanks for helping me work through so many issues between the chair and the keyboard. Duplicati database failed on me earlier today, so have been a bit frantic trying to sort things out and find a replacement that won’t hurt me again. Nevertheless this has been very helpful in learning how all the parts work together in Duplicacy.

#12

No problem, and welcome to the club :slight_smile:

Duplicati has no stable version for quite a while – they EoLed stable one long ago and the new one is in permanent beta. Who does that!? That’s a dealbreaker right there. When I was testing stability of various tools for myself once CrashPlan was sunset – duplicati failed almost instantly. Great marketing though – it invariably a first result on google for a backup tool… After trying almost every tool on the planet I coverged to duplicacy. Among many other benefits – database won’t fail, because it does not have one :wink:

Duplicacy Web GUI has some rough edges, but the CLI engine is what is brilliant. Simple, robust, and performs well. I use CLI version myself with the OS scheduler, but web UI has its place too.

1 Like
#13

[OT]

This is an off-topic comment, but I think it’s worth commenting once you’re starting to use their storage:

#14

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.