Server side duplicacy install vs. SFTP only?

Support
#1

We (rsync.net) have been very slow to adopt or attempt to support duplicacy but I would like to explicitly support it … people increasingly ask for it.

My loose understanding is that SFTP is a full-fledged, fully supported transport for duplicacy, but that there is additional value in having the duplicacy CLI executable on the server side … is that correct ?

What can users do when we host the duplicacy binary for them on our end that they can’t do with just plain old SFTP ?

Related to this: is the duplicacy CLI an actual binary executable or is it actually a python script or something like that ?

Thank you very much.

#2

I’m just a user, but here is my understanding:

Duplicacy is written in golang and hence its’ a self-contained native executable with no dependencies (outside of trivial) on each supported platform. All it needs from the backend to work is file access API with ability to upload, download, rename, and delete – and that’s pretty much it; all work is done on the client. There is no “server” mode functionality, like you might find with Borg. It’s strictly client-side software.

There are however some use cases that may benefit from being able to run another instance of duplicacy client closer to the storage, e.g. on the server – for example deep check with hash verification (that would otherwise result in full download to do rehashing via SFTP; some backends support that via API however), or replication from one storage service to another (to avoid download and re-upload).

Those are however all “nice to have” things and not required for backups to work (and is mostly cloud compute type of deal; it can be done on any other cloud platform that has fast pipe to your storage.)

With that in mind I’m not sure what this

would entail: if you make storage available via one of the supported protocols, (sftp is one of them of course) – duplicacy would work, I don’t see how more explicit that support might be.

What are users asking for specifically? Because your understanding is correct – if SFTP is supported – duplicacy will “just work”

1 Like
#3

OK, thank you - the key part is what you said about:

“There is no “server” mode functionality, like you might find with Borg. It’s strictly client-side software.”

That is what we were confused about. borg does, indeed, benefit greatly from having the borg executable installed server-side - and so we do that.

However, if duplicacy (like restic and duplicity and so on) is fully functional as a client against a plain-old-dumb-sftp-server then I think we will leave it at that.

Thank you - appreciated.

#4

Every said by @saspus is correct. I just want to add that there is another use case that requires a server-side Duplicacy installation. That is, use Duplicacy as a sync tool – run Duplicacy on the client side to back up large files to a storage directory on rsync.net, then run Duplicacy on an rsync.net server to restore to the destination directory. The advantage of this over a plain sync tool such rsync/rclone is that you can use as many threads as you want.

I doubt you will ever support checking for corruption on the server side or copying backups to a different storage, but this sync workflow may be of interest to you.

#5

I’m having an issue where duplicacy is backing up incremental system images generated by macrium reflect (windows backup software). Duplicacy takes HOURS to scan/do whatever it is going to do for the backup every time a new incremental image is added to the backup (backup destination is linux box on gigabit lan via ssh for testing). Is this a scenario where having duplicacy installed on both client/server would help or is this just a function of it not handling these incremental images well?

#6

Nope, server-side wouldn’t help here, but are you already using fixed-size chunks for your storage? That would certainly help.

1 Like
#7

Why are you creating incremental compressed deduplicated versioned backup of incremental compressed deduplicated versioned backup?

Just replicate macrium datastore directly.

2 Likes
#8

Eh, it was just easier and one less thing to break. Replicating everything stored on my local nas with duplicacy to my offsite nas was the goal. The more random crap I have that can break itself the less reliable my backups will be in the long run. But yeah, plan B is to just have macrium replicate them although I don’t know how efficient their implementation is, I suppose I’m about to find out…

#9

I’m not sure, I’m using whatever the web interface defaults to. I’ll look into that now and see if it helps at all. Thank you for the suggestion!

#10

It could be worth providing some guidance on recommended settings for Duplicacy storage on rsync.net.

For example noting that enabling erasure coding is not recommended as the data integrity guarantees provided by the underlying ZFS storage make this pointless, plus an example something like this to create a fully encrypted storage:

# create destination directory for backups
ssh <username>@<hostname>.rsync.net mkdir <directory>

# generate key for RSA encryption of chunks
openssl genrsa -aes256 -out private.pem 4096
openssl rsa -in private.pem  -pubout -out public.pem

# init  storage
duplicacy init -e -key public.pem <snapshot id> sftp://<username>@<hostname>.rsync.net/<directory>
#11

HI @saspus

My pet topic with duplicacy is server side verification of the data. I’ve still not gotten a great way to do it. You mention a couple use cases in your post here. Maybe I could ask a question?

Is there a use case that you know of where I can run duplicacy locally (or on the server) to verify that all my remote users have restorable storages? I’d like to check chunk consistency/validity and presence… but of course I don’t have all their passwords

Thanks for any thoughts!!

#12

If you turn on Erase Coding, all chunks as well as the revision files will be stored in a format that makes it possible to check for corruption without the storage passwords. However, this is currently not supported by Duplicacy, although it is quite easy to write a simple tool to do so.

A downside of this approach is that it can’t detect missing chunks.

1 Like