Duplicacy demolishes duplicati and rclone for large backups

Hi all,

We have a lot of data to back up (8TB+ and growing) and I’ve recently been looking for a better solution than CrashPlan that we’ve been using that hasn’t been able to keep up at all. It’s quite appalling how bad CrashPlan actually is - and if you think it’s slow backing up, just wait till you have to restore (if you can even get the file listing to load).

I wanted our new backup solution to be cheap, unlimited, and fast.

After doing a bunch of research, I selected Box @ $45/mo for unlimited transfers and storage. Later, I found that Google Drive for Gsuite Business offers the same but for $12/mo (they say you need min 5 users for this, but it’s been working for myself and others with just 1 user). I’m going to use both as that’s only $57/mo - very cheap for unlimited.

Anyway, I started testing with rclone but quickly realized that the 2.5mln files we have (and growing) would take days or even weeks to sync (at least without rclone+restic) because cloud operations are expensive and slow.

I then moved on to duplicati which was much better, but still had a large number of issues and was more confusing that it needs to be. Particularly, while backups were relatively fast (2mln files ~500GB in a day), restores to other hardware in case of a catastrophic loss were extremely slow at best and broken at worst. There are lots of experimental and unstable versions, and the last beta was really buggy and slow too. It’s kind of a mess.

Thankfully, I then found duplicacy on someone’s recommendation. After figuring out how it works, I set it up on the command line (duplicacy web is, unfortunately, still unstable and crashed for me during a restore) and wrote a script to run and log the backups. It’s been running pretty much flawless so far.

But let’s talk about speeds. I posted this comparison on Twitter, but I’ll repost it here.

Some numbers for a ~400GB initial Linux backup with ~2mln files.

rclone (without restic): it’s been 3-4 days and it’s only like 25% done. Unacceptable performance due to file-by-file copying.

duplicati: 24 hours.

duplicacy: 4 hours.

duplicacy is the clear winner.

Update on subsequent incremental backup speed on the same data set with whatever new files got uploaded since yesterday:

rclone: N/A as it never finished the original upload

duplicati: 1 hour 6 minutes

duplicacy: 4 minutes

duplicacy destroys the competition yet again.

So yeah, I’m very excited about duplicacy because it’s robust and very fast both backing up and restoring. I feel much better about a potential disaster recovery.

Thank you, devs!

5 Likes

And has another advantage: an open format. If you decide to change your backup provider/storage tomorrow, simply move your existing files. With Crashplan and others you get stuck in a proprietary format.

I’m a very satisfied Rclone user too, but remember: it is basically a synchronization tool, not backup, even with the --backup-dir option.

Yeah, but I do mention it because it’s a viable alternative. rclone+restic are supposed to be killer, but duplicacy is essentially that, but in one tool.

FWIW, here are the instructions:

Yeah, I saw that post, feels very similar to duplicacy.

I’ll be looking into setting it up as a redundant backup going to Box, since duplicacy doesn’t yet support Box natively.

Just as a side note — there is way to make Crashplan very fast. It boils down to disabling client-side deduplication. If your data is mostly incompressible and unique you don’t benefit from deduplication anyway and it’s a pure waste of resources, logarithmically increasing with the backup set size. It’s like a natural throttle. I described more here: Optimizing CrashPlan Performance

Duplicacy is still an order of magnitude faster — but at $10/month for another full service backup seems worth it. Personally I too use three — Crashplan, Duplicacy to B2 and synology HyperBackup to another synology.

1 Like