We have a lot of data to back up (8TB+ and growing) and I’ve recently been looking for a better solution than CrashPlan that we’ve been using that hasn’t been able to keep up at all. It’s quite appalling how bad CrashPlan actually is - and if you think it’s slow backing up, just wait till you have to restore (if you can even get the file listing to load).
I wanted our new backup solution to be cheap, unlimited, and fast.
After doing a bunch of research, I selected Box @ $45/mo for unlimited transfers and storage. Later, I found that Google Drive for Gsuite Business offers the same but for $12/mo (they say you need min 5 users for this, but it’s been working for myself and others with just 1 user). I’m going to use both as that’s only $57/mo - very cheap for unlimited.
Anyway, I started testing with rclone but quickly realized that the 2.5mln files we have (and growing) would take days or even weeks to sync (at least without rclone+restic) because cloud operations are expensive and slow.
I then moved on to duplicati which was much better, but still had a large number of issues and was more confusing that it needs to be. Particularly, while backups were relatively fast (2mln files ~500GB in a day), restores to other hardware in case of a catastrophic loss were extremely slow at best and broken at worst. There are lots of experimental and unstable versions, and the last beta was really buggy and slow too. It’s kind of a mess.
Thankfully, I then found duplicacy on someone’s recommendation. After figuring out how it works, I set it up on the command line (duplicacy web is, unfortunately, still unstable and crashed for me during a restore) and wrote a script to run and log the backups. It’s been running pretty much flawless so far.
But let’s talk about speeds. I posted this comparison on Twitter, but I’ll repost it here.
Some numbers for a ~400GB initial Linux backup with ~2mln files.
rclone (without restic): it’s been 3-4 days and it’s only like 25% done. Unacceptable performance due to file-by-file copying.
duplicati: 24 hours.
duplicacy: 4 hours.
duplicacy is the clear winner.
Update on subsequent incremental backup speed on the same data set with whatever new files got uploaded since yesterday:
rclone: N/A as it never finished the original upload
duplicati: 1 hour 6 minutes
duplicacy: 4 minutes
duplicacy destroys the competition yet again.
So yeah, I’m very excited about duplicacy because it’s robust and very fast both backing up and restoring. I feel much better about a potential disaster recovery.
Thank you, devs!