Duplicacy with millions of small files

Support
#1

Hello,

I’m looking to possibly move from Crashplan to Duplicacy using Backblaze B2, and use the GUI for Windows Server.

The Windows server has approx 30,000,000 small files in a 1.8TB partition. There are about 8,000 files added daily, and we do prune the overall files periodically so basically our total overall backup hovers around 30 million.

We currently use Crashplan, but its most recent versions and changes have made it problematic at best.

I was wondering if Duplicacy would be appropriate for this situation? We can throw lots of CPU and RAM at the problem, but I just don’t have any real-world stats on if Duplicacy would be a good fit. I’m using Backblaze Personal for home backup, and I’ve experimented with the Duplicacy CLI.

Best,

John

#2

The only issue will be the memory usage. Currently Duplicacy needs to load the entire file list into memory and my estimate is that it probably needs about 20G memory to do that.

1 Like
#3

Thank you.