How to get Duplicacy to utilise 100% of resources?

Support
#1

I’ve got a 10gbit network end to end, backing up from any machine to a NAS, Duplicacy uses very little resources and does not utilise full available bandwidth. While running Duplicacy in a Win 10 VM i tried manipulating VM parameters and found that at 2 vCPU cores there was maximum 60% utilisation, at 4 vCPU, no more than 20% and so on… ram usage is no more than 1.5GB, even though 32GB is available.

Maximum write speed to the NAS should be 200MB/s yet most of the time I get no more than 40MB/s regardless of thread count. I tried 1, 2,4,8,16 threads, found that going over 4 is meaningless.

Is there a way, some setting to crank Duplicacy to 11 so that it utilises 100% of available resources ?

#2

What is source media? It is likely limited by that, and/or seek latency on your nas, and/or transfer protocol on the nas. Check performance stats of stuff around duplicacy of all subsystems involved, starting with disk io queue size.

FWIW I actually throttle duplicacy down with CPULimit to prevent it from maxing out CPU.

Btw increasing number of threads beyond 1 will be detrimental to performance unless both source and target are flash arrays.

#3

Source media is another nas that has avg 700MB/s read speed

#4

Sustained max sequential transfer bandwidth is meaningless and unimportant metric in most case (exceptions are — sustained sequential transfer on a freshly formatted media — aka unicorn scenario). You need to measure seek latency of opening, reading/writing, closing, indexing, etc of a multitude of small files. That’s what causes waits and limits performance. Even on SSD arrays.

To assess that look at IO load (queue size) on the disk subsystems on each nas, as I suggested above. Do you NASes have resource monitors? Start there.

(Furthermore, why are you trying to make it use all resources in the first place? It’s a backup tool, shall be unnoticeable background process; duplicacy is efficient so you can throttle it further down than most tools)

#5

I work in film industry where we got a lot of propriatery IT equipment, bespoke setups and legacy systems that don’t work with common backup tools, Duplicacy is the only tool out there that can do what I need, becasue of how simple it is both on a technical level and politically (Most big name products have very strict enterprise licensing and hardware support terms that do not cover our use case).

I haven’t tested IOPS performance or seek latency on my NAS, but if it’s good enough for multiple editors to edit and render high datarate 4k footage, then it’s probably good enough for Duplicacy…

#6

That’s predominantly sequential access

Please do; and if you confirm this to be the culprit, depending on the type of data you backup you can improve performance by increasing duplicacy chunk size: this will reduce amount of random IO and therefore performance reducing latency

See

   -chunk-size, -c <size>          the average size of chunks (default is 4M)
   -max-chunk-size, -max <size>    the maximum size of chunks (default is chunk-size*4)
   -min-chunk-size, -min <size>    the minimum size of chunks (default is chunk-size/4)
#7

Duplicacy does a lot more than just transfer data from one point to another. It has to chunk, compress, and encrypt that data, see if it matches a hash, and then upload it.

There may be ways to increase core parallelsation beyond raw transfer threads, but that might add significant complexity, in what is supposed to be an application that should prioritise data integrity. 40MB/s is pretty impressive for a backup application of its type (incremental, de-duplication across multiple source machines/repositories, compressed and encrypted, easy to prune etc.)…

So as long as backups run and complete on the scheduled intervals, what does it matter? And why should it use all 32GB (leaving no room for file system caching) when 1.5GB is actually enough?