Hello folks,
New to Unraid as well as Duplicity - I Googled and search this forum with regards to Backup performance, learned about -threads and -limit-rate, etc. nothing seems to matter for my upload speeds which are currently 1.2mb/s. Backup up is for my local HD array with nvme write cache only to Backblaze B2 (also new for me).
Any suggestions / recommendations appreciated.
Thank you!
Greg
As usual, determine what is a bottleneck.
Is it your array read performance? Seek latency? Upstream connection bandwidth? Upstream connection latency? Bufferbloat (that can result in dramatic (3-4 orders of magnitude latency increases under load)?
Without measuring all that it’s impossible to advise…. Which is to say, the advice is to measure it — and then you’ll know exactly what to fix.
All great point, would love some direction as to how to determine / isolate the bottleneck
Is it your array read performance? I think it’s 132 MB/s (based on what I see in Unraid) - all new WD 7200 RPM drives
Seek latency? Not sure how to check
Upstream connection bandwidth? 30 mb up (1GB down) cable
Upstream connection latency? as in ping latency to host? if so 11ms
Bufferbloat (that can result in dramatic (3-4 orders of magnitude latency increases under load)? Not sure how to check
Should add nothing else is running on the server at this time as it’s new .
Thanks for the help!
Greg
This is sequential read/write best case. When there are many small files disks spend most time repositioning heads to the correct track and then waiting for the correct sector to fly by. This is whats seek latency is referring to.
You can use tools like Crystal Disk Mark or Amorphous Disk Mark to measure performance of your mounted network drive: you will get better numbers due to caching, but even those better numbers will be very low. It is not unusual to see 0.01MB/s IO performance.
On your storage server there shoudl be some tool to display either IO rate hitting the array or disk queue length, or “disk busy” metric, or “storage utilization”, depending on the vendor; some show it in percent. See if it is pushing 100% during transfer – then this is you bottleneck.
Another thing to check is CPU utilization. does it push 100% per core? e.g. 25% on 4-core CPU, etc.
Ah, fellow Comcast customer with Technicolor modem, I presume? The bufferbloat is horrible there, I keep SQM on at all times for the past 7 years at least.
This is unloaded latency. What latency do you get when you are saturating your upstream?
go to Speed test - how fast is your internet? | DSLReports, ISP Information and run the test. if you get anything lower than B for buffer bloat – you need gateway with SQM. More on bufferbloat here: https://www.bufferbloat.net
This can also mean that the server being new is aggressively indexing all content, fully saturating array performance not leaving anything for anything else.
Unraid, specifically, is famous for horrible array performance as it is, doing anything else on it would compound the issue.
Side off-topic comment, since you say it’s a new server: why Unraid, when TrueNAS exists, objectively better in any way that matters?
All test performed while backup is running on server (in background)
CrstalDisk Mark Test on same share mounted
Ping when backup is running
Sorry had to split up due to new user limitations 
I chose Unraid vs TrueNAS simple becuase I had issues with the AMD Threadripper stability / performance on BSD vs Linux (I ran on Fedora)…
Still chuggin along at 1.16MB/s
Everything looks good indeed!
However the upload rate Duplicacy is reporting on your screenshot is 1.16MB/sec, which is 9.28 Mb/sec, not 1.1mb/sec from your post:
And this completely changes the situation.
How many upload threads are you using? One?
Backblaze provides 10Mbps per connection; clients are expected to use it multithreaded. You have 30Mbps connection – you should be able to saturate it with 3 threads. (note, 30Mbps is about 3.5MBps).
I would set 4 threads, and call it a day, this should be saturating your upstream, at 3.5MBps (30Mbps).
GOT IT!
I’ve tried 1, 2, 3, 4, 8, 16 threads the number is always the same BUT I’ll lock it in at 4 from now on just in case.
Greg
Oh, so with 4 threads you are still getting very low speed? That’s not good. Maybe you are far from the datacenter/some outage?
What do you get here: https://www.backblaze.com/speedtest/ ?
Bizarre.
Check backup log – in the beginning of the log duplicacy lists how many threads is it using. Confirm it’s not 1.
I am at a loss… Summoning @gchen for input on why else could duplicacy upload so slowly?
Can you also run duplicacy benchmark Benchmark command details ?
Hi Greg,
Based on the screen grab you posted of the UNRAID system info, the ASUS mainboard in your UNRAID box has dual embedded NICs – an Intel I211-AT (1 Gbps) and an Aquantia AQC-107 (10 Gbps).
However, your UNRAID box is listing both NICs as capable of 1000 Mbps so it’s not clear which NIC is currently mapped to “eth0”. Could you post the shell output from “lspci” and “ip a s”? If I’m reading the screen grab correctly, there’s only one Ethernet cable plugged in.
Recapping the results from your Backblaze “Check Your Home Bandwidth” speed tests:
Backup Down Up Ping Jitter
Running (Mbps) (Mbps) (ms) (ms)
--------------------------------------
No 496.27 38.20 121.60 153.83
Yes 440.75 35.09 114.40 26.82
The Waveform bufferbloat test you ran clocked your download speed at 839.8 Mbps and upload at 24.4 Mbps. While it’s quite a difference compared to the Backblaze test results, it’s the jitter that looks odd because it went from bad to good while a backup was running (in general, 30ms or less is ideal for VoIP, video conferencing and other near real-time applications).
The CrystalDiskMark and ping tests were clearly from a Windows machine, but were the Backblaze speed tests also from the same machine or were they from the UNRAID box? (If the tests results were all from a different computer, it’s fine for diagnosing your broadband connection, but unfortunately leaves a lot of unknowns with your UNRAID box.)
In situations like this, it’s best to test with a barebones setup. Head over to https://system-rescue.org/ and make a bootable CD or USB flash drive. Boot your UNRAID box from the SystemRescue live CD/USB. Select the option to buffer the image into RAM so that you can unplug the USB drive (if that’s what you’re using).
(Based on your earlier posts on this thread, I’m assuming that you know enough about Linux for me to skip some details. If I’m wrong, please let me know.)
In SystemRescue, setting the user password will allow remote shell access via SSH. It’s not the only way, but it’ll be more convenient for copying-and-pasting the output from the commands below.
At the shell prompt, dump some hardware info with the following commands:
dmesg | grep ' sd '
lspci
ip a s
Although unlikely to be an issue based on your CrystalDiskMark results (the random 4K read test at 10.95 MB/s is more than 2x your upload speed), if you’d like to test the WD drives in your UNRAID box try the following command on the first detected SATA/SCSI drive (assuming there’s no RAID hardware):
hdparm -t /dev/sda
(Repeat the speed test at least 3 times to get a decent average.)
Assuming that your UNRAID box already got an IP address via DHCP, test the network upload speed:
iperf3 --time 60 --interval 5 --client la.speedtest.clouvider.net
Next, check the network jitter:
iperf3 --time 60 --interval 5 --client la.speedtest.clouvider.net --udp --bitrate 10M
That’s all for now. The next steps depend on what’s in the collected output.
Hi Gadget!
To clarify, this configuration runs well on linux (fedora) bare metal install and all tests 100% inline with HW specs, I ran it this way for 2 years.
The question I posted was to see if Duplicacy on unRAID backing up to 1.12MB/s with -threads 4 is expected results or if I need to tweak something to speed it up as from my testing none of the parameter options seem to effect the 1.12 MB speeds.
Also I have 2 ethernet adapters 1 is 1GB and the other 10GB, but my home router is only 1GB, also my cable modem upload is on 30MB while down is 1 GB (optimium).
I’m not concerned with the performance on the system as a NAS or Hybervisor, both are doing great, it’s just that with Duplicacy running it’s barely putting a strain on the allocated HW and from the looks of it network, so I was curious if I had to tweak something for this configuration.
again many thanks for all of your time,
Greg
As @saspus mentioned, running the benchmark command is the best way to figure out what the bottleneck is.
Open the docker console and run these commands:
cd /cache/localhost/0
/config/bin/duplicacy_linux_x64_2.7.2 benchmark -storage BackBlazeB2FilesBackup -upload-threads 4