SFTP performance tuning

Support
,
#1

I am using duplicacy 2.1.0 on my qnap with 2 arm cores.
My upload speed is 3,1MB/s (25Mbit/s)
I am running a VPS on which I want to upload my files using duplicacy.
The upload only has 1,5MB/s in average, exactly half the speed I could actually use. It never even reaches 3,1MB/s as peak.
What it did to find the problem:
running iperf between server and client -> full speed 3,1MB/s possible
uploading sftp files from my windows client to my VPS -> with 5 parallel streams I get 3,1MB/s
using duplicacy between my 2 local NASes gave me 6MB/s

So my conclusion is that no throtteling is active (if so my windows pc wouldn’t be able to gain the upload speed) and also every computer in the setup can reach this speeds.
What I tried with duplicacy:
increasing chunk size from 4M to 8MB -> 1,5MB/s
increasing threads to 5 -> 1,5MB/s

That I discoreved that the Ram usage of duplicati is very high:

performance
performance1276×665 129 KB

top

top
top733×262 9.36 KB

Any ideas how to increase the upload speed?
I am using encryption, is there any way to lower the compression level?

edit:
I could lower the ram usage by using chunk size 1M and DUPLICACY_ATTRIBUTE_THRESHOLD=1

Nevertheless the speed with 4.5 and 6 threads was as low as before.

SFTP backup speed is slow and slows more over backup time
#2

If you lower the RAM usage and still get the same upload speed, then the RAM isn’t the bottoleneck.

Did you mean when you ran Duplicacy on the same qnap and backed up to a local NAS the upload speed was 6MB/s?

#3

yes (sorry for the late anwser)

#4 
./duplicacy_linux_arm_2.1.1 benchmark
Storage set to sftp://u179509@u167409.your-storagebox.de:23/xxx/xxx
Generating 244.14M byte random data in memory
Writing random data to local disk
Wrote 244.14M bytes in 2.30s: 106.07M/s
Reading the random data from local disk
Read 244.14M bytes in 1.90s: 128.66M/s
Split 244.14M bytes into 50 chunks without compression/encryption in 19.79s: 12.34M/s
Split 244.14M bytes into 50 chunks with compression but without encryption in 23.11s: 10.57M/s
Split 244.14M bytes into 50 chunks with compression and encryption in 44.24s: 5.52M/s
Generating 64 chunks
Uploaded 256.00M bytes in 248.00s: 1.03M/s
Downloaded 256.00M bytes in 627.75s: 418K/s
Deleted 64 temporary files from the storage

Same as I had with my upload 1.03M/s for the upload

#5

For webdav:

duplicacy benchmark
Storage set to webdav://u1xxx9@uxx.your-storagebox.de:/xxx
Generating 244.14M byte random data in memory
Writing random data to local disk
Wrote 244.14M bytes in 2.13s: 114.88M/s
Reading the random data from local disk
Read 244.14M bytes in 1.05s: 232.34M/s
Split 244.14M bytes into 50 chunks without compression/encryption in 19.23s: 12.70M/s
Split 244.14M bytes into 50 chunks with compression but without encryption in 22.65s: 10.78M/s
Split 244.14M bytes into 50 chunks with compression and encryption in 43.34s: 5.63M/s
Generating 64 chunks
Uploaded 256.00M bytes in 114.91s: 2.23M/s
Downloaded 256.00M bytes in 837.02s: 313K/s
Deleted 64 temporary files from the storage

Download seems to be slow upload is fine