Upload speed decreasing until Failed

Support
#1

Dear,

Thank you for the great software. I really look forward in using it for many years to come.

Setup:
I have a Synology NAS of type DS218play with about 3.5T of data. I have 6 different volumes on it, of which I want to backup to Backblaze 4 of them. I am backing them up one by one on different directories on the same storage.

Problem:
My upload speed starts being reasonable 15MB/s, and then steadily goes down until some 100s of KB/s until finally and always failing. Restarting the backup through the web UI results again in 15MB/s (10 threads) to then again slow down until failure.
NAS CPU and memory usage looks fine, but when upload speed slows down, also my NAS becomes more and more unresponsive. If I kill the process through command line on the server, the NAS becomes responsive again.

Benchmark:
Interesting benchmark result is:
Split 256.00M bytes into 52 chunks with compression but without encryption in 78.64s: 3.26M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 61.47s: 4.16M/s
Generating 64 chunks
Uploaded 256.00M bytes in 16.79s: 15.25M/s
Downloaded 256.00M bytes in 6.37s: 40.22M/s
Deleted 64 temporary files from the storage

I have been investing possible cause for weeks now, but I still haven’t find it. Can anyone please help me point in the right direction to get this solved. I can live with a 15MB/s upload speed if it does not constantly slows and fails.

Thank you!
Maarten

#2

What is the reason for the failure? What’s in the backup logs?

How does memory utilization looks right before failure? How does disk utilization look as the NAS becomes more and more unresponsive?

How much ram in total is on device? How many files are in the backup set?

I suspect you have a lot of small files, and as backup progresses, more and more file metadata is displacing disk caches, slowing evening down or potentially running out of memory, resulting in OS killing the duplicacy.

#3

Hi! Thank you so much for your brief response!

Memory is around 50-60% the whole time, no significant increase, of 1GB in total physical memory.
CPU constant at 20%.

Your reasoning makes a lot of sense. There are 460K files in the volume that I want to back up. And inspecting it I saw that 200K of them are of the #recycle folder, that I am now excluding from the backup. I will fine tune that using ncdu tool as I continue testing.

I will update here with further info.

Happy Thanksgiving,
Maarten

#4

That’s very little. NAS performance is mostly affected by disk latency; Linux tries to keep metadata in ram whenever possible to avoid drives needing to do extra seeking. That cache resides in unused ram. Anything that uses ram takes it away from the disk cache and decimates performance. You might want to consider running apps, including duplicacy, on other hosts and use NAS just as a file server. This will improve performance.

That’s not many at all.

Have you checked the backup log, to find the reason for the failure, and disk utilization in the resource monitor when things slow down?

#5

Thanks for your feedback!

Memory usage stays stable at 50%. There is also a 2GB swap space, but I guess that does not help.
Disk usage oscillates around the 15MB/s read speed, but as process slows down the GUI of the NAS gets stuck, so I would have to find a way to log.

Error message is:

2023-11-24 11:32:09.410 INFO INCOMPLETE_SAVE Incomplete snapshot saved to /volume1/duplicacy/repositories/localhost/0/.duplicacy/cache/JoNAS/incomplete_snapshot
Failed to upload the chunk 9b597b783d24f0c9d995d21342eb72512971549ef68e8f41f5f4f7ded62aad96: Post "https://pod-031-2023-17.backblaze.com/b2api/v1/b2_upload_file/54d0f7e930cdc8678ab60d15/c003_v0312023_t0036": write tcp 192.168.0.129:58326->45.11.38.78:443: use of closed network connection

So why is memory a limiting factor as it is only used up to around 50%? I thought there was enough margin, and thus not being the problem, however the NAS clearly slows down.

#6

Does Synology show IOPS (Input/output operations per second) as opposed to megabytes per second? The latter is useless: reading 15MB per second in one file is effortless, but reading 15MB worth of 1kb files will halt the disk to the grind. It will likely only be able to read at 0.1MBps, because most time will be spent seeking to the file beginning as opposed to transferring data. Mechanical hard drives can only support about 200 IOPS. That’s why having unused ram is so important — after few file access filesystem metadata (information about file location) will be cached in ram, and this almost halves the amount of IO requests hitting the disk, thus improving responsiveness. Disk cache does not “use” ram, as in the amount of space it takes does not count when determining amount of free ram for applications; if applications need more memory — disk caches will get evicted.

I think that’s what’s happening — you have few folders with massive number of small files, Duplicacy tries to read them, this generates a lot of IO to the disks, 50% of free ram means you only have 512MB of cache available and it’s really nothing. The problem is exacerbated by duplicacy needing to store information about files it found in ram, that further reduces available memory for the disk caches

If you look at resource monitor (or whatever it is called now) there is a graph called “volume utilization”), in percents. Approaching 100% means disks is completely overwhelmed and can’t catch up.

From my past experience with Synology — to get any decent system performance as a file server you want about 8GB of ram. If you also want to run applications on the NAS — then more. Performance kept improving until about 24GB. Going above 32GB was bringing diminishing returns.

On your NAS memory is not upgradable, so that’s a no go. The remaining approach is to run as many apps as possible on other hosts, and let NAS just serve files. 1GB is still way too little — but on such a resource constrained system every little bit of extra freed ram will have measurable effect on performance.

This looks like this: use of closed network connection · Issue #37 · kothar/go-backblaze · GitHub but maybe it’s a secondary issue, a result of performance grinding to a halt and stuff starting timing out.

#7

Hello saspus,

First of all I want to thank you for such detailed explanation. I learned so much.

I followed your suggestion to use a host for duplicacy that is not my 1GB memory NAS.

Using my Mac connected close and connected through the same router as my NAS, I get high upload speeds around 35MB/s (4 threads).

However, I need my Mac to move around for work, so I cannot rely on that. I have a Linode server in London, that I could use, however setting up duplicacy on that server I get the following speeds:

NAS mounted via NFS => 1,3 - 1,4 MB/s
NAS mounted via SSHFS => 1,2 - 1,5 MB/s

As a benchmark I scp’d a large video file and that gave me around a 5,7 MB/s upload speed (I both tried uploading and downloading).

What is the bottleneck of a setup using Linode in this case? Because the difference between 35MB/s and 1-2MB/s is very big. Is it latency, protocol encrypting?

Thanks for orienting so I can further finetune my setup.

Best regards and thanks again!
Maarten

#8

Hi Maarten,

No problem, any time!

Yes, it’s likely latency. Every file duplicacy reads involved back and forth over the ssh or nfs over the internet (I assume NFS was over some VPN connection, right?)

Maybe backing up whenever your Mac is around will be frequent enough?

Or you can buy some raspberry pi with 8GB of ram and run duplicacy there, backing up over NFS. Or upgrade NAS :slight_smile:

Network failure causing re-indexing which takes hours