Rate-Limiting for Backup and Copy Commands

[dave.engelsma]

I’ve used the rate-limiting options when using the ‘backup’ and ‘copy’ commands. In either case, no rate-limiting has occurred. I verified this using the ‘iftop’ command and checking the traffic on my network interface. I am using CLI version 2.1.2 with Ubuntu Linux 18.10 x64 on systems that are basically idle (very little, if any network traffic and disk activity).

The command I used for backup:

sudo ~/duplicacy backup -limit-rate 20 (I would assume an upload rate of 20 KiloBytes/second here)

And for the copy command:

sudo duplicacy copy -upload-limit-rate 150 -from default -to offsite-wasabi

Any ideas as to why the rate limiting is not working in either command?

Thanks in advance for the help!

[dtownsend]

I see this too. I set a limit of 3000 and duplicacy consistently uploads at a rate of around 6MB/s which is overloading my internet connection.

[saspus]

While this is likely a bug (ignoring throttling requests), I wanted to make a slightly tangential comment:

You should be able to let duplicacy fully saturate your upstream without affecting any other applications on the network. If you do see that everything else dies when you fully utilize your upstream (what I think you meant by “overloading”) then you are likely experiencing bufferbloat – undesired latency spikes due to network equipment buffering too much – which effectively prevents you from fully utilizing bandwidth you are paying for!

Managing client devices and services bandwidth to attempt to “fix” this is a losing battle, because ultimately the bandwidth is not a problem, latency is.

To confirm that this is what you are actually experiencing – start pinging google.com in one window and then start multi-threaded backup with duplicacy at full speed. Watch the ping. It shall not change. If it does change (and it may increase drastically, 1000x is not unusual) you just confirmed that your issue is indeed caused by the buffer bloat.

You need network equipment that can manage the queue to prevent buffers from filling up. (that is usually achieved by SQM algorithms, such as fq_codel). There exist a number of devices, both commercial (Ubiquiti EdgeRouters and USG) and free (OpenWRT) that support that rather well.

How well? Anecdotally, I had 12/1 MBps (yes, 1 Megabit per second) upstream for a very long time, and I was uploading non-stop (of course, it used to take forever to transfer anything) with various tools (backup, sync, all other services); essentially my connection was saturated at 100% all the time while the ping never exceeded 10ms and users did not see any impact on their browsing or other activities.

What I’m trying to say is that instead of limiting your backup speed and effectively under-utilizing the connection you are paying for it would be more productive to address the root cause of the issue, which has nothing do with bandwidth utilization.

[Droolio]

I’d just like to add, that while SQM may fix this issue, I don’t think it would be accurate to describe this as bufferbloat. Bufferbloat is about latency-sensitive traffic (gaming, VoIP etc.) being affected by long queues. (I suppose it depends on if the poster above is doing any of that, but the fact is all download traffic will be affected…)

Really, at its simplest level, this is most probably caused by an asymmetric connection (similar to what you describe; with a very small upstream) being throttled by itself - the upstream saturated so much that ACK packets can’t get out in a timely fashion.

The result is the downloads slow down because the TCP/IP connection at the other end isn’t receiving those ACKs, so it purposefully slows down the connection thinking you can’t handle more packets to download, even when you’ve got plenty of downstream spare.

Personally, I haven’t got around to re-flashing my Archer C7 back to OpenWRT, might do that this week. But for the last few years I’ve been using cFosSpeed traffic shaper for Windows, which does a wonderful job of prioritising ACKs and various applications to appropriate levels. However, I highly recommend OpenWRT and getting some SQM and bufferbloat-busting action on the go. Being able to browse/game while streaming or otherwise heavily downloading on the rest of your network, is really quite nice.

[Droolio]

I see this too.

Are you using just the one thread?

[dtownsend]

I have tested with one thread or multiple threads.

[dtownsend]

Thanks, this was helpful and after configuring my router I no longer need to rate limit.