Pruning with threads 20 => Internet connection mostly locked up

Support
#1

I had to free some backup space as my B2 costs were running away too quickly. So, I manually started a prune run to delete some old revisions I no longer need.

I run with -threads 20 to get some speed on the purge run. I saw the list of revisions being deleted, and after the final one, the command ran for a couple of hours and mostly blocked my NAS.

There is no further output of what Duplicacy is doing there. And the NAS has 64GB of RAM and a fast CPU. System resource reports didn’t show any extensive load. But my internet connection was mostly unusable. Maybe the router struggles to deal with that many parallel requests.

So, I had to kill the pruning process and hope to continue/pick up when rerunning it.

Some questions:

  1. What does Duplicacy do after deleting the last revision?
  2. Is there anything I can do to speed things up without killing my internet connection (1Gb/s)?
  3. Do I have to take care of anything when I continue/repeat to delete the revisions?
#2

It’s likely bufferbloat. One devices maxing out internet bandwidth shall have minimal to no impact on other devices.

Here is test to confirm:

Here is further reading:

https://www.bufferbloat.net/projects/

To fix you want a router that supports SQM. There are some examples at the above sites.

#3

Thanks for the tip. Never heard about this. But doesn’t seem to be a problem, tested via WiFi, not even LAN cable:

image
image1920×1438 115 KB

#4

Wow, this is pretty magnificent!

Is your connection cap is 430 mbps? Or gigabit?

If your wifi is slower than what isp provides the test would not have been able to fully load the connection and therefore results are bogus: they are just indicative of your WiFi quality (modern wifi includes its own QOS).

This is good to know, but not applicable to the issue at hand :slight_smile:

Please test from the wired connection, even though such high speed internet connections rarely have bufferbloat indeed. Still, to rule this out.

You can also try Netflix speed test at fast.com:

ignore the test that auto starts, let it finish, then click Show More Info, set min and max number of threads to 30 and min and max test duration to 30sec, and save.

#5

It’s a 1Gb/s symmetric fiber connection.

I will repeat the test tomorrow using a LAN cable connected to a 10Gb/s switch, which is connected to the ISP router (1Gb/s, I’m waiting for 10Gb/s).

1 Like
#6

Answering this: when duplicacy prints a log message “ deleting chunk” or something along those lines it does not actually delete the chunk that instant. Instead, it simply add them to the list and actually deletes everything in the end, while being stoically silent. Rather confusing behavior that sent me to the wrong rabbit hole in the past.

Since it does it one by one, there are a lot of B2 calls being made, and it’s possible that it’s your router indeed that can’t support such a concurrency. Isp issued routers are not exactly famous at being overspecced…

As a workaround you can reduce number of thread — there really is no hurry pruning; duplicacy is fully concurrent, if pruning takes five days, so be it

#7

I could see the B2 transactions going up from around 2 million to 4 million. So, yes, there seem to be many requests done.

I think it’s an excellent time to get the newest ISP router, which is two generations ahead. And then start with a burn-in test :slightly_smiling_face:

I have a regular backup & pruning job running every night. But this is following a timed pattern.

I need to update that pattern so that not so many old bits stack up in the cloud. Looks like I keep too many old backups, which I never need anyway. 7 days and 1 per week for 4 weeks should be enough.