Feature Suggestion: Limit CPU usage

Feature
,
#1

I think if would be helpful to be able to limit CPU usage during backup, at least in GUI version.

The problem that this will solve: Eliminate bursts of high CPU usage and spread workload over longer period of time.

Affected use case: Imagine watching a movie in the evening at low volume on a laptop. Once scheduled duplicacy backup run starts the fans spin up and continue spinning for the duration of backup. (MacBook Pro 2015, very annoying)

Compromise: Backup time will increase, however with the --vss support that does not matter. In addition, duplicacy is already ridiculously fast – average backup on my home folder takes from 40 sec to 3 min depending on amount of new data so making it even 10x longer would not matter while improving usability.

Some other backup solutions support that (CrashPlan, Arq)

4 Likes
Possible to disable deduplication?
Running duplicacy as root from OS X launchd with Keychain access
Watch file system in real-time aka File System Notifications
#2

I have the same high-CPU, fan whirring issues on my macbook air. It gets very loud and hot when a backup is running.

#3

I suppose a solution to this would be lowering the upload speed. That will make everything slower hence reducing cpu usage.

#4

When backup is scheduled frequently enough, e.g. hourly, there is very little uploaded. Most of the time is spent looking for changes. That’s when the fans spin up.

#5

oooooooooooh, i see

#6

If that is the case, then it is the file indexing phase that causes the burst. I’m not sure if it is cpu intensive – more likely disk i/o intensive. The only solution I can think of is to add some intentional delay to slow it down.

1 Like
#7

Well, it is CPU utilization that heats up the cores, so it is a code that runs on a CPU, regardless of what sybsystem or driver it belongs to.

Looking at spindump during the indexing stage: out of 10 seconds this thread unsurprisingly took about 7 seconds of CPU time:

   Thread 0x1dc823           1000 samples (1-1000)     priority 31-46 (base 31)  cpu time 7.107s (24.7G cycles, 33.6G instructions, 0.74c/i)
  992  runtime.main + 519 (duplicacy + 196007) [0x402fda7]
    992  main.main + 21694 (duplicacy + 8262430) [0x47e131e]
      992  github.com/gilbertchen/cli.(*App).Run + 1617 (duplicacy + 3513441) [0x4359c61]
        992  github.com/gilbertchen/cli.Command.Run + 1905 (duplicacy + 3523649) [0x435c441]
          992  main.backupRepository + 1471 (duplicacy + 8218735) [0x47d686f]
            992  github.com/gilbertchen/duplicacy/src.(*BackupManager).Backup + 1236 (duplicacy + 7707156) [0x4759a14]
              992  github.com/gilbertchen/duplicacy/src.CreateSnapshotFromDirectory + 988 (duplicacy + 7973420) [0x479aa2c]

But it is how that time is used is what is interesting. Going further down the stack those 992 samples are split between following stacks: 30% regex matching and 70% APFS kernel code:

One

            566  github.com/gilbertchen/duplicacy/src.ListEntries + 2665 (duplicacy + 7844649) [0x477b329]
              416  github.com/gilbertchen/duplicacy/src.(*Entry).ReadAttributes + 201 (duplicacy + 8139065) [0x47c3139]
                376  github.com/gilbertchen/xattr.Listxattr + 77 (duplicacy + 7633837) [0x4747bad]
                  372  syscall.Syscall6 + 48 (duplicacy + 729408) [0x40b2140]
                    [ truncated calls into kernel and APFS driver ]

Two

            226  github.com/gilbertchen/duplicacy/src.ListEntries + 4279 (duplicacy + 7846263) [0x477b977]
              208  github.com/gilbertchen/duplicacy/src.MatchPath + 746 (duplicacy + 8132506) [0x47c179a]
                208  regexp.(*Regexp).MatchString + 85 (duplicacy + 1250789) [0x41315e5]
                  208  regexp.(*Regexp).doMatch + 177 (duplicacy + 1237281) [0x412e121]
                    [ truncated user-mode stack all the way ]

Three

            195  github.com/gilbertchen/duplicacy/src.ListEntries + 462 (duplicacy + 7842446) [0x477aa8e]
              190  io/ioutil.ReadDir + 124 (duplicacy + 1757084) [0x41acf9c]
                190  os.(*File).Readdir + 62 (duplicacy + 831966) [0x40cb1de]
                  155  os.(*File).readdir + 446 (duplicacy + 832734) [0x40cb4de]
                    155  os.Lstat + 77 (duplicacy + 864125) [0x40d2f7d]
                      155  os.lstatNolog + 92 (duplicacy + 865740) [0x40d35cc]
                        155  syscall.Syscall + 54 (duplicacy + 729302) [0x40b20d6]
                          [ another call into APFS, also truncated ]

So yes, I guess if we could insert throttling in the loop in ListEntries it would help tremendously improve user experience: there is absolutely no hurry to burn CPU to backup ASAP, especially with filesystem snapshot support already in place. (If anything, even Time Machine takes ages to backup – I think it is deliberate for the similar reasons.)

2 Likes
#8

Sounds like an easy, global option to add: go - how to change process (application) priority from Normal to Low Programmatically in Golang - Stack Overflow

I think this would be worthwhile in general. But I realize that if your laptop is not busy doing other tasks, lowering the priority could still consume a lot of CPU (and kick-in the cooling fans).

#9

Process or thread priority only affects OS scheduler behavior; it does not throttle CPU for anyone.

If there were other processes in the system competing for CPU lowering Duplicacy priority would result in those processes being scheduled more often/for longer, making them progress faster, but it would not affect total CPU utilization in any way. That should be the default for backup tools in general. For the record, I run my copy of Duplicacy under nice For exact same reason - I don’t want it to interfere with anything else I do on the machine.

If there are no other processes - and some process wants to do work - well, it’s the one that gets scheduled. There is nobody else to compete with so it gets scheduled every time.

So yes, while it is useful to lower priority for Duplicacy,

  1. It does nothing to addres the issue on hand
  2. I don’t think Duplicacy should manage priorities itself in the first place. Every task scheduler supports setting priority, some go as far as not only managing CPU time but also IO (see launchd) and every decent OS has tools to run these things interactively outside of schedulers (see nice). Duplicacy need to focus on backing up, it just needs to learn to pace itself :slight_smile:
1 Like
#10

I agree, backup tools should all default to run at “idle priority”, though there may be times when you would want to adjust it.

I would more think that forcing a backup utility to “slow down” is contrary to the best usage/intentions. The idea is to reduce the risk of data loss, and that partially equates to getting files backed up sooner rather than later.

It sounds like you are a pretty savvy PC user. Wouldn’t it be better to only schedule backups when you are not watching a video, or when the laptop is plugged in, etc.? You are probably increasing your risk a little, but it should easily produce the results you are after, no?

#11

This would be just relocating the problem to the user’s shoulders: forcing the user to still “throttle” duplicacy, but instead of it being on a OS scheduler level and automatically this would now be on a global level: effectively pausing it when user wants silence and unpausing it afterwards. This will be forgotten 50% of the time, as any menial manual task.

This must be automatic set it and forget it type of things. Especially when maintenance tasks such as backup are concerned.

Very good example is Time Machine. I don’t know (in a sense that I don’t feel any impact of it on my workflow) wen it runs, but everytime I check on it - last backup was completed within past hour. That’s the behavior I’m looking for.

I would more think that forcing a backup utility to “slow down” is contrary to the best usage/intentions. The idea is to reduce the risk of data loss, and that partially equates to getting files backed up sooner rather than later.

I sort of disagree. Backup is usually scheduled every hour. or every four hours.

Duplicacy however completes backup pass in about 40 seconds (thumbs up for that!) burning 100% CPU on the way.

If it instead took 50 minutes - backup frequency will remain the same (so possibility of data loss does not change), it will still be every hour, however this will avoid burst CPU workload: it would not heat up, fans won’t spin, and this will also result in additional power savings (hot CPU burns more power at the same load (that’s how semiconductors work) – so even though it is accomplishing the same task over longer period slowing it down will result in power savings.)

#12

Well, we’re still talking about YOUR use-case. Backups do not have to be every 4 hours or every hour. If that’s how you prefer it, it’s fine, and your example fits.

You’re also referring to some specific hardware. I have laptops that do not have ANY fans. I have PCs and servers that could use 1-2 cores and not materially increase fan usage. (I’m also not sure how many additional watt-hours you’d use over the course of the year for “power savings”, I would be surprised if it were material…it would seem academic.)

I will respectfully bow-out of your thread, as you have a Feature Request for the devs. I didn’t mean to troll or be negative; I was trying to offer an alternative that turned-out to not be acceptable to you. Best of luck to you.

#13

On the contrary, thank you for contributing to the discussion. It’s always good to have diverse opinions and experiences.

Still, I’ll clarify:

Backups do not have to be every 4 hours or every hour.

Not really. In the GUI version the hourly backup is the default (as it should be). And most users will leave it at that. So this use case must be optimized the most. Power users that run it on servers can and should configure things per their requirements, but the out-of-the box configuration shall be polished.

Well, we’re still talking about YOUR use-case.
I have laptops that do not have ANY fans.
I’m also not sure how many additional watt-hours you’d use over the course of the year for “power savings”, I would be surprised if it were material…it would seem academic.)

Yes, this is about a specific use case, but it is not unique to me in any way: I feel there are plenty of people using laptops with batteries who totally don’t care about cost of power per kW, but absolutely treasure the battery life.

Also consider this: when the battery level is low the user could have worked on the remaining few percents for a while (editing text or watching a video consumes very little power) however 100% CPU hog kicking in for no reason will force almost immediate power alarm and sleep (as useful capacity depends on a current). Not ideal, to say the least. And it is a UX disaster in my view.

I’m more than most paranoid about avoiding feature creep, but I strongly feel that [controllable] throttling is paramount to ensure good user experience. (And user experience is arguably above everything else on the importance scale).

P.S. Actually for myself I’ve concocted lldb script that injects sleep after each call to “regexp.(*Regexp).doMatch” and few other places discussed above and I launch duplicacy that way (did not want to modify the source and then to have to maintain my changes) – while this solves the issue for me the solution is obnoxious and users should not need to jump through these hoops.

That’s all my [strong] opinion of course based on my personal experience and I do not deny existence of other views - hence I"m expressing it on a public forum and not in an email to duplicacy staff :slight_smile:

2 Likes
#14

All good then.

I definitely second a FR for an option “Do not run on battery” (and this does not completely fix your problem, but could help some use-cases).

As I said, best of luck to you! :slight_smile:

#15

Small update: Just stumbled upon wonderful little utility GitHub - opsengine/cpulimit: CPU usage limiter for Linux that works by monitoring the process (group) CPU usage and sending SIGSTOP followed by SIGCONT in rapid succession to pause/unpause the process thereby limiting CPU time.

I’m not sure how safe/reliable/appropriate it would be to expect the utility to do that to the software that actively touches network – but I’ll try to stress-test it for a few weeks. It would be still safer to throttle in the app itself, at the controlled barriers, (i.e. I don’t think it would be a good idea to pause the process in the middle of network transaction") but if this works - it may be viable solution, without need to change Duplicacy, which is awesome.

Running duplicacy as root from OS X launchd with Keychain access
#16

You’d think that the amount of resources the utility would need to monitor a second process and then issue the signals to the OS would be almost as much as just letting the original process run!

Interesting find though.

#17

It takes about 2.8-3% of the CPU on my machine (iMacPro 3.2GHz). Fairly acceptable.
Also my concerns about possible network implications are unfounded – the delays introduced are insignificant.

I’ll start using it immediately, and will report if anything goes wrong.

Controlling and limiting CPU usage
#18

So what is the state of this feature request? If I understand things correctly there is the cpulimit tool as a workaround for Linux. But nothing for Windows or OSX. In any case, we agree that duplicacy should discipline itself, right? So is @gchen planning anything like that?

High CPU usage till backup gets killed
#19

Ping.

The CPULimit is not an option with DuplicacyWeb… we do need CPU throttling for the scanning phase please: it runs full speed saturating single core 100% since the SSD is not a bottleneck since forever.

3 Likes
#20

Ping.

Apparently forum does not allow to just write one word as a comment, so I had to write this sentence, even though all I meant to do was to bring back this discussion and remind about this…

Fans flaring up periodically is annoying to the point of me considering to move back to using command line utility with launchd under throttler