Parallels jobs, frequency, recommended settings

Support
#1

Hi, I have configured several jobs in the web edition:

  • backup to local storage, hourly
  • copy to wasabi storage, every 4 hours
  • check local storage, daily
  • check wasabi storage, daily
  • prune local storage, daily
  • prune wasabi storage, daily

My questions:

  1. can I configure all these jobs to run in parallel or do I risk corrupting backups etc by doing so? if yes, can any of them run in parallel without causing problems or only some specific jobs?
  2. what is the recommended frequency for each job type considering a local storage and a mirrored cloud storage?
  3. are there any recommended flags/settings for each job type? I am already using -vss (I’m on Mac and it seems this is supported) and -threads 1 because my understanding is that it’s better to use a single thread when using spinning disks for backups. What else?

Thanks a lot in advance!

#2

I would run backup and copy sequentially, just to minimize amount of time data only exists in one place.

You can run checks concurrently, because they work on different storages
You can also run prunes concurrently, for the same reason.

I would not run checks and prunes concurrently though since prune may confuse checks but yanking the carper from under the snapshots so to speak.

Recommended frequency is up to you. I run backup every 15 minutes, but I’m paranoid. Once an hour seems to be a commonly used frequency. I’m also using CPU throttler not to let Duplicacy blast fans on my laptop every 15 minutes; since it’s using VSS and snapshots I don’t care how long does the backup take. (Scan takes 40 seconds on my almost terabyte home folder if allowed to use full CPU, so I throttle it down to 10% and I don’t mind that it slowly chomps on it for 7 minutes instead – with snapshots enabled that does not matter)

For wasabi I set 10 threads, because why not? :slight_smile: You can experiment to see what gets you the best performance, but with the backup tool, unless it is an initial backup, performance does not matter much – I’d argue that backing up slower and not interfering with other tasks on the machine is more important.

For rotational media you are right, single threaded will get you the best performance.

#3

Copy to wasabi should be run after local prune. In this way you may copy less files!

#4

Thanks both :slight_smile:

@saspus I like the idea of a continuous stream to both the local storage and the remote storage (via copy), but the problem is that for example yesterday I added ~100 GB to the repository and the copy job is still uploading that new data. In the meantime the backup to the local storage was still running at regular intervals - it seems to me that backups can run on a storage while copying from it to another storage, or should I expect problems by doing this? This way at least it keeps updating the local backup while the copy job is taking a log time to upload many changes. If I had backup and copy running sequentially I wouldn’t have a backup at all of the work I do while the copy job is still running, rifght? So which way is better, have both copies updated at more or less the same time but backups may be delayed, or a constantly updated local backup but the second, remote copy may be delayed?

I will change the schedules so that checks and prunes to the two storages can run concurrently. I’ll also merge checks and prunes in single jobs so that they do not run at the same time if this can cause issues as you suggest.

Also, one thing I am still confused about is, can prunes and checks run on a storage (with default settings) while backing up to it? Or is it absolutely required that no backups or other operations run while prunes are running? If the latter then it’s a bit difficult to rely on schedules because I can’t know if a backup is still running when a prune job starts…

As for the multiple threads, wouldn’t more than 1 thread still cause problems/slow things down also when reading from a backup on a spinning disk as a source?

@TheBestPessimist Good point, I hadn’t thought about that.

Thanks both again! :slight_smile:

#5

Multiple threads are used for uploading stuff. Just 1 thread is used when reading from the disk.

#6

Ah ah! Thanks, I will update the job. What about the checks/prunes? Can they run while a backup or copy is running as well?

#7

I think check is fine.
About prune: (ref: Prune command details) i think you’re also fine as long as you’re not using -exclusive and/or -exhaustive

#8

Wonderful! Thanks a lot for your help! :slight_smile:

#9

How do you make sure that the jobs don’t clash?
I have a similar setup
30 mins: local backup
4 hours: copy to b2
daily: check local, check b2, prune local, prune b2

But there comes a a time when the 4 hours jobs clash with the 30 mins and it makes either of them fail, and also the daily jobs clash with the other two and since we are backing up, copying, checking, pruning- it clashes and the job fails