Trying Duplicacy for first time, have some questions

Support
#1

Hi!

Previously, I always used Arq. I want to try out Duplicacy now to see if I maybe prefer it over Arq.

I have setup Duplicacy, using the Web GUI, and I like the overall setup. There are a bunch of questions I don’t find any answers to though, I wish there would be a bit more documentation.

  1. The Quick Start Guide mentions “Starting from version 1.1.0, Duplicacy Web Edition can be installed as a Windows service”. Should I do that, or should I not do that? What is the difference, what is the advantage, what is the disadvantage of installing it as a service?

  2. The default seems to be to not use VSS on Windows. I know Arq by default uses VSS. Why does Duplicacy not use VSS by default? Is there a reason why I should keep VSS disabled with Duplicacy? Is it not stable to use with Duplicacy?

  3. I have configured a backup to Google Cloud with Duplicacy. I selected my whole C drive, which is an NVMe SSD. I run the backup with the -threads 20 option as part of an hourly schedule. That leads to Duplicacy uploading with a constant 25 MBit’s. I have tried different values for the -threads option, but it never goes above 25 MBit’s. What is limiting it? Duplicacy only uses very little CPU, RAM and Disk activity.

I have then tried backing up a different disk simultaneously, that also uploads with 25 MBit’s, so then I get a total of 50 MBit’s, so the cloud storage and my internet can definitely do 50 MBit’s, but why can the individual backup task not?

  1. Then I tried something where I expected it would probably not even allow me to configure it that way, but it did: I added a second schedule, with all the same settings as the first schedule, that again also works on backing up my C backup. So I have two different schedules that back up the same “Source” to the same “Destination”. And indeed, now when I run those two schedules simultaneously, I can nicely backup my C drive with a total of 50 MBit’s.

Is that how I am supposed to do that? Why do I need two schedules for that, just to be able to reach my full upload speed? It seems kinda weird to do it that way. I am surprised that it works to backup the same disk simultaneously two times. Can that lead to any issues, or is this how you’re supposed to do it to get the maximum speed? Can it happen that those two tasks end up basically uploading the exact same files twice, as they’re working on them simultaneously?

  1. Does Duplicacy automatically exclude things like the pagefile from the upload, or do I have to manually configure that?
#2

Most people use the service because the service can run backups without a user login.

If files to be backed up won’t be locked by other processes then you don’t need VSS.

20 threads may be too many. Try a different number like 4 or 8 to see if it helps.

It is not recommended to run the same backup from different schedules. This doesn’t speed up the backups. At the end only one revision will be created and the other will be overwritten.

No, you’ll have to manually configure that.

1 Like
#3

Thanks for the reply!

I don’t quite understand what “without a user login” means? So the service doesn’t have any disadvantages over not using the service? How comes the service is not the default then?

I do have some files that are locked by other processes sometimes, so I would like to use VSS, and that’s why I’m wondering if there’s any problems with it? If there are no downsides to using it, it would probably be the default setting, right?

How does Duplicacy deal with locked files when VSS is not used? Does it error on those, or does it skip those? Can it happen that Duplicycy locks a file, and the software that normally accesses that file then can not access or modify it?

I have tried many values between 1 and 20, and I was never able to get above 25 MBit’s. All values between 1 and 20 leads to basically the same upload speed, I don’t feel any difference. To me it almost feels like there’s a hard cap of 25 MBit’s for an individual backup task, which I understand does not exist, but that’s how it feels.

Ah, hm… too bad. I was happy to have found a (weird) but easy solution to make Duplicacy use my whole bandwidth.

Ok, I’d recommend to by default exclude such files. Arq is doing that, and I think that makes sense, as many users probably don’t know about what a pagefile is and that it would make sense to exclude it… Would make Duplicacy a bit more user friendly to non-tech-savy people.

2 Likes
#4

@gchen I still have that issue. An individual task is always capped at 25 MBit’s upload, no matter if I use 4 or 8 or 10 or 15 or 20 threads, and no matter if its an SSD or an HDD. Only if I have 2 tasks upload in parallel, I get my full 50 MBit’s.

Is there anything else I can set to get 1 task to upload with the full 50 MBit’s speed?

#5

You can try to find the bottleneck with the benchmark command. To run a benchmark with the web GUI, enter these commands in a terminal:

cd ~/.duplicacy-web/repositories/localhost/all
../../../bin/duplicacy_osx_x64_2.7.2 benchmark -storage <storage name>
1 Like
#6

Ok, thanks! I have tried that now, with a lot of different values. The benchmark is same capped at those ~25 MBit’s, both in Upload and in Download. Different values for threads and chunk size or count do not make much difference:

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.24s: 1071.46M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4340.35M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.29s: 199.12M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 1.77s: 144.23M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 1.80s: 141.91M/s
Generating 64 chunks
Uploaded 256.00M bytes in 130.79s: 1.96M/s
Downloaded 256.00M bytes in 102.04s: 2.51M/s
Deleted 64 temporary files from the storage

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup -upload-threads 15 -download-threads 15
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.23s: 1094.29M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4130.28M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.33s: 192.21M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 1.76s: 145.83M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 1.83s: 139.93M/s
Generating 64 chunks
Uploaded 256.00M bytes in 118.36s: 2.16M/s
Downloaded 256.00M bytes in 91.98s: 2.78M/s
Deleted 64 temporary files from the storage

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup -upload-threads 15 -download-threads 15 -chunk-size 8
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.24s: 1045.23M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4267.08M/s
Split 256.00M bytes into 28 chunks without compression/encryption in 1.33s: 192.17M/s
Split 256.00M bytes into 28 chunks with compression but without encryption in 1.79s: 142.94M/s
Split 256.00M bytes into 28 chunks with compression and encryption in 1.88s: 136.01M/s
Generating 64 chunks
Uploaded 512.00M bytes in 252.52s: 2.03M/s
Downloaded 512.00M bytes in 180.67s: 2.83M/s
Deleted 64 temporary files from the storage

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup -upload-threads 15 -download-threads 15 -chunk-size 4 -chunk-count 128
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.49s: 524.76M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4198.06M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.45s: 176.85M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 1.86s: 137.63M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 1.95s: 131.56M/s
Deleting 128 temporary files from previous benchmark runs
Generating 128 chunks
Uploaded 512.00M bytes in 195.69s: 2.62M/s
Downloaded 512.00M bytes in 171.16s: 2.99M/s
Deleted 128 temporary files from the storage

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup -upload-threads 40 -download-threads 40 -chunk-size 4 -chunk-count 128
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.42s: 606.03M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4064.83M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.33s: 192.46M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 1.76s: 145.44M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 1.82s: 140.86M/s
Generating 128 chunks
Uploaded 512.00M bytes in 191.96s: 2.67M/s
Downloaded 512.00M bytes in 171.90s: 2.98M/s
Deleted 128 temporary files from the storage

C:\ProgramData.duplicacy-web\repositories\localhost\all>C:\ProgramData.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage Backup -upload-threads 40 -download-threads 40 -chunk-size 4 -chunk-count 256
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 0.25s: 1008.19M/s
Reading the random data from local disk
Read 256.00M bytes in 0.06s: 4198.17M/s
Split 256.00M bytes into 52 chunks without compression/encryption in 1.30s: 197.06M/s
Split 256.00M bytes into 52 chunks with compression but without encryption in 1.76s: 145.13M/s
Split 256.00M bytes into 52 chunks with compression and encryption in 1.83s: 139.80M/s
Generating 256 chunks
Uploaded 1024.00M bytes in 380.51s: 2.69M/s
Downloaded 1024.00M bytes in 344.02s: 2.98M/s
Deleted 256 temporary files from the storage

I have also tried running two benchmarks in parallel. Both still achieve the same speed as before then, so I do again get my total of 50 MBit’s if two run in parallel, both in Upload and in Download.

Is this some kind of bug in Duplicacy that it only uses half the available bandwidth per task?

#7

For comparison, you could try the benchmark command with a local backup storage - instead of Google Cloud - perhaps on a separate drive to where your repository resides (especially if it’s not an SSD). That way, you might see if the bottleneck is CPU or bandwidth related.

From what I understand, -threads is only for upload threads, but data has to be chunked sequentially, then compressed and encrypted. What CPU do you have btw?

#8

The NVMe SSD implies a relatively new motherboard and thus a modern CPU.