Azure Performance/speed

[zaheer22]

I am attempting to test this for possible business purpose. In this case, a fast restore would be needed. When I attempt to use Azure cool storage with 4 threads to upload several hundred GB, I am only getting about 4MB/sec.

Introducing Azure Cool Blob Storage | Azure blog and updates | Microsoft Azure

states that performance should be the same as the hot tier. I got the same performance using 2 threads on dropbox for business.

My network connection can hit 800mb/sec using AZcopy so I know this is not a network issue. Any suggestions here ? I am using the latest web edition for Mac. Would B2 storage be any better? or is this just the general speed of the product?

[saspus]

You can try running a benchmark on a storage:

Benchmark command details

The benchmark command is used to test the upload and download speeds for a specific storage and disk access speeds for your repositories. Click here for a list of related forum topics. Quick overview NAME: duplicacy benchmark - Run a set of benchmarks to test download and upload speeds USAGE: duplicacy benchmark [command options] OPTIONS: -file-size <size> the size of the local file to write to and read from (in MB, default to 256) -chunk-count <count> the numb…

@zaheer22

My network connection can hit 800mb/sec using AZcopy

Duplicacy is not just a copy. It is a scan/read/hash/split/pack/encrypt and then upload. It can only do the latter as fast as the former.

Depending on your source data (i.e., few huge or many small files, SSD, or HDD) disk performance is likely limiting factor. Look at resource monitor when backup is running to see whether it is CPU, memory, or disk queue bound. Often it’s the latter: reading a lot of small files from the rotational media can be very slow, we’re talking kilobytes per second, because of seek latency: the disk spends most of the time moving heads and waiting for the right sector to fly by as opposed to actually transferring data.

Worst offenders are browser caches and other transient files. If you have included them in the backup – exclude. If your dataset legitimately consists of a multitude of small files – then you’ll have to wait it out. Subsequent backups will be faster, because duplicacy would only need to read metadata. This is not specific to duplicacy btw.

@zaheer22

or is this just the general speed of the product?

Definitely not. It’s one of the fastest in the industry. (I’m a user, not related to the company, so this is not bragging, but praising)

[zaheer22]

This is fair, but these backups are all coming from RAID or NVME disk is definitely not the issue. Even with those operations I would expect better, our disk speeds are in the order of 5GB/s.

[zaheer22]

Fair fair, Using the web edition, I am unsure as to how to access the CLI on a Mac to run the benchmarks, trying to find out how to do that.

[saspus]

@zaheer22

RAID or NVME disk

Raid OF NVME disks, or do you have HDD RAID array and nvme in your local mac.

@zaheer22

our disk speeds are in the order of 5GB/s.

HDD? This is sequential read rate.

What is the disk queue length on your raid array when backup is in progress?

[saspus]

@zaheer22

the CLI on a Mac to run the benchmarks, trying to find out how to do that.

The CLI binary is located under ~/.duplicacy_web/bin/duplicacy_osx_x64_2.7.2

Open the logs of the backup, in the first few lines there would be a path to local cache: e.g. ~/.duplicacy_web/repositories/localhost/0/ or something along those lines. Path to the “repositories” you can look up in WebUI on the settings page.

CD to that directory and call duplicacy binary by full path from above.

e.g.

cd ~/.duplicacy_web/repositories/localhost/0/
~/.duplicacy_web/bin/duplicacy_osx_x64_2.7.2 benchmark [command options]