Extremely slow restore operation from Azure

How-to
#1

I setup store my storage in Microsoft Azure (Performance/Access Tier = Standard/Hot). I then copied a backup from local storage to Azure and I was able to get ~ 17MB/sec upload speed. Just to validate if my data is intact, I am trying to restore and am getting only 1.6MB/sec resulting in estimated download complete time to be ~4.5 days

When I ran the benchmarking - I am getting much better results - what could be the problem here?

> Generating 256.00M byte random data in memory
> Writing random data to local disk
> Wrote 256.00M bytes in 0.54s: 477.85M/s
> Reading the random data from local disk
> Read 256.00M bytes in 0.08s: 3368.68M/s
> Split 256.00M bytes into 52 chunks without compression/encryption in 1.58s: 162.27M/s
> Split 256.00M bytes into 52 chunks with compression but without encryption in 1.94s: 132.20M/s
> Split 256.00M bytes into 52 chunks with compression and encryption in 2.05s: 124.99M/s
> Generating 64 chunks
> Uploaded 256.00M bytes in 70.47s: 3.63M/s
> Downloaded 256.00M bytes in 63.31s: 4.04M/s
> Deleted 64 temporary files from the storage

#2

Are both commands (backup and restore) using the same number of threads?

#3

I didn’t set any threads option explicitly, so unless the commands have different defaults, they should be using same thread count.

I just restarted restore operation with “-threads 10” and the download speed seems to have gone up slightly to 3.5 MB/s - but still way slower than what I was expecting :frowning:

#4

I was going to point you to the topic below but I just noticed that you already found it…

#5

Yeah - increasing the # of threads wasn’t too successful and resulted in timeout errors.

Now I am back to 1 thread and download speed went back to ~1.7 MBps - which is very disheartening :frowning:

#6

I’m not a Duplicacy customer but came across this thread when searching google for “slow azure restore”. We’re using Azure File Sync and have an azure file share (standard/hot) and two on prem full replicas (no cloud tiering). Agents on each of the file servers maintain synchronisation. We have a file share that’s about 1.2TiB and contains around 1.5 million files. We use azure backup to backup the cloud share (it’s basically just scheduled snapshots in the same storage group as the cloud file share).

I want to document the process for recovering from a potential worst case scenario ransomware attack in which we need to restore the entire share. There appear to be two options 1) restore to original location 2) restore to new location. The downside towards restoring to original location is that it retains all encrypted files so you’d need to go on a mammoth cleanup operation post restore. If you restore to a new location you get a clean copy of the file share but you then have to build a new file server or mount a new volume to copy back the data… Anyway,both options are horribly slow and i can rule out anything on prem or transport related. The first stage in restoring to a new location is copying the azure snapshop to a new Azure storage account so this is all happening withing the azure cloud (same region East US 2). I kicked off that restore this morning and 6 hours later it’s still running and only a 1/3 way through. Assuming it completes in 18 hours i then have to copy this share back on prem and i’ve seen speed similar to what you are when doing previous testing so it could take days. Admittedly, Azure FIle Sync can copy the name space to the on prem file servers which gives users the impression the data is on prem when in fact it’s just a pointer to the Azure file share location but even building the namespace takes hours and accessing data in azure is very slow compared to on prem.

Either i’m missing something or Azure isn’t going to come close to meeting our recover time objective.