Local backup extremely slow

Support
#1

Configuration: Windows 10, GUI edition, Intel 11th gen CPU, 32GB RAM, PCIE SSD on host machine. Backing up to a 2.5" portable HDD formatted in exFAT.

I’m running a local backup of about ~700GB of files on my SSD, backing up to a USB hard drive. I started it last night with 1 thread (as per recommendation on this forum) and the speed was initially good - I checked Task Manager and I was getting about 45MB/sec writing to the local drive. I went to bed and woke up (8 hours later) and it had only copied 150GB of data and was now dragging along at a paltry 3MB/sec. I checked in Task Manager to look at disk IO speed and in some cases it was just sitting there at 0KB/sec even though Duplicacy said it was backing up.

I tried stopping/restarting the backup job and tooling around with the # of threads and it still behaves oddly, sometimes pushing a normal 40-50MB/sec and other times just sitting there at zero seemingly not doing anything. CPU usage consistently hovers around 10-15%.

This is very odd behavior not only because it’s so slow but because even Duplicacy itself has been much faster backing up to CLOUD in the past. I backed up another ~350GB data set which finished in several hours which is reasonable, and that’s over the Internet, not local.

If I copy files to the local HDD in Windows Explorer I get a consistent 70MB/sec easily.

I have seen suggestions to run benchmark from the CLI however I am using the GUI version and do not know how to install the command line.

What is going on here?

#2

This is the problem right here.

First of all, writing to the disk starts at outer track and progresses inward, so as data is written the speed will at least halve.

Next, the usb to sata controllers are notoriously horrific garbage, so I would not put past them to just fail for a multitude of reasons.

And lastly, your drive itself is likely dying; those 2.5inch disks are always a steaming dung.

I’m not sure how to best do it on windows (I’ll look it up for you in a minute) but I would start with filling the entire disk surface with zeroes as a test and means to test writability. Likely there are bad sectors that just take forever to process, causing stalls.

Everything aside — even if you use erasure coding backup on a usb drive, let alone 2.5 inch one is worthless. It’s enough for a bad sector to develop where your config file is located to render entire backup worthless. Backup you can’t rely on is counterproductive waste of time. Just don’t do that. You seem to already have cloud backup — that’s enough.

If you want local backup in addition for some reason — build or buy a nas that provides data integrity guarantees. TrueNAS or Synology with BTRFS are good options. Once you look at cost — you’ll realize it’s much cheaper to just backup to another cloud than trying to replicate worse version of it at home: with depreciating things like data storage facilities it’s always better to rent than to own.

#3

To zero out the disk on windows, you can use format utility. Open commadn prompt (Win+x,a) and use format z: /p:0

This will format the disk Z: and fill it with zeroes.

See if the process competes in the expected time or also stalls out.

#4

As I mentioned on my earlier thread, my particular backup strategy calls for an offline local copy to be maintained. This is an additional layer of protection to my cloud and NAS backups which I am already highly confident in. I have no interest in eliminating local offline backup copies from my backup strategy. I know you seem to feel very strongly about this particular matter, but I am not looking for advice on my backup strategy, I am looking for advice of how to successfully execute my backup strategy with Duplicacy.

I can safely rule out issues with the drive and controller as I have run backups on the same drive using both Macrium and Duplicati in the recent past (looking to move away from both of those suites altogether) where the copy speeds are consistently 50-75MB/sec. Direct sustained file copies (hundreds of GB) can also sustain this speed just fine. So the issue seems to be Duplicacy, or at least how I’m currently using it/setting it up.

Any advice regarding how to improve copy speeds would be greatly appreciated.

#5

To run the benchmark command, go to C:\Users\username\.duplicacy-web\repositories\localhost\all and run:

C:\Users\username\.duplicacy-web\bin\duplicacy_win_x64_2.7.2.exe benchmark -storage <storage_name>
#6

I don’t see the reference to the previous thread; it’s unreasonable to expect everyone to be aware of entire post history of a specific user.

Often the best advice on “how to do X” is “not to do X in the first place”. Also, I do continue to provide advice on how to solve your issue anyway, so I’m not sure what’s the problem here.

More importantly, I’m not against local backup; I’m against backup to media with no consistency guarantees. If you feel your NAS/Cloud backups are insufficient to the point that you want “additional layer of protection” – backup to yet another nas or cloud. Using vastly less reliable media to supplement highly reliable one makes no sense. Eventually you will be posting “duplicity cannot restore because chunk fails to unpack” or “the config file is corrupted” – what do I do? The answer would be “use reliable media next time”.

I would not. What happened in the past is irrelevant. Disks can fail any time. What you are saying is “My disk is fine today because it was fine yesterday”. By that logic, disks can never fail. This is not the case obviously.

As I said before, fill the disk with zeroes to rule out disk failure. Before you do that – doing anything else is a waste of time.

This has nothing to do with duplicacy. Average default chunk size is few MB, so even accounting for the seek latency, 3MBps is way too slow for the healthy disk to exhibit at that chunk size.

This means you are not bottlenecked by the CPU; this could have been the case of aggressive deduplication – where duplicacy would prepare chunks that happen to already exist on the disk.

This is irrelevant for the suspected failure: sectors you copy to are ok. Your duplicacy backup also started fast, did not it?

To reiterate:

  1. Ensure your disk is healthy by filling entire surface with zeroes, and then checking S.M.A.R.T data for reallocated sector count to confirm. Scrap the disk if that counter is nonzero.
  2. When the slowdown occurs – look at the size of disk IO queue, and amount of free ram on the machine – but from your description I get a feeling you have already ruled that out. If not – that would be the next step.
  3. I would also use NTFS as a filesystem instead of exFAT; this would provide marginal performance improvement, but not to degree of a degradation you are seeing.
#7

Thanks for the advice. For #1, would running an extended SMART test achieve basically the same thing? I ran a quick SMART which shows no bad sectors, for what it’s worth.

Great pointer about NTFS - I read the same after starting this thread and now am starting to think that exFAT may be the culprit. I wiped the drive, did NTFS format and started a fresh backup in Duplicacy and it’s already written 250GB at seemingly normal speeds. I’ve read that exFAT struggles with large numbers of small files which is exactly how Duplicacy works. I’m holding my breath until the whole backup finishes, but it seems like that may have been the issue.

#8

Not really. Extended SMART test simply reads from the entire disk surface. It does not reallocate bad sectors (unless read fails and then subsequent retry succeeds) and does not test writes, (which is what seemingly fails for you). While not likely, but it’s possible for reads to succeed but writes to fail.

Writes on the other hand will reallocate failed sectors and update corresponding SMART parameters.

Short SMART test selectively reads a few sectors. It only proves that the drive is capable from reading from a few sectors, which is a very low bar to clear.

It would be bottleneck-ed by the CPU though, if the file count in the directory was an issue.

Another possibility however is that the actual FAT happens to be on a bad-ish or failing sectors and updating (rewriting) it is what stalling. NTFS does not have FAT in a fixed location, it uses so-called MFT, which can be anywhere (pointed to by a special read-only record), there is no fixed location. This may explain why you see the difference.

It’s all guesswork, of course, but if you see such a drastic performance difference between filesystems, I would definitely try to “break-in” the disk before trusting it with any data.

BTW, is the disk SMR by a chance? Then seeing initially good throughput followed by tragic sustained write speed is not unexpected

#9

Providing an update on this topic - the short answer is that the exFAT format seems to have caused the issues and they have been resolved by using NTFS.

I wiped the disk, wrote zeroes to the whole disk, ran an extended SMART test and checked SMART stats and chkdsk - all of which came back with perfect results.

With the disk now in NTFS format, I ran a local backup job (internal SSD > external USB drive) which copied at roughly ~50MB/sec or so - around the expected range.

The mystery eludes me as to why exFAT would make such a dramatic and negative impact to Duplicacy’s behavior - with exFAT it was almost as if the program was sitting and doing nothing for extended periods of time and then working in short bursts. With NTFS it sustains writing activity at more or less a consistent speed. Again, I will reiterate that exFAT did not pose similar issues with two other backup suites (Macrium and Duplicati) and exFAT exhibits normal speeds with direct file copying via Windows Explorer, so this seems to be a Duplicacy issue… at least in my experience. I am unsure if others are able to directly reproduce the same issue.

In any case - I am happy now because my backups are working, I’ve thoroughly tested my disk (and test restores of my backups!) to rule out a bad disk or controller, and I’m going to keep going with using NTFS for this drive.

Thanks for everyone’s help with this

1 Like
#10

This is indeed so bizarre.

I have two ideas though:

  • writing zeros pass healed the disk by reallocating bad sectors. This is precisely how I recently revived an old SSD that threw all sorts of smart errors after sitting dormant for 2 years on a shelf. One pass with zeroes — a bunch of relocated unrecoverable sector and bam — clean smart results. I’m very skeptical if the same was possible for HDDs though, blocks just don’t heal themselves.
  • when something unexplained going on it usually worth checking ram. Memtest86 is my go-to tool. It did save the day a few times on the past, in the no less mysterious circumstances. It’s a low risk low effort high reward thing to try when you have a chance and nothing else to do.
#11

I think the only way to really know for sure is to reproduce the issue with a second (known working) drive on exFAT and see if I have the same problems.

I’m highly doubtful that the disk was ever bad to begin with. It had shown no signs of failure with very recent usage prior to Duplicacy and even with the exFAT format, I could sustain copy speed of 50-75MB/sec when copying directly - and that was after the Duplicacy install too. Like you said too, a HDD wouldn’t just heal itself. I also believe that if Duplicacy was trying to write to bad sectors on the drive and was failing, that the disk would have marked those sectors as bad at that time - which didn’t happen.

If I have time this weekend I’ll do a test run with another disk on exFAT just to see if I have the same issue. If that’s the case then it could be something related to my configuration (Duplicacy, my RAM, my PC, Windows install, etc.) In any case it would just be an experiment at this point as I have no intention of using exFAT + Duplicacy together after this experience.

Thanks for your help during this process

1 Like
#12

(Technically speaking there’s a difference between how drive manufacturers and programmers talk about units of storage, e.g., KB vs KiB. Drive manufacturers like to say that 1MB = 1 million bytes, which isn’t the same definition used in file systems where it’s expressed in binary, but to keep the example below simple I’m temporarily ignoring it.)

At the most basic level, exFAT was designed (and optimized) for flash media, so on a HDD, the performance can be degraded.

Not knowing the make/model of your 2.5" HDD, I’m making some general assumptions based on details you’ve already mentioned earlier. Since your Duplicacy backup is ~700GB, the HDD is going to be a minimum of 750GB (in order to account for formatting overhead). And with write throughput of 50-75MB/sec, the USB interface must be at least USB 3.0.

  • This mean that the HDD very likely has a 4,096 byte block size instead of the older 512 byte block size at the hardware layer.
  • exFAT’s default block size varies depending on the volume size. If the disk partition on the HDD is 1TB, the default is a 256KB block (512KB for 1-2TB and up to 32MB for larger volumes.)
  • A 256MB file copied to the exFAT volume on the HDD will be diced into 1,000 chunks to occupy 1,000 exFAT blocks.

So, where it gets messy is if each exFAT block is 256KB while the HDD’s native block size is 4KB, it means that 64 disk blocks are required to hold a single exFAT block. A 256MB file is therefore chopped up twice (first by Microsoft Windows and then by the disk controller) before it’s written onto the drive platter(s).

Because Duplicacy dices a snapshot into chunks, your ~700GB backup could easily consist of over 200,000 chunks if there’s very little deduplication. If every chunk was 4MB, there would be 175,000 chunks in the snapshot. 4MB / 256KB = 16 exFAT blocks = 1,024 disk blocks. That’s 1,024 write operations for each chunk, or just over 179 million write operations for the snapshot.

In comparison, NTFS has a default block size of 4KB, which aligns with the native block size of modern HDDs (and SSDs). Having matching block sizes streamlines the read/write process. One of the original reasons manufacturers migrated from 512 to 4K byte blocks on HDDs was faster read/write speeds that better aligned with the default block sizes used by modern file systems.

Part of the reason for the 50-75MB/sec write speed when copying files directly to the HDD is because there were likely a smaller number of much larger files instead of a lot of small files, so fewer IOPS.

Other potential weak link is the USB-to-SATA bridge in the external drive enclosure. It might have been optimized for larger files rather than lots of small files.

I also wanted to quickly mention that the bad sector reallocation on HDDs is done automatically while the bad block mapping at the file system level might or might not be automatic depending on the particular file system. When zeroing a HDD, bad sectors are handled automatically by the HDD, but there’s a fixed number of spare sectors set at the factory. At some point the pool of spare sectors will run out and the HDD will no longer be able to remap bad sectors. A similar process applies to SSDs, except that some SSDs support a variable bad block pool size, e.g., Samsung SSDs can be reconfigured to sacrifice storage capacity for data integrity.

3 Likes
#13

This is brilliant. Thank you very much for writing this out.

It is strange, though – if the massive number of blocks being written was the culprit – should not this be not the case:

Or is that pure CPU overhead slicing and dicing the data at the filesystem level?

#14

This was my thought as well, but I did not notice any spike in CPU usage during the disk “idle time”. Perhaps you are right though, but maybe the slicing and dicing just isn’t very CPU intensive in the first place, so the CPU usage wasn’t noticeable. If I recall correctly CPU usage was hovering around 10-20% the whole time. This is a Core i5 1135G7 so not beastly but definitely a modern CPU.

I also wouldn’t rule out issues with I/O delay, but unfortunately I did not monitor it at the time.

Thanks for the very detailed explanation.

This is accurate. I think it also speaks to a fundamental difference between the way that tools like Macrium and Duplicati work. Macrium creates one very large file for each run and writes to it while Duplicati breaks things up into chunks of set sizes (in my case 512MB) so I was probably seeing better performance with those tools because of the fact that they deal in fewer, larger files. With Duplicacy the chunks can be quite small.

The external drive in question is a 4TB Western Digital external on USB 3.0. I do not remember what the chunk size was set at when it was formatted, but the explanation of it being something other than 4KB is highly likely.

#15

Some of the disk idling is due to disk caching in Windows and other buffering so disk writes often aren’t immediate. There’s the buffer in RAM, possibly a small buffer in the USB-to-SATA bridge, and the buffer in the HDD itself. Windows can only see up to the USB interface so sometimes a drive can be busy even when Windows says it’s idle.

A benchmark utility such as CrystalDiskMark (https://crystalmark.info/) is very handy for diagnosing transfer speed issues. CrystalDiskMark tests how a drive performs under 4K, 128K and 1M file sizes (using the same drive in different enclosures is also a good way to test USB-to-SATA bridges).

You’re welcome.

With 512MB chunks in Duplicati, it’d require less than 1,400 chunks for a snapshot. Definitely a lot less I/O compared to transferring 100,000+ chunks with the default chunk size in Duplicacy.

I didn’t have a spare 4TB HDD handy to test with, but with a 2TB HDD, Windows disk manager selected a default cluster size of 128KB on Windows 7 and 512KB on Windows 10 for exFAT, so it’s probably a 1MB cluster size for a 4TB drive when formatted on Windows 10 / 11.

A few years ago I was copying about 6.5TB of data for a research project. Almost all of the files were around 2MB or less but there were over 21 million files across thousands of folders. The data dump was going onto a pair of NTFS-formatted 5TB 2.5" HDDs. Initially the transfer speed was pretty decent at around 40MB/s (limited to USB 2.0 due to the NAS the data was archived on) but then slowly dropped until it was hovering below 500KB/s – at that rate it would have taken over 3 months for the transfer to finish. After a few benchmarking tests I ended up reformatting the drives as ext3 (it’s native to Linux but there are tools for Windows) and was able to complete the transfer in a few days. At least from my experience, NTFS isn’t a great choice when it’s a combination of a large number of small files + SMR media (I think at least part of the problem was the journaling in NTFS).