OneDrive slow with nesting

Support
#1

Hi there,

I have a short question:
I am using OneDrive for Business plan for my backups. This is working really great. The speed is ok and it seems really good.

But as OneDrive does not allow much files in a single folder (and has a limitation), I turned on nesting. But as nesting uses more folders, the check command now takes about 2-3 hours for my 5TB archive (from personal internet connection with 32MB/s down and 5MB/s up). It seems like the iteration through the folders is taking the most.

Is there any possibility to speed things up during the check? The main reason is, that I am doing the backing in this order which means I am loosing 6-9hours for checks and about 30-60 minutes for the backup and prune process:

  • Check
  • All Backup plans
  • Check
  • Prune
  • Check (i know this can be skipped)

I generated the storage inside OneDrive with this command:

duplicacy_win_x64_2.7.1.exe init -erasure-coding 5:2 -e -c 64M -min 32M -max 128M -storage-name QnapDuplicacy QnapDuplicacy odb://Duplicacy/QNAP

Nesting file:

{
“read-levels”: [2],
“write-level”: 2
}

About OneDrive:
Useful link with all information about illegal filename characters, max filename length, … can be found here (german): Ungültige Dateinamen und Dateitypen in OneDrive und SharePoint - Office-­Support

Information about the file limit per folder cannot be found here, but I found some posts where people complain about a 20’000 files per folder limit.

#2

It looks like the limit of 20K files per folder is there only when you need to sync the storage with local computers. If you don’t sync then the limit is much higher.

Even if it was only 20K, it would be enough for your 5TB storage with a nesting level of 1. You’ll need 20K * 4M * 256 = 20T to hit the limit.

#3

hm strange, the last time I tried to backup my 5TB with default settings (no nesting, default filesizes) it was not working. I always got a message about too much files in folder. That is why enabled nesting and also increased the filesize to avoid that in the future. (Last one seems to also increase the upload speed with only one thread.)

#4

The default nesting level in recent versions is 1. There used to be no nesting at all for very early versions but that was changed a few years ago.

#5

Hm ok but I want to stay with nesting to avoid hitting the limit again. And I started with the backup some months ago, so there was always a nesting :-).
(I have a 5TB nas which I backup with about 5MB/s upload possibility. As of problems in the past I now uploaded this 5TB about 5 times, and one of the last errors was this file limit in OneDrive - that is why I do not want to settle back without nesting - it takes too long to upload everything again.)

But in general, do you think it can be made faster? I would really appreciate that!
Already did some research on my own - it is the wrong one but maybe it helps :slight_smile: :
Someone said all files in a folder can be found recursiv by this command:

Find-PnPFile -Folder $Folder -Match *

Sources: