Is anyone else having restores take quite a long time because of the listing? I got about 35 TB on SFTP (rsync.net) and it seems to take quite a long time for the list to be generated before i can start a restore. Wondering if there is anything I can do to make this a bit faster
List generation translates to chunk file enumeration. Unless the server is configured to cache metadata this will result in a massive amount of IO to the data disks that can only handle 200-300 iops. Clearly, rsync.net servers aren’t, hence the problem.
The solution is to move to the storage provider that does not live in 1990 and has solved the metadata caching. Or maybe there is a way to ask them to mount cache on your pool.