Restoring is sloooow

Support
,
#1

Is it normal that it takes hours to list backups, and then hours to list files, and then again a lot of time to restore some files ?

I use Windows GUI with a folder containing 1,100,000 files 210,000 folders 149 Go with 15 days of history with a backup every 30min with hubic storage.

What is the bottleneck ? Number of files ? Size of files ? Number of revisions ?..

It makes restoring files very complicated and I suppose it will worsen in the future with more and more files backed up…

Thanks in advance.

To restore a tiny file, Duplicacy downloads a huge amount of data
#2

If you run the CLI version to restore it will be much faster:

cd \path\to\repository
c:\Users\username\AppData\Local\Duplicacy\duplicacy_win_x64_2.0.9.exe" list
c:\Users\username\AppData\Local\Duplicacy\duplicacy_win_x64_2.0.9.exe" restore -r 1

The CLI executable can be found under c:\Program Files (x86)\Duplicacy if you installed Duplicacy for all users.

The CLI version will be much faster because 1) it can use the same cache in the repository and 2) there is no need to pass the entire file list from the CLI version to the GUI version and to populate the file list widget.

That being said, another factor in your case is the hubic storage. It is known to be slow because their servers do not allow https clients to reuse connections so that there is a significant amount of overhead associated with each https request (although you can mitigate the effect to some degree by using multiple threads).

#3

Thanks for your answer.

Maybe I was wrong with the time listing the snapshots.
I made more tests comparing CLI and GUI:

  • listing snapshots (90 of 5-6 ko each): 18min with CLI / around 30min with GUI
  • listing files (files: 1097902, total size: 154501404237, file chunks: 30975, metadata chunks: 84): 3min with CLI / around 1h45 with GUI (1h20 for list command, 2min for Parsing command output, 23min for Populating file list)
  • restoring files (111k files, total 106 mo): around 3min with CLI and GUI

Everything takes a lot of time but especially listing files with GUI (vs CLI) no ?

I have two more questions/comments:

  • How can I specify a target destination for restore command (as with the CLI) ?
  • Would it be complicated to add progressions indicators (at least for the CLI) ? It seems easy to count snapshot files in storage or files indexed during last backup…
#4

Hubic is the slowest among all storages supported by Duplicacy. I guess at least the CLI version will speed up a lot if you use a different storage.

You can’t specify a target destination for restore command. It always restores to the current repository. However, you can initialize a new repository under the destination where you want to restore to, with the same repository id and the same storage url, and then run the restore command there.

The -stats option of the backup and restore commands will give the upload speed, percentage, etc, after every chunk is uploaded/downloaded.