Restore command details

[TheBestPessimist]

SYNOPSIS:
   duplicacy restore - Restore the repository to a previously saved snapshot

USAGE:
   duplicacy restore [command options] [--] [pattern] ...

OPTIONS:
   -r <revision>                 the revision number of the snapshot (required)
   -hash                         detect file differences by hash (rather than size and timestamp)
   -overwrite                    overwrite existing files in the repository
   -delete                       delete files not in the snapshot
   -ignore-owner                 do not set the original uid/gid on restored files
   -stats                        show statistics during and after restore
   -threads <n>                  number of downloading threads
   -limit-rate <kB/s>            the maximum download rate (in kilobytes/sec)
   -storage <storage name>       restore from the specified storage instead of the default one
   -key <private key>            the RSA private key to decrypt file chunks
   -key-passphrase <passphrase>  the passphrase to decrypt the RSA private key
   -persist                      continues even if errors occur

The restore command restores the repository to a previous revision. By default the restore procedure will treat files that have the same sizes and timestamps as those in the snapshot as unchanged files, but with the -hash option, every file will be fully scanned to make sure they are in fact unchanged.

By default the restore procedure will not overwriting existing files, unless the -overwrite option is specified.

The -delete option indicates that files not in the snapshot will be removed. This option is ignored if any pattern is specified.

If the -ignore-owner option is specified, the restore procedure will not attempt to restore the original user/group id ownership on restored files (all restored files will be owned by the current user); this can be useful when restoring to a new or different machine. This option is available in a version later than 2.0.9.

If the -stats option is specified, statistical information such as transfer speed, and number of chunks will be displayed throughout the restore procedure.

The -threads option can be used to specify more than one thread to download chunks.

The -limit-rate option sets a cap on the maximum download rate.

When the repository can have multiple storages (added by the add command), you can select the storage to restore from by specifying the storage name.

The -key and -key-passphrase options are needed if the storage is encrypted by RSA encryption.

Unlike the backup procedure that reading the include/exclude patterns from a file, the restore procedure reads them from the command line. If the patterns can cause confusion to the command line argument parser, -- should be prepended to the patterns. Please refer to the Include/Exclude Patterns section for how to specify patterns. Note particularly that even in Windows you must use ‘/’ as folder delimiter, and letters are case sensitive.

Example

Restore the folder C__Users_link\crist\Documents\StarCraft II\

duplicacy.exe restore -r 113 "C__Users_link/crist/Documents/StarCraft II/*"

[Christoph]

Here is an interesting little hidden “feature”: duplicacy’s restore procedure is incremental:

Preserve Duplicacy cache across restore runs

Duplicacy already does incremental restore. When the file to be restored exists locally, Duplicacy will split the existing file into chunks first and then compare them with those to be restored. Only changed chunks will be downloaded.

The file available locally is the best cache.

[TheBestPessimist]

i still don’t understand how incremental works as when i have different files (or just cancelled restores), i must always use -overwrite otherwise duplicacy complains that the files already exist.

[petrom]

I have the same question - actually I’m not able to continue restore process after interruption. I got following error message:

Restoring /mnt/hdd/backups to revision 6
File *** already exists.  Please specify the -overwrite option to continue

My storage is big enough and it takes few days to download all data from the cloud. So, it’s very important for me to get incremental restoration to work.

[petrom]

Ok, seems that it works after manual removal of file mentioned in the error message…

[nopro404]

I still don’t understand how incremental works as when i have different files (or just cancelled restores), i must always use -overwrite otherwise duplicacy complains that the files already exist.

This is correct. Say you have half the file, duplicacy still needs to overwrite the file to make it whole (even by using half the data you already have). By specifying -overwrite you are acknowledging that duplicacy will change existing files to make them identical to the snapshot being restored.

seems that it works after manual removal of file mentioned in the error message…

This will cause redownload of the data if it is not in the cache. If you left it and specified -overwrite it would have…

split the existing file into chunks first and then compare them with those to be restored. Only changed chunks will be downloaded.

…making the existing files match what was in the snapshot.

You can see the code here:

gilbertchen/duplicacy/blob/master/src/duplicacy_backupmanager.go#L1219

[dreamflasher]

The overwrite behiavour of this command does not match the expectation.
By default the restore procedure will not overwriting existing files, unless the -overwrite option is specified. But this is not happening. It fails with File *** already exists., while it should just ignore existing files and continue restoring.

[dreamflasher]

The documentation didn’t help me in finding out how to specify multiple patterns, what one apparently needs to do is to provide them after the -- in seperated quotations. Eg. -- "+include/*" "-exclude/*".

[hiavi]

@nopro404 - are you suggesting that the “-overwrite” is only applicable to partially downloaded chunks/files?

I thought Duplicacy allows one to stop (Ctrl+C) and restart the operations without having to redo entire operation from scratch. Is this really true or am I mistaken?

Because

• When I retry the restore after I had to interrupt, I am getting the error saying I need to use "-overwite options

• As per the documentation, “By default the restore procedure will not overwrite existing files, unless the -overwrite option is specified.”. To me this means the command will overwrite all the files that were downloaded before the operation was stopped and hence this means restart the operation from scratch - which is tedious for large restore operations.

So, it feels like Duplicacy is not capable to restarting operations from the place it got stopped.

Please help clarify.

[nopro404]

@hiavi

@hiavi

are you suggesting that the “-overwrite” is only applicable to partially downloaded chunks/files?

No, it would also apply to files that have changed on disk compared to the backup being restored. -overwrite is to acknowledge you are allowing existing files to be changed to whatever backup you are restoring.

@hiavi

I thought Duplicacy allows one to stop (Ctrl+C) and restart the operations without having to redo entire operation from scratch. Is this really true or am I mistaken?

It will resume any partially restored files if -overwrite is specified.

@hiavi

“By default the restore procedure will not overwrite existing files, unless the -overwrite option is specified.” . To me this means the command will overwrite all the files that were downloaded before the operation was stopped and hence this means restart the operation from scratch

It will check existing files with the backup being restored. If they are different only the chunks needed to make it identical to the backup will be downloaded. If the files match they will not be re-downloaded or otherwise touched.

@hiavi

So, it feels like Duplicacy is not capable to restarting operations from the place it got stopped.

False.

[hiavi]

Thanks @nopro404 for the very helpful clarification.

It will check existing files with the backup being restored. If they are different only the chunks needed to make it identical to the backup will be downloaded. If the files match they will not be re-downloaded or otherwise touched.

When my previous “Restore” operation had to interrupted, the last outline line was

Downloaded chunk 266236 size 1817426, 17.92MB/s 2 days 09:12:20 25.8%

When I restarted the “Restore” operation, it restarted from 0% and I was hoping Duplicacy might have to do quick verifications and it will speed up to last interruption point very quickly - but even after 3+ hours of running, it is still at 7.7%

Downloaded chunk 58046 size 1915759, 21.74MB/s 1 day 18:56:28 7.7%

So, while your explanation holds true - it feels like Duplicacy is re-downloading (or copying in case of local) the chunks from storage URL and doing most of the work. I was hoping Duplicacy could quickly race up to 25.8% and then keep making progress at its regular pace. Thats not what I am seeing.

I am sure there is a good enough reason why Duplicacy is doing this - I don’t understand the tech to make any comment on that - all I am saying is that by reading the forum statements, I had a slightly different expectation w.r.t time taken to completion of the operation when restarted and it feels like one needs to be super careful to ensure the “Restore” operation doesn’t get interrupted - OR else expect the command to rerun for the full period of time again to complete the operation.

[nopro404]

@hiavi

I was hoping Duplicacy could quickly race up to 25.8% and then keep making progress at its regular pace.

The 7.7% in your example is on top of the base you have restored already. Say you had 100G to restore and you restored 25.8G, you have 74.2G left to restore Resuming the restore will start from 0% and count up to 100% for that 74.2G – not the full 100G.

[hiavi]

Thanks for the clarification @nopro404

Even the chunk #s started from 1 along with % started from 0 % - so it seemed like the restore operation started from scratch.

It would be good if the output messages represented the real % because as a user I just ran a command and I would like to know much of the work I requested (restore a given snapshot’s revision #) is remaining.

[nopro404]

@hiavi

It would be good if the output messages represented the real %

It is the real % . If you want more info during restore or backup - use -stats

[hiavi]

It is the real %

To elaborate a bit - there are two different ways to look at and both are theoretically correct - the only question is what is more apt.

1. Perspective of the Command: Taking your example, say you had 100G to restore and you restored 25.8G, you have 74.2G left to restore. When one kicks off the “Restore” command again, from the command’s point of view - it is going to be see what work it needs to do and realize that it needs to restore 74.2G and gets started starting at 0% and hits 100% once it is reaches the 74.2G.

2. Perspective of the User: A user running Restore command wants to restore a given repository to a given end point. If a portion of the work is already done, wonderful … the command could quickly progress to 25.8% and then slowly progress from there on to 100%.

I feel both are “accurate” - but the question is what is more helpful to the user. May be there are more involved usages that I am not seeing the bigger picture - but just wanted to share a different perspective.

If you want more info during restore or backup - use -stats

I am using -stats. Is there anyway to get progress % without using -stats option? Running without -stats option didn’t show any progress %.

I am nube w.r.t Duplicacy - so apologies if my point of view is naive.

[saspus]

I think both approaches miss the point.

Restore, and many other operations, are effectively atomic: until it’s done completely you can’t really rely on intermediate results. Therefore 100% carires drastically different importance than any other percentage value.

In other words, percentages are means to answer the question of when will whatever it is it’s doing complete.

“54% done” is meaningless and non-actionable. It’s only useful indirectly, an aid in a process of watching progress bar, taking first derivative and arriving to completion time.

“3 days 16 hours remaining” — is extremely useful and avoids the ambiguity you highlighted entirely.

that’s what Duplicacy should report. And there is no need for a progress bar either — because nobody cares about the progress outside of done/not done but everyone cares about ETA.

TLDR: I would remove progress bars altogether and replace them with ETA in medium weight type.

[hiavi]

I would agree with you that ETA is very important.

But I feel % done has its own importance - say an operation is 99% done and will take 3 more hours to complete (for whatever reason). I would hold off any sort of disruption as most of the work is done.

But if same thing is 0.1% done and will take 3 more hours to complete - I might be completely ok to interrupt and rerun the operation later.

In both the cases the ETA is same - but based on how much work is done, the approach one takes might be different.

[Droolio]

I agree adding an ETA would be useful, but ETAs can fluctuate widely over an internet connection, and removing percentages would put you more in the dark. For example, your transfer could actually hang and your ETAs change, but you’d be making no progress.

Yes it could show speeds as well but these are all averages and I, as a human, can better judge from a percentage reading over a greater time span, if we’re getting anywhere. Removing important information is not a good idea.