Restore command details

How-to
#21

This is the problem exactly. To attempt to get accurate ETA or a full % complete, the initial startup time to calculate that would be immense. Starting from a cancelled restore would waste this calculation. It would have to scan every file checksum locally, download all the remote chunks and compare. This is a huge waste of time and resources - which IMHO is why the way it is - and the way similar software has worked for decades.

I was using rclone yesterday and the ETA said 2 minutes for near an hour because it calculated the size of the remaining files, but because there were 60,000 of them it did not take into account the latency of starting all of those transfers.

Anyway - this has really gone into the weeds now.

1 Like
#22

Thanks for the clarification @nopro404.

#23

This doesn’t seem to be true.
I just run: duplicacy -d restore -r 16 -overwrite -hash -delete
This is the relevant part of the output:

...

Parsing filter file /home/qwuy/.duplicacy/filters
There are 0 compiled regular expressions stored
Loaded 4 include/exclude pattern(s)
Pattern: +desk/*
Pattern: +dl/
Pattern: +docs/*
Pattern: -*
Listing 
.cache/ is excluded by pattern -*
.config/ is excluded by pattern -*

...

common-lisp/ is excluded by pattern -*
desk/ is included by pattern +desk/*
dl/ is included by pattern +dl/
docs/ is included by pattern +docs/*

...

Clearly I did not specified any filter and yet restore read the filter specified in the file.

For completeness, I am using v. 2.7.2 compiled from source on ArchLinux

#24

Is there a reason why -delete is ignored when a pattern is specified? Couldn’t it act on the pattern?

For example, using

duplicacy -d restore -r 16 -overwrite -hash -delete "docs/*"

would restore exactly as “docs” it was on revision 16.
What happens now, since -delete is ignored, files are either added or overwritten and nothing extra gets deleted.

#25

Because that would delete files that are included in the backup but excluded by the pattern, which isn’t a safe thing to do in normal circumstances (what if a wrong pattern is specified by mistake?).

#26

What do you mean by that?
Maybe that the filter file specifies, say, two directories and i restore just one with duplicacy restore .... "docs/*"?
If yes, than why couldn’t duplicacy just delete files for the directory used in the command line?

#27

Sorry to quote an old message.

So if I understand this properly. If I have part of the file, so let’s say I modified just something in it (so I have a previous version of the file) , the use of -overwrite is mandatory.

Basically i do not use -overwrite only if I do not have the file or part of it or a previous version of the file at all?

If yes, how can i tell if the file i’m restoring with -overwrite is exactly the same or if i’m intervening on a part of the file? Hope the question is clear.

#28

Not entirely clear, but could you be getting at the fact that Duplicacy does incremental, delta-like, backups which means only the changed parts get uploaded? And that a restore might be doing the reverse and rely on a known state of that file?

Well, -overwrite is only needed if you’re doing an in-place restore - i.e. changing the original file in the same location as the one being backed up.

Doesn’t matter on the state, since Duplicacy maintains a complete snapshot of each file regardless of what chunks were uploaded and when.

If your concern is that you don’t know what’s changed, the history command can help with that, but you can also simply restore into a temporary location, choosing not to overwrite the original. Then you could use a third-party tool (I like Beyond Compare) to examine the differences.

2 Likes
#29

Thanks!

I performed some restore tests and noted that the actual content of the folder i am restoring into is not deleted, so the files I restore are added to the files that already are in the destination folder.

So if I understand this correctly, the restore does not restore an actual snapshot (intended as a perfect mirror of the backup) in the destination folder, but “just moves” my files to the destination folder.

Correct?

#30

A bit of topic, but ability to mount snapshots is way overdue. I like the way Kopia implemented it (also golang app btw) - you run kopia mount --browse And bam - Finder opens with a folder full of snapshot, just like Time Machine presents its.

This would solve the all restore and exploration related user experience issues. “Just mount and use your favorite tool”

1 Like
#31

Haven’t yet tried the mount tool but if it presents as a normal file system - i.e. all metadata - Beyond Compare should work extremely well, and instantly highlight the differences between a backup snapshot and original backup folder structure.

Definitely wanna try it under Windows at least…

#32

A restore of a snapshot will not delete existing files that were not present when it was created unless you pass -delete . It doesn’t move anything.

It will restore everything within it or what was requested to be restored by using pattern filters.

Hope that helps

1 Like
#33

It helps a lot! Thank you.

It’s not clear tho why if using patterns, the -delete flag is ignored.

The -delete option indicates that files not in the snapshot will be removed. This option is ignored if any pattern is specified.

#34

I don’t see what the use case of that would be. My guess is that since implementing it would require more code no-one found a reason to do it.

Duplicacy default behavior is to not be destructive by requiring -overwrite and -delete to to opt into possible bad behavior.

If you really wanted to do that I would restore the files you wanted outside of the tree you want then rsync -avxP --delete /source/dir /target/dir

2 Likes
#35

A dry run option would be nice for this. I have a scenario where I want to know what’s missing and run the command to restore only chosen folders (that which I didn’t mean to delete recently). diff almost works but isn’t quite the same. It won’t show me exactly what the restore command would do and you can’t specify a folder to diff.

Just a thought. Not a problem/ dealbreaker

2 Likes
#36

Hi,
i do not fully understand how the part in parenthesis matches with the first part that says that using the flag the user/group id will not be restored.

  • “all restored files will be owned by the current user” – the current user that’s running Duplicacy? Not sure i got this right.
#37

Yes. (Post must be at least 20 characters long so I had to add this nonsense here).

#38

Thanks, but I do not understand.

If I do not use the flag I would expect to restore the files with their exact previous owners (are permissions also restored?). But instead here is said that the user running Duplicacy will take the ownership. In my case root. And I don’t want to assign root to the files I restore.

Also it seems to be the opposite to what is said before the parentheses.

How can I avoid this? What owner would be assigned if using the flag?

#39

When the restore command creates a file to be restored, the owner of the file is the user that runs the restore command. After the file is restored, the uid and gid will be restored from the meta info. But if the -ignore-owner option is specified, the uid and gid won’t be restored so the owner is still the current user.

1 Like
#40

@iocularis, there are two important nuisances:

  1. Unless you are running duplicacy as root, it won’t be able to change ownership of the files.
  2. There is no guarantee that on your new machine you are restoring to the user id and group id will be the same as before; then restoring files even as root will result in files you can’t read; you would need to adjust permissions manually, as a super-user.

In other words, if you backed up data of user john(502:1000), machine died, you reinstalled the OS and re-created user john – it is possible for john to have differnt ID now, e.g. (501:1000). Now if duplicacy restores permissions – John is screwed. You would need to do sudo chown -R john /Users/john or something along those lines to let user john access their restored files.

3 Likes