I have been testing the restore function after performing a few weeks of backups. However, I noticed that the latest revision does NOT have all of the directories and files listed in my backup set. But Revision 1 does. This leaves me to question whether I’ll actually be restoring all files from any revision I choose. Is this normal?
How did you check that specifically?
You should always test restore regardless of which backup tool you use. Unless you actually try to restor you can’t be sure that it’s possible in the first place. So please do check that.
But the idea is that yes, restoring files from revision N will bring back the state of the repository (or subset of files if you specified filters) to how it was when the backup of that revision started.
I go to the “RESTORE” option in the GUI. Below you can see the two revisions I referenced.
Revision 1 includes the AppData/Local folder.
Revision 41 (latest) does NOT include the AppData/Local folder.
I initiated the first backup a little over a week ago and have made absolutely no changes to the Include/Exclude filters.
Wow. That’s… unusual.
Few possibilities:
- Stupid question, but I have to ask – is “c:\ProgramDara\New User\AppData\Local” folder still on disk?
- Could it be that enumerating folders process was terminated/aborted somehow and it shows you partial list? You can use command line duplicacy tool to list files in both revisions (depending on how did you install duplciacy – for all users as a Windows service or for current user only the location of the instructions will differ)
- Add
-dglobal flag to your backup task, backup, and watch logs. Do you see New User/Local folder actually being picked up? if it is skipped – you will see the reason why
Yea, the folder is still on disk. For option 2, what would be the syntax I would need to use in the command line (I’ve only used the GUI)? I installed as a service.
I figured it out. Somehow the path “+Users/New User/AppData/Local/” was not included in the filters. Not exactly clear how it changed, but since I seem to be the only person with this issue, I’m going to mark it as user error. Thanks for the help!
Note: I would still really like to know the syntax I can use to display a list of of files in both revisions. I think that may be helpful in troubleshooting in the future.
Great. I found that “selective exclude” works better than “selective include”, which will miss newly added folders and locations. Maybe something to consider for the future.
And stuff in Local I would generally exclude, with a few exceptions for misbehaving apps that put important data there. Roaming profile on the other hand you likely want to include, with a few exceptions for misbehaving apps that put trash there…
When you run any command in WebUI you can open logs; the first line in the log will be “Running <command> from <folder>”. Make a note of that folder, it is a temporary location that has Duplicacy CLI repository initialized.
Start CMD, go to that folder and run duplicacy command line tool from there. Command you are looking for is List command details
Another helpful one is Diff command details to see the differnce between snapshots. It’s useful to find and exclude transient temporary data for example to minimize data turnover at the destination, save bandwidth, etc.
Thanks saspus. So maybe you could clarify the “selective exclude”. Below is what I would like to backup:
C:\ProgramData\.duplicacy-web\duplicacy.json
C:\Users\New User\AppData\Local\Google\Chrome\User Data
C:\Users\New User\AppData\Local\Intuit
C:\Users\New User\AppData\Local\Microsoft\Outlook
C:\Users\New User\AppData\Roaming\Microsoft\Excel
C:\Users\New User\AppData\Roaming\Microsoft\Office\Recent
C:\Users\New User\AppData\Roaming\Microsoft\Outlook
C:\Users\New User\AppData\Roaming\Microsoft\Templates
C:\Users\New User\AppData\Roaming\Microsoft\Windows\Recent
C:\Users\New User\AppData\Roaming\Microsoft\Windows\SendTo
C:\Users\New User\AppData\Roaming\Microsoft\Word
C:\Users\New User\AppData\Roaming\Google\Chrome\User Data
C:\Users\New User\Contacts
C:\Users\New User\Desktop
C:\Users\New User\Documents
C:\Users\New User\Favorites
C:\Users\New User\Links
C:\Windows\Fonts
C:\Windows\System32\drivers\etc\hosts
C:\Windows\Tasks
And here is the filter I am currently using:
# include Google Chrome profile but nothing else from Google
i:(?i)/AppData/[^/]+/Google/$
i:(?i)/AppData/[^/]+/Google/Chrome/$
i:(?i)/AppData/[^/]+/Google/Chrome/User Data/
e:(?i)/AppData/[^/]+/Google/
+Users/
+Users/New User/
+Users/New User/AppData/
+Users/New User/AppData/Local/
+Users/New User/AppData/Local/Intuit/*
+Users/New User/AppData/Local/Microsoft/
+Users/New User/AppData/Local/Microsoft/Outlook/*
+Users/New User/AppData/Roaming/
+Users/New User/AppData/Roaming/Microsoft/
+Users/New User/AppData/Roaming/Microsoft/Excel/*
+Users/New User/AppData/Roaming/Microsoft/Office/
+Users/New User/AppData/Roaming/Microsoft/Office/Recent/*
+Users/New User/AppData/Roaming/Microsoft/Outlook/*
+Users/New User/AppData/Roaming/Microsoft/Templates/*
+Users/New User/AppData/Roaming/Microsoft/Windows/
+Users/New User/AppData/Roaming/Microsoft/Windows/Recent/*
+Users/New User/AppData/Roaming/Microsoft/Windows/SendTo/*
+Users/New User/AppData/Roaming/Microsoft/Word/*
+Users/New User/Contacts/*
+Users/New User/Desktop/*
+Users/New User/Documents/*
+Users/New User/Favorites/*
+Users/New User/Links/*
+ProgramData/
+ProgramData/.duplicacy-web/
+ProgramData/.duplicacy-web/duplicacy.json
+Windows/
+Windows/Fonts/*
+Windows/System32/
+Windows/System32/drivers/
+Windows/System32/drivers/etc/
+Windows/System32/drivers/etc/hosts
+Windows/Tasks/*
-*
What changes would you suggest to make this better?
Hmm. What is the goal of your backup? I mean, what scenario are you protecting against?
Line by line:
Duplicacy skips .duplicacy-web folder unconditinaly. Instead, copy that file elsewhere if you want to save it; but what for? it’s a configuration of a backup program itself; to restore it you need to first configure it. What’s the point in putting a key in the box it is locked with? An alternative would be to keep it (along with keys for storage) in your password manager, that is inherently replicated to multiple machines and destinations, e.g. 1Password, or what have you. And yet, I would not bother, It’s trivial to configure. What I would save is filters file.
This folder is almost pure transient trash. There are mostly cache files with a lot of turnover. Instead have it keep a copy of your bookmarks in your google account, if that’s important to save. (or selectively add only bookmarks database to backup, not everything else there. Can’t help you with specific file – don’t use chrome, but I’m sure it’s discoverable)
I don’t know what’s in that folder, but I’d think that your tax and other financial documents would be in Documents folder. I doubt saving application state is of any use, but I could be wrong.
From outlook you want .pst files. (and you have to user -vss flag to be able to back them up). Not sure where does outlook keep those file these days, I think its somewhere under Documents, but yes, that would be a good candidate for backup.
That folder does not seem to exist on my machine, but same idea as in the one in Local. Transient browser crap.
If you have installed additional fonts – backup those additional fonts from the source location. there is no point in backing up system data.
I would do something like this:
+Users/
-Users/New User/AppData/Local/*
-Users/New User/AppData/LocalLow/*
+Users/*
-*
Then explicitly exclude stuff from Documents that some apps like to dump there – I have seen a game from a reputable publisher who would dump 2GB GPU shader cache file into Documents folder. That file would change every launch (why!?) and get re-uploaded to OneDrive every time. that happens.)
If you need some specific data from AppData/Local or LocalLow because some misbehaving app stores non-transient data there – add an exclusion for it the way you did, (and maybe email the vendor to tell them to stop storing user data there). E.g.
...
+Users/New User/AppData/Local/
+Users/New User/AppData/Local/MisbehavingApp/
+Users/New User/AppData/Local/MisbehavingApp/ImportantData/*
...
Note, with duplicacy * matches everything, including path separator.
Then run the backup in -dry-run mode and see what folders and files are being picked up.
On the other hand, depending on how much bandwidth and storage you are willing to trade in exchange for not needing to dig inside AppData to hunt down important files – include everything, run backup few times, and then exclude the worst generators of transient trash. Those would likely be your browser profiles and AppData/Temp folder. Exclude them by explicit exclusion rules. Unimportant data but with small turnover is best left included: efforts to maintain the exclusion list is not worth saving couple of Mg of bandwidth.
This way you will never miss any important data that newly installed app may place into unorthodox locations, and few GB at the destination is a small price to pay for not needing to sift through your data on the regular basis and keep duplicacy rules up to date, risking missing something important at every turn. At least that’s what I do myself.
1 Like
I would also include Roaming profile as well. Nobody pays attention anymore and garbage is everywhere.
I’ve just checked on my PC: there is c:\Users\me\AppData\Roaming\NVIDIA\ComputeCache folder, with tens of gigabytes of what appears to be compute shader caches. The one of a very few things that with 100% certainty is temporary, specific to local machine hardware, and in under no circumstances should be roamed anywhere. If even Nvidia does not care, why anyone else would?
Thanks. This is very helpful.