How to restore a symlink repository?

TheBestPessimist · 11 January 2020 14:09

I’m in a kind of stupid situation now. I want to test-restore last revision of all my backups but It seems i cannot without risking to overwrite all my existing data.

All my backups use the symlink-repository pattern, and on windows a symlink can only be created with admin rights.

Right now i’m afraid that if i run the restore with admin rights, will create the symlinks then overwrite all my current (real) files via the restored symlinks.

Here’s what restore tells me:

PS C:\duplicacy-repositories\tbp-restore> .duplicacy/z.exe restore -threads 32 -stats -ignore-owner -r 9
Storage set to G:\My Drive\backups\duplicacy
Loaded 0 include/exclude pattern(s)
Parsing filter file \\?\C:\duplicacy-repositories\tbp-restore\.duplicacy\filters
Parsing filter file C:/duplicacy-repositories/tbp-fusion/.duplicacy/duplicacy-utils/filters/filters_general_usage
Ignoring duplicate pattern: e:(?i)^PerfLogs/ ...
Ignoring duplicate pattern: e:(?i)\.cache$ ...
Loaded 198 include/exclude pattern(s)
Restoring C:\duplicacy-repositories\tbp-restore to revision 9
Can't create symlink C__Users_link/All Users: symlink C:\ProgramData \\?\C:\duplicacy-repositories\tbp-restore\C__Users_link\All Users: A required privilege is not held by the client.
PS C:\duplicacy-repositories\tbp-restore>

And here’s how a normal repository looks for me:

C:\duplicacy-repositories\tbp-fusion>dir
 Volume in drive C is os
 Volume Serial Number is 9C1C-D5F4

 Directory of C:\duplicacy-repositories\tbp-fusion

Sun, 05 Jan, 2020  19:12    <DIR>          .
Sun, 05 Jan, 2020  19:12    <DIR>          ..
Sun, 05 Jan, 2020  19:12    <SYMLINKD>     C__all_link [C:\all]
Sun, 05 Jan, 2020  19:12    <SYMLINKD>     C__Users_link [C:\Users]
               0 File(s)              0 bytes
               4 Dir(s)  523,475,296,256 bytes free

C:\duplicacy-repositories\tbp-fusion>

Can someone advise me how should i proceed in restoring? What i am interested in is that all the symlinks in the original repo should be created as normal folders in the “restore-repo” (in my example: C:\duplicacy-repositories\), then --if this were a real restore-- i would copy everything back in place manually.

dgcom · 11 January 2020 16:27

The top-level symlinks are recreated as regular folders if you restore in empty/new repository - I do this all the time.
I don’t know abut other symlinks, but since is not following them, it should just recreate them and continue.

TheBestPessimist · 11 January 2020 18:27

hmm, then why does want admin rights, especially since i run with -ignore-owner?
or, hmm

Can't create symlink C__Users_link/All Users: symlink C:\ProgramData \\?\C:\duplicacy-repositories\tbp-restore\C__Users_link\All Users: A required privilege is not held by the client.

It wants admin to create a symlink named All Users (with target C:/ProgramData) which is placed in the folder C:\duplicacy-repositories\tbp-restore\C__Users_link\ ?

dgcom · 11 January 2020 18:49

Since this is not a top level symlink, it should create it without any issue.
But did you run backup with admin rights? If so, check permissions on your C__Users_link folder wonder if permissions were restored - normally C:\Users does not allow users to write there.

TheBestPessimist · 11 January 2020 21:35

maybe it needs admin rights to symlink C:\ProgramData, since it’s a special folder?

dgcom · 11 January 2020 21:46

I tried this on my Windows 10 install and I was able to create symlink to that folder in temp folder as user:

C:\temp>mklink /d "All Users" C:\ProgramData
symbolic link created for All Users <<===>> C:\ProgramData
dir
...
01/11/2020  04:40 PM    <SYMLINKD>     All Users [C:\ProgramData]

Can you create some folder manually in your C__Users_link folder?

I’ll see if I can reproduce this behavior myself, should be easy test

dgcom · 11 January 2020 22:59

I did some testing… In short - I was able to restore test symlinks with user. I did not try actual folders yet - because I run into another issue.
For whatever reason, is not logging any action performed with symlinks, even if I specify -v -d, making it impossible to verify if symlinks are even backed up!
I found this while trying to create a filter for All users folder to just backup small subset of files - neither -dry-run nor -enum-only showed that All Users being seen by the program.
So, I created a test source tree with all 5 different types of files and folders:

normal file
normal folder
normal symlink to file
hardlink (file only)
Directory symlink
Junction

And backed this up. The backup output only showed real files and folders and hardlink. Same is true for list -files command - it does not print symlinks either.

But when I run test restore, all symlinks were restored, but with one issue - Junction was restored as directory symlink. We already know that has some issues with Junctions, but this behavior is a bit dangerous - if you restore backup with junctions, you will have them converted to symlinks without your knowledge!

11/20/2017  05:14 PM            20,748 test-hardlink
01/11/2020  05:31 PM    <SYMLINKD>     test-junction [c:\Duplicacy\testfolder]
07/31/2018  11:46 PM         3,313,940 test-normal-file
01/11/2020  05:24 PM    <DIR>          test-normal-folder
01/11/2020  05:31 PM    <SYMLINK>      test-symlink [c:\Duplicacy\testfile]
01/11/2020  05:31 PM    <SYMLINKD>     test-symlinkd [c:\Duplicacy\testfolder]

The backup log file does show symlink handling:

Symlink test-root-symlinkd/test-junction updated
Symlink test-root-symlinkd/test-symlink updated
Symlink test-root-symlinkd/test-symlinkd updated

And hardlinks seems to be treated as normal files.

I think, there is plenty of room for improvement in symlink handling and I’d be very careful using it if source may contain Junctions - for example, I would not backup c:\Users\Default User folder, because it is Junction and not directory symlink.

TheBestPessimist · 12 January 2020 06:41

Just to be sure i understand: you managed to create a symlink via cmd (not powershell) using a non-admin cmd? Or what are you trying to tell me there?

I have tried reproducing what you did, but as non-admin it fails:

C:\duplicacy-repositories>mkdir temp

C:\duplicacy-repositories>cd temp

C:\duplicacy-repositories\temp>mklink /d "All Users" C:\ProgramData
You do not have sufficient privilege to perform this operation.

C:\duplicacy-repositories\temp>

dgcom · 12 January 2020 06:46

And the final test:

Created clean folder for source repository
Created symlink mklink /d C__Users_link "c:\Users\"
Created a filter to exclude most molders under `c:\Users’ (for testing, include filters don’t seem to work)
Run the backup

I left the following not excluded:

C:\Users\Default - normal folder
C:\Users\All Users - symlink
C:\Users\Default User - junction

As I mentioned already, there is absolutely no mention of symlink or junction being backed up - only real files and folders show up in verbose/debug logs. Same applies to list -files command.
I was able to restore data as regular user. As expected, top level symlink was restores as regular folder as well as C:\Users\Default.
Both symlink C:\Users\All Users and junction C:\Users\Default User were restored as symlinks

I would NOT recommend backing up these folders with because:

C:\Users\Default User is not restored correctly
None of NTFS permissions seems to be backed up and restored

This means that if you hope to be able to restore windows profiles correctly, this will not work.
It is fine to backup data onside profiles, keeping in mind that restore will mark them with inherited permissions from the folder where restore is performed and all original permissions will be lost.

dgcom · 12 January 2020 06:52

That’s exactly what I was telling you - you do not have permissions to modify content of your C:\duplicacy-repositories\temp - this is because it was created as administrator.

NTFS permissions are a bit tricky… When you do something with elevated process (ex. command prompt, run “As Administrator”), things you create will have Administrator as owner and regular user won’t have modify permissions to them.
When you work with folders like that, you can ether make a choice to continue working with folders and files as admin or reset ownership and permissions to allow user to modify files and folders and inherit ACLs. Still, any new sub-folder you create with admin will have rights for admin only…

Usually, backup and restore need to be run with elevated privileges, so they can properly backup files and restore permissions or write files to places where regular user cannot.

TheBestPessimist · 12 January 2020 06:57

Another interesting thing: during restore with admin right shows info about symlinks:

2020-01-12 08:51:54.299 TRACE SNAPSHOT_PATTERN Pattern: e:(?i)(^|/)Network Trash Folder/
2020-01-12 08:51:54.300 DEBUG LIST_ENTRIES Listing
2020-01-12 08:51:54.307 INFO RESTORE_START Restoring C:\duplicacy-repositories\tbp-restore to revision 11
2020-01-12 08:51:54.316 TRACE DOWNLOAD_DONE Symlink C__Users_link/All Users updated
2020-01-12 08:51:54.319 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default User updated
2020-01-12 08:51:54.331 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Application Data updated
2020-01-12 08:51:54.341 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Cookies updated
2020-01-12 08:51:54.345 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Local Settings updated
2020-01-12 08:51:54.347 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/My Documents updated
2020-01-12 08:51:54.355 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/NetHood updated
2020-01-12 08:51:54.358 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/PrintHood updated
2020-01-12 08:51:54.365 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Recent updated
2020-01-12 08:51:54.372 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/SendTo updated
2020-01-12 08:51:54.374 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Start Menu updated
2020-01-12 08:51:54.381 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Templates updated
2020-01-12 08:51:54.387 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/AppData/Local/Application Data updated
2020-01-12 08:51:54.393 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/AppData/Local/History updated
2020-01-12 08:51:54.409 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Documents/My Music updated
2020-01-12 08:51:54.411 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Documents/My Pictures updated
2020-01-12 08:51:54.417 TRACE DOWNLOAD_DONE Symlink C__Users_link/Default/Documents/My Videos updated
2020-01-12 08:51:54.431 TRACE DOWNLOAD_DONE Symlink C__Users_link/Public/Documents/My Music updated
2020-01-12 08:51:54.433 TRACE DOWNLOAD_DONE Symlink C__Users_link/Public/Documents/My Pictures updated
2020-01-12 08:51:54.436 TRACE DOWNLOAD_DONE Symlink C__Users_link/Public/Documents/My Videos updated
2020-01-12 08:51:54.452 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Application Data updated
2020-01-12 08:51:54.453 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Cookies updated
2020-01-12 08:51:54.458 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Local Settings updated
2020-01-12 08:51:54.463 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/My Documents updated
2020-01-12 08:51:54.472 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/NetHood updated
2020-01-12 08:51:54.478 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/PrintHood updated
2020-01-12 08:51:54.486 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Recent updated
2020-01-12 08:51:54.490 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/SendTo updated
2020-01-12 08:51:54.492 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Start Menu updated
2020-01-12 08:51:54.500 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Templates updated
2020-01-12 08:51:59.031 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/AppData/Local/Application Data updated
2020-01-12 08:51:59.032 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/AppData/Local/History updated
2020-01-12 08:51:59.891 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Documents/My Music updated
2020-01-12 08:51:59.892 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Documents/My Pictures updated
2020-01-12 08:51:59.894 TRACE DOWNLOAD_DONE Symlink C__Users_link/crist/Documents/My Videos updated
2020-01-12 08:52:02.822 TRACE DOWNLOAD_START Downloading C__Users_link/desktop.ini
2020-01-12 08:52:02.823 DEBUG DOWNLOAD_PREFETCH Prefetching C__Users_link/desktop.ini chunk 8239f50e223b2b1cc76ddaea25e5e4859046227680412bfcebeb190a867b4545
2020-01-12 08:52:02.831 TRACE DOWNLOAD_INPLACE Updating \\?\C:\duplicacy-repositories\tbp-restore\C__Users_link\desktop.ini in place
2020-01-12 08:52:02.903 INFO DOWNLOAD_PROGRESS Downloaded chunk 1 size 1530351, 1.46MB/s 07:00:02 0.0%
2020-01-12 08:52:03.200 TRACE DOWNLOAD_START Downloading C__Users_link/Public/desktop.ini
2020-01-12 08:52:03.200 TRACE DOWNLOAD_INPLACE Updating \\?\C:\duplicacy-repositories\tbp-restore\C__Users_link\Public\desktop.ini in place

your C:\duplicacy-repositories\temp - this is because it was created as administrator.

So, I’m not really sure how I created c:/duplicacy-repositories (ie. admin or not) but i have the feeling that said folder and also all other folders inside (ie. temp) were created via windows explorer (CtrlShiftn). It’s weird that all my subfolders are admin-only :-?

Testing this admin issue by creating a totally new folder C:/aaaaaaaaaaaaaa by right clicking in Windows explorer -> selecting new folder. Here’s my permissions:

Does this mean that folders already created in C:/ don’t give me full rights over them?

dgcom · 12 January 2020 07:08

Yes, symlinks are only mentioned during restore, otherwise they are completely hidden.

Yes, that is correct. If you check Advanced settings, you will see that these permissions are probably inherited from C:. Also, it may depend on UAC settings how these folders are created.

I prefere using command line tool icacls to check and control permissions - it is a bit more transparent and detailed

bruce · 17 March 2024 21:46

This is not a permissions issue, and has nothing to do with top-level vs not top-level directories.

In Windows, you have to use an elevated (Admin) command-prompt to create symlinks. This includes SYMLINKD (directory ‘mklink /D’) and SYMLINK (file ‘mklink no-option’). You can turn on Developer mode in Windows to remove this restriction, but don’t depend on this for delivered software.

Under normal mode and Developer mode, you can create Junction Points AKA JUNCTION (directory ‘mklink /J’) and file hardlinks (file ‘mklink /H’) without an elevated/Admin window.

I found this restriction, to be blunt, VERY STUPID OF MS. How stupid is it? The default (no option) is a SYMLINK, which doesn’t work under normal circumstances, but adding an option (/H) does work. Let me repeat - the default option doesn’t work under default circumstances!

Hope this helps (including the rant).