How to ignore symlinks (cant restore backup)

Support
#1

Helllo, I have two questions:
1.) How do I restore without creating symlinks (ignore them?)
2.) If that’s not possible, how to set symlinks not to be saved during backup?

My user case:
I use Duplicacy for fast frequent backup of C:\users folder on Windows, specifically appdata folders with -vss

What I’m trying to do:
I would like to restore the entire c:\users backup to another disk, so I can explore what’s big in the backup, etc. I want to restore only data without symlinks, so that the real size of the data in the backup can be reliably calculated and there is no risk of affecting the host system and data

My approach and problem:

  1. I created new exFAT partition for restore - because exFAT don’t support symlinks

  2. IN GUI I started restore to exFAT partition

  3. I get error and nothing is restored (only empty root folders)

The restore command encountered an error:
Can't create symlink All Users/Application Data: symlink C:\ProgramData \\?\W:\appdata_26.11-8110-restore-exfat\All Users\Application Data: Incorrect function.

Exit code: 100

and this error in log:

ERROR RESTORE_SYMLINK Can't create symlink All Users/Application Data: symlink C:\ProgramData \\?\W:\appdata_26.11-8110-restore-exfat\All Users\Application Data: Incorrect function.

I tried some parameters - same problem

-persist
or
-ignore-owner

If I try run Duplicacy without elevation, I get:

2022-11-26 12:12:29.308 ERROR RESTORE_SYMLINK Can't create symlink All Users/Application Data: symlink C:\ProgramData \\?\W:\appdata_26.11-8110-restore-exfat-user\All Users\Application Data: A required privilege is not held by the client.

Any idea how to ignore symlinks during backup or restore?
Is there an equivalent to the Duplicati setting symlink-policy ?

Thanks

#2

Sorry there isn’t an option to skip symlinks. You’ll have to exclude those symlinks (I guess there aren’t too many?).

#3

Inside every user folder is aprox 10 symlinks (appdata,documents etc)

But, if I need to restore content of C:\Users\user\AppData\Local folder, I cant restore that folder in Duplicacy because:

  • Duplicati will restore full path, so I cant exclude symlinks. If I want restore folder C:\Users\user\AppData\Local then I get error because parent folder is symlink
Can't create symlink user/AppData/Local/Application Data: symlink C:\Users\user\AppData\Local \\?\W:\appdata_29.11-8110-restore-exfat_local\user\AppData\Local\Application Data: Incorrect function.
  • And I cant restore all subfolders without root folder …I can restore “Local” folder subfolders, but I would have to repeat the restore 60 times for each subfolder to complete the restore… (because web gui don’t support selecting multiple folders)

I’ve found a lot of mentions on the forum about symlinks and problems with recovery because the “ignore errors” parameter is missing, but I wouldn’t have thought the situation was that bad. Personally, I will reconsider using Duplicacy in the future because one thing is super backup and the other is recovery options.

This symlink problem could also be seen as a security flaw. Even non-admin users can create symlinks in Windows, even to external drives. And if symlinks can’t be batch ignored during Duplicacy restore, I imagine it will be possible to block Duplicacy restore using symlinks to non-existent locations…

#4

Is this specifically a Windows problem? I am able to successfully restore a directory tree containing a broken symlink, with the backup and restore operations happening on different Linux machines. The symlink is restored and still points to the correct nonexistent location, which is how I would expect/want it to behave.

#5

Windows unfortunately puts a total of 10 symlinks into the user’s home directory. And they are always absolute symlinks, not relative ones. So it would be handy if there was an option to automatically skip all symlinks during duplicacy backup.

#6

Duplicacy always follows first level symlinks and never follows second and lower level symlinks.

So you could configure backup of parents directory (C:/Users) and only include your user’s folder, excluding all others.

+fritzmg/*
-*

Another solution is to create a completely separate folder where to initialize your repository in, and symlink C:/Users/fritzmg into there. (Along of possibly other things that are separated by nature — such as network drives or other local disks — but are handy to have in the same backup)

1 Like
#7

Ah, I see, good to know! I ended up only backing up specific folders from my user home anyway.

#8

Btw I stopped using Duplicacy on Windows computer because restoring C:\Users\ folders is so problematic. And I’m quite sorry that there is no change since 2022.

#9

The solution was described above: to exclude symlinks during backup (preferred) or skip errors during restores.

There is no automatic way to skip all symlinks because that would be dangerous. There are only 5 or 7 patterns of those symlinks. Duplicacy could ship with them excluded by default but it isn’t, and it’s probably not worth anybody’s time maintaining that ruleset . Arq attempted this (to provide default exclude list) and its is almost immediately obsolete and needs to be augmented from time to time. That’s a chore you need to put on your calendar. Part of maintenance. Annoying, but any alternative is worse.

If the parent folder is a symlink then obviously you should not have backed it up, let alone restore anything from it. Restore the data to original location instead.

Windows creases a lot of symlinks for compatibility with ancient software. Absolutely all of them can be ignored.

Tangentially, I would be very careful backing up APPDATA. 90% of stuff there is garbage, and the remaining 10% almost always misplaced becuse nobody reads documentation when developing windows software it seems. I have seen user data put in transient folders and temporary caches into users documents. It’s all over the place.

You would need to selectively pick up stuff that matters anyway, so this problem would solve itself. Or the alternative is what you do now — backup all that trash and take a hit on performance, storage, and mild annoyance during restore

#10

Thanks for the reply.

Just to clarify — the issue I’m describing is not during backup, but during restore.
The backups already exist, some of them several years old. What I need is to restore a snapshot to another location (for example on a separate disk) in order to inspect or compare its contents.

Since Duplicacy doesn’t currently support mounting a snapshot version for browsing, the only practical way to explore or analyze a backup is to perform a restore.
That’s where the problem appears — if the target location is on exFAT, or if symlinks cannot be created, the restore fails completely, even if all I want is to restore the regular files.

Suggesting to “exclude symlinks during backup” doesn’t really help in this case, because the backups already exist, and this restore limitation is not documented anywhere.
It’s not reasonable to expect users to have excluded specific Windows system symlinks years ago — especially when Duplicacy ships with default exclude filters for C:\Users folders, which implies that these paths are intended to be backed up.

I’m not asking for Duplicacy to follow or recreate symlinks — only for an option to ignore them during restore, so the process doesn’t stop entirely because of one system link.
A simple --skip-symlinks or --ignore-errors flag would make restores much more reliable in these cases.

For comparison, other backup tools such as Duplicati or Restic don’t abort the whole restore if a symlink can’t be created — they just log a warning and continue restoring the rest of the files.
That kind of “skip and continue” approach would make a big difference here. It’s much better to recover 99% of data and skip a few system links than to have the entire restore fail.

Thanks again for your time and for moderating the forum. Have a nice day : )

#11

You don’t have to restore to compare files — see duplicacy diff

I’m wondering if adding a -persist flag to restore will help here. It allows to ignore certain failures and proceed. I’m not sure if failure to create a file is one of such ignorable failures. But it’s worth a shot.

  -persist                      continue processing despite chunk errors or existing files (without -overwrite)

https://github.com/gilbertchen/duplicacy/wiki/restore

#12

The issue I have is that it still backs them up - the symlink itself I mean, not the content it is pointing to. During restore, those symlinks become invalid.

#13

This is by design. On most filesystems backing up and restoring symlinks makes perfect sense.

Those symlinks you refer to, however, are probably directory junctions and to restore them properly admin rights are needed at the very least and maybe some other special handling.

I’m not familiar with windows enough to comment on this further.

If they become invalid after restore it’s a bug. They shall either fail to restore or restore correctly (as long as you restoring to the same path).

Iirc windows uses absolute paths in symlinks so if you are restoring to a different directory — of course the symlinks will be broken.

#14

All you need is the SE_CREATE_SYMBOLIC_LINK_NAME permission, which you can add to your user with a tool like Polsedit. And they weren’t junctions, they are directory symlinks ("<SYMLINKD>").

It does look like an odd bug, or some kind of other permission issue. I will try to investigate further.

Windows is capable of creating relative symlinks (Windows not being able to do that is a myth caused by a quirk of the respective API function) - but in this case they were indeed absolute symlinks. The backup was restored into the same directory as before.

1 Like
#15

Looking at it again I see now that the symlinks were restored as file symlinks (<SYMLINK>) instead of directory symlinks (<SYMLINKD>) - but I think the reason is that the symlinked target was not included in the backup as well. So when the target was actually restored otherwise (the target were PHP dependencies installed via Composer), the symlink still wouldn’t work as it was the wrong symlink type.