Expanded Symlink Support

[santacruzskim]

CURRENT IMPLEMENTATION
Symlinks at the root of the repository path will be followed, traversing to their target and backing up the contents of the target, even if that target lives outside of the repository. Symlinks anywhere else in the repository (or residing inside of symlink targets) are completely ignored. They are not backed up as objects or traversed.

I am grateful for this “feature,” but it simply uses symlinks as a workaround for another issue; it does not directly address the needs of those with symlinks in the folders they wish to backup.

FEATURE REQUEST
Allow the user to decide how to handle symlinks, either per each backup operation with a “option” flag (preferred, eg. -follow symlink), or maybe per repository?

FreeFileSync implemented and explained it the most simply, so I’ll include their documentation below. Duplicati also allows for the user to determine how to handle this. I was using Duplicati to do just this for over a year before switching to .

WHY?
…why not? In trying to find a solution to my own issues, I have seen plenty of others either in my predicament, or just plain confused on how symlinks and the like are handled. As long as its not the default it won’t get people into trouble. Not many people are going to search this out and use it, and if they do, and they create path loops or other issues, it will be immediately apparent why that is occurring.

It will also clear up a lot of inconsistency around how interacts with symlinks, specifically in the include/exclude dialog box where they are presented to you and you can interact with them as if they were folders, including selecting them for inclusion/exclusion, and even more confusion when trying to use custom filters (not sure about regex yet, but I’m assuming it’s the same). By letting the user specify how to handle symlinks, can use that decision to choose how it displays, filters, etc. symlinks. Either that, or use some sort of overlay icon or other indicator that lets the user know what they are interacting with isn’t a normal folder and won’t be treated as such.

For me personally, I backup all my local machines to a server using UrBackup, which uses hard and soft links extensively, mainly for the purposes of deduplication. I’d like to back up these backups offsite using Duplicacy, but instead of backing up the 2,000 file | 2.8GB folder I tested with, it only pulled 22 files | 8.2MB, skipping the vast majority of it. There are other ways to handle my issue that are specific to UrBackup so I don’t want to bog this down any further with a discussion on that, but suffice it to say, the other options are equally bad.

FURTHER READING (from freefilesync manual)

Symbolic Link Handling

FreeFileSync lets you choose to include symbolic links (also called symlinks or soft links) when scanning directories rather than skipping over them. When included, you can select between two ways to handle them:

1. Follow: Treat symbolic links like the object they are pointing to. Links pointing to directories are traversed like ordinary directories and the target of each link is copied during synchronization.
2. Direct: Evaluate the symbolic link object directly. Symbolic links will be shown as separate entities. Links pointing to directories are not traversed and the link object itself is copied during synchronization.

Note

• Under Windows the symbolic link options apply to symbolic links, volume mount points and NTFS junction points.
• Copying symbolic links requires FreeFileSync to be started with administrator rights.

–end

thanks!

[santacruzskim]

to add to my own discussion, alasdair requested the following in a bug report thread, #4 being the one of most interest for my sake:

Excluded symlinks are still included in backup

1. Duplicacy should correctly display and differentiate symlinks from files and folders in the include/exclude browser. (including support for both hard and soft links)
2. Duplicacy should be consistent. If something is going to be shown as a directory, then it should be excluded with directory syntax.
3. Duplicacy should create correct include and exclude patterns for symlinks
4. Duplicacy should provide some configuration for handling symlinks.
   • backup as symlink rather than following it
   • backup symlinks as directory
   • ignore symlinks which link outside backup root
   • ignore symlinks entirely

[ChrisN]

This would be perfect for my situation. We already use FreeFileSync, and it handles our archive symlinks beautifully.

Right now, I am backing up the archive files separately, and fortunately I don’t have a complicated setup so this works fine. However, symlink support would allow for more seamless restorations, since I wouldn’t have to figure out if the file is in our live or archive storage before I can restore it.

Add my +1 to this feature…

[Christoph]

For anyone who hasn’t seen it yet:

Proposal: following symlinks by pattern

This is to address issues brought up by recent posts such as Expanded Symlink Support and Excluded symlinks are still included in backup My proposal is to add a new field to the preference: ... "symlink_follow": ["^dir1/link.*", "^dir2/[^/]+$"] ... The value is a list of regex patterns. If a symlink matches any regex pattern in the list, then the symlink will be followed. If the target is a file, the target file will be backed up. If the target is a directory, the target directory…

[santacruzskim]

Thanks for the heads up @Christoph. I still need to wrap my head around what the proposed feature update will mean for my own use cases, but on the surface I don’t think it addresses the fundamental request of allowing the user to flip a switch and let traverse symlinks. It seems to only let users specify specific instances of symlinks via filters which, though theoretically more configurable, doesn’t include the basic option of “follow all symlinks” and seems unnecessarily complicated.

If you have 100+ symlinks with no underlying similarities, I’m not yet sure if this will help. Also, if symlinks were simply treated as normal folders, then they would also follow include/exclude rules so you could still drill down and explicitly control individual instances to your heart’s content.

I’ll add my feedback to the proper thread once I’ve had time to digest and make sense of it all. Leaving the thoughts and ramblings over here in my feature request Feel free to straighten me out if I am misunderstanding things.