Did a basic test backup and restore on a file with a single ADS, and there was not ADS on the restored file. Does Duplicacy support backing up/restoring files/folders with ADS streams ?
If not, can you add this feature, as you can sometimes lose a lot of relevant data which is stored in ADS stream(s).
I don’t think ADS is supported. I’d argue this falls under extended file attributes, and 
doesn’t support even much more basic attributes (this is not uncommon for backup solutions in general). Semantics is also not obvious for restoring to non-NTFS filesystems.
More importantly, I think this is very, very, niche. Not that I believe that there are absolutely no non-malicious use cases for that, but I personally have never seen one.
Don’t agree.
ADS kind-of falls under the extended attributes umbrella, and a lot of extended attributes aren’t necessary for backup/restore (edge-cases excluded), but some app’s DO store relevant/important info in ADS/extended attribute’s, which mean’s you are losing information in your backup. A simple example to prove the point (yes, I know it’s not realistic, but backup’s should be reliable, and if important information is lost, then it’s not doing it’s job)- suppose some software uses ADS to store an important person’s B’day, and everyone would die if it wasn’t remembered - If my PC blew up and I restored it using Duplicacy - Everyone is dead!
Some attribute’s, like Owner aren’t as important, as although this info might be lost, it can be restored.
NTFS ADS backup is definitely NOT “VERY” niche - you are literally, potentially losing important info. without knowing it.
Most decent backup/copy/move app’s copy ADS as well - File Explorer included. Good one’s use portable method to backup/restore, when dest/source don’t support ADS.
Should be supported in Duplicacy, IMO.
PS. Re. restore to unsupported file-system’s, a lot do support extended attribute’s, so they should be used in most Standard way (ie. use native file systems extended attribute’s to store ADS) - if not, and there’s not a good way of restoring them, don’t restore them,BUT, log/display a message saying they couldn’t be restored, along with some info. that data might’ve been lost, but the original backup still has that data.
niche: adjective
I am still pretty sure ADS usage checks out as very, very niche. So IMO limited development resources are better spent elsewhere, especially considering this is likely to be a non-trivial change. But hey, this is not my product. Also, source code is available, so if this feature is critical for your use you can always contribute a patch, this will significantly improve chances of ADS support getting incorporated into the mainline.
While you obviously have access to a dictionary, and have correctly quoted a reasonable definition of the word “niche”, you don’t seem to be applying it in the correct way.
Is the Window’s OS niche ? I would say no, given it’s currently one of the most used consumer OS by far. The built-in file manager, File Explorer (which is basic, but provide’s standard, most used functionality), has probably been designed to cater for the basic, non-niche usage, but gets the job done in most cases.
Given that, why does this basic, non-niche, built-in tool, support ADS by default when copying a file ? Could it be because it actually isn’t that niche (end-user’s don’t need to be aware of it to be non-niche) ?
Could implement it myself and contribute a patch but, at the same time, it should be a default feature of a backup app (like-wise with other certain file system attributes which cannot be re-created without the source), and ultimately, the devleoper is being paid to do it.
Implementation should actually be relatively trivial given that ADS’s are simply other file’s that are normally hidden from the end-user, and the OS natively supports them, so you don’t have to do some wizardry to access them. One quick Cross-platform/portable suggestion would be to store them in a ZIP (.7z would be better, which should also work with cloud storage + de-dupe). During backup it would check if source/repo is NTFS - if so, then check each file/folder for ADS’s, and back them up in ZIP/7z. On restore, if dest is NTFS, just restore ADS’s (make optional), otherwise decide whether to resore in source’s extended attributes.
The term “Backup” is subjective, but if potentially important data is lost, then it isn’t a true backup solution - why backup with a tool when you lose information/data which cannot be re-created without source ?
As with everything, everyone has their own opinion though.
I will add a 2nd voice to the argument that backing up ADS would be a worthwhile endeavor. While I agree that it is probably not a crucial feature for basic users (since they probably don’t even know what ADS is), lacking support for it means that backups are incomplete. Furthermore, I imagine that a very large majority of users are probably backing up NTFS partitions with Duplicacy which means that they are almost certainly losing information in their backups (whether that information was needed/valuable is of course another argument… but certainly in some cases it is). The issue with NOT backing up ADS is that often you don’t actually know that valuable information was being stored in an ADS until you restore a backup, at which point it is too late. Additionally, I doubt it would take much effort to implement ADS support. Accessing ADS streams is certainly very simple… However, I obviously cannot speak to the complexity of integrating another stream of backup data into Duplicacy. As @sevimo stated, Duplicacy is open source so I guess if it really is a feature I value, I should look into contributing a patch myself. I may just do that… Need to take a look at the code base to see how difficult it is to comprehend (and therefore how much time it will take).
If I’d be a project maintainer (which I am not), I’d be very hesitant to include a feature like that in the mainline. I can support all and any additional storage backends - the more the merrier, because generally speaking, if you don’t use them, they don’t impact you at all. These patches can be buggy as hell, and the vast majority of users won’t notice because they are not impacted.
ADS support would be a very different story. What you’re changing is a definition of a fundamental unit of backup - a meaning of file. Right now, it is just a sequence of bytes with a name attached to it, which is pretty much standard POSIX filesystem semantics, not even attributes are supported. Now, to support ADS this basic unit needs to become a complex entity, with internal structure, multiple streams, stream names etc. This is not just a question of supporting reads from multiple streams on backup. This most likely necessitates changes in the backup format, in restore code, special handling of restores into different filesystems with each filesystem likely to be handled separately, moving away from POSIX semantics. All this is in the absolute core functionality that will be in use by everyone who uses . Any bugs in this code space could be pretty catastrophic, and this is exactly NOT what you want to hear about a backup solution. This could be worth it IF there would be a strong use case for it, but there really is not.
IMHO, the risk is not worth the reward, not by a long shot. Complexity is the enemy of code that has to be robust and reliable. YMMV.
I can certainly understand your view on the matter and I actually agree with all the points you made. I think by default, duplicacy should be file system agnostic, ignoring all non-standard (non-POSIX) file system attributes/peculiarities (other than basic attributes such as timestamps). However, that doesn’t mean that additional optional modules/features couldn’t be added to support file system specific features which could make it a more complete/robust backup solution. For example, I’m sure many would appreciate the ability to backup file/directory permissions, or even ACL data, on unix based platforms. Similarly, some Windows users might want the ability to backup ADS data (or file/directory permissions as well). By making such features non-default/optional, you’re mostly mitigating the risk of such functionality affecting the entire user base while still providing options to those who want it. I’m not suggesting that such features should be a priority or are crucial features that must be implemented but there are plenty of stable/reliable backup products which do support such optional features, making them a more robust/complete package for users who desire said features. That’s just my 2-cents though… You’re free to disagree.
Hi all! o)
I came across this topic, while searching for advanced backup solutions.
Well… o) I think most people are not even aware that they are making use of ADS every day. If you download a file from the internet or from a network location with your regular browser, it will write an ADS to the file you downloaded. It’s also known as MOTW (Mark of the Web) and Zone-Identifier and it’s done by Chrome, Vivaldi, Edge, Brave etc… all major browsers.
Windows “tags” files downloaded from untrusted locations by writing this MOTW in ADS. So it is a very common thing. I use it to automatically sort downloaded files e.g., unfortunately Linux browsers don’t write MOTW yet (in extended attributes e.g.), so a downloaded file in a Linux machine, will not trigger the “untrusted source!” warning if opened in Windows (which is unfortunate, as security is lower and there also is no automation on Linux possible without the MOTW).
If you have a file manager which has some advanced features like “Directory Opus” on Windows, you can even read and process these ADS to your liking, they are very good supported in this file manager. You can list the MOTW internet urls you downloaded from e.g. by enabling columns which read the MOTW directly (very handy!).
The NTFS Alternate Data Streams are around for 25 years on Windows and used by various applications to store additional data, exactly what they were invented for. The general support on Windows of ADS is quite good already.
To make a long story short:
Alternate Data Streams are a first class citizen on Windows. If I cannot backup these ADS thingies, I experience data loss.
All my ratings and tags on files would be gone. The MOTW will be gone, the Windows indexer will re-index and re-hash files if I restore without the ADS, etc. Backing up files from my Windows machines without ADS is a no no. o)
This is just to spread the word for ADS some more! o) Many people still think ADS only has to do with malicious software, but it’s actually a really useful feature (and Linux basically has the same with “extended attributes”).
It helps to play around with ADS to get a better grip on it, you can easily create and edit an ADS with “Notepad.exe” on Windows e.g.
Notepad.exe MyFile.txt
Notepad.exe MyFile.txt:my-description
Thank you! o)
Continuing here, because I was not allowed to post two embedded images in my first ever posting here! o)
I hope this works, this is not spam, this is valuable information for the internet! o)
The mentioned file manager above also makes use of ADS to store “rating”, “color labels”, “descriptions” and “tags” into ADS for all file types, which do not have native support for that (like a regular *.txt file e.g.). It also uses ADS if you rate or tag a folder. If you list the number of AD streams, you can see the number “2” in the column “Streams” for all rated directories (because they use ADS).
I hope this helps to understand the seriousness behind the urge to be able to backup ADS.
