ability to mount backup history as a virtual drive. It’s nice and frequently requested feature on duplicacy forums.
support to run server side component to facilitate ACLs.
small server space usage
Disadvantages of Kopia:
does not support extended attributes driven exclusions (for example to exclude all Files skipped by Time Machine on a mac). My suggestion to implement that on the forums was met with complete silence.
rather complex multilevel datastore structure that if corrupted not clear how to recover.
no stable version yet — so not suitable for production
very limited support for cloud destinations (no *Drive type services. Can be workaround with rclone destination but this adds a layer of complexity).
mutable chunks(blobs). the chunks store will be gc and reorganised to decrease server storage usage
unclear future and support model — most questions and requests are answered by “here is source, pull requests are welcome”. The dude works at google and develops it on the weekends. At which point will he throw in the towel?
Generally, not very friendly support forum — this is subjective of course.
duplicacy
Advantages of duplicacy
backup configuration is stored on the client. this is convenient if you have multi clients and copy one from another
erasure coding making possible use of ureliable storage (hard drives) as backup destination a
simple datastore structure. Easy to recover from corruptions
tons of cloud destinations supported, including google drive
support for asymmetric RSA encryption (different backup and restore keys)
rather simple to make it work with immutable storage
super fast exclusion patters, including regex
out of the box support for excluding known transient files (such as TimeMachine exclusion by xattr) — this drastically reduces the need to write custom exclusion patters.
has stable version and exists for a long time.
outstanding suppprt where the developer responds quickly and fixes issues users face even while using the products in most weird and unsupported scenarios (I’m one of those users)
friendly community.
Disadvantages of duplicacy:
most advanced settings are not in UI, and some UI areas require a lot of rework (exclusion patters, restore workflow)
does not support mounting the backup history as volume
Thanks for a comparison!
Unless I misunderstood the Duplicacy state, there’s another advantage to Kopia. It’s actively developed. This partly offsets the ‘no stable version’.
There goes your “advantage of active development”.
Another tidbit: support for filesystem snapshots on macOS. With Kopia you have to write scripts to make it work. Duplicacy supports it out of the box.
I wanted to like kopia, I played with it. On paper it is a superset of duplicacy – both in functionality, flexibility, and complexity. In reality – it’s a mishmash of features, and looks more like a playground for developers and their egos than a user-centric product; it’s needlessly over-engineered, while being permanently not ready. (Duplicati, anyone?). I do appreciate the overkill approach to development though. Slack, discourse, CI, docs, whole nine yards. Looking at recently closed bugs – it won’t be stable for a very long time. I gave up waiting.
Duplicacy on the other hand – simple like a rock, clear architecture, very supportable. Does not have a million features (like fancy inheritable exclusions, mountable history, and compression selection by types) but has a few that actually matter to the end users. If I corrupt Kopia datastore, as I did a few times while playing with it, I’m at a loss. With Duplicacy – it’s trivial to fix. Both of them are CAS, and yet, all that complextity translates into unsupportability.
Amazing, I started writing this to defend kopia and yet…
I echo your words. I went through the exact same process (playing around with kopia - and other products too, over the last few years) and came to exactly same conclusions. Duplicacy’s design/architecture, based only on the filesystem, is rock solid, and that’s what matters in a backup software. I don’t know of another similar solution on the market, and I’ve looked around a lot. Yes, it lacks some features, but it’s a more than acceptable trade-off.
The day duplicacy has the ability to mount a backup to the file system is the day (in my eyes) this product becomes complete… until then… i’ll continue using it and paying for the privilege because in my personal limited testing… it’s still the best tool out there for my needs.
I’d put Archival Storage support first, Mounting Backup – second.
In fact, it does not even need to be an actual virtual filesystem, as originally envisioned. Today most OSes provide API to implement a file provider allowing to natively integrate to the rest of the system (example) without the need to mess with VFS and kernel extensions and all that horrible unstable nonsense this has been classically approached (see Box.com client (very good) vs Google Drive client (horrible crap, that only got worse with migration away from fuse, something nobody thought was possible (I mean, getting worse, not migrating))). In fact, I would not use it myself unless it is implemented via modern and lightweight OS interfaces.
At the very least – wrapping existing functionality into a nice usable native UI will accomplish most of what users actually want: searching for a file, seeing its version history, comparing, restoring it by dragging (see how Arq accomplishes this).
This is a dead end, not worth spending efforts on. Custom kernel extensions are discouraged and are being deprecated across pretty much all oses. Therefore fuse based solutions are nonstarters.
I won’t pretend to know much about this, but Rclone uses fuse for its mount command, and works exceedingly well. Are you saying there’s a better alternative that’s cross-platform and has a readily available Go library?
This is a bit of a tangent from the original post, but since we’re already discussing Duplicacy being able to mount its snapshots as a regular filesystem…
FUSE, while being a kernel extension, the virtual filesystems that interface with the kernel module are not (hence why the acronym is short for “filesystem in user space”).
The original intent behind FUSE was to allow kernel developers to test changes to existing filesystem modules and to develop entirely new filesystems in user space without having to recompile their running kernel → reboot → test → rinse and repeat each time, and avoid the risk of crashing their development systems (20 years ago virtual machines weren’t commonplace and a VMware Workstation license was $299-$329).
After FUSE was officially committed into the mainline Linux kernel (2.6.14 in October 2005) a lot of creative developers have released all kinds of handy tools. In addition to Rclone, some others include…
bindfs – Make directory and files readable by a dedicated Duplicacy user without altering the original file permissions.
mp3fs – Store your original music files in FLAC format for Duplicacy to back up while streaming them on-the-fly in MP3 format.
Besides Linux and a number of other Un*x systems, FUSE implementations are available for Windows, macOS, Android, Chrome OS, etc. The last time I checked there were over 800 FUSE modules available including Oracle’s ODFS (Oracle Database File System) product. I’m pretty certain there’s no risk of FUSE being deprecated anytime soon.
No, on the contrary. Apple continuously deprecates kernel extensions in favor of OS supported interfaces for various services. This is the right approach because installing third party software into protected environment is not wise. I’d say, it’s reckless. Modern macOS boots from a readonly protected snapshot, and installing kexts is increasingly and deliberately made difficult. Deprecated Kernel Extensions and System Extension Alternatives - Support - Apple Developer
There is Apple Developer Documentation to support cloud storage. Big players (I’m familiar with box and google drive, so focus on these) moved away from osxfuse – box took advantage of the file provider api, but google drive for some reason decided to serve data over local SMB server (what? yes. It’s buggy.). Box before relied on their own kernel extensions and both performance, stability, and user experience was horrible: It’s not easy to write kernel software. Now box.com client is one of the best I’ve ever used. Stable like an iceberg, lightweight and “just works”.
This is the way forward. Third party software that is increasingly migrating to that new API: ExpandDrive is one of them: the ExpanDrive is fuse based product, and StrongSync (horribly named) is FileProvider based one. Performance and reliability difference is day and night.
The only caveat is that sparse files and streaming is not supported. You have to gulp the whole file at once. This is not a critical requirement, if the alternative is kernel extensions and stability nightmare.
There was a feature request for rclone to implement that https://github.com/rclone/rclone/issues/5158. It did not go anywhere. I stopped using rclone before that for other reasons, but this was a deal breaker going forward.
You might say that this is macOS specific – yes, it is. I’ll elaborate in the next comment on other OSes.
disclaimer: below is a strongly opinionated comment.
The problem is not that the fuse will disappear – as you described, its serves its purpose well, there is continued development, and well-defined uses. But its applicability is limited to these niche uses.
Those are irrelevant in the context of commercializing user facing backup program that needs to appeal to wide audience: it must be user friendly, stable, with great user experience, and commercially viable. Now lets look at target audience: (I’d love to see actual breakdown of duplicacy revenue by OS – if gchen could share, so guesstimating):
I believe vast majority of end users who will end up buying duplicacy don’t use unix: they use macOS or Windows. macOS users are more likely to pay for (quality) software (I’ll find citation), and windows users due to sheer numbers also may generate sufficient revenue. ChromeBooks – forget it. People buy them because they are cheap. They may justify it in many different ways, but ultimately – it’s just price. Everything else is awful there, starting with user experience. Users don’t care. They won’t pay for a backup program either. They’ll use google sync and call it a day. Or use some other stuff – like duplicity+DejaVue.
Now between these two:
MacOS deprecates kernel extensions, in favor of purpose built OS interfaces, so osxfuse not only does not have a future there, it lost the present: installing kernel extension, rebooting the mac, configuring security exceptions – all just to be able to see your files in a backup program (and paid one at that!!) is the opposite of good UX. If I attempt to install an app and it tries to sneak in a kernel extension – in trash it goes right away. So, on macOS fuse is effectively dead.
ExpanDrive released SugarSync to retain users, but that software is not a paragon of good UX for other reasons.
Windows was moving into similar direction last I checked (about a decade ago; I’m not well versed in the state of affairs on windows today, so cannot comment on details), for similar reasons, and maintaining fuse stability will be cat and mouse chase as well.
Therefore, for duplicacy to be successful, it needs to provide native integration into os-specifc services and be a good OS citizen. Fuse does not have a place in that universe.
Good question. And end users could not care less about an answer. If there is no go library – that part needs to be implemented natively. There is no requirement that the whole thing has to be written in one language. Maybe it’s easier for the vendor to maintain one codebase – but again, as a user – not my problem. AgileBits made that choice with 1Password8 – and lost at least one long time paying customer. At least.
In fact, go’s goals were completely different. It was designed for services, not user interaction. And it excels at that. Duplicacy CLI benefits from from it, on every platform, and it’s great. Duplicacy-web, on the other hand, is horrible on all OSes. Maybe eventually there will be natives bridges from AppKit to go, but who is going to wait?
So, ultimately, the answer to this question is irrelevant. What matters is – users won’t install kernel extensions (and those that would and don’t see a problem with that – are those who definitely should not install those) and therefore Fuse based solutions today is a dead-end.
The whole point of fuse-based vfs is that it can be used by any application that is only aware of regular filesystems. This opens up specific backends to applications you won’t be able to use otherwise - scripting, file managers etc.
This is why rclone mounts are so great - all these cloud and composite backends can be used by any application that can work with regular files. And it works very well on *nix and Windows. Can’t say that I care about macOS.
You’re living in cloud cuckoo land if you think a single developer should embark on writing native code and UX for a backup tool that’s already comfortably cross-platform. The principle alone reminds why I thoroughly disdain Apple products - they don’t believe in common standards and they go out of their way to make it difficult for power users to control their own systems. Yea, nah. I don’t particularly care about macOS either, but Duplicacy and fuse is cross-platform, and duplicacy mount only needs to be read-only.
I hesitated which response would be best here, so here are all three:
If you want good UX – it has to be native. No (easy, feasible) way around it, as illustrated by a multitude horrible electron apps flooding the market.
Does not need to be the same developer, it can (and likely should!) be outsourced to a reputable UX design firm. Rarely a single person combines both system and UX development talents, and that’s OK.
In spite of the previous bullets, some competitors do singlehandedly pull that off, and rather successfully.
Those who discover cross platform frameworks (QT, Electron, and similar nasties) and believe that this will solve all their problems, end up with smallest common denominator, software horrible on each platform, because in the name of questionable short term gains in development time trade quality for quality and lose customers.
Of course we are talking here strictly about desktop, user-facing software. Not services, tools, utilities (including duplicacy CLI!), servers, databases, etc, which greatly benefit from being cross platform in general, and go in particular.
In other words, I strongly oppose cross platform frameworks that create and illusion that system specifics don’t matter. I’m willing to die on this hill
With all respect, you have probably haven’t used macOS in the past 15 years, because neither of these two statements reflect reality. In fact, they can’t be further from truth, for quite a number of objectively verifiable reasons. This popular on reddit notion that “apple does not let me do whatever I want with the system” and “it creates incompatible gadgets just to screw tie the rest of the world” is dated, to say the least, and probably is not worth arguing over here.
Fair enough. I, on the other hand, only care about macOS. (Yes, I have used/use windows, linux (Fedora and Debian), and FreeBSD for a variety of reasons as part of personal and professional life, and it’s ridiculous to me to even imagine that someone would willingly pick any of them as their main desktop OS!. Whether duplicacy works on 5 different other platforms is irrelevant to me, as a user. It’s all implementation detail: are there three different separately designed products from ground up or code shared with some weird frameworks partially or entirely – does not matter. On MacOS FUSE is dead for all intents and purposes when wide adoption is concerned, simply because it requires third party kernel extension, so if duplicacy wants traction on macOS – there is no other way than a native client. Google and Box stopping using it should be a rather obvious sign that the times have changed.
Now, whether duplicacy should care about macOS users – it’s not up to me or you to decide. But I hope it does.
I’m not arguing virtues of VFS, I fully understand their merits, benefits, purpose, and drawbacks.
I’m arguing that relying on Fuse on macOS is not an acceptable way forward if wide adoption of the software is expected, outside of few geeks.
While today it’s only macOS, in the future this will extend to other OSes, including Windows, that care about security. Simple as that. FUSE, therefore, is not future proof for end users apps that want wide adoption. Again, pointing to Google and Box who stopped using FUSe in their end-user facing products.
For a start, what I want is a decent, cross-platform, backup tool - not a pretty little UX that has to be constantly maintained after each major OS upgrade breaks something, or forces you to comply with a new set of material guidelines. Functionality matters most over form.
Electron is the worst example but you could hardly say it’s ‘losing customers’; look at Discord. Anyway, these two aren’t the same in terms of performance - QT has some quality apps behind its belt, and might’ve been a good choice for Duplicacy when the legacy GUI was abandoned.
Regardless, Duplicacy Web UI, is a different beast. Contrary to what you say, we’re not strictly talking about desktop here - with the latter, you can control a remote instance on a NAS or server. Probably around 50% of the Duplicacy licences I’ve bought are for servers! This isn’t a small sector, what with docker and web apps increasing in popularity.
The decision to migrate Duplicacy GUI to a web app was absolutely the right call. May not be slick, and the framework might not be the best choice, but the fact it brought *nix users into the fold, works with NAS/servers, as well as reducing most of the dev work to a single code base, made utter sense.
With all respect, I’m talking about all Apple products and its ecosystem. I’ve owned an iPhone - it cannot do what I do very easily on my Android. One great example is taking control of my own data; using SyncThing with KeePass is a fantastic mess on iPhone and you’re expected to “deal with it” or use their proprietary cloud bs. Hard no.
The fact fuse is harder to work with, only further demonstrates my point. Increasingly, users are given less options and, because it’s a walled garden, have nowhere to go, too invested to escape.
I don’t see how are these two related, let alone mutually exclusive. If one part of the software I see and understand (user interface) is sub par, why should I have any confidence in the parts I don’t see nor understand (backup engine). In fact, I dismissed duplicacy years ago simply on the account of atrocios UI (in pre-webUI days). It took me the whole 5 seconds to move on. I came back to try it again due to sheer luck. How many more potential paying users got filtered out at that stage?
Exactly. For system admins and geeks, text or web based remote access is the paragon of UX. Duplicacy does it, these demographics are already used to unusable repulsive interfaces all across windows and linux, so they will be happily using it. I too, in fact, in that role, preferred ssh + duplicacy CLI when managing backups on remote servers.
This group however is a minority, and the subset of them who cares about a web front end for a backup tool is even smaller.
The group of home users on the other hand, who want a big button – backup my data to the cloud and an easy and intuitive way to restore (see Time Machine), with the learning curve you can’t see behind the horizontal axis – want simple and familiar user experience.
Nobody said that there can only be one interface. Needs of remote system admins and home users differ, hence they need different UI. IN fact, when wearing my admin hat, I’d love to have a native tool on a Mac, that will talk remotely to the instance of the service I’m managing. Not a copout in the form of a web page.
This is strictly not my problem (I’ve put on my user’s hat now). I’m gleefuly not using the product, developer happily saved time developing. I’m a developer myself, and therefore I can say those things. Never compromise on user experience. Of course, if the goal is to make quick buck and then abandon the product – sure, optimize returns. Crappy unified UI with a single codebase will be a way to go. I would not, and did not do it to my customers myself.
I’d love details, but it sounds like you are tryign to do somethign that may not be such as good idea. I’m using an iphone (and before - windows phone, before that – blacbkerry, and before that – BREW based phones) and I don’t feel constraine in any way.
It’s a keepass and syncthing problem, that they can’t attract developers that know what they are doing.
1Password happily syncs via dropbox, icloud, and local wifi, but you are right – why bother when built-in keychain works and with the privacy and security far superior due to hardware integration?
I don’t get this attitude. Why do you prefer to waste time with synthing and other thrid party products if the ecosystem already provides all you need, and it just works, becuase someone else is paid to make it so.
Anyway, iOS is not a point here, macOS is.
Emmm… yes and no. Users are given less options by default, to protect said users. I don’t want my grandma installing fuse anywhere. Seriously. I don’t even want her to have admin access. Power users on the other hand are not restricted by anything. Indeed, if the user cannot figure out how to disable SIP – they have no business disabling it.
And I love walled gardens. I don’t want to configure, debug, and triage software I use daily, mess with configuration, tweaks, and mods. I want stuff to work, let me do what I want, and go on with my life. I don’t want vendors pointing fingers at each other when something goes wrong. Enter walled garden. Everything works, across devices and applications, flexible, top-notch security, and I don’t feel restrained in any way. Literally, anything I could think of doing – not once hit I any wall. (Except PC-only games, who has not yet compiled macOS version. For that—I still have a bootcamp partitions and crossover).
Anyway, the point being there are much fewer geeks and admins combined compared to general population. And general population needs a good, easy to use, performance, and reliable backup software, with native UI and tight integration into the OS.
We want Time Machine in the cloud, so to speak. Duplicacy hits all points here except the “easy to use”, “native UI”, and “tight integration”
If that is accomplished — I would happily pay $100 annually for a license to use it, and will tell my friends. Admins and geeks can keep using ugly web UI.
P.S. (because it’s not relevant to a discussion but I can’t not say it)
Yea, I don’t want to escape. And yes, there is nowhere to go. There was Windows phone — Microsoft killed it. Android is not an option. (I worked with android on an inside, specifically, graphics/display pipeline, with one of the SoC vendors). I refuse to have anything to do with it. I’m not kidding, I changed employer purely on this basis. Yeah, today it’s probably better, yada yada, but still, as you said – hard no.
Wow… You are indeed VERY opinionated… Can’t say I agree with any of your opinions. macOS does indeed sound like the right OS for you though if you like everything entirely locked down. I for one like to be able to do what I want to do with the hardware I own. Obviously, I VERY strongly disagree with your comment about Apple having “changed” in some way in this regard in the last 15 years. As you implied though, I don’t think it a worthwhile endeavor to debate this point since I’m sure neither of us will change our viewpoint.
While I am certainly no fan of the direction Windows has taken in recent years (in fact, I despise it), it still remains my daily driver on my primary machine for gaming/productivity. Everything else I own runs Linux. Again, I disagree with your sentiment that Linux based OS’s are somehow inferior to macOS. If that is your subjective view, fine… but needless to say, there are millions that disagree (and to be fair, millions that DO agree).
As I’m sure I’ve already made very clear, I completely disagree with your sentiments about cross-platform frameworks. There are a variety of cross platform frameworks which do their job well enough and absolutely serve the purpose they were created for. They save developers time, money, and effort. Not every app needs a slick, modern looking UI (not that some of the frameworks cannot do that). Many just need a functional interface that can serve as a user friendly alternative to their command line utility. I will admit that I’m not the biggest fan of Electron… but it has gained popularity for a reason. Not every developer can afford the time/money/effort to write a custom UI for every platform… nor in most cases would that be a wise investment of said resources.
In regards to FUSE and rclone… Again, your arguments against them seem shortsighted and lacking in substance. There is a reason rclone has gained enormous popularity and personally, I think it is a great utility along with its FUSE implementation. Being able to implement a file system in userspace, using a framework that simplifies its implementation is invaluable.
I’m sitting here debating whether I should even post this now after re-reading your posts… It is clear that you are a VERY typical Apple user/fanboy. I do not mean that as a bad thing either… I’m just saying that you make it very clear that you’re of the opinion that if you want to do something that Apple does not let you do on their devices that it must “not be such a good idea.” I’m sorry but that is so typical of the Apple fanboy paradigm. You’re literally arguing that the only code worth running is that which Apple deems “worthy.” That is so antithetical to my view that it is clear that trying to have any debate with you is going to be a complete and utter waste of time. Again, I have no problem with people who use Apple devices. They work well for people who just want an easy to use, simple experience for doing normal, every-day tasks. However, acting as if the number of technical/savvy users, who want to do more with their devices, is insignificant or unworthy of even considering is ridiculous. You enjoy your walled garden and I will enjoy running the software of my choosing and be responsible for my own security. Again, don’t really even know why I’m posting this at this point other than adding another voice to the argument that strongly disagrees with almost every point you made. Good day and good luck though. Certainly no hard feelings from my direction. I just simply disagree.
Advantages of kopia:
Disadvantages of Kopia:
or use their proprietary cloud bs. Hard no.