Lack of updates indicates stability, which is a good and desirable thing you want from a backup program.
What activity are you referring to? The forum is pretty active, roadmap is published somewhere too…
Please clarify 
Edit: links for reference:
2 Likes
If you referred to the web GUI, the latest version 1.5.0 was released January this year. This version has been pretty stable and no critical bugs have been found, which I’m happy with.
For the CLI, I just released this PR last month . As you can see from the PR, it is a really big change (basically a complete rewrite of the entire backup engine). I wanted to put a hold on the official release (which will become CLI 3.0) until I receive enough feedback.
Generally speaking, for Duplicacy I have been taking a no-rush approach, mostly because as @saspus just pointed out this is a backup program and the last thing you want from a backup program is frequent ‘immature’ updates that kill the stability and the reputation. Every feature or fix should be well-thought and thoroughly tested, to the point that it makes the development ‘slow’, but I think this will benefit everyone in the long run.
14 Likes
Thanks for your thoughts. It was never my intention, but maybe I was too rough.
Regarding the CLI, it seems pretty stable, and that was not the reason for the post. The web GUI has some annoyances that I assumed the would be fixed sooner than later. I was not aware of the roadmap, btw. I’ll post my ideas on the Feature category.
Thanks.
2 Likes
I’m planning a new web GUI release on January, which will be a minor update. Some planned features/fixed will be include, while others will be deferred until the next major update.
3 Likes
And I think that this is the core of the issue and I agree with them. I myself made a detailed post re. desirable quality-of-life improvements in April of 2020, but I’m not sure that many were adopted. Obviously nobody owes me or any other user to adopt their suggestions.
In the meantime I’m occasionally wondering what the target market and the future for Duplicacy really is. To me it seems right now to be geared primarily toward the enthusiast user, as it required a certain acquaintance with the OS to fix things, when they go out of whack. Perhaps that “gearing” is not intended, because I’d think that more users (and thus money) might be found in the small business market. But unfortunately it does not quite have the polish that competing products have, even though those might be inferior in the core backup functionality.
But the polish is what sells (when your customer isn’t technically competent enough) to evaluate each product on the technical merits. And since a user interacts most with the software in the initial setup stage, flaws in polish become apparent very quickly (and might persuade a user to discard Duplicacy for a technically inferior, but more polished product).
I don’t know how much of their time @gchen is able to devote to this project and how successful it is economically, so I’m not sure, if what I’m looking for is simply out of reach. Perhaps it is. But I hope at least some of that is still within reach, because I want Duplicacy to be successful in the long run (if for no other reason, that I bought lifetime licenses 
).
So I’d suggest to expand the current roadmap and add a section that shows those items that are intended for the upcoming release. Also a faster update cycle would be desirable at least for non-critical UI-features, e.g. two releases per year. But again this of course depends on how much time @gchen can devote to Duplicacy. And I understand @saspus ’ concerns about stability, but I’m more concerned about that in regards to the engine than around some changes in the Web UI.
Anyway just my 2 cents. It’s a great discussion and thanks to @duplicacity for starting it, even though the style might have been a tad polemic.
3 Likes
The next web GUI release is to fix bugs so none of the improvements you suggested in that post will be addressed. I’ll respond to each one individually after next release.
I think less of the target market because I strongly believe that when you build a great product users will naturally come. There is still a killer feature that I conceived during the early development of Duplicacy and haven’t been able to find time to implement since then. This feature will be the game changer and will dramatically expand the user base.
I have other projects such as Acrosync and Vertical Backup if you don’t know. There is almost no development work for them at this time, although customer support can be demanding from time to time. In addition, I have been ‘secretly’ working on a new project, not related to backup or sync, that I hope to release in a couple of months. So my time on Duplicacy is limited, but it should become better once I got that one out.
Business wise Duplicacy isn’t successful considering all the time I spent, but it is still sustainable in the sense that if the revenue (from all my projects) can remain at the current level I should be able to keep doing what I’m doing now. My goal of course is to grow the business so that I can at least hire someone to help me. Certainly this will happen, I just don’t know how long it would take. In any case, Duplicacy is a long-term project and I won’t call it an end until that killer feature is implemented.
3 Likes
Well, that is a bit of sad news. Not because these were my suggestions, but because the UI part of the Web UI is IMHO sorely in need of improvement.
While I can relate to that sentiment (Build it and they will come) to some degree, however the questions are always “Great for whom?” and “Will they recognize the greatness?”. For example right now Duplicacy is not suitable for medium/larger businesses who’d require a central infrastructure to manage the backups on individual computers and perhaps even would want things like key recovery. I also don’t think a non-sophisticated computer user will find this easy to set-up and use, e.g. I wouldn’t trust one of my siblings to get this running and properly configured on their computers. So by default (whether a conscious choice or not) you have for the most part limited your target market to the people who recognize the technical superiority of the backup engine and are willing to put up with the clunkiness of the Web UI. Or those who have friends or family members like that.
Yes, I’ve noticed that you have those projects going as well, I just wasn’t sure how much they overlap and thus how much improvements on one benefit the others. Re. your new project, I wish you much success with that one, though it’s a bit sad to see that my impression (and apparently the OP’s as well) that Duplicacy is getting somewhat neglected has some basis in reality. I understand the temptation to start something new and exciting, because polishing an existing thing feels much more tedious, yet as a user of that existing thing I’d love to see some serious polish action.
I am sorry to hear that Duplicacy isn’t super-successful. It always feels bad, when you put a ton of time into something and it really doesn’t take off as much as you’d like to see. Though I’m glad to hear that things are at least sustainable.
P.S.: Thanks so much for sharing the state of the products and of the business side so openly, I really appreciate that. And I wouldn’t have said anything, if I believed that Duplicacy is a bad product. Quite the contrary. I believe that it is a good product that is on the cusp of greatness, but not quite there yet. And I’m still hoping that it’ll get there one day. Of course as always that’s just my opinion and at the end of the day it’s your opinion that matters. Thanks again.
6 Likes
Unfortunately, this is while necessary is far from being sufficient. There were quite a few technically brilliant projects including from large players with huge marketing budgets over years that flopped.
As @tangofan pointed out – for the product to be successful it needs to target the majority of average users, not just enthusiasts. Yes, the technically good product will be driven by mavens for a while (using terminology from “The Tipping Point” by Malcolm Gladwell, who describes this phenomenon really well) but then it should be usable by “normal” users to get wide adoption and tip. For duplicacy, this means implementing a few low-hanging, very boring, but necessary features in the UI, that will allow somebody’s grandma to use it, without the need to hang out on forums and write blog posts about the experience and share tricks.
I use duplicacy myself for the past few years, and through the magic of working in tech, I appreciate its technical brilliance and have personally recommended and helped set it up to quite a few of my colleagues and friends. Some bought the license, some used the personal CLI version. That last bit is important. I had to sell them on it and provide support in configuring it. At the same time when recently my parents needed a backup tool – I bought them [other competing polished to death native on the OS tool that does not even have a forum but “just works”] licenses. They have set it up themselves, it just works, they can backup, restore and do everything on their own in a predictable way with no ramp up or messing with OS internals.
I can perhaps summarize those missing features in a separate post, but I’m sure they were already discussed before. For example, as of today if more than one user is using a Mac setting up duplicacy to backup both users requires jumping through many hoops (figure out SIP permissions and launchd, environment variables, change ports if ran separately for each user, configure logs, and caches location; then add manual exclusions for Logs and Caches folders (because they are not marked with TM extended attribute) using the terrifying filters editor that requires learning and understanding the exclusion syntax for it to even work correctly. And then come restore time – how many forum posts were there where people could not figure out how to navigate directory structure? How many just gave up without posting anything?) that any brilliant solutions inside the engine become irrelevant. Nobody is ready to finish the journey to enjoy those.
I’m not sure if I ever mentioned that – but when I was looking for a replacement for CrashPlan a few years ago and was testing various backup tools I rejected duplicacy after looking at UI for 3 seconds. Only because every single other tool I looked at failed in actually doing backup reliably I lowered the bar and revisited each tool again, this time focusing solely on the robustness of the backup engine itself. Only this time I got a chance to actually test duplicacy CLI and I’m glad I did. Think about it – the only reason I (and a bunch of friends) are using duplicacy today is that all other tools failed tests that most people won’t bother running in the first place (aggressive stability testing essentially). That’s a very restrictive filter.
BTW, Web UI did not exist then. But today it does, and I understand that it’s based on the off-the-shelf “control panel” component with probably limited customizability and adjusted expectations accordingly – it’s not really most users’ concern.
It’s disheartening to hear, but not unexpected, and for me, the reasons are painfully obvious (and in line with @tangofan’s comments above)
Duplicacy is successful in a small circle of tech enthusiasts who bothered to do a lot of research, recognized the gem behind the rough exterior, and are willing to sacrifice creature comforts and invest time in making it work. Now it needs to be made usable by everyone else, so it can be recommended to their non-tech friends with the chance to be accepted and usable. It needs to be able to pass the grandma test (literally).
I understand this feature is going to be in the engine, and therefore many users will not get a chance to experience it because they will get filtered out a the UX onboarding stage. I’m sure this feature will appeal to and will attract a few more enthusiasts, but that problem does not need solving. Enthusiasts are already onboard. Even doubling their number won’t help. And yet, maybe getting even 1% more of general users might provide more bang per buck - because there are so many of them.
10 Likes
As a non-technical user finally settling on Duplicacy after procrastinating with leaving CrashPlan, the one thing that scares me after reading the forums is (based on my understanding) when Duplicacy will back up happily and report no problems despite referencing zero/invalid/unreferenced chunks. I don’t know how often this problem occurs, but what I appreciate about competitor options is their seeming management of archive integrity; the last thing I want is to use Duplicacy for years only to realise should the time come where a restore operation is necessary that my data is irrecoverable.
Even if the average layperson learnt how to configure backups and run check/prune/restore, would they be expected to know what actions to take and how to perform them if check found problems with chunks/hashes? I’m used to programs just re-uploading the data automatically in the background, or picking up where things left off if a process is interrupted, to the point where I would feel more secure in simple cloud syncing with long retention periods for changed/deleted files.
1 Like
I don’t think this ever happened with well behaved remotes. I.e. if duplicacy send the file and remote confirmed that the full file was transferred and saved when in reality it wasn’t — there is not much duplicacy or any other software can do. You have to trust remote at some point.
I don’t think it was ever reported for anything but pcloud and some obscure ftp servers running out of space.
From personal anecdotal experience using duplicacy for few years — I never lost data. All the issues were superficial — like interrupted prune that left half-deleted snapshots in the datastore that later caused failed check. It’s still a duplicacy bug — but it does not cause data loss, on the contrary, it fails to delete what it should. But then I purposefully avoided using flaky storage: if targeting NAS — it must have checksumming filesystem and periodic scrub; if targeting cloud — no crappy cheap providers. I have used/am using as targets Google Drive, Backblaze B2, and Amazon S3. No data loss in all these years. It has been solid.
It is not unexpected though that your backup is only as reliable as underlying storage — so there is that
2 Likes
May I ask which polished alternative you went with?
I’m also one of those somewhat technical users who appreciates duplicacy for the things it does well but would also like to chime in that i’m also disappointed by the lack of features and usability updates to the web interface. It completely lacks any form of discoverability, expects the user to pass through command line options to achieve many desired basic functions and is just generally very confusing to use and understand. I was forced to purchase a license for Arq for my parents machines and push other friends that direction as well simply due to the increased polish of the interface. I understand the author has their own opinion on these things but I suspect the majority of potential users would greatly appreciate a fully revamped GUI of some sort. If this were to happen I could easily recommend duplicacy to everyone I know. As things are now i’m forced to look at things such as Arq, kopia, and even acronis.
7 Likes
Thanks for writing this. I couldn’t agree more. I’ve been monitoring the activity on the forum as an indicator for duplicacy’s user base. I’ve been patiently waiting for that tipping point that you mentioned, i.e. for the moment when the user base starts to increase significantly, but it hasn’t happened. I found that somewhat puzzling, given the quality of the product, but your post solves that puzzle.
BTW, there was a question for you in the post right above, in case you missed it.
2 Likes
I wasn’t sure if advertising a competitor is appropriate here. But heck it was already mentioned. So why not. It’s Arq 7.
I had massive issues with arq5 — in my tests (where I ended up with duplicacy) it would not even complete a backup; arq6 was a disaster with electron ui I did not even bother trying, but with arq7 it’s a compete 180 from that. Its literally my favorite program. Not favorite backup program — just favorite program. I’m running it in parallel with duplicacy targeting a mix of hot and archival storage in AWS, (waiting patiently for duplicacy to catch up…)
The attention to details there is phenomenal, it “just works”, does not need a forum; heck it even supports automatic backup and restore to and from Amazon Glacier Deep Archive with cost monitoring, and little things like user impersonation to read data from per-user mounts and user mode filesystems. On macOS it uses native APFS snapshots, something that duplicacy should have done long ago (I’ve actually suggested here that few years back). This requires talking to Apple and getting an app reviewed to get approvals to access snapshotting API.
It’s quite a bit slower than duplicacy in scanning the filesystem, but as we’ve discussed before — speed is great but far from being instrumental for the product success. For me it takes 5 min vs duplicacy 50 seconds, and I’ve slowed it down further to 30 minutes. No need to hurry.
The only advantage that Duplicacy has today besides speed is concurrent backup from multiple machine to the same datastore. For most users this is not a dealbreaker whatsoever.
Here are some of the features that I need to work out of the box; and those that duplicacy needs to have to compete:
- Handling multi-user environments
- Pause-resume on battery
- Control CPU usage
- Support for archival storage (backup to hot storage exclusively is counterproductive for vast majority of users)
- Backup of user-mounted filesystems when running under super user account (user impersonation)
- Restore workflow. Right now duplicacy does not have a restore UX at all for all intents and purposes
- Filtering that does not require reading manuals and UI controls that never leave filter list in a bad state. Today in vast majority in scenarios it is not possible to create an exclusion pattern using ui control. It’s simply broken from day one.
I perhaps forgot some — but those are basic features that have to be there for a fighting chance at this point and on this stage of product maturity. Most of these improvements were suggested by various people over the years on this forum. It’s not a revelation really.
2 Likes
I also tested arq in 2016, I was on Windows at the time, and I believe they had just released the first version on Windows. Your post triggered me to check my notes from back then to see what caused me to reject it. I didn’t write down much, which indicates that it became clear pretty soon that it was a lost case:
arq problems
- need to set up multiple backups in order to have different backup frequencies for different folders
- If you’re backing up to the same provider these backups will be labelled identically (e.g. Amazon Cloud Drive) and there is no way to tell them apart
- Email messages look identical, regardless which backup set is being reported on (it is a global setting, not a per backup set setting)
- error when trying to access an archive under restore in arq:
wishlist for arq
- set archive bit when file is backed up
- allow exclusion based on last modified date (and other file attributes)
I seem to remember that I blamed it on it being mature on windows yet.
In any case, thanks for sharing the good news about version 7. I’d probably switch right away, but my mind is still obsessed with deduplication across machines. It just makes things so much easier when I don’t have to worry about the cost of backing up the same files from different machines across Linux, Windows and MacOS. And I’m also willing to give duplicacy some more time.
I’m curious, why do you have such a significant overlap between machines that justifies worrying about cost?
Is that from sync services? I.e. photo library in the cloud replicated locally?
Myself — I have one Mac at home that is always on, always syncs to iCloud and backs up all uses to aws. The rest of machines and phones sync to iCloud and all data is stored there. In fact, iCloud is the primary place where data is stored. There is nothing else worth backing up on those other machines, no backup software runs there, not even time machine.
I’m not sure how significant the overlap is, but without duplicacy I’d have to start figuring that out. I have multiple external harddrives that I used on different machines and I have a home server to which I move stuff for whatever reason. A lot of these files are media files (i.e. big files).
With duplicacy I don’t have to worry that any of these moves will blow up my storage, because, apart from some marginal chunks, most of them are already there.
You might say: well, how much extra would it cost to backup the same files twice and perhaps you are right, it may be more a question of not wanting to spend more than necessary. If I can save 0.5-1 TB, it may only save me 5-10 USD per month, but as a principle, I calculate subscription costs per year, not per month (otherwise any subscription looks like “almost free”) and then we’re talking about spending 60-120 USD extra per year… But, as I said, it’s not the amount, it’s about not wasting money.
Then you seem to be an ideal user for this feature
It is 10x more than what the storage of a terabyte shall cost.
If you calculate per century it will be completely unaffordable!
For media files though — why not copy them as is (e.g. with rclone) to, say S3 Glacier? They are immutable and large so shredding them to chunks only adds to api costs.
So you have multiple backup jobs from each machine that may have any drive mounted at any time with separate backup tasks per mounted drive? Either I don’t fully understand the setup or it is unnecessarily complicated… how do you track what is connected to where if you need to restore specific file?
I’ve been lurking here a while as well, but this prompted me to post. Quite a few of the sentiments struck a chord with me, and I want to phrase this in a supportive way that stimulates further development which then attracts further adoption.
I chose and purchased Duplicacy after hunting around for quite some time after being dissatisfied with crashplan. I too administer an account on behalf of family members machines and the crashplan costs were getting out of hand compared to data stored. However I have not yet migrated my family as I have concerns on usability for them.
Duplicacy is a commercial product so thus there must be a customer in mind. Existing and future customers have problems in backup. Solving these customer problems will help the product appeal to a wider customer base, and then “they will come”.
I agree that adding some general UX improvements through the GUI will be more likely to bring on more paying customers than adding more technical features. I’m speculating here, but expect that the these unreleased technical features may also appeal more to existing techy users rather than bringing in new users. It may not even be a deciding feature for new techy users.
I have a vested interest - I have bought a licence and would like to buy a few more so I can have all my families machines backed up (I use B2) with the ability for me to quickly be able to confirm the whole family is backed up with no issues (alerts for any form of failure). I would love for Duplicacy to solve this problem for me (without me reading a bunch of manuals/forums).
At work I had colleagues who would check and do test restores for our backups every week. In my personal life, I need for this to be closer to ‘set and forget’ and accept a little more risk.
I hope that from the multiple projects gchen has running, they generate the revenue to allow gchen to appoint another dev who could then take on the task of polishing the existing products which then allows gchen to break ground on shiny new projects. Failing this, spending some time doing the less exciting polishing jobs may well bring in the customers and revenue to hire the dev and break out.
I’ll finish with this thought: I’ve recently moved in to Software Product Management - the feedback in the preceeding posts is gold! I would trade a months revenue to get this kind of candid feedback. The fact that people are taking the time to write it out demonstrates that they value the software and want it to grow and succeed.
7 Likes
I haven’t read through the whole thread here, but i read some comments with interest about the target market, enthusiast, normies etc.
I think this project is great for enthusiasts and power users atm, but the doco is lacking. I think what it mostly needs to get more normies on board is the webUI needs a pretty major overhaul (i’ve made suggestions a while ago there) and the documentation needs… to exist I tried it on Linux yesterday after a couple of years away (atm, i only use the webUI on Windows PCs, and i use Borg on my Linux devices) and the doco basically just says “doesn’t need to be installed”. And that’s about it. Doesn’t even say that the file even needs to be made executable after download Doesn’t say you need a systemd or cron job or provide examples ready-made.
This is a great blog post by senior (boss?) KDE developer Nate about different ‘classes’ of users and how it should affect interface design:
I couldn’t find it, but i recall another article from somewhere else that talked about the danger of attracting ‘normies’ to FOSS projects. But i don’t think that applies as much for Duplicacy. I suspect it wasn’t done for this reason, but what i think this software has going for it is the small paywall. It keeps those users away who are not remotely technically inclined, who demand free, and demand their needs be met without contributing in any way other than their idea (which may even be a good one). I call it a paywall, because while the CLI is free, that will not appeal to the noob and they won’t generally come here for it unless they have the right mind-set.
So yeah, i also agree that improving the WebUI is of vital importance to getting more moderately technical users interested to become paid users. I’ve set Duplicacy up on 3 PCs via the WebUI. 95% of the settings are the same. I would be great to be able to import/export for example. It’d be great to only have to input the email details once (needs a separate area for email info which can be stored). I know i could grab files from somewhere, but i’d probably have to ask on the forum, because the doco is inadequate.
I’m not a noob, but i’m not a Linux administrator either. I’m an in-between user. There’s a lot of us as potential customers
Sadly, i do not know the product well enough to contribute to the documentation. And i’m not encouraged to either (can i do a merge request via GitHub?). I did such a thing on Borg doco - i suggested an edit to improve clarification via GitHub, and it was accepted.
https://borgbackup.readthedocs.io/en/stable/
So good!
Here’s some of my previous suggestions or contributions to others’ (i don’t think they’re on the Roadmap):
1 Like