Awesome! That’s exactly what I meant by
I should have been more insistent on getting an answer 
Duplicacy is fully concurrent. No need to stop anything for anything else. There is one corner case but it won’t apply to your scenario (IRRC it’s over 7-days long backup and concurrent exhaustive prune)
That I’m not sure about. Try it.
hmmm. was going on advice from somehwere… good to know. BUT, does that mean I can run prunes concurrent with the backup that is being pruned??
Tried it and seems to be the case. Been running consistently overnight…
Yes, you can, outside of the corner case mentioned.
The snapshot file is uploaded in the very end, so until backup is finished it does not exist.
Prune also does not delete stuff immediately, it fossilizes chunks first. If the chunk happens to be needed later it will be recovered.
This is also why you if you run check you should run it with -fossils flag. That should have been a default but historically isn’t.
More here: duplicacy/duplicacy_paper.pdf at master · gilbertchen/duplicacy · GitHub
Damn. First I’ve heard that. I’d like to suggest that the guides need some serious updating and padding with additional detail. There’s a notable lack of organized info on these settings and practical configs. It’s feels like an old Microsoft help file.
and while I’m ranting, let me also note that no way to edit a prune command once it’s scheduled is a PIA.
You can edit prune command. Just click on the options list. Not obvious, but possible
omfg. Let’s create a low-contrast, light colored interface and only provide a “-” as an indicator… SMH!
My checks are now fossilized… 
Oh yes. This is probably not gchen’s fault directly – he is using off the shelf UI component but nevertheless I agree that this is not very user friendly.
Dark mode has been requested and promised. (Why does not that component support dark mode is also beyond me)
I personally feel that cross-platform UI is an invariably horrible idea, and duplicacy-web is not an exception. The whole point of an UI is to create familiar native behavior on the target OS. Attempting to webify it is almost always results in an unified experience across platforms, erasing all differences and using supported overlapping subset of features making end result equally foreign on each OS. Doing so only helps developer. And maybe.
Unfortunately many companies decided to do that recently (1Password 8 comes to mind) and I don’t mind switching to alternatives as this happens (and stopping paying money to those who don’t understand this simple concept, goodbye AgileBits)
UI must be native, period. On macOS this means AppKit/SwiftUI. On windows and other OSes – corresponding native frameworks. Leave web for simple web sites.
Using native frameworks also allows to provide deeper integration into the OS and enhance user experience as a result, with less efforts – native frameworks do tons of work for you, and there is no need to simulate native behavior.
Anyway, it’s my little pet peeve. I despise cross platform UI frameworks in general and web-ui, including electron, in particular. It’s evil, and shoudl never be used if quality of user experience is important.
We share that peeve. I commiserate with you.
Okay replying to my own post here to summarize in the interest of posterity. I think I’ve managed to resolved this (with everyone’s help). The problem was caused by faulty logic and a poorly thought-out “max runtime” setting, both entirely my fault.
In an effort to avoid conflicts between checks/prunes and an ongoing backup, I had set a backup routine to start at 0600 and set max runtime to 23:45. I set the check and prune schedule to run from 05:45 for max runtime of 14 minutes. This probably would have been fine (although unnecessary, I’m now aware), except that I had manually stopped my backup a couple of times to change the threadcount. When I restarted the backup, the max runtime started anew which, of course, screwed up the genius coordinated schedule I had created. SO, max runtime not only starts counting at the start time set in the schedule, but also starts over counting anytime the the SCHEDULE IS MANUALLY STARTED. Logical? maybe. Intuitive? no.
I now understand that I can run these actions concurrent with a backup, which removes the need to stop the backup schedule before running the prunes & checks schedule.
As I have no further need for a max runtime at this time, I set the max runtime to 00:00, which I assumed would disable it altogether. It’s been running uninterrupted now for over 8 hours and my most recent checks & prunes schedule ran as expected without incident. I am cautiously optimistic…
New 16GB sodimm arrived yesterday. NAS booted right up. Now running 20GB. Yay!
I think it is better to run check independently of any other command.
Otherwise, the check results won’t be so accurate.
For small shops and independent devs that is rarely possible.
For me cross os compatibility is a must.
The choice of golang + web-iu is spot on.
It’s just the ui that needs a bit of work.
Mainly better design.
But it has huge potential.
It would be a bug. In what scenario does check malfunction?
I strongly disagree. Being small is not an excuse not to focus on quality. ArqBackup comes to mind. Native app on every supported platform. Also one-man dev team, also same industry. There are other examples but this one fits best.
Not customizing for each platform is a problem not limited to UI: for example, designated temporary, cache, and log files locations vary between OSes. Dumpling everything to the root folder, as duplicacy does today, it definitely uniform, but equally wrong on every single platform. To rectify this users are forced to do more work configuring paths and creating symlinks when configuration is not possible. Wasting users time to save developer time is never acceptable. Moreover, most users would either not think to do it in the first place, or simply never change defaults (this is generally a good practice) and UX of their entire computing will suffer as a result: for example, another backup solution may be picking up mountains of transient stuff from duplicacy temp folder, wasting space, time, power, and bandwidth, for many users, simply because the developer decided to put in the wrong place by not customizing for the specific OS.
These are not mutually exclusive: application can be cross-os compatible and yet native on each platform.
This is an irrelevant, for the users, implementation detail.
I knew this would come up.
I actually was an Arqbackup user.
Core functionality was broken for many many years: Extremely high CPU and RAM usage.
Yet the app was slow as hell.
Security was awful as well (weak on data and non-existent on metadata).and no global deduplication.
The truth is that now is much better but it has taken many years to get here.
We should give duplicacy at least the same time.
Arq has other gret features like:
-Easier and better gui
-Better S3 support
-Budget support
Yes, you are right.
But for me cross-OS compatibility means Windows+Mac+Linux+BSD.
Arq offers no Linux or BSD support
Not at all. The programing language of choice will determine performance and resource usage, OS support as well as limitations. Golang gives Duplicacy some of its greatest features.
(high performance, scalability, compatibility and low resource usage) but also its limitations: poor GUI support.
For me, an app must nail core functionality.
Duplicacy CLI has almost everything that I value in a backup application.
The only missing core features are:
-Repository search: which could be implemented with Meilisearch and a centralized (or synchronized) index in the storage
-Glacier support: which would require splitting metadata chunks to hot storage, and a new restore flow.
The web iu needs a redesign using a more modern web toolkit and a few features:
-Better security
-Better restore flow.
-Decentralized control and monitoring of multiple instances using gRPC.
-Repository search (could be implemented as a web-ui only feature.
That would increase the value of a web-ui license.)
-Budget support.
But we are close and all of this is doable. And that is great!
Not a bug.
I might be wrong but if you are currently backing up and performing a concurrent check, the reported storage will not match actual storage since the backup operation will add more data.
This is not a bug and does not affect the main purpose of the check command.
Reported usage will simply temporarily be out of date.
Excellent point. Arq5 was a massive pice of crap. They gave up on me literally, sayin “no other customers as seeing issues you are reporting”. That’s when I started using duplicacy in the first place. Arq7 on the other hand is my favorite thing. It’s perfect – functionality wise, performance, integration, features, and support.
And we just did, exactly the same about of time, did not we? For the time it took Arq to do complete 180, duplicacy accomplished very little. The development is very slow, important features stay unimplemented for months and years. This is so painfully obvious, and was brought up before, in threads like this. I like the product, I’m still hangin out here after all, but I would not hold my breath. I’m actually using Arq7 for a while now as my main backup solution, waiting patiently for duplicacy to catch up, reading every announcement thread. I’d like it to succeed. Heck, look even at Kopia. Where it started and where is it now. It literally incorporates all of the benefits of duplicacy and then added some killer features on top. Also a go project, with web-ui by the way. The only thing its missing is Time Machine exclusions support, which is one short PR away, if not already merged.
These are not difficult features to implement, maybe except archival support, but they massively improve quality of users life. I work on a laptop, and when duplicacy would kick it, it would deplete my 4% remaining battery life immediately. I hated that, and started running it under cpulimit. I spent my own time writing scripts to make duplicacy bearable on macOS.
Then Arq is not for you. Arq was just an example. For me, anecdotally, cross-platform compatibility is not important – I work on just one OS, and neither is cross-machine deduplication, but you are right that requirements vary, and one should pick the tool that satisfies most of the requirements and works well. The latter being more important that the former: duplicity [sic] for example is an old program and exists on all platforms imaginable but I would not recommend using it.
Understandable, but this is a bare minimum, and a very low bar to clear. For me the app must be polished. User experience is more important than features. Let it have less features, but better implemented and better integrated.
WebUI need scrapping and rewrite from scratch. Today usability is not there and liklely won’t ever be. Its simply broken beyond repair as that component wasn’t designed to be a fluid UI engine.
It’s a third party universal crappy control panel bolted together with sap and twigs. It’s not sustainable to try to massage it into a usable product. It simply does not fit.
That should have been done long ago. It wasn’t. I wish I shared your enthusiasm, but I had ran out of mine, unfortunately.
What actually needs to happen here is outsourcing/delegating of GUI development to outside UX design firm, that specializes on creating user experiences on each target OS. That’s what I would have done had it been my product. Duplicacy CLI engine is well designed piece of software, but let’s be frank here – neither old GUI nor Web GUI are not something that can be taken seriously, and they cast bad shadow on the otherwise excellent backup engine. I, in fact, rejected duplicacy at the first pass after giving one look at the GUI when picking a replacement backup tool few years ago, at the same time I rejected Arq5. Only because other contenders failed to actually do backup properly and reliably (including Arq5) I had to lower bar, re-visit rejects and became paying customer of duplicacy. This is what actually happened. Not the most ideal way to acquire new customers.
I still disagree here. One can create magnificent software in java, objectve C, python, and Cobol. Programming language is just a preference, and its choice is driven by libraries and support availability, somewhat language features (e.g. goroutines are cool, but the same can be accomplished in other languages, albeit with extra efforts) and mostly robustness of development environment, testing, etc etc.
I don’t think it’s fair to blame go for poor gui support – the purpose of the language as it was conceived was entirely different. It all still boils down to libraries, and I don’t envision Apple or anybody else creating AppKit bindings for go. Why would anyone want to do that, when you can create UI in SwiftUI or Objective C natively already?.
I like go just as much as the next guy, the design decisions they made in the language are short of magnificent (I’m coming from several decades of software development mostly in C-based languages), and I do agree that it’s a great if not the best choice for the tool like duplicacy CLI. But it’s a horrible choice for UI. Nobody forces to write entire product on one languages though.
Yes. For me too. It used to be ahead for a while. But by now so do a bunch of competitors, that have a benefit of also having polished UI, proper OS support, and don’t require me to reading source, writing blog posts and hanging out on forums to overcome tools’ shortcomings just to be able to run it on my OS.
In my opinion Duplicacy stagnated, and it’s heartbreaking to witness. Unless priorities change (or, lacking further development interest, releasing it under bsd-like or MIT license so that others can meaningfully contribute could be next best thing) I don’t see a future here.
Am I being negative here? Maybe. Maybe I’m being realistic, and don’t want the tool with great potential to fail. But potential, unless realized, spoils. It does not have a very long shelf life. Both Kopia and Arq demonstrated much faster development pace and while both are quite opinionated tools in different direction, both now provide competitive advantages over duplicacy. Especially Kopia, as it’s almost drop-in replacement with all the features we asked for years implemented.
Note again, I’m not on kopia forums, I’m here, and I really want duplicacy to succeed. This negativity is intended to be constructive.
Understood. Yes, the stats can be wrong, depending on the interpretation:
“how much storage do existing backups take” is still correct: the backup in progress should not be counted, it’s not there yet, and not even guaranteed to be there ever. Orphaned chunks will also likely not be accounted for, IIRC.
But if you mean "how much data is in “Duplicacy” folder on the target – then your are right, check is probably wrong tool to answer that.
You guys are way more deep-dived into this than I care to be, but I can offer you and Duplicacy, if they’re listening, a limited perspective (or narrow rant, as you prefer);
I use Linux, exclusively, every day on my desktop & laptop. After decades of Windows use and support (I’m talking v3.1 > v10) as a MS SMB partner, I abandoned Microsoft completely and with a vengeance. I still run the odd Windows VM for research and troubleshooting and I still support my tech-obsessed mixed OS family, but otherwise, I shun all things Microsoft. If an app does not support Linux, I do not use them, full stop. Software like Arq is useless to me. Ditto anything from Adobe.
Being a dedicated Linux user, I’m comfortable with the cli, but it is not my preference. This is 2022, people, and I want a GUI. I want a gui that gives me the option to set, config, enact, and monitor everything that I need/want to do… simples. I run dual 27" monitors in HD resolution. My graphics card has more ram than my laptop. I’m done with monospace fonts and black-screen CLI environments. I think most of modern society is too.
THIS 
. I passed over Duplicacy numerous times before now due to it’s cli focus. Now that I’m here, I’m impressed by the features and functionality, but DEPRESSED by the interface. In my head, I keep thinking; “I paid for this?” I can think of magnificent cli-based software applications that have outstanding GUI interfaces. Grsync comes to mind, as does putty. A decent and full-featured os-specific gui is utterly doable, so it just needs doing. I can live with web interface on my Syno for now. But I will never commit to using Duplicacy on my family’s systems. Despite the great backup Duplicacy provides, I’d rather pay twice as much for a mature application with a full-featured, platform specific gui than to live with shitty interface just to save a few cents.
And, this interface is shitty; it’s bloated and takes up way too much screen space for little info it provides. It’s not intuitive at all. The “Dashboard” takes up two full screens and the “Activities” section is a tragedy. While “functional” to an extent, it feels like an afterthought. The kindest word I can use to describe it is “primitive”.
I’m no dev, so I don’t know the ins and outs of the process required to get from point A to B. But, I do know that many, many others with one-man shows or small teams have managed to get great OS-specific UI’s for their applications and I think it’s not unreasonable to expect Duplicacy to do the same. ESPECIALLY if you’re asking me to pay you for the privilege of using it. I’ll pay my $20 and give Duplicacy a year. But, when my renewal comes up, if there hasn’t been some improvement in this area, I’m probably done.
Not the most ideal way to keep customers, either.
