I believe Kopia does use central databases, or what they called index. I briefly looked at how it worked when it just came out. In my opinion their way of handling the databases is too complicated, even more complicated than other competitors.
This has changed, I think around 0.9, now indexes are append-only and also live in CAS.
Can’t comment on arq6. It was very short lived, electron based UI, experiment. Not even a contender. It was rightfully scraped shortly after.
I doubt it had anything to do with a backup tool. Any backup solution is as reliable as underlying storage. On the other hand, with encryption enabled with any tool, including Arq and duplicacy, there is no way for you to end up with bad data even if storage rots. You either get back exactly what was backed up or nothing.
There are two factors that affect reliability (assuming your ram and cpu are working correctly): storage and networking.
For storage it’s easy: Use reliable storage that guarantees data consistency. Repairing data rot is a job of a filesystem, not applications. Datastore corruption is an impossibility.
Duplicacy does support erasure coding, so it can tolerate some rot — but the problem is, if even one byte is allowed to rot, who is to say that it will be limited to that one byte? Ultimately, the storage either guarantees data integrity or it does not. In the former case you don’t need to worry’s about corruption. In the latter — you don’t have any guarantees, so don’t use that storage.
The other factor — resilience to network interruptions — is indeed important, and both tools handle it very well, according to my limited synthetic testing and years of use.
Performance is never a selling point of a backup tool. It’s simply irrelevant. Yes, duplicacy is very fast, which is nice, but I take CPU throttling over any super-speed backup utilizing full resources. There is no reason to hurry up to finish backup in 5 minutes to then wait for 12 hours for another backup.
Duplicacy benefits from a very fast regex engine, this helps a lot with scanning when exclusions are specified. Arq5 used to be very, ridiculously slow for the reasons I did not look into. I don’t know about arq7, because I no longer use manual filters. Everything gets backed up indiscriminately.
This statement applies to any piece of software on earth 
Arq 7, sometimes, does nothing for hours, no CPU load, no disk IO, no full RAM, then it goes back to its slow speed until the next shutdown
Have you got to the root cause of this? Apps don’t just do nothing, when they appear to be not doing anything with all stacks idle they are likely waiting for some asynchronous request, network, or disk IO. This is not specific to ark or duplicacy or Microsoft word. Nobody inserts delineate delays in the code
For the VMDK corruption with the support we think the corruption was during restore. We search but never find why.
I make a new test just few minutes and with last version of Arq 7 this seems working. If i reinstall the old Arq 7 that i use when i have the troubles, problem occurs again with a new one,
if I launch several backups taking care to have modifications in the VMDK (just start then stop the VM) in one snapshot out of two I have no files displayed and therefore nothing to restore.
This doesn’t seem to happen with the latest version, I couldn’t find anything interesting in the changelogs that might relate to this issue but it’s not very reassuring.
So the easiest thing for me was to go with Duplicacy, much less obscure, very practical in its CLI version, and just as little to work under linux, which allows me to have a single backup format for my three platforms.
For performance I don’t agree, nowadays it’s not up to humans to wait for tools, so I prefer a reliable and fast tool, than a reliable but slow tool. When I launch a new backup, I need the time needed to achieve it to be realistic, Arq 7 has improved on this point but not enough for my taste and given the articles found on the internet I am not not the only one to think that.
But fortunately there are many tools, this makes it possible to satisfy more people and find the tool best suited to their needs.
PS: I really like restic too which seems a very good tool 
3 Likes
Ha! This is very interesting. Maybe some weird versioning incompatibility. If this is repeatable – i.e. you backup a file, restore a file and it’s different – send that file to arq, they should be able to debug.
Can you clarify this a bit? are you saying arq does not detect changed files? Or does vm solution change files without changing timestamp?
Absolutely. Simplicity and robustness is what I value in duplicacy as well. This is great for servers. But:
- for workstations and personal laptops I value creature comfort more.
- Support for Archival storage is still missing. I don’t want to pay 4x in storage fees for no reason. for 4TB this is literary difference between $50 and and $200/year. I can find better use for $150.
If that was about anything else I would agree. But backup specifically is a background task, there is no hurry ever, nobody is waiting for it to end. If you backup once every hours – it has the whole hour to make that incremental backup.
Just like when some people don’t use dishwashers because “I can wash dishes faster!”. Well, guess what you are not doing when the dishwasher takes 2 hours to do that – you are not washing dishes!.
Also, when I’m running on 5% battery remaining, and duplicacy decides to complete backup in 30 seconds, fans blaring, immediately sending my laptop to sleep, I strongly dislike it. I’d prefer it to take 3 hours and not disturb me.
Two comments here:
- initial backup is a one-time operation, and therefore it does not matter how long it takes.
- I’m yet to see a backup tool that is limited by anything but network. Granted, I don’t run them on raspberry pis…
This is a bit sideways comment… but I increasingly grew to take stuff written on the internet with a giant boulder of salt In vast majority of cases there is some sort of undisclosed factor making the whole ordeal pointless and misleading. The remaining genuine testimonials are so far and between that they are not not worth the efforts to sifting though mountains of garbage. So, I don’t read reviews anymore, no longer ask questions on forums, and don’t form any opinion on anything, unless I’ve looked and tried things myself. Expecially when technology is concerned.
Yes. Competition is good, and nothing is perfect.
Restic is great. Borg is also fine. Some people get away with using git for backup. My main day-to-day backup is, in fact, Time Machine. Why? Because it’s integrated into the OS and provides excellent UI. It takes 4 hours to complete an incremental backup – because it’s a background service and it stays out of the way. It supports exclusions if needed, but I don’t use them, storage is cheap. Arq to AWS is for disaster recovery, I never expect to need to restore from it.
At some point it becomes too much to keep pouring over one solution, and you want something that “just works” without the need to read forums and write scripts. Time Machine and Arq allow me to do that, (so I can obsess over other things )
LoL dupliacy.com doesn’t even comply with GDPR, good luck getting sued.
(C) 2019 is just sad, who doesn’t auto update that.
You handle customer data, where are your privacy policies?
Even with external payment processing, you still have at least the name and email of the person.
https://duplicacy.com/privacy_policy.html
Constructive criticisms are welcome, but labeling my paid software as a scam (while continuing to use my free software) is highly unprofessional, to say the least.
5 Likes
There are no links to it from anywhere I looked — including the homepage of duplicacy.com. It’s impossible to find it. Per GDPR it shall be prominently displayed and easy to find.
Completely agree.
while continuing to use my free software) is highly unprofessional, to say the least.
Yes, ofc it is highly unprofessional. As a company, it is also highly unprofessional to completely ignore basic consumer rights and to interpret law in a very strange way. That’s the cost of doing worldwide business. GDPR is not an opt-out law, like install a browser plugins to disable Google Analytics in your policy. It’s the complete opposite, the customers have to actively agree and only after that you can send their data away. If the link is not accessible from the site, the policy basically doesn’t exist.
Yes, calling your product a scam is overdramatic, but there are some shady things going on and everything I touch feels deprecated.
The constructive part of the criticism above is the following.
Either redesign the whole site and make the paid version actually feature complete with a good gui or pay someone to do it.
1 Like
This is for the forum: Privacy - Duplicacy Forum. You can access it by clicking the three line icon on the top right corner and then clicking Privacy. It is just the default Discourse template and was set up before there was GDPR. Certainly it needs an update.
What was your opinion/reasoning on Borg? The main reason I ask is because I used duplicacy for some time, but ended up switching back to Borg because:
a) Before using ZFS as a target for backups, I could use Borgs append-only mode to protect against hacks on the client-side (as in, if server/client was compromised I could guarantee storage side would not be also impacted)
b) The cache in duplicacy seems to blow out to prohibitively large proportions
c) I can mount Borg as a browsable filesystem
I like the 1-repo-many-clients aspect of duplicacy though, and would like to know (from your opinion/experience) if there is any compelling reason to reconsider duplicacy over borg?
TLDR – I like Borg, but I trust duplicacy
Duplicacy never modifies data on the target, it’s essentially always in append-only mode. So you can set permissions to allow read and write, but not modify or delete. Or, indeed, just take periodic filesystem snapshots, so this makes the point moot.
Hmm. I haven’t noticed. I was using it on a Mac to backup all local users, total about 4TB. I thought only chunks for the snapshots are cached – and those represent very small amount of space. Under what usecase do you see unbounded cache growth?
This is a feature I suggested long ago, Mount duplicacy snapshot as a virtual filesystem, and it was recently implemented by a fellow forum member, see at the end of that thread, so you can build it and use if you want, but personally I now consider it just a gimmick. I would not install FUSE on a Mac, and the feature while nice is not a dealbreaker in any way, but does add complexity and unnecessary dependencies.
Yes, the reasons are the same as before – it’s stable as a rock and simple enough for me to trust the implementation. I’m prepared to pay with creature comfort for such assurances.
Thanks for the response!
Is this also an option in terms of SFTP access though, for example? I.e. borg serve means that I can associate the SSH key with an append only flag; is this also an option in duplicacy (and if so, I have missed that)? Or are you just referring to standard options like object lock etc on B2/S3 compat. systems?
The backup size I was using was ~1.9TB at the time, and the cache was somewhere between 150-200GB, which was a far cry larger than borg’s 2GB for ~3TB atm. I tried recreating a few times but re-encountered this same growth.
Yeah I agree re: not installing FUSE on a Mac. The only reason it was more of an “issue” for me with duplicacy was because I found some inconsistencies/difficulties in restoring compared to borg (and this is presumably on me; I had just hoped at that time for a “mount” option to quickly work around the issues I was having restoring - so that I could just use rsync instead).
duplicacy is serverless, all it needs just a basic file access api. Object lock on S3 storage is not supported (there is a FR for it), but you can accomplish the bulk of this with server-side permissions; here is a relevant recent discussion with some details: Using CLI on headless linux server - #2 by saspus, there are a few details.
I’m wondering if there were a lot of revision chunks that got downloaded? Were you doing just a backup or also check/prune? Just the backup should not have used cache at all.
(Arq keeps about 40GB of caches on my Mac, which is quite a bit more than Borg, but indeed not third of a terabyte. its weird)
Yes, essentially you need to figure out exclusions patterns to filter what to restore, and it’s not straightforward.
I’m thinking of this as a last resort backup and I never expect to need to restore. So difficulties restoring are pretty much irrelevant – if I find myself in a situation where duplicity backup is my last hope – I won’t mind a few minutes of reading documentation. In the day-to-day life I’m relying a Time Machine to a remote (over ZeroTier) server – because of excellent UI, reliability, and system integration. Time machine keeps a bunch of local snapshots as well, and usually those are more than enough to rectify mishaps. If not – Time Machine bundle on my server is already almost 9 years old, and I’m not planning on starting over any time soon
Ideally duplicacy could implement a FileProvider client, instead of a fuse based vfs, to be a first class macOS citizen, or, to make it cross-platform and compatible with anything without the need of ugly kernel extensions – local NFS server service duplicacy’s CAS storage. Mountain duck takes this approach. And it can probably be even done outside of duplicacy as a separate utility, or via go api (if duplicity offers any). This of course all does not exist today, and should not be used in making a decision on which backup tool to use.
What in Borg made you reconsider other choices?
I will take a read of this - thanks!
I had checks + prunes, but they were done via Web UI scheduler at that point, and it was a couple of years ago so I cannot attest to their success at the time.
Fair point - though for me, as part of my backup procedure I also do regular restore testing, so I’d not feel comfortable until I’d “figured that out” so to speak.
Agreed - TimeMachine is great, though these days work is all cloud-based storage, so everything important goes there (and then I control the backups of that to various other storages for the company). I only own ipad + iphone for personal devices, so I run TrueNAS for my actual storage needs (and plug into that with FileBrowserGo on the devices).
No dissatisfaction - quite the opposite, I am quite satisfied. However, I know borg is releasing a new version sometime soon, which will be a breaking change. Old version will be supported for some time but I will eventually need to re-upload my entire content. I am in Australia so my internet is not the greatest, and so hand-in-hand with this I thought I’d just revisit duplicacy and see if it makes sense to switch (or not).
Right now I get my easy-access backups from raw encrypted ZFS-send’s to a remote, so from that perspective my lack of “easy restore” would be covered for duplicacy here. I do also like the idea of being able to backup without leaving the “restore” key on the server, though realistically this doesn’t matter drastically given an attacker has server access anyway and can read files on source. I’ve partially mitigated this with borg anyway by loading the password from 1Pass CLI integrations via service account. The main protection I’m afforded this way is that if the server is physically stolen, the ZFS pool remains encrypted, and the borg password is not in plaintext on the server (so I can revoke the 1Pass token before the thieves got a chance to read the borg password out of that, for example).
At any rate, I’m rambling a little - but tl;dr is that I’m just re-assessing options, and from reading various posts here your opinion seemed informed, so thought I’d get another perspective!
1 Like
Sorry, dead thread revival. I just wanted to defend the project.
I’ve only just read this thread. Your tone is not polite and I think you were alarmist. Out of interest, did you actually pay for it? This isn’t a corporation, it’s a small business. And sometimes small business can’t keep up with what’s going on in the world of regulation, especially international regulation. I bet @gchen is also not familiar with the various and varied consumer protection laws around the world. The important thing is whether or not, once aware, gchen makes the improvements needed. Ironically, the dominant corporations comply with GDPR, but that hasn’t stopped them mining the crap out of our activity. What data does gchen have and share? I agree this should be prominently available.
As for calling this a “scam”, that’s a truly awful thing to allege. In some countries, it’s perhaps possible to be sued for defamation (i wouldn’t have a clue). Just be polite, highlight issues and assess the response.
For what it’s worth, i’m a paying user for a few years and i’ve been very happy with the reliability of the product. I’ve had the need to personally ask gchen to address something in the past (including, as it happens, something related to GDPR) and he’s been very helpful and addressed it quite promptly.
The webUI sure does need some love (i provided some suggestions years ago), but i did not purchase Duplicacy with an expectation of rapid product development. I had a need for a reliable backup solution for Windows and my unRAID server and this has proven itself that.
TL;DR
- It’s not a scam, it’s a small business
- The webUI really needs some improvements, it is disappointing, but it’s not a scam.
- the privacy policy needs to be easily found, it’s not terribly easy to find. I found it by going to the FAQ, then selecting the button from there. It could be put with FAQ, About etc here:
2 Likes
Who do you think you are coming in here, tone policing in a two year old thread?
Yes, of course I paid for duplicacy even for multiple years after this thread.
You don’t know where I live, I can call this scam if I think it’s a scam. I provided more than enough reasons why it’s absolutely not worth paying for, as some features have been broken for years.
What’s in the FAQ now vs. what has been at the time of writing are two completely different things.
Feel free to start a new Thread in the Praise Category to share your experiences.
Such a sadly immature response.
“not worth paying for” (in your opinion) does not equal “scam”. You must know this.
3 Likes