Please describe what you are doing to trigger the bug:
I am not doing anything specific. However, another process had a partial file lock on a file that is causing the backup to abort which makes this backup software kind of useless for me.
Excluding this specific file is very much hackish and not a solution for my use case. I would always lack confidence in the backups if I relied on excludes, because some other file might have a lock that I wasn’t aware of ahead of time, and it might be a week or more before I even notice. I don’t check Duplicacy multiple times a day to make sure the software works, I expect it to work. Also, I do need this file backed up. It’s not always locked, just most of the time.
I have googled and seen that the 0 bytes read abort backup behaviour is intentional, however I think you should re-evaluate, or at least check for this specific error code and only issue a warning like you do with other cases when a file cannot be read (like “file open by a different process”).
Please describe what you expect to happen (but doesn’t):
I would expect the backup to succeed.
Please describe what actually happens (the wrong behaviour):
The backup fails. Here is a message from the log…
2024-11-04 06:00:51.149 ERROR CHUNK_MAKER Failed to read 0 bytes: read \?\C:\redacted.dat: The process cannot access the file because another process has locked a portion of the file.
Failed to read 0 bytes: read \?\C:\redacted.dat: The process cannot access the file because another process has locked a portion of the file.
Thank you so much for the quick reply, I appreciate it. I’m not sure -vss is what I was looking for. VSS requires administrator privileges and is using filesystem snapshotting to include these locked files that were causing an abort. I was suggesting that an abort condition becomes a warning condition, or at least a command line option to relax some of these failures and allow them to be warnings. Like a lot of people, I would rather have an incomplete backup instead of no backup. In my case, one file is causing the entire backup to fail. As long as the warnings are included in the logs, they can be reviewed, identified and resolved at a later date.
I restarted web edition in administrator mode and tested my backup and now it fails with a different “0 bytes read” error. This error is related to cloud services. Normally when backing up a folder on a cloud service like dropbox, icloud, etc, files not existing locally will be downloaded. Now obviously, the VSS snapshot is read only, and this behavior now fails when the cloud service can’t download the file. Here is the error message:
“2024-11-05 12:10:18.383 ERROR CHUNK_MAKER Failed to read 0 bytes: read \?\C:\Users\redacted.duplicacy-web\repositories\localhost\1.duplicacy\shadow\Users\redacted\iCloudDrive\some-redacted-file: The cloud operation is not supported on a read-only volume.”
This is a different 0 bytes read error that I really wonder if it should be aborting the backup. I haven’t been able to find much information on what kind of support Duplicacy has for backing up files on cloud services, so maybe there is another option I’m not aware of that will fix this.
This is slightly off topic, but I like the side effect that all files get downloaded locally when running a Duplicacy backup. This is great for my use case, but I can also see this can be a problem for people with small amounts of local storage and large amounts in the cloud – causing hard drives filling up and excessive network traffic. This might also be a place where there should be a command line option to control if remote files are downloaded.
Unfortunately my backup fails with and without VSS. They both fail with 0 bytes read abort, but for different reasons: file lock and read only volume. I still stand by my initial suggestion that some of these abort conditions are too strict.
Unless you sync all data of *drive services locally fully, no selective download, you shall exclude iCloud drive, Dropbox, OneDrive etc from backup. Duplicacy does not support backing up dataless files. There is a feature request for that but it hasn’t been planned.
I, on the contrary, prefer complete coherent backup period. if full backup is not possible — no backup. Partial backup is useless in vast majority of cases.
Being unable to read a file user expects to be readable is a catastrophic failure and justifies abort, to investigate. Warning is fine, but most people ignore warnings.
I have pointed out a bug with VSS/Cloud services and get told I’m using the software wrong or I shouldn’t back up my files. Out of the box, it doesn’t work for a large number of Windows configurations. I’m sure it’s great for a lot of people (especially non windows users, because windows seems to be where a lot of the problems are). An hour of reddit and google searches will show you thousands of people who have given up on Duplicacy.
Your statement about partial backups is just silly. You literally told me not to back up my files, which by definition is a partial/incomplete backup. Also, the default configuration of Web Edition, you will see hundreds of files not backed up because “it is being used by another process”. Those are warnings that happily let the backup continue, again opposite of what you advocate for – and your statement kind of implies Duplicacy is doing the right thing, when it’s only aborting some of the time. It’s so inconsistent. Can’t open up a file? No problem, let’s continue the backup. Oh, I read 0 bytes?? ABORT!!! Get the team to make the “being used by another process” a fatal error too, then you will get your preferred no backup and also see windows adoption drop to 0%.
Your profile says you aren’t affiliated with Duplicacy, but your name is the one I see on so many of the bug reports I have read, so I don’t know what to make of that. You certainly come across as a spokesperson of the company. Do you use Duplicacy on Windows? I kind of have a feeling you don’t. If I’m wrong, my apologies, but maybe try walking in someone else’s shoes, You may see that there really are some major shortcomings with the Windows version and us people that are trying to bring problems to people’s attention and offer solutions are not crazy wingnuts.
It doesn’t take much looking on the Duplicacy forums to see that the core community and development team are not open to criticism and suggestions. It’s not a great look.
I hoped everything I read about this project was wrong, because it looks like there is great potential here for an amazing rock solid product at a great price. But when it doesn’t even work out of the box and you are told “working as intended”, that’s unfortunate. This backup solution is obviously not for me. I’m glad I found this out now, because I was just getting ready to commit and buy some licenses. The fact that it’s impossible to backup my computer because both -vss and not using -vss fail helped make this decision too.
TLDR: Duplicacy does not have many features and does not support many configurations, but what it does do and support – it does better than any other backup solution today in terms of reliability, resilience, and performance. It’s the only backup solution that survive in my (albeit aggressive) stability testing. People who decide to use it sacrifice features for stability. That’s pretty much the underlying reasoning of the rest of my reply here.
A lot to unpack here. First of all, this is not a bug. Support of dataless file is a feature. Duplicacy does not support that. The only backup software that does is Arq7. That feature is nice, but not common, and has other issues. Another issue with this, especially on windows is how those dateless files work – via reparse points. Reparse points on windows are filesystem extension. Their behavior is application specific. Attempting to support that is a huge task. Yes, Arq did that. No, duplicacy did not. So, if you absolutely need to backup your iCloud or OneDrive in sparse mode – duplicacy is a wrong tool for you, use Arq.
But again, it’s not a bug. it’s a lack of a feature.
That said, with the lack of dateless files backup, the right thing to do is to exclude the dateless datasets from backup. Nothing weird about it, albeit it seems to at first glance. I can elaborate on that more if needed.
Right, but this is irrelevant. You shall decide for yourself if the solution fits your needs.
It may appear so at the first sight.
No, I said fully backup your files. And if you can’t – fix the issue, so you can fully backup your files. Partial backup would be leaving your data in inconsistent, broken state. You don’t want broken backup, do you?
Web UI is a hot mess, and I don’t use it either. You may be interested to review this thread: Dont buy this scam
I fully agree. This is a bug. The key is to ensure backup does not generate warnings.
“the team” is just one dude. Shall it be fixed? Yes. Is it high on the priority list – evidently, not, because it’s still not fixed. Does it sacrifice backup reliability – no.
It’s very simple. I am a user. I see many shortcomings of this tool, and don’t shy away from pointing them out. I also having tested all other backup tools still believe this one is the best at what mattes most – making reliable backups. So, I still use it. I also use Arq, because of the features duplicacy lacks, and because it is so user friendly it’s unbelievable - works out of the box, and supports many things duplicacy does not, such as Archival storage, and dataless files, and rich filters that make sense. But if Arq backup datastore will get corrupted – I will probably lose backup. If duplicacacy datastore gets corrupted – I am very confident in being able to recover vast majority of data. So I use Arq for conveniences, and duplicacy to save the day.
Also see this (read that whole topic if you have time, it will explain why the state of duplicacy is the way it is)
I apologize for that, it’s unintentional, and this happens so much that I had to put that in my profile.
Correct. I don’t use windows at all. But the issues you point out are not specific to windows (reparse points are, but partial backups due to lack of dataless files support – are not)
Correct. There are plenty of shortcomings. If those are dealbreakers - then duplcacy is definitely not for you.
For example, many people today use cloud storage as place where their data lives, and use computers as disposable tools access that data. I personally have 3TB of data in iCloud, and my main desktop machine has only 512GB SSD. This means I cannot use duplicacy to backup my files, because they are part of that 3TB dataset, that does not fit on my device. So duplicacy is 100% useless in this scenario. I posted feature request. And I’m using Arq on that machine. it works.
Oh, they are open. It may seem this way for two reasons:
When something breaks, like google drive changed the policies around the API use, or a bug was discovered in STORJ library that was crashing 32 bit systems, developer addressed the issue in the matter of days, if not hours.
I would say you have a pretty solid picture. Duplicacy in my opinion is a gem with rough exterior, and requires some work to make it work in the specific environment, but this is the only tool that actually does backup in a stable and reliable way. Of course, some users want “brilliant out of the box” experience, who would not, but I personally not prepared to pay for that in reliability. Unfortunately this is the choice you have to make today.
To summarize,
Excluding the partially synced cloud storage from backup and using -vss will accomplish what you need. I can explain in more details why backing up partially synced storage is a bad idea. If duplicacy supported dataless files this would not have been a problem, but it does not.
And that’s OK. I recommend exploring at Arq7. I also recommend exploring duplicacy beyond the rough exterior, because you may be repulsed by it just as I was few years ago, and yet, I’m still here, relying on it to protect my most important data, despite all shortcomings in the feature set.
Again, I have nothing to lose nor gain from you using or not using duplicacy, just sharing my experience and why am I still using it, while so many other tools do so much more, at least on paper.
Oh, and I don’t use WebUI, at all. I find it useless, clunky, and unusable. I use duplicacy CLI with OS scheduler. BTW, the CLI is free for personal use. So it’s a win/win of sorts.
Heeey, I was just wondering why I got notified about this. I created the “Don’t buy this scam” thread.
I share your frustration, it shouldn’t be this hard. Not only that, but I have seen more and more hostile behavior across the self-hosting community in general towards backups. Specifically on the UNRAID discord server where official helpers tell completely new users to shut up about requesting a simple backup integration because writing your own backups scripts is so easy.
But there is hope on the horizon, as the backup noobs voted a backup solution to the top of the new features poll. I am still using the duplicacy web ui to this day, while semi regularly checking if the restore functionality is still working. I don’t like it, but it works.
Unfortunately, it is hard. It’s extremely rare for a single individual to possess the skill set required to create a reliable service software, such as a backup engine, while simultaneously delivering an exceptional user experience, as each of these domains requires a lifetime to master.
This does not mean it can or shall not be done; it just means user experience could be outsourced, and as it was discussed in that magnificent thread you’ve started, it’s absolutely necessary for duplicacy to gain wide adoption, beyond the circle of tech enthusiasts. I mentioned there, for my parents I’ve setup Arq. Because it works out of the box and is user friendly.
oh wow. That’s not only wrong, but also quite reckless.
This, unfortunately, is a big wad of nothing: amount of time it will take unraid to create and then stabilize backup software is immense, but users need backup now. Besides, the outcome is not guaranteed:
We have an example: Synology. With every nas they ship native HyperBackup application, that has amazing UI (I’m serious, it’s almost perfect), runs on a significantly resource constrained devices (also a massive feat they had to accomplish because most of their NASes have 2GB or ram and yet must shovel terabytes of data), it exists for decades, so had enough runway to polish bugs, and it “just works”. Mostly. Until one day, when stars align, and network interruptions happens in the wrong moment and your backup becomes completely unusable. You lose all history and have to start over. At this point you could not care less how user friendly UI was, if you can’t access years of version history of your data.
So world is not perfect, and we have to compromise. IN the world of backup, data durability is one and only priority. And at that duplicacy is far ahead of competition. The CLI is actually not bad, it’s the GUI that sucks, but this is counteracted by the lack of necessity to use it. This of course does not help with wider adoption – but I’d assume developer needs to balance other priorities.
Maybe some folks will start an open source project to create a native-on-the-platform backup GUI, that will support multiple backup engines, including duplicacy. That sounds like a feasible and likely outcome. The problem is – it would be very difficult to monetize, and people who possess skills to create such software don’t mind using command line tools for backup…
