Please describe what you are doing to trigger the bug:
I am not doing anything specific. However, another process had a partial file lock on a file that is causing the backup to abort which makes this backup software kind of useless for me.
Excluding this specific file is very much hackish and not a solution for my use case. I would always lack confidence in the backups if I relied on excludes, because some other file might have a lock that I wasn’t aware of ahead of time, and it might be a week or more before I even notice. I don’t check Duplicacy multiple times a day to make sure the software works, I expect it to work. Also, I do need this file backed up. It’s not always locked, just most of the time.
I have googled and seen that the 0 bytes read abort backup behaviour is intentional, however I think you should re-evaluate, or at least check for this specific error code and only issue a warning like you do with other cases when a file cannot be read (like “file open by a different process”).
Please describe what you expect to happen (but doesn’t):
I would expect the backup to succeed.
Please describe what actually happens (the wrong behaviour):
The backup fails. Here is a message from the log…
2024-11-04 06:00:51.149 ERROR CHUNK_MAKER Failed to read 0 bytes: read \?\C:\redacted.dat: The process cannot access the file because another process has locked a portion of the file.
Failed to read 0 bytes: read \?\C:\redacted.dat: The process cannot access the file because another process has locked a portion of the file.
Thank you so much for the quick reply, I appreciate it. I’m not sure -vss is what I was looking for. VSS requires administrator privileges and is using filesystem snapshotting to include these locked files that were causing an abort. I was suggesting that an abort condition becomes a warning condition, or at least a command line option to relax some of these failures and allow them to be warnings. Like a lot of people, I would rather have an incomplete backup instead of no backup. In my case, one file is causing the entire backup to fail. As long as the warnings are included in the logs, they can be reviewed, identified and resolved at a later date.
I restarted web edition in administrator mode and tested my backup and now it fails with a different “0 bytes read” error. This error is related to cloud services. Normally when backing up a folder on a cloud service like dropbox, icloud, etc, files not existing locally will be downloaded. Now obviously, the VSS snapshot is read only, and this behavior now fails when the cloud service can’t download the file. Here is the error message:
“2024-11-05 12:10:18.383 ERROR CHUNK_MAKER Failed to read 0 bytes: read \?\C:\Users\redacted.duplicacy-web\repositories\localhost\1.duplicacy\shadow\Users\redacted\iCloudDrive\some-redacted-file: The cloud operation is not supported on a read-only volume.”
This is a different 0 bytes read error that I really wonder if it should be aborting the backup. I haven’t been able to find much information on what kind of support Duplicacy has for backing up files on cloud services, so maybe there is another option I’m not aware of that will fix this.
This is slightly off topic, but I like the side effect that all files get downloaded locally when running a Duplicacy backup. This is great for my use case, but I can also see this can be a problem for people with small amounts of local storage and large amounts in the cloud – causing hard drives filling up and excessive network traffic. This might also be a place where there should be a command line option to control if remote files are downloaded.
Unfortunately my backup fails with and without VSS. They both fail with 0 bytes read abort, but for different reasons: file lock and read only volume. I still stand by my initial suggestion that some of these abort conditions are too strict.
Unless you sync all data of *drive services locally fully, no selective download, you shall exclude iCloud drive, Dropbox, OneDrive etc from backup. Duplicacy does not support backing up dataless files. There is a feature request for that but it hasn’t been planned.
I, on the contrary, prefer complete coherent backup period. if full backup is not possible — no backup. Partial backup is useless in vast majority of cases.
Being unable to read a file user expects to be readable is a catastrophic failure and justifies abort, to investigate. Warning is fine, but most people ignore warnings.
I have pointed out a bug with VSS/Cloud services and get told I’m using the software wrong or I shouldn’t back up my files. Out of the box, it doesn’t work for a large number of Windows configurations. I’m sure it’s great for a lot of people (especially non windows users, because windows seems to be where a lot of the problems are). An hour of reddit and google searches will show you thousands of people who have given up on Duplicacy.
Your statement about partial backups is just silly. You literally told me not to back up my files, which by definition is a partial/incomplete backup. Also, the default configuration of Web Edition, you will see hundreds of files not backed up because “it is being used by another process”. Those are warnings that happily let the backup continue, again opposite of what you advocate for – and your statement kind of implies Duplicacy is doing the right thing, when it’s only aborting some of the time. It’s so inconsistent. Can’t open up a file? No problem, let’s continue the backup. Oh, I read 0 bytes?? ABORT!!! Get the team to make the “being used by another process” a fatal error too, then you will get your preferred no backup and also see windows adoption drop to 0%.
Your profile says you aren’t affiliated with Duplicacy, but your name is the one I see on so many of the bug reports I have read, so I don’t know what to make of that. You certainly come across as a spokesperson of the company. Do you use Duplicacy on Windows? I kind of have a feeling you don’t. If I’m wrong, my apologies, but maybe try walking in someone else’s shoes, You may see that there really are some major shortcomings with the Windows version and us people that are trying to bring problems to people’s attention and offer solutions are not crazy wingnuts.
It doesn’t take much looking on the Duplicacy forums to see that the core community and development team are not open to criticism and suggestions. It’s not a great look.
I hoped everything I read about this project was wrong, because it looks like there is great potential here for an amazing rock solid product at a great price. But when it doesn’t even work out of the box and you are told “working as intended”, that’s unfortunate. This backup solution is obviously not for me. I’m glad I found this out now, because I was just getting ready to commit and buy some licenses. The fact that it’s impossible to backup my computer because both -vss and not using -vss fail helped make this decision too.
TLDR: Duplicacy does not have many features and does not support many configurations, but what it does do and support – it does better than any other backup solution today in terms of reliability, resilience, and performance. It’s the only backup solution that survive in my (albeit aggressive) stability testing. People who decide to use it sacrifice features for stability. That’s pretty much the underlying reasoning of the rest of my reply here.
A lot to unpack here. First of all, this is not a bug. Support of dataless file is a feature. Duplicacy does not support that. The only backup software that does is Arq7. That feature is nice, but not common, and has other issues. Another issue with this, especially on windows is how those dateless files work – via reparse points. Reparse points on windows are filesystem extension. Their behavior is application specific. Attempting to support that is a huge task. Yes, Arq did that. No, duplicacy did not. So, if you absolutely need to backup your iCloud or OneDrive in sparse mode – duplicacy is a wrong tool for you, use Arq.
But again, it’s not a bug. it’s a lack of a feature.
That said, with the lack of dateless files backup, the right thing to do is to exclude the dateless datasets from backup. Nothing weird about it, albeit it seems to at first glance. I can elaborate on that more if needed.
Right, but this is irrelevant. You shall decide for yourself if the solution fits your needs.
It may appear so at the first sight.
No, I said fully backup your files. And if you can’t – fix the issue, so you can fully backup your files. Partial backup would be leaving your data in inconsistent, broken state. You don’t want broken backup, do you?
Web UI is a hot mess, and I don’t use it either. You may be interested to review this thread: Dont buy this scam
I fully agree. This is a bug. The key is to ensure backup does not generate warnings.
“the team” is just one dude. Shall it be fixed? Yes. Is it high on the priority list – evidently, not, because it’s still not fixed. Does it sacrifice backup reliability – no.
It’s very simple. I am a user. I see many shortcomings of this tool, and don’t shy away from pointing them out. I also having tested all other backup tools still believe this one is the best at what mattes most – making reliable backups. So, I still use it. I also use Arq, because of the features duplicacy lacks, and because it is so user friendly it’s unbelievable - works out of the box, and supports many things duplicacy does not, such as Archival storage, and dataless files, and rich filters that make sense. But if Arq backup datastore will get corrupted – I will probably lose backup. If duplicacacy datastore gets corrupted – I am very confident in being able to recover vast majority of data. So I use Arq for conveniences, and duplicacy to save the day.
Also see this (read that whole topic if you have time, it will explain why the state of duplicacy is the way it is)
I apologize for that, it’s unintentional, and this happens so much that I had to put that in my profile.
Correct. I don’t use windows at all. But the issues you point out are not specific to windows (reparse points are, but partial backups due to lack of dataless files support – are not)
Correct. There are plenty of shortcomings. If those are dealbreakers - then duplcacy is definitely not for you.
For example, many people today use cloud storage as place where their data lives, and use computers as disposable tools access that data. I personally have 3TB of data in iCloud, and my main desktop machine has only 512GB SSD. This means I cannot use duplicacy to backup my files, because they are part of that 3TB dataset, that does not fit on my device. So duplicacy is 100% useless in this scenario. I posted feature request. And I’m using Arq on that machine. it works.
Oh, they are open. It may seem this way for two reasons:
When something breaks, like google drive changed the policies around the API use, or a bug was discovered in STORJ library that was crashing 32 bit systems, developer addressed the issue in the matter of days, if not hours.
I would say you have a pretty solid picture. Duplicacy in my opinion is a gem with rough exterior, and requires some work to make it work in the specific environment, but this is the only tool that actually does backup in a stable and reliable way. Of course, some users want “brilliant out of the box” experience, who would not, but I personally not prepared to pay for that in reliability. Unfortunately this is the choice you have to make today.
To summarize,
Excluding the partially synced cloud storage from backup and using -vss will accomplish what you need. I can explain in more details why backing up partially synced storage is a bad idea. If duplicacy supported dataless files this would not have been a problem, but it does not.
And that’s OK. I recommend exploring at Arq7. I also recommend exploring duplicacy beyond the rough exterior, because you may be repulsed by it just as I was few years ago, and yet, I’m still here, relying on it to protect my most important data, despite all shortcomings in the feature set.
Again, I have nothing to lose nor gain from you using or not using duplicacy, just sharing my experience and why am I still using it, while so many other tools do so much more, at least on paper.
Oh, and I don’t use WebUI, at all. I find it useless, clunky, and unusable. I use duplicacy CLI with OS scheduler. BTW, the CLI is free for personal use. So it’s a win/win of sorts.
Heeey, I was just wondering why I got notified about this. I created the “Don’t buy this scam” thread.
I share your frustration, it shouldn’t be this hard. Not only that, but I have seen more and more hostile behavior across the self-hosting community in general towards backups. Specifically on the UNRAID discord server where official helpers tell completely new users to shut up about requesting a simple backup integration because writing your own backups scripts is so easy.
But there is hope on the horizon, as the backup noobs voted a backup solution to the top of the new features poll. I am still using the duplicacy web ui to this day, while semi regularly checking if the restore functionality is still working. I don’t like it, but it works.
Unfortunately, it is hard. It’s extremely rare for a single individual to possess the skill set required to create a reliable service software, such as a backup engine, while simultaneously delivering an exceptional user experience, as each of these domains requires a lifetime to master.
This does not mean it can or shall not be done; it just means user experience could be outsourced, and as it was discussed in that magnificent thread you’ve started, it’s absolutely necessary for duplicacy to gain wide adoption, beyond the circle of tech enthusiasts. I mentioned there, for my parents I’ve setup Arq. Because it works out of the box and is user friendly.
oh wow. That’s not only wrong, but also quite reckless.
This, unfortunately, is a big wad of nothing: amount of time it will take unraid to create and then stabilize backup software is immense, but users need backup now. Besides, the outcome is not guaranteed:
We have an example: Synology. With every nas they ship native HyperBackup application, that has amazing UI (I’m serious, it’s almost perfect), runs on a significantly resource constrained devices (also a massive feat they had to accomplish because most of their NASes have 2GB or ram and yet must shovel terabytes of data), it exists for decades, so had enough runway to polish bugs, and it “just works”. Mostly. Until one day, when stars align, and network interruptions happens in the wrong moment and your backup becomes completely unusable. You lose all history and have to start over. At this point you could not care less how user friendly UI was, if you can’t access years of version history of your data.
So world is not perfect, and we have to compromise. IN the world of backup, data durability is one and only priority. And at that duplicacy is far ahead of competition. The CLI is actually not bad, it’s the GUI that sucks, but this is counteracted by the lack of necessity to use it. This of course does not help with wider adoption – but I’d assume developer needs to balance other priorities.
Maybe some folks will start an open source project to create a native-on-the-platform backup GUI, that will support multiple backup engines, including duplicacy. That sounds like a feasible and likely outcome. The problem is – it would be very difficult to monetize, and people who possess skills to create such software don’t mind using command line tools for backup…
This is new information (which probably would’ve been picked up sooner if pathnames in your error message hadn’t been fully redacted, which is understandable yet unfortunate). You’d have had a more definitive answer, with less walls of text, and probably a workaround by now…
The answer is simple - Duplicacy doesn’t support Window’s ‘on-demand’ files and few backup programs do, because it requires specific API support. This isn’t a bug.
It also makes total sense, since in order for Duplicacy to work it has to hash the content of files which ‘demands’ all cloud files be downloaded, which nobody wants. Even Microsoft recommends to exclude OneDrive files from third-party backup tools. (-vss was a misnomer here, although it does help with ‘locked’ files - just not ‘on-demand’.) And yea Duplicacy could ignore such files and not error out, but as a Linux+Windows CLI+GUI user myself of Duplicacy for 6+ years I would much much much prefer Duplicacy smack me in the head and tell me I’m using it wrong, than silently ignore similar errors triggered with non-on-demand files!
To me, this is a perfectly acceptable limitation - just as I don’t expect Duplicacy to do full system image restores. You may have a different opinion, but then you’re always free to (kindly) request it as a feature (to support the on-demand API) for the consideration of the developer, or contribute code yourself since it’s source-available. Beware however, even CrashPlan doesn’t backup ‘offline’ cloud files, so it’s not exactly lemon squeezy.
Now, as a workaround, you could always mount your cloud drive with Rclone (mount or serve) and point Duplicacy at it. It’d necessitate downloading files at least once, and subsequently every time a file is modified - not greatly efficient, but it does work. Personally, I don’t trust the cloud enough to not to have a full (raw-files) local copy, from which Duplicacy will backup directly anyway.
Redacted file names had nothing to do with this. You can read my messages again if you like. My original post (and title of the bug report) was about reading 0 bytes of a locked file and aborting the backup. It had nothing to do with iCloud or OneDrive. Your suggestion was to use Windows VSS. I tried your suggestion/workaround *wink* *wink* and backups failed for a different reason with different types of files (cloud files vs. locked files).
With VSS enabled, the Shadow Copy is read only, but Duplicacy attempts to download the remote files from cloud providers. This obviously fails, and then the backup fails. I have been told this a feature request, but I consider it a bug. Why is Duplicacy attempting to download a file to a read only snapshot? I’m sure I could fix the code in a few minutes, but this is not an opensource project and I have little faith the pull request would be honored because my fix wouldn’t honor the false belief you have in the strictness of Duplicacy. I also don’t want to spend any more time on this. My goal is to get good backups working for my wife and children, not discussing and trying to provide good feedback to a product I won’t be using.
I have bad news for you. Duplicacy already excludes lots of files from windows backups with only a warning in the logs. The reason for this is obvious. There would be hardly any Windows users using this software if this was a fatal error. Windows does things different than macOS / Linux / BSDs. If another process has a file open, Duplicacy cannot open it and read it. Since this is likely to happen on close to 100% of windows machines in the default configuration of Duplicacy Web Edition. Someone obviously made the decision to change the behavior to continue the backup, even though hundreds, or in my case thousands, of files from every single windows backup are missing. I pointed out how inconsistent this is in a previous message. It’s why I suggested a change to make the locked file error a warning instead of a fatal error. Or the less strict behavior I described above could become a fatal error. At least make it consistent. I also suggested a command line flag to toggle this, similar to how gcc has strict flags that makes something that would normally be a warning an error. I respect your use case, and I agree with it in lots of circumstances. However, not everyone uses the software the software the same and some people understand the risks involved with their decisions.
It’s clear the majority of users on this forum providing support to Windows users don’t use Duplicacy on Windows and don’t understand some of the nuances. I think they expect it to work exactly like it does on their linux box or Mac. Maybe Windows support was added as an after thought, or maybe the software just works so much better on linux/unix/macOS that those users gravitated here.
Thank you for this suggestion, I really appreciate the time you took to reply to me. This looks like it might be a good solution for the Windows users in my family. My preliminary evaluation is good. SFTP backup to the home linux server, VSS support, it handles OneDrive/iCloud/etc folders in a sane way. We will see how it pans out later on. I’m not against paying for software and would have gladly payed for Duplicacy if it fit my needs. I will certainly keep an eye on it and check in from time to time.
This is in no way specific to windows. It depends on how the file is open and with what flags. backing up live filesystem on any OS is a horrible idea because you end up with incoherent dataset. You need to first create a filesystem snapshot, so all data is grabbed at the same time. And ten backup that snapshot. That’s exactly what -vss flag accomplishes.
IN fact, VSS stands Volume Shadow copy Service, and its windows-first feature, hence the name. Support on macOS was added later, and it of course has nothing to do VSS, because it does not exist on macOS: there, and on other OSes exist other tools to create filesystem snapshots. Duplciacy is agnostic of that, and is universal. It helps on windows as a courtesy.
So, in your case excluding iCloud folder and adding -vss would have solved all issues. I believe WebUI should have enabled that by default on windows – but as I hope it’s painfully clear now, WebUI is not perfect.
It does not. It sees the file as presented by the filesystem and reads it. The filesystem filter handles the downloading of the file. If anything, it’s a bug in iCloud fileystem filter – it shall not attempt to download to a readonly snapshot. Please submit bug report to Apple.
Why would you fix something that is not a bug?
I was kernel developer on windows of 16+ years in the past. Believe me, I understand nuances. If you have specific questions.
No problem, I’m happy to help however I can, I don’t want anyone to lose there data, and I’m quite passionate about this. I still think the duplicacy (and the rclone workaround, if you have to, is the way to go), but it’s up to you. Test each solution you pick in variety of circumstances, try to break network connection, corrupt datastore, etc, make sure it can recover your data.
BTW, if you exclusively use windows you may also want to consider Veam Agent For Microsoft Windows. It’s polished to death app, that among other things can create bootable backups, including all the network and storage drivers injected in the image. It’s free to use in a standalone mode. Free Windows Backup Solution for PCs and Endpoints
Be careful, arq 7 is quite bad on windows over time, performance quickly collapses, sometimes some files are not saved without any explanation. I had a lot of problems with the Windows version while the Mac version works wonderfully.
So many things we see differently, and so many things you are choosing to ignore in my messages. Or, you are just not reading them.
And I worked on BSD with Keith Bostic, et al and one of my first jobs was updating/creating a backup system at a university that backed up 10k+ users daily to reel to reel tape drives with retention policies and off site storage. Don’t even get me started with the ebcdic to ascii shit I had to deal with on the AIX systems. 100% restore success, not a single file lost over the years I worked there. Can’t do a proper backup with snapshotting? ZFS/AFPS/JFS2/WAFL were pipe dreams and linux didn’t even exist. If you have specific questions.
I didn’t know providing credentials was part of discussing Duplicacy. I don’t know what your point is, but we are obviously talking past each other and it’s not productive and I am moving on, so feel free to get the last word. I see and respect your point of view, but it’s clear you are unwilling to see others, or acknowledge there might actually be 1 or 2 nuggets of truth in what I’m talking about. I’m glad you are so smart that that you can dismiss everyone else. This is what I meant when I said this community is not open to discussions or criticism.
To provide point of reference, that some users, including the one still engaging with you, do.
But you are right, we are going in circles.
I think by now I expressed my point clearly enough, provided you with a few approaches how to get what you want working, explained the current feature set. You want support for dataless files, duplicacy does not offer it. I don’t know what else to tell you. I’ve also went above and beyond and directed to other solutions that may fit your use case better. So I’m not sure why are you so upset.
Ignoring this baseless ad-hominem attack – I’m one dude. I’m not representative of the whole community in any way. I disagree with your assessment of the situation. If you are requesting a feature – great, add your vote to that post, link is in my first or second response to you.
But if a new user comes and starts blasting accusation that everything is broken without trying to maybe check if they are doing something wrong – I immediately assume they did not do the homework, and try explain the basics. Be that here or on any other platform, workplace or elsewhere.
again, more concisely:
use -vss, for coherent in time backups and access to locked files.
exclude iCloud/OneDrive/DropBox/etc folder, unless you fully sync it, or use other cloud mounter that does not rely on filesystem filters, like rclone or mountain duck – whether they support iCloud it’s on your to research.
use Arq, if you have to have dataless file support.
Lack of a specific feature is not a bug. Your complaint that reading data over unsupported reparse point fails is missing the point: it’s not supported. Nobody promised it to work.
Materializing files to a readonly snapshot is bug of a filesystem filter, report it to Apple, they listen.
Definitely not on purpose, but please point out specific bits I missed, I’ll address it, if you want to continue constructive discussion.
You redacted the pathnames - this is pretty damn obvious. Had this not been done, we’d have seen you were trying to back up OneDrive or iCloud. And we’d have told you to exclude that location (as well as use -vss). As I said, understandable yet unfortunate, but my suggestion wasn’t wrong given the circumstances.
I don’t know why you feel you need to go off on a long “shit should just work” rant before bothering to understand Duplicacy’s (and many other tools’) current limitations. You’re not immune to reality, or common decency. The context to which our suggestions (as users here) are given, is to simply help solve your problem in good faith, and most users on this forum accept it without the crap attitude!
I have news for you. You can’t backup / without errors either. Most backup programs I know advise you to exclude shit like /dev/, /proc/, /sys/ too.
Again, Linux+Windows CLI+GUI user here of 6+ years, and 4 decades experience with numerous backup tools on all platforms. I manage just fine - using the right tool for the right job - Duplicacy works exceptionally well compared to most other software, once you know its capabilities and know how to use it.
I have complaints about some of Duplicacy’s shortcomings. For instance, GUI installation is poor, and plenty suggestions have already been put forth… the difference is, I remain respectful to the sole developer, and try to help users solve problems where I can, with what’s presently feasible, without bitching through a sense of entitlement (having bought 3 personal lifetime licences and a good number of commercial down the road).
