Using VSS when backing up files (Windows)

[saspus]

@Droolio

Duplicacy already aborts a backup, with an error, when 0 files are backed up.

This is a bug that needs to be addressed. I’ve reported it before, years ago.

@Droolio

What about when 10% is excluded, or 90%? And Duplicacy happily creates partial backups - potentially forever - without generating an error!

Yes, because there is no error. All files that were possible to back up were correctly backed up. It is not possible to back up files that are unreadable. Kind of obvious.

What we are actually discussing here is whether there is some threshold of failures that should warrant notifying the user.

Not aborting backup, just notifying the user. That courtesy note I mentioned above, that duplicacy can issue when count of files in the dataset changes drastically. Or maybe even if at least one file was skipped: this way to prevent constant nagging, users ultimately would have to write an exclusion pattern for those files, thereby making any message from duplicacy meaningful (zero-warning policy)

@Droolio

Not relevant to what you said:

@saspus

@Droolio

Making the next incremental take eons

That would not be the case.

Which is technically incorrect, since Duplicacy has to rehash all the data again.

For most users, transferring data takes time and is expensive. Compute resources are free and infinite.

@Droolio

Untouched data that was already in a previous backup, but is now disregarded because of an intermediate fs hiccup. That’s simply not correct behavior.

You are trading the mild inconvenience of spending extra CPU cycles rehashing content for risk of skipping important, readable files. Makes zero sense to me.

The correct behavior is to protect as much of user data as possible. That’s the goal. If that means re-hashing everything next time around – so what? CPU time is worthless, user’s data is priceless.

[Droolio]

@saspus

@Droolio

Duplicacy already aborts a backup, with an error, when 0 files are backed up.

This is a bug that needs to be addressed. I’ve reported it before, years ago.

Not a bug, in fact a quick n dirty workaround to address the exact problem we’re talking about.

@saspus

The correct behavior is to protect as much of user data as possible. That’s the goal. If that means re-hashing everything next time around – so what?

The rehashing is just undesirable behavour. Not the first it’s come up. Either way, I wouldn’t call a 30-fold increase in backup time, acceptable behaviour - given that it can safely abort and retry on the next schedule.

The dangerous behaviour I’m more concerned about is a proposal to make Duplicacy NEVER fail when encountering a disk error! i.e. Silently ignoring read errors. I chose Duplicacy because of its reliability - not to do silly stuff like that. Perhaps if you want to do that, the backup command could have a -persist option too?

I agree with your idea that Duplicacy could revert to using normal file access if VSS fails, although there’s no evidence that’d help; archon will probably have to exclude the binlog file anyway. Not such a big deal for a new repository.

[saspus]

@Droolio

Silently ignoring read errors

Not silently. Notifying the users, as suggested above.

I guess we argue about two different things:

• You say duplicacy shall not ignore failures. I agree. It shall report them, and keep nagging the user and have him/her configure exclusions in a way that no exceptions are reported during normal backup run.
• I say it shall nevertheless continue attempting to backup whatever else it can, not abort on first error, simply because there in the queue could be a document that user spent hours editing, and it shall be given a chance to get backed up. If this comes at the cost of 30x longer next backup – well, you just salvaged user’s data from the disintegrating system. It’s a very small prices to pay. The machine is dead anyway.

[saspus]

@Droolio

The rehashing is just undesirable behavour. Not the first it’s come up.

This is backwards. Backing up empty dataset is not an error. What should have been detected as an error instead – is VSS failure, or network failure, or mount point missing. If duplicacy can’t distinguish between these failures – it’s a bug; checking for zero count of files is a dirty hack of a workaround, that needs to be undone, once the correct error handling is implemented (which is sending a message such as “selected mountpoint is missing”, or “VSS timed out”, or anything else.

This problem is not unique to duplciacy, and approach to solving it can be borrowed from other tools, that do exactly what suggested – report that mount point is missing and carry on with backing up the rest of the stuff.

[Droolio]

@saspus

Not silently. Notifying the users, as suggested above.

How is the user notified something urgent needs to be fixed with their backup jobs if there can never be such a thing as a failed backup when disk read errors occur??

I see small numbers of WARNs in my logs all the time - for junction points and locked files (with VSS). It’s a reasonable balance. Harder errors like disk reads will just get lost amongst a sea of green marks in the UI. Nothing is amiss if you don’t look at the logs. Unless you fail the backup and don’t save the snapshot.

Not saying things can’t be improved here. i.e. Try non-VSS when VSS fails. Exceed 100 skipped files? Error out.

In fact it’d be nice if Duplicacy could adopt the incomplete snapshot concept for incrementals as it does for an initial backup. (And make the incomplete data accessible.) That’d solve everyone’s needs.

@saspus

I say it shall nevertheless continue attempting to backup whatever else it can, not abort on first error, simply because there in the queue could be a document that user spent hours editing, and it shall be given a chance to get backed up. If this comes at the cost of 30x longer next backup – well, you just salvaged user’s data from the disintegrating system. It’s a very small prices to pay. The machine is dead anyway.

Well I guess if you want that behaviour, Duplicacy could always offer a -persist option.

[saspus]

@Droolio

How is the user notified something urgent needs to be fixed with their backup jobs if there can never be such a thing as a failed backup when disk read errors occur??

Via the same mechanism used for notification about failed backup. Or I don’t understand your question.

In the email sent, there would be three sections:

1. Fatal errors: Could not connect to the target, could not transfer data, could not upload snapshot, etc.
2. Warnings: Unexpected failures to read few files: bad sectors, VSS crap, SIP, other stuff that shoudl work but did not, etc; list those files.
3. Notes: Expected failures to read few files: reparse points, symlinks, other stuff that is known unsupported but was included, etc; list them too.

Successful backup means no fatal errors. User is supposed to review the warning and notes and fix them. I would also not send an email at all if there are no warnings, no notes, and not failures. All is good – don’t say anything. There is enough spam as it is.

(Separate discussion is whether backup tool developers should maintain exclusion list for mainstream OSes and not put the burden on the user)

Also, none of what’s listed in Warnings or Notes should preclude completing the remaining backup. Only the user knows which failures and what files are important. Duplicacy cannot take it upon itself to abort backing up a set of files because the other set of files failed. I keep repeating this, but this is very critical: safety of user data is paramount here. CPU usage is not.

@Droolio

I see small numbers of WARNs in my logs all the time - for junction points and locked files (with VSS).

Fix it. Add exclusions. See next comment.

@Droolio

It’s a reasonable balance. Harder errors like disk reads will just get lost amongst a sea of green marks in the UI.

It’s not. Because, just like you said, critical warnings will drown in the sea of unimportant ones. Empty logs are happy logs.

@Droolio

Nothing is amiss if you don’t look at the logs. Unless you fail the backup and don’t save the snapshot.

How will you know it failed the backup? By receiving a notification.
That same notification can just list warnings instead, without failing the backup and missing files that are readable just fine.

@Droolio

Exceed 100 skipped files? Error out.

Why 100? Why not 10? or 1? Any magic number here will be wrong. What is my mount of external filesystem got mounted with wrogn permissions? All 30000 files are now unreadable. Shall back up halt and skip backing up my document I just edited in the home folder? Answer – heck no!

@Droolio

In fact, it’d be nice if Duplicacy could adopt the incomplete snapshot concept for incrementals as it does for an initial backup. (And make the incomplete data accessible.) That’d solve everyone’s needs.

Oh, yes! Another related killer feature would be to continue taking local file system snapshots at the scheduled cadence, even if backup cannot be started or completed for other reasons (such as no network, or running on battery) and when connectivity is restored – process and transfer data for these multiple snapshots all at once. I don’t think anyone besides Time Machine does it today. This would be a huge selling point for folks like myself.

@Droolio

Well I guess if you want that behaviour, Duplicacy could always offer a -persist option.

That’s a copout. Offering the multitude of configuration options only moves the burden of making a choice (the developer could not make!) to the user. The developer needs to decide on the right approach and stick to it, and optionally, teach the user the correct way.

[Droolio]

@saspus

User is supposed to review the warning and notes and fix them.

Herein lies my problem with your logic. As said previously, WARNs can happen almost every single backup. Perfectly normal behaviour. VSS isn’t perfect. Now you expect to flag disk read errors as a WARN and let the user sift through all that? Never an error.

What you’re advocating for is ‘silent failure’, however you want to put it.

@saspus

@Droolio

I see small numbers of WARNs in my logs all the time - for junction points and locked files (with VSS).

Fix it. Add exclusions. See next comment.

Not possible. Locked files. These are random.

Quite funny really. You expect me to add exceptions for junction points but not for others to do the same with troublesome binlogs and other .lock files VSS can’t deal with.

@saspus

I keep repeating this, but this is very critical: safety of user data is paramount here.

Couldn’t agree more, which is why I think your proposal is unsafe. The underlying issue may mean a complete snapshot is never created again + the last known good backup is no longer protected from pruning + the user is not properly informed anything is wrong.

@saspus

The developer needs to decide on the right approach

Apparently, he has, and agrees that it should fail on such disk read errors.

I’d never use a -persist option either but definately don’t want the default behaviour to change. I very much favour incomplete for incrementals as the better compromise, although recognise implementation could take a fair bit of effort.

[fisowiw784]

I’m the type of person who always, without fail, checks every file that isn’t backed up. If a file fails to backup, for whatever reason, I still expect my other files to be backed up. Give me those warnings and errors; I will deal with them. I absolutely do not want any ‘silent failures’ in any way!

Back up everything, but tell me what failed. Every backup program I’ve used follows this behaviour, even IDrive.

I also think incomplete should definitely be implemented for all backups, not just the initial backup, if only to make resuming aborted/cancelled backups faster.