Graceful Handling of Failed and Incomplete Snapshots

[Christoph]

I’m curious why the most popular suggested workaround of creating a pre-backup script cannot also be the solution. That is, to bake the same “check” on the repository into the default behavior of the backup command itself. I have prodded at this from various angles, but for this solution in particular, what are the downsides? Maybe I’m missing something that could really cause issues for people?

From what I understood, the problem is that your pre-backup script will work because it is written for your specific setup/environment. To bake such a script into the backup command itself would require it to work in all setups/environments, which is very difficult (or impossible?) for reasons that others can explain better.

[santacruzskim]

I suggested this method specifically because it works on all platforms, at least Windows and Linux-based (maybe there are lots of users on platforms where this isn’t an option?). The syntax is slightly different of course, but they accomplish the same goal of asking a folder, file, mount-point, etc., “are you there? can I access you?” and it issuing a yes/no response which duplicacy can then act on.

My lack of knowledge on how something like this would be implemented may be showing though. I’m assuming there would be a way to detect the platform and issue the relevant command, or at least get creative…

Regarding pre-backup scripts in particular, you can send both Windows and Linux commands and have no downside that I can see - on Windows if I create a script called pre-backup.bat it will run if it’s in the scripts folder. If it’s called pre-backup (not an executable file extension), it is totally ignored, but on Linux it would have run, correct?

These commands appear to be very simple too, where Duplicacy would only need inject the root path(s) to the source/“repository.”

And if all of this is complete nonsense to the folks much more knowledgeable than myself, the option of just skipping a backup if there are 0 files detected would seem to get the job done and be platform agnostic, no? This would also prevent 0B / bad backups from getting factored into prune calculations where some pretty common options would leave you with dramatically less revisions containing actual data.

[leerspace]

I suggested this method specifically because it works on all platforms, at least Windows and Linux-based (maybe there are lots of users on platforms where this isn’t an option?)

If you’re referring to this post where the script only checks if a directory exists, this won’t work for all mount types. For example, on one my Linux VMs I have several NFS mounts that just look like empty directories when they aren’t mounted.

So a pre-backup script that just checks if the path exists would say that it’s okay to run the backup, even if the NFS share isn’t mounted.

Regarding pre-backup scripts in particular, you can send both Windows and Linux commands and have no downside that I can see

It’s a maintenance headache. You now have two scripts to maintain and keep in sync with any improvements (one for each platform), multiplied by the number of backups you want to have pre-backup scripts check before running.

the option of just skipping a backup if there are 0 files detected would seem to get the job done and be platform agnostic, no?

Agreed. Is there an edge case you think these two suggested changes from earlier in the thread wouldn’t cover?

1. Make errors accessing the root directory fatal, so the backup fails before uploading an empty snapshot
2. For mounts that only look empty when unmounted, maybe add an option to backup like -no-empty-root or something that causes an empty root directory to also throw a fatal error

[santacruzskim]

Thanks for the follow-up.

My script checks to see if the source exists because at least on Windows, this appears to be all I need (it worked with symlinks pointing to offline volumes too). I was assuming (possibly incorrectly) that a viable Linux script would be similarly easy to accomplish; same logic, different syntax.

To put some context around the scripts, I look at these as inspiration for how something so simple seems to address the problem. I was/am assuming the same tactics could be built into Duplicacy natively using the tools and language Duplicacy is built on. Not knowing what tools are actually available to address this, I speak with willful ignorance and the intent of inspiring someone who actually knows more than me and can make that leap to an actual solution.

@leerspace 's suggestions make perfect sense to me. As I see it, my suggestions above are exploring how to implement #1:

1. Make errors accessing the root directory fatal, so the backup fails before uploading an empty snapshot

Looking at my log of a successful backup of an offline volume under Windows, I see 3 (overlapping) things Duplicacy is claiming knowledge of that should cause the backup to abort, rather than complete “successfully”: (1) Can’t find the path (2) can’t list contents (3) errors accessing directory (in this case the root of the repository)

#1: WARN LIST_FAILURE Failed to list subdirectory: open \\?\I:\FiTB Archive: The system cannot find the path specified.
#2: WARN SKIP_DIRECTORY Subdirectory  cannot be listed
#3: WARN BACKUP_SKIPPED 1 directory was not included due to access errors

I’m pasting the entire log at the bottom for context.

For item #2 of @leerspace 's solution:

2. For mounts that only look empty when unmounted, maybe add an option to backup like -no-empty-root or something that causes an empty root directory to also throw a fatal error

I see no possible side effects and it has my vote and 100% support. Depending on how this is implemented, I wonder how would respond to a situation like mine where it can’t find the directory. Curious if there is a distinction in its logic between “empty” and “inaccessible” since the resulting behavior is currently the same under both circumstances.

I would even like to take things a step further and provide another option like -skip-empty-backup where if there is 0B to back up for any reason, the backup is skipped instead of completing successfully. I would never recommend this be a default behavior though. I just like that it feels super easy to implement and would be an interesting alternative solution for my particular backup needs.

…and here’s the full log of my empty, successful backup of an offline volume:

2020-03-29 13:23:52.652 INFO REPOSITORY_SET Repository set to I:/FiTB Archive
2020-03-29 13:23:52.653 INFO STORAGE_SET Storage set to gcd://(e)/DUPLICACY_FITB_01
2020-03-29 13:23:56.403 INFO BACKUP_START Last backup at revision 27 found
2020-03-29 13:23:56.403 INFO BACKUP_INDEXING Indexing I:\FiTB Archive
2020-03-29 13:23:56.403 INFO SNAPSHOT_FILTER Parsing filter file \\?\C:\Users\Administrator\.duplicacy-web\repositories\localhost\0\.duplicacy\filters
2020-03-29 13:23:56.403 INFO SNAPSHOT_FILTER Loaded 26 include/exclude pattern(s)
2020-03-29 13:23:56.403 WARN LIST_FAILURE Failed to list subdirectory: open \\?\I:\FiTB Archive: The system cannot find the path specified.
2020-03-29 13:23:58.325 WARN SKIP_DIRECTORY Subdirectory  cannot be listed
2020-03-29 13:23:58.336 INFO BACKUP_END Backup for I:\FiTB Archive at revision 28 completed
2020-03-29 13:23:58.336 INFO BACKUP_STATS Files: 0 total, 0 bytes; 0 new, 0 bytes
2020-03-29 13:23:58.336 INFO BACKUP_STATS File chunks: 0 total, 0 bytes; 0 new, 0 bytes, 0 bytes uploaded
2020-03-29 13:23:58.336 INFO BACKUP_STATS Metadata chunks: 3 total, 8 bytes; 0 new, 0 bytes, 0 bytes uploaded
2020-03-29 13:23:58.336 INFO BACKUP_STATS All chunks: 3 total, 8 bytes; 0 new, 0 bytes, 0 bytes uploaded
2020-03-29 13:23:58.336 INFO BACKUP_STATS Total running time: 00:00:03
2020-03-29 13:23:58.336 WARN BACKUP_SKIPPED 1 directory was not included due to access errors

[gchen]

With this commit, Duplicacy will error out if the repository root isn’t accessible, or if there are no files under the repository to be backed up.

[santacruzskim]

woah. major thank you @gchen. I can’t wait to test this!

[leerspace]

When is the next planned release that would include this change?

[gchen]

A new CLI release is planned for this week, then followed by a new web GUI release next week.