Lance, if a backup fails with any error, it won’t run the post-backup script, so you’ll know something is wrong.
I’m all for feedback so please comment/correct where I’m wrong. (I am new to Duplicacy)
So, in essence to backup my family and have decent monitoring in place I’ve come to the conclusion that the Duplicacy GUI is not sufficiant. As such I think I can do what I want with the command line version and a small program around it.
Please take a look at the flow if you dont mind and give me your feedback:
https://www.lucidchart.com/documents/view/8d385416-1dc1-46e0-b6f7-ef08fe930aa8/0
I’d probably code this in something cross platform like python with a .yaml file for all the variables, but I think this would do what I want…
I’m not quite clear on why you’d want to check the date of last snapshot file.
If you get a success code (exit 0) from duplicacy, what situations would there be where the backup didn’t actually succeed? And if such cases do exist, wouldn’t that be a bug?
Maybe I am mistaken but i’ve observed a few conditions:
ERROR when File should be accessible but is not
WARN when a file simply is not accessible
And backup completes with snapshot when WARN condition occurs… See the below from my logs
01:25:27.848 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\Application Data: Access is denied.
01:25:27.849 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\###\Cookies: Access is denied.
01:25:27.849 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\Local Settings: Access is denied.
01:25:27.850 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\My Documents: Access is denied.
01:25:27.850 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\###\NetHood: Access is denied.
01:25:27.850 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\PrintHood: Access is denied.
01:25:27.851 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\Recent: Access is denied.
01:25:27.851 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\SendTo: Access is denied.
01:25:27.851 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\Start Menu: Access is denied.
01:25:27.852 Failed to read the symlink: readlink C:\Users\####\.duplicacy\shadow\Users\####\Templates: Access is denied.
01:25:29.635 Failed to read the symlink AppData/Local/Packages/#########_b6e429xa66pga/LocalCache: Unhandled reparse point type 80000018
01:25:29.863 Failed to read the symlink AppData/Local/Packages/Microsoft.Microsoft3DViewer_8wekyb3d8bbwe/LocalCache: Unhandled reparse point type 80000018
<snip>
1:29:08.712 Backup for C:\Users\#### at revision 41 completed
01:29:08.712 Files: 64863 total, 38,638M bytes; 6544 new, 1,690M bytes
01:29:08.712 File chunks: 8074 total, 40,886M bytes; 114 new, 663,321K bytes, 322,280K bytes uploaded
01:29:08.712 Metadata chunks: 9 total, 21,866K bytes; 9 new, 21,866K bytes, 7,410K bytes uploaded
01:29:08.712 All chunks: 8083 total, 40,907M bytes; 123 new, 685,188K bytes, 329,690K bytes uploaded
01:29:08.712 Total running time: 00:03:47
01:29:08.712 12 files were not included due to access errors
01:29:08.789 The shadow copy has been successfully deleted
Now, when I look at those specifically, they all are not really an issue for me and they should be excluded;however , it does tell me that there are “Failed” conditions that the backup engine accepts as “WARN” conditions in the logs and completes the backup.
or am I missing something obvious?
Thanks!
What is the exit status of this backup? While it finished, was the exit code zero or non-zero?
Honestly, I’m unconvinced that this is an error anyway. After all, if you don’t have access to files that you tried to back up, but everything you did have access to backed up successfully, I’d call this a “good” backup. If you want to flag it for errors, I guess you can (grep for “not include due to access errors”). Personally, I’d just look at the logs from time to time (once every few weeks, maybe), even on success, to trap this sort of thing. Or you might want to include, in the success E-Mail, the last 10 lines of the log or so (making it super trivial to just look at the back log sent via E-Mail).
I back up files as my user account (non-privileged). If I try to files that I don’t have access to, I’d expect this error. I’d rather have this than run with privileges; I always minimize what runs with privileges for obvious reasons.
What is the exit status of this backup? While it finished, was the exit code zero or non-zero?
Honestly, I’m unconvinced that this is an error anyway. After all, if you don’t have access to files that you tried to back up, but everything you did have access to backed up successfully, I’d call this a “good” backup. If you want to flag it for errors, I guess you can (grep for “not include due to access errors”). Personally, I’d just look at the logs from time to time (once every few weeks, maybe), even on success, to trap this sort of thing. Or you might want to include, in the success E-Mail, the last 10 lines of the log or so (making it super trivial to just look at the back log sent via E-Mail).
I back up files as my user account (non-privileged). If I try to files that I don’t have access to, I’d expect this error. I’d rather have this than run with privileges
I dont have the specific exit codes as that was a GUI scheduled backup as a service – based on the published documentation, it should have been 0: Exit codes details
My next step would be to validate this for these types of situations because if the exit code on fully successful was zero, on successful backup with warnings is something like 1, that makes the automation of the monitoring much easier – but also unpublished “features” 
While I agree with the general concept of unprivileged, shadow copies and filesystem snapshots are fantastic tools to get consistent backups. For example, if a very large file changes while backing up, without some sort of snapshot tech in place, your backup could end up with 1/2 of the old version and 1/2 of the new version as a single file and not know better – entirely possible duplicacy handles this condition as well, but unknown for me. Other files are almost always locked but are important like .pst files.
In any case, Duplicacy handles the portions that require elevated accounts, while what I am planning on coding is some automation around detection of failures and non-running backups.
For the first MVP, i’d probably do exactly that and simply output the status as email – however, with a goal of a lot more “fire and forget” type of automation such that the emails are reduced to digest type of communications for success and regular for error/warning.
I’ve got a github setup where i’m starting to store my artifacts, but it’s very minimal at the moment. I’ll post back as I capture exit codes and initial code.
I’ve cobbled together some python to launch the command line version of duplicacy and found this (last line is python reporting the exit code of duplicacy)
12 files were not included due to access errors
The shadow copy has been successfully deleted
Command exit status/return code : 0
go.py is my testing code and above is the repo i’ll start developing my wrapper.
Hey Lance,
You may want to take a look at my cross-platform “Go” program to run Duplicacy backups. I need docs (and will work on it if someone needs it next), but I’m using this for my stuff pretty successfully.
It was originally just going to be for Mac OS/X, but I decided in the end to write it in “Go”. Partly this lets me learn “Go”, and partly this gives me portability. I test on Windows but primarily run it on OS/X.
If you have questions, send me E-Mail or post an issue to the repository.
I just got an E-Mail framework going. It has log files, log file rotation, short summaries (printed on screen, will be body of E-Mail message). All of this was written to be completely portable. “Go” is a great language for that, as there is a rich library of packages (both in Go itself and elsewhere).
1 Like
ooh- I will take a look at this too!
Looks like I could use a tiny bit of help to get started. I too am on OS/X.
Not sure how to begin to try out your code… can you give me a 3-step quick start? Thanks very much!!
Looks like I could use a tiny bit of help to get started. I too am on OS/X.
Not sure how to begin to try out your code… can you give me a 3-step quick start? Thanks very much!!
I had the same need: send myself emails when no backups were received for 3 days, and again after 7 days.
I created a small PHP script that uses the Duplicacy CLI to check all the backups found in specific folders, and use the list command to find the latest snapshot available for each, and send myself an email when it was too old.
If anyone wants to use it, I published it here: https://gist.github.com/gboudreau/91b8866e3adb19965897ecd80a12525b
See instructions at the top of the script.
Example output:
[2018-05-17 20:33] Last backup from Guillaumes-MacBook-Pro-gb-gb was 0 hours (0 days) ago...
[2018-05-17 20:33] OK.
or when it fails:
[2018-05-17 20:45] Last backup from Guillaumes-MacBook-Pro-gb-gb was 1 hours (3 days) ago...
[2018-05-17 20:45] Oh noes! Backup from 'Guillaumes-MacBook-Pro-gb-gb' is missing for more than 3d!!
[2018-05-17 20:45] Sending email notification to you@gmail.com
Comments / questions / Pull Requests are welcome.
Cheers,
- Guillaume
PS Tested on Linux; pretty sure it would work as-is on Mac. On Windows: meh!
Great! I have been trying to put together something like this too… my biggest problem is that my sftp server is a QNAP and QNAP doesnt support any sort of command line mail utility (sendmail etc)… and that is REAL irritating.
PS- I too am a crashplan refugee and trying to get the same features that I am used to from Duplicacy I now have about 8 people backing up to my basement using it…
Kevin, please contact me via E-Mail, and I’ll be happy to give you help getting started. You can get my E-Mail from any of the commit messages in my repository. I also sent you E-Mail at your E-Mail address, you can respond to that.
Note that one advantage of Go, as a language, is that it makes self-contained executable files. The capability to send E-Mail, for example, is totally built into the executable and requires no command line utility for E-Mail.
Knowing that others are interested, I’ll start getting to work on the README.md file.
If you’re interested in duplicacy-util, I’ve documented it’s usage. I’ll keep it up to date as I add new features.
If you try it and have any issues or questions, please raise an issue on GitHub. I’ll respond promptly and either answer your question, update the documentation to include your answer, or both!
For the record, I was using Crashplan years ago, but moved to alternatives a few years back because Crashplan didn’t scale well. As your backups got larger and larger, Crashplan took more and more memory and suffered problems at an increasing rate. After I felt like I was “baby sitting” Crashplan far too often, I aborted it and looked elsewhere. I’ve been off of Crashplan for about 3 years now so, when they cancelled their services, it didn’t really affect me.
HI Jeffaco, this is great. I guess I am too stupid to find your email address in your commit’s. And I didnt get a message from you…
From the point of view of server side monitoring- Gilbert is working on some stuff that I think is important… although I dont know when we might see it. He was going to add a feature to duplicacy that uploaded a list of chunks that should be in each snapshot along with some info that would allow at least a tiny bit of server side data integrity monitoring. I hope that gets in at some point and if so- maybe it could be added to your util.
I am hoping to try out your stuff soon…
Hi Kevininv. Apparently, GitHub hides the E-Mail address pretty well. It’s kind of funny, too, since my E-Mail address is on every commit. But to see that, you need to download the repo and then do a git log, and then you’ll see it.
At this point, docs are in the repo, so if you have comments or questions, I prefer them in GitHub as issues. That way, I can track the issues and they get closed as I resolve them (by commit or whatever). So this is actually better now.
But that said, if you want to reach me, my domain is taltos dot com and my user is jeff (done this way to avoid scraping for email addresses).
Hi @kevininv Just FYI, I’ve spent a good chunk of time on documenting duplicacy-util, including how to set up server-side monitoring with Gmail. With this setup, Gmail itself will notify you if duplicacy-util stops working (for whatever reason).
I’ve also documented how to automatically run duplicacy-util from Mac OS/X (my platform), Linux, and Windows.
@lance I’ve designed duplicacy-util to be extendable. If you wanted to use it and found it useful, I could trivially modify it to watch for specific messages if you wanted different behavior for E-Mail notification.
Jeffaco, I hope to check it out soon!!!