Backing up stuff on a different hard drive

Support
#1

I have a data storage drive which contains almost everything I want to back up. This drive is separate from my system drive, which contains the \Users\lphelan folders and files. I also want to back up these folders and files.

  1. From reading around here, it seems that using symlinks is the only way to accomplish this within one Duplicacy job. Is this correct?

  2. If I don’t use symlinks, and instead make two jobs, one for backing up my data storage drive, and another to back up \Users\lphelan on my system drive, can I get both these backups to run automatically if I schedule them? If so, can both jobs run without interfering with each other? (As a test, I manually started two jobs at the same time. They ran sequentially; the second job started up as soon as the first job completed. I think this is acceptable behavior, since after my initial backup is done and only incremental changes will need backing up, both jobs will have plenty of time to complete on a regular basis.)

  3. Are there any other options for accomplishing what I want to do?

How to automatically run as administrator at startup?
Duplicacy vs Duplicati
#2

The symlink approach was recommended as a work around when the GUI version could have only one repository. Now with the multiple repository support in 2.1.0, you don’t really need to create symlinks.

Yes, in 2.1.0 you can have multiple jobs each of which has its own schedule. Only one job will be run at a time so a job whose scheduled time has been reach will have to wait until the current job finishes. If there are multiple waiting jobs then they are sorted by their scheduled times (whichever is supposed to run the earliest will become the next running job).

#3

Thanks, gchen. So, I set up my first “real” (not just testing) backup, consisting of two jobs. One job backs up some folders from my data drive; the other job backs up my \Users\lphelan folder on my system drive.

The data drive job completed successfully, but the \Users\lphelan job had some errors. Here are the error-related log lines (I skipped the log entries that looked normal to me).

2018-01-24 18:16:36.385 INFO STORAGE_SET Storage set to b2://HP13B2
2018-01-24 18:16:38.834 INFO BACKUP_START No previous backup found
2018-01-24 18:16:38.838 INFO VSS_CREATE Creating a shadow copy for C:
2018-01-24 18:16:41.788 INFO VSS_DONE Shadow copy {EDA8EF7F-F296-40F7-A126-892501247D5F} created
2018-01-24 18:16:41.789 INFO BACKUP_INDEXING Indexing C:\Users\lphelan
2018-01-24 18:16:41.789 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2018-01-24 18:16:41.791 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Application Data: Access is denied.
2018-01-24 18:16:41.791 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Cookies: Access is denied.
2018-01-24 18:16:41.791 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Local Settings: Access is denied.
2018-01-24 18:16:41.792 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\My Documents: Access is denied.
2018-01-24 18:16:41.792 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\NetHood: Access is denied.
2018-01-24 18:16:41.792 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\PrintHood: Access is denied.
2018-01-24 18:16:41.792 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Recent: Access is denied.
2018-01-24 18:16:41.793 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\SendTo: Access is denied.
2018-01-24 18:16:41.793 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Start Menu: Access is denied.
2018-01-24 18:16:41.793 WARN LIST_LINK Failed to read the symlink: readlink C:\Users\lphelan.duplicacy\shadow\Users\lphelan\Templates: Access is denied.
2018-01-24 18:16:42.325 WARN LIST_LINK Failed to read the symlink AppData/Local/Packages/Microsoft.Microsoft3DViewer_8wekyb3d8bbwe/LocalCache: Unhandled reparse point type 80000018
2018-01-24 18:16:43.880 INFO BACKUP_LIST Listing all chunks
2018-01-24 18:16:43.992 INFO BACKUP_THREADS Use 4 uploading threads
2018-01-24 18:16:43.994 INFO PACK_END Packed .recently-used.xbel (218)

…a zillion …Packed… and Uploaded chunk… lines

2018-01-24 21:19:50.864 INFO UPLOAD_PROGRESS Uploaded chunk 1639 size 5644942, 763KB/s 01:04:56 73.8%
2018-01-24 21:19:50.864 ERROR CHUNK_MAKER Failed to read 0 bytes: read C:\Users\lphelan/.duplicacy\shadow\Users\lphelan/AppData/LocalLow/Google/GoogleEarth/Cache/unified_cache_leveldb_leveldb2/003311.sst: The device is not ready.
2018-01-24 21:19:50.889 WARN VSS_DELETE Failed to delete the shadow copy: 80042308
2018-01-24 21:19:50.900 INFO INCOMPLETE_SAVE Incomplete snapshot saved to C:\Users\lphelan/.duplicacy/incomplete
[21:19:50] C:\Users\lphelan Backup operation returned an error:
ERROR Failed to read 0 bytes: read C:\Users\lphelan/.duplicacy\shadow\Users\lphelan/AppData/LocalLow/Google/GoogleEarth/Cache/unified_cache_leveldb_leveldb2/003311.sst: The device is not ready.

I see lots of suspicious looking things: “Access is denied” - I had Enable Volume Shadow Copy selected. Error Chunk Maker. Incomplete snapshot. Backup operation returned an error. Device not ready.

Do you know what went wrong?

Edit: Sorry the log lines are formatted so poorly. Is there a way to make them more readable, should I need to post log entries again?

#4

You can enclose log lines with three back-ticks. See the example at the bottom of this page.

This is the error that aborted the backup:

2018-01-24 21:19:50.864 ERROR CHUNK_MAKER Failed to read 0 bytes: read C:\Users\lphelan/.duplicacy\shadow\Users\lphelan/AppData/LocalLow/Google/GoogleEarth/Cache/unified_cache_leveldb_leveldb2/003311.sst: The device is not ready.

GoogleEarth didn’t implement the VSS writer correctly so Duplicacy couldn’t read the content of the file. Since this file under a cache I assume you can safely skip it with an exclude pattern.

As for those symlink warnings, if you run without VSS will you be able to read them?

#5

I decided to exclude the whole cache directory, not just that one file. It fixed the Google Earth problem, and the new backup did not get aborted.

I also turned off Enable Volume Shadow Copy, but I still got those “failed to read the symlink” messages in the log. Also, I got a bunch (48, according to the log) of new “the process cannot access the file because it is being used by another process” messages.

#6

Sometimes even an administrator may not have the permissions to read a user folder depending on the configuration. Can you check the security setting of c:\Users\lphelan to see if the administrator can read this folder? The effective permissions tool is very useful for this purpose: http://www.thewindowsclub.com/effective-permissions-tool-windows

#7

The Effective Permissions Tool appears to be no longer available. On the link you supplied I scrolled down to the Effective Permissions Tool section, then clicked on the Effective Permissions Tool link, which does not land on the link URL but instead forwards to https://www.microsoft.com/en-US/download/details.aspx?id=53314, which is a “retired content” page that contains only a PDF file. Is there another way to get this tool, or an alternate tool?

#8

Sorry I wasn’t clear. It isn’t a separate tool. If you right click a file in File Explorer, select Properties, switch the Security, and click on Advances, you will see the “Effective Access” tab, where you can check if a particular user or an administrator has the required permission to access the file.

#9

I checked the c:\users\lphelan folder.

User SYSTEM, user lphelan (7700T\lphelan) and user Administrators (7700T\Administrators) are all the same: ‘Allow’ is checked for all the rows (full control, modify, read & execute, etc., except the last row, ‘special permissions,’ which is not checked. ‘Deny’ is not checked for any row.

Those three are the only users in the list. 7700T is the computer name.

#10

Wait - there’s more. I looked in my \Users\lphelan folder. I found that all the folders that gave ‘Failed to read symlink’ errors (Cookies, Local Settings, My Documents, PrintHood, SendTo, Templates) are not present in the \Users\lphelan folder. So the questions may be, why did the backup process give errors for these non-existent folders? Perhaps this is a non-problem, and I should ignore these errors if they recur.

#11

Never mind. I unhid protected operating system files, and the missing folders all appeared. But they show as shortcuts to folders in another location. Anyway, I checked permissions on the Cookies shortcut, and they are the same as I listed for the \Users\lphelan folder two posts up in this thread.

#12

Can you check the permissions on the targets pointed to by those symlinks to see if the administrator can access them?

#13

How do I find the target of one of these symlinks? I right click, and see nothing relevant except Properties. In the Properties dialog, the Target field is blank. I tried a command prompt run as administrator (just in case), but a dir listing of the User\lphelan folder does not list the symlinks at all.

I noticed that in the Security tab there is another user, ‘Everyone,’ that has no permissions. Sorry I didn’t mention this before. I deleted this User from the list in this tab. Now, I can click on the, for example, Cookies symlink and it takes me to the actual Cookies folder. But it doesn’t make sense. The Cookies folder is right where the Cookies symkink is, in other words Users\lphelan\Cookies.

Now that I can go inside the Cookies folder, I checked the permissions of a folder, and a file, inside. They are the same as the permissions I listed above: ‘Allow’ is checked for all the rows (full control, modify, read & execute, etc., except the last row, ‘special permissions,’ which is not checked. ‘Deny’ is not checked for any row.

#14

My next step was going to try another backup, but I ran into other problems:

  1. Despite having installed Duplicity as administrator and installing the service, I’ve always had to run it as administrator from its shortcut. Right clicking the tray icon and clicking Open does not give an administrator access instance, so I don’t get the jobs I set up when running as admin, and Enable Volume Shadow Copy is grayed out. (This makes me wonder if the schedule would be able to run, since my jobs require admin level.)

  2. I found that today I was unable to get the admin level privileges even when running as admin from the shortcut. I figured I’d uninstall and reinstall Duplicacy, but I had trouble with that too.

  3. I found that the Uninstall shortcut in the Start Menu doesn’t work. It appears to complete, but does nothing. I tried running the Uninstall shortcut as admin, and that didn’t work either. What worked was uninstalling from Windows Control Panel - Programs and Features. I noticed that only this time did I get the UAC warning, which I allowed, and uninstall really uninstalled. Also, some time during this process I tried stopping the service, and quitting the tray icon. I don’t think these actions helped, but I’n not sure. So, I finally got Duplicity reinstalled, and I’m now on Beta 8.

#15

After the reinstall, when I run Duplicacy as admin from its shortcut, I can enable Volume Shadow Copy again. I put in the Storage info again and started the backup. I got the same ‘Failed to read the symlink’ errors as before.

By the way, I stopped the backup after I saw the ‘Failed to read…’ errors appear in the log, and I got a Duplicacy error dialog that said the program exited with status code -1. I suspect this is not really an error and occurred because I stopped the backup. If it’s not really an error, I don’t think users should get such a message.

#16

Another thing. I have a month left before CrashPlan ends. I had set its client to back up \Users\lphelan also. It never gave any errors about these symlink folders, but as a test I restored all the ‘Failed to read symlink’ folders from a CrashPlan backup to a folder on my desktop. They are all empty. Apparently, CrashPlan is unable to back up the contents of these folders also; it just remained silent about it.

I don’t really need to back up these folders. I keep my Documents on my data drive, and as far as I can tell the other folders are of no value. We can stop this troubleshooting. But if you want to try to solve it, I will continue the troubleshooting work per your instructions.

(Oh, I did also try doing the Duplicacy backup with the CrashPlan client not running, in case it was locking these folders somehow. It made no difference - still got the ‘Failed to read symlink’ errors in Duplicacy.)

#17

Can you run dir /a or dir /ah under Users\lphelan in DOS prompt? This will list detailed information about each symlink.

I found that today I was unable to get the admin level privileges even when running as admin from the shortcut. I figured I’d uninstall and reinstall Duplicacy, but I had trouble with that too.

That is because you need to exit the current running instance first. To do that, right click the system tray icon and select Quit. Otherwise, even if you attempt to run ad admin it will just bring to front the current instance without elevated privileges.

#18

I ran dir /a. In the list, all the problem symlinks appeared like this one

2017-12-21  06:07 PM    <JUNCTION>     Cookies [C:\Users\lphelan\AppData\Local\Microsoft\Windows\INetCookies]

Also,

…you need to exit the current running instance first. To do that, right click the system tray icon and select Quit.
Otherwise, even if you attempt to run as admin it will just bring to front the current instance without elevated privileges.

Which leads me to another question. Why was the current instance running without elevated privileges? I have Duplicacy set to start at login and start service on boot. The service gets started, but Duplicacy starts without elevated privileges. How can Duplicacy run my scheduled jobs in this case, since volume shadow copy is not enabled and my job tabs are not present in the GUI?