Error: ERROR VSS_CREATE Error while mounting snapshot: exit status 73

[gecko]

Please describe what you are doing to trigger the bug:

Trying to run a backup on an SFTP storage using the CLI, running on MacOS Mojave 10.14.6. I am using the -vss option to enable the use of APFS snapshots, as well as -vss-timeout 400 . The CLI is successfully creating the snapshot, but is unable to mount it, and so errors out with ERROR VSS_CREATE Error while mounting snapshot: exit status 73

Please describe what you expect to happen (but doesn’t):

-vss creates an APFS snapshot and then mounts it from /tmp/snp_*

Please describe what actually happens (the wrong behaviour):

The snapshot is successfully created, but is unable to be mounted.
tmutil listlocalsnapshots /
shows that a snapshot was created at the time of the duplicacy backup attempt. There is also a snp_* folder created in /tmp/ with the same time created as the backup attempt.

For whatever reason, duplicacy was unable to mount the local snapshot in the tmp directory, even though it was successful at creating the snapshot and creating a folder in the /tmp/ directory.

Even stranger - the Web-UI is not having the same issue. Running a backup from the Web-UI using the same -vss and -vss-timeout 400 options results in a successful APFS snapshot being mounted and the backup being performed using it.

I have both the Web-UI and the CLI binary in the MacOS Full Disk Access list. I have tried running the CLI as sudo. It doesn’t make a difference.

[saspus]

Can you try mounting the same snapshot manually and see what happens? (don’t use the timeout for now; adjust the snapshot name for Mojave – I think it won’t have the .local suffix)

e.g.

/sbin/mount -t apfs -o nobrowse,-r,-s=com.apple.TimeMachine.2020-08-08-082719.local / /tmp/test

That’s what Duplicacy does.

If that fails as well, next I would verify the volume. Find out your APFS data volume (diskutil list or diskutil ap list), e.g disk1s2 and do diskutil verifyVolume disk1s2. this will attempt to verify the volume in live mode, along with all snapshots. The machine will be unresponsive throughout the process, so avoid clicking around until this complete. Alternatively, you can boot to recovery and do that from there.

What kind of Mac is this? I the disk hard drive or fusion drive by a chance? (Judging by Mojave, when Catalina has been released quite a while ago it’s an old one that cannot be upgraded to Catalina; which makes it likely to have HDD, which is very likely to rot and as result make snapshot unmountable. Just speculating here).

[gecko]

@saspus
Thanks for such an informative reply! Ok, so firstly, it’s a MacBook Pro mid-2012, equipped with a Samsung EVO SSD. Catalina is supported on this model, but I chose not to upgrade from Mojave because I heard so many bad things about Catalina.

I did as you suggested. I was unable to mount the apfs snapshot manually using your command in the terminal. I was getting the error mount_apfs: mount: Operation not permitted

So I ran your verifyVolume command on the root volume, and that was fine.

I looked in the console to see if I could get more information about the failure to mount, and I saw that when I tried to manually mount the snapshot, the follwoing error was being output: Sandbox: mount_apfs(2073) System Policy: deny(1) fs-snapshot-mount /private/tmp/test

I tried running duplicacy backup again, and saw in the console that it is the same error when Duplicacy atttempts to mount the snapshot:
Sandbox: mount_apfs(2108) System Policy: deny(1) fs-snapshot-mount /private/tmp/snp_485348242

So apparently it’s something to do with the system protection and sandboxing.

Is there a reason that this happens with the CLI and not the with the Web-UI?

[saspus]

This is interesting.

Neither duplicacy nor duplciacy_web have any explicit entitlements (you can dump entitlements with codesign -d --entitlements :- /Applications/Path/To/The/Bundle/Or/Executable. (see EXAMPLES section in man codesign).

The child process however inherits the parent’s entitlements, so if Duplicacy_web can successfully backup with -vss flag set that means it has the right entitlements for children to inherit that allow mount_apfs to access the snapshot and likely must has full disk access manually granted; just as you indicated in the first message, so no surprise there.

Confirm that the Duplicacy_cli executable you are running manually also has full disk access – because there are quite a few duplciacy cli executables laying around after web gui is used for a while, so it may happen that you have granted access to one but using one of the others?

You can also temporarily grant Terminal or iTerm (whichever you are using) full disk access, restart it and try mounting again: it would succeed this time. This would be a good proof of concept test.

(I personally disable Secure Boot and SIP first thing on every Mac, but I like to mess with low level stuff and it would be terrible advice to suggest that to everyone, that is to say I can’t verify everything I’m saying here myself without revering it all back and possibly breaking the os)

[gecko]

@saspus

You can also temporarily grant Terminal or iTerm (whichever you are using) full disk access, restart it and try mounting again: it would succeed this time. This would be a good proof of concept test.

A-ha, progress! Adding iTerm into Full-Disk Access list enabled me to run backup with -vss and the CLI was able to mount the snapshot and complete the backup! Thank you for this advice! But, like you said, I would rather this be temporary and wouldn’t want this to be permanently granted Full-Disk Access, so advice on how to proceed would be much appreciated (I also don’t want to disable SIP etc)

@saspus

Confirm that the Duplicacy_cli executable you are running manually also has full disk access

Yep, as I’m new to Duplicacy, there is just the one CLI executable in the .duplicacy-web/bin/ directory, and that’s what I have added to the Full-Disk Access. But like I said before, that didn’t make a difference, but adding iTerm like you suggested did. In fact, taking the CLI out of full-disk access and only leaving iTerm in still enabled the backups to mount the snapshot.

[saspus]

Great. It must be wrong executable (or path) added to the full disk access or something else getting stale so that the entitlement does not get applied there.

Try the following:

• Remove Duplicacy CLi from the full disk access list.
• in terminal type open ~/.duplicacy-web/bin. Finder Window will open showing the folder with the duplicacy_cli_whatever_its_names
• drag it specifically to the Full Disk Access list.

Then try backing up using explicitly that same executable including the path:

• in terminal cd to the root of your repository
• drag the Duplicacy executable from that finder window to the terminal window to drop the path.
• append backup -vss and press enter.

This will guarantee unambiguously that we run the literally same executable from the same path that has been granted access. I would be ridiculously surprised if that did not work.

[gecko]

Unfortunately that didn’t work
I did exactly as you advised, step by step, but nope, same error
Error while mounting snapshot: exit status 73

Seems like the only way the CLI can use -vss if is the terminal app it runs in has Full-Disk Access, but I don’t think that’s a particularly safe permanent resolution

[saspus]

Fascinating. I’ll try to reproduce this on the other Mac once I get home. I can’t think of any other reason for this behavior.

What is the specific version of your macos? (sw_vers in terminal — the two digits/letter/more digits, e.g. 18G6020)

[gecko]

@saspus

What is the specific version of your macos? ( sw_vers in terminal

Output of that is
ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G6020

Look forward to hearing more about this after you try and reproduce it Thanks again for your help

[saspus]

Ok, I’ve reproduced this issue on Catalina.

And I found this thread about which I have forgotten, describing your exact issue and this seems to be the culprit: -vss not working with Catalina

@arno

I’m working through the same thing right now and my understanding (one source ) is that simple binary executables (the CLI app) can’t be added to the Full Disk Access list. Terminal

It appears you cannot add naked binary executables to full disk access, only app bundles. Which, given that the UI allows you to do that, seems to be a bug. I would report it to Apple.

The bunch of workarounds are described in the article linked from that same comment, and also here:

@timnn

For those who are in a similar situation, I finally managed to make it work using the fdautil shipped with LaunchControl , which was written specifically for granting Full Disk Access to launchd entries.

As a proof of concept I’ve added an automator action to the new workflow as follows:

on run {input, parameters}
    do  shell script "cd ~/source; ~/.duplicacy-web/bin/duplicacy_osx_x64_2.6.1 backup -vss --dry-run"
end run

And saved it as “~/Test.app”. Then added that Test.app to full disk access. Then initialized an empty repo:

mkdir ~/source
touch ~/source/file
mkdir ~/target
cd ~/source
~/.duplicacy-web/bin/duplicacy_osx_x64_2.6.1 init test ~/target

Then run the Test.app. Backup succeeded.

So, this seems to be a bug and the workaround is to write a wrapper around duplicacy CLI engine for various tasks… or disable SIP.

This by the way seemed to reproduce another issue: the duplicacy CLI said:

Packed file (0)
The snapshot contains an error: The file file ends at chunk 0 while the number of chunks is 0

This seems to be due to the only file in the repository being 0 size. Adding content there removes the issue. It’s a duplicacy bug - summoning @gchen.