No such file or directory error when using -vss on macOS during backup

Support
#1

We are backing up with the command line version 2.52 and using the vss option. We are able to see the snapshot mounted in “/tmp” during the backup.

During the backup errors like

Failed to open file for reading: open /tmp/snp_388793941/Users/mb/Library/Application Support/com.xericdesign.earthdesk/Desktops/1-7-7789-23313-4294967295 (4124139A-430F-4093-83A5-36BB0A33740B).jpg: no such file or directory

are logged. The file does not exist in the snapshot so that is triggering the error message. The big question is, why is the file on list of files to backup. To my understanding duplicacy should make a snapshot, then perform the analysis in the snapshot tree, perform the backup and release the snapshot. If this is implemented correctly an error of this kind should not happen - the reason for using snapshots is to prevent this kind of error in live systems during a backup…

#2

Is Duplicacy granted full disk access privilege in Securty section of System Preferences?

#3

Yes, it has. The problem are files that are temporary in nature and vanish after some time. If they would live in one of the /tmp directories they would be filtered away but the developer of the app in question decided to put them in some other directory…

#4

Oh, I see. I think you make a valid point. I have misread your post.

This does look like the files were enumerated before the snapshot was taken. Let’s wait for @gchen to comment.

#5

The files are enumerated after the snapshot is taken. See this:

@michael1, is it possible to check the snapshot manually while the backup is in progress?

#6

Just some remarks about the observed behaviour:

Our backups are relatively slow (> 24 hours) as the internet connection is via GSM and not the fastest ;-(.

All files we saw the problem occurring are sort of temporary files that are not identifiable as such and pass our regex’s to filter temporaries. The one I mentioned in this post is a cached image of planet earth redone every some minutes to be used in a screen saver. The same error has appeared with log files and similar kind of files.

We did check the snapshot manually during the backup: After the error happened we inspected the mounted snapshot and indeed the file was not in the snapshot. So our “educated guess” was that the sequencing of generating the snapshot and traversing the file tree was in trouble.

We are not really worried about these files as they are temporary in nature and the system can usually regenerate them if not included in. a backup. Bigger problems might arise if running virtual machines show the same behaviour during backup.

#7

I found this: macos - apfs snapshot changing after creation? - Ask Different

So maybe one of the parent directories is a symlink.

#8

I checked the file tree up to the files (which are also no symlinks) but no symlinks inside there. But a good point - so its impossible to do a save snapshot of a directory hierarchy containing symlinks because even multiple snapshots wouldn’t be in an atomic transaction…

Blockquote
❯ ls -lad “/Users/mb/Library/Application Support/com.xericdesign.earthdesk/Desktops”
drwxrwxrwx 62 mb staff 1984 May 27 16:47 /Users/mb/Library/Application Support/com.xericdesign.earthdesk/Desktops
❯ ls -lad “/Users/mb/Library/Application Support/com.xericdesign.earthdesk”
drwxr-xr-x 7 mb staff 224 May 27 16:48 /Users/mb/Library/Application Support/com.xericdesign.earthdesk
❯ ls -lad “/Users/mb/Library/Application Support”
drwx------@ 199 mb staff 6368 May 27 13:11 /Users/mb/Library/Application Support
❯ ls -lad “/Users/mb/Library/”
drwx------+ 100 mb staff 3200 May 17 16:11 /Users/mb/Library/
❯ ls -lad “/Users/mb”
drwxr-xr-x+ 91 mb staff 2912 May 27 16:50 /Users/mb
❯ ls -lad “/Users”
drwxr-xr-x 7 root admin 224 Sep 29 2019 /Users

Blockquote

#9

Maybe the difference here is that duplicacy should be backing up from [snaphot of] /System/Volumes/Data/Users/usename and not /Users/usename, which is “fake”?

The latter one literaly resides on a separate APFS volume (see root vs /System/Volumes/Data/

me@obsidian ~ % mount 
/dev/disk1s2 on / (apfs, local, read-only, journaled)
/dev/disk1s1 on /System/Volumes/Data (apfs, local, journaled, nobrowse)
...

me@obsidian ~ % diskutil list
...
/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +3.8 TB     disk1
                                 Physical Store disk0s2
   1:                APFS Volume Admin - Data            1.6 TB     disk1s1
   2:                APFS Volume Admin                   14.8 GB    disk1s2
...
#10

For testing purposes I was creating snapshots with tmutil, mounting them and analysing the branch of the directory in question again without running duplicacy. Looking with the finder at the directory tree I saw a “~/Library/ApplicationSupport” symlink pointing to “~/Library/Application Support” (! the missing SPC is not a typo !). Looks like a command line application did not like file paths with an embedded space, iTerm was known to create this sort of symlinks in former releases and I am using it for a while. So I think gchen was right suspecting symlinks to cause the trouble, duplicate will follow the file tree once over the regular subdirectory and once via the symlink looking not at the snapshot but the live filesystem, causing the trouble. Defaulting for not following symlinks and a command line switch to follow symlinks would make live easier - there might be a couple of symlinks in big filesystems hidden like this…

2 Likes
#11

Good catch.

AFAIK Duplicacy however does not follow symlinks. To be precise: it only follows symlinks that are in the root of the repository to enable scenarios where you create one repository folder and symlink a bunch or stuff to it from random locations. It should not follow symlinks elsewhere and should backup symlink as a symlink.

#12

Then we would be back to square zero.