Docker folder not being backed up

[JohnAtl]

I’m not sure if this is a filter problem, or something else. I’m using the GUI.

I have a drive attached to my iMac Pro thus:

(base) Johns-iMac-Pro :: /Volumes/Data » tree -L 1
.
|-- Archive
|-- Docker
|-- MATLAB
|-- Research
|-- School
|-- ScreenFlow
|-- Sync
|-- Sync_Home_75744.btskey
`-- test.txt

7 directories, 2 files
(base) Johns-iMac-Pro :: /Volumes/Data » tree -L 2 Docker
Docker
`-- DockerDesktop
    |-- Docker.raw
    |-- console-ring
    |-- guest.000005f5 -> 00000003.000005f5
    |-- guest.00000948 -> 00000003.00000948
    |-- hyperkit.json
    `-- log

2 directories, 5 files
(base) Johns-iMac-Pro :: /Volumes/Data »

My filters are thus:

However, the only file being backed up from the Docker folder is .DS_Store:

Based on searches here on the forum, I went to the /Volumes/Data folder, and tried to test the filters:

(base) Johns-iMac-Pro :: /Volumes/Data » duplicacy -d -log backup -enum-only
2021-10-07 16:14:35.160 ERROR REPOSITORY_PATH Repository has not been initialized

but it looks like the GUI works differently than the CLI as far as initializing repositories and storing filters. I do see the file structure inside ~/.duplicacy, but don’t know where to go from here.
Thanks for any help!

[saspus]

Questions:

• Are you running Duplciacy Web manually under your user account or as a superuser via launchd?
• What are permissions on docker files? Run ls -alt /Volumes/Data/Docker/DockerDesktop. Are they readable by the user duplicacy runs as?
• Are these files marked for exclusion for time machine? Run e.g tmutil isexcluded /Volumes/Data/Docker/DockerDesktop/Docker.raw
• do you pass -vss flag to backup job OR is docker shut down when backup runs? (-vss shall be default – I can’t think of any reason why anyone not want to always use it)

If still inconclusive – add -d global flag to backup job and then check the logs to see which files does it actually try to pick up and what happens to them.

As a side note – you likely don’t want to backup docker files inside that folder to begin with. That data is disposable, as are containers. Instead, backup external volumes (folders) that mount into containers

[JohnAtl]

Thanks for the reply.

• Duplicacy Web is running as me.
  501 550 1 0 Wed07AM ?? 2:42.03 /Applications/Duplicacy Web Edition.app/Contents/MacOS/duplicacy_web_osx_x64
• I am the owner of all the files, with rw permissions.

(base) Johns-iMac-Pro :: /Volumes/Data » ls -alt /Volumes/Data/Docker/DockerDesktop
total 39089312
-rw-r--r--  1 john  staff  137438953472 Oct  2 17:52 Docker.raw
drwxr-xr-x@ 8 john  staff           256 Oct  1 17:52 .
drwxr-xr-x  4 john  staff           128 Aug  8 12:32 ..
-rw-r--r--  1 john  staff         65536 Aug  8 12:25 console-ring
lrwxr-xr-x  1 john  staff            17 Aug  8 12:25 guest.000005f5 -> 00000003.000005f5
lrwxr-xr-x  1 john  staff            17 Aug  8 12:25 guest.00000948 -> 00000003.00000948
-rw-r--r--  1 john  staff          2432 Aug  8 12:25 hyperkit.json
drwxr-xr-x  2 john  staff            64 Aug  8 12:25 log

• Yes, the whole drive says [Excluded]. I don’t currently use TimeMachine.

(base) Johns-iMac-Pro :: /Volumes/Data » tmutil isexcluded /Volumes/Data
[Excluded]    /System/Volumes/Data/Volumes/Data

• I do pass -vss, but Duplicacy ignores it:
  2021-10-07 12:47:04.262 WARN VSS_PATH VSS not supported for non-local repository path: /Volumes/Data
  /Volumes/Data are two SSDs running RAID0 as created by Disk Utility.

Thanks for the advice about backing up the Docker files.

At this point, my concern is more for learning why this folder was excluded (except for .DS_Store), so that I can ensure the things that matter are being backed up.

[saspus]

@JohnAtl

Yes, the whole drive says [Excluded]. I don’t currently use TimeMachine.

Duplicacy, like many other backup tools, now honors Time Machine exclusions (specifically, a special extended attribute applications can set on a file they think should not be backed up; this, as opposed to centrally managing exclusions, allows decisions to be made by entities best suited for the task). This is configurable, and is by default enabled when using Web UI.

A little demo:

% rm test
% touch test
% xattr test
% tmutil isexcluded test
[Included]    /System/Volumes/Data/Users/alex/Downloads/test
% tmutil addexclusion test
% tmutil isexcluded test
[Excluded]    /System/Volumes/Data/Users/alex/Downloads/test
% xattr test
com.apple.metadata:com_apple_backup_excludeItem
% xattr -c test
% xattr test
% tmutil isexcluded test
[Included]    /System/Volumes/Data/Users/alex/Downloads/test
%

@JohnAtl

/Volumes/Data are two SSDs running RAID0 as created by Disk Utility.

Doh! I confused it with /System/Volumes/Data/… Ok, so this is an external drive. It will be excluded from time machine, but not via extended attributes; so as long as those are not set on the files (I’ll be pleasantly surprised if Docker marks its discardable files with the right attributes, but please do check) they should be picked up.

If Docker is running however these files are locked and may not be readable by other apps – perhaps that’s the root cause. Run backup with -d flag to see whether Duplicacy revisits those files or why does it skip them. This is one of many reasons why back-up shall be reading from a snapshot (-vss, naming from windows world)

In your case however -vss does not work on external drive because duplicacy is asking time machine to create APFS snapshot for it (tmutil localsnapshot /); in order to use filesystem snapshotting API directly application developers are expected to get Apples’s permission. (Acrosync did not reach out to Apple; ArqBackup did, and they use native snapshots for quite a while now and are not reliant on Time Machine)

[JohnAtl]

Okay, added -d in the global options, and it says:

...
2021-10-07 22:44:40.457 DEBUG LIST_ENTRIES Listing Docker/
2021-10-07 22:44:40.458 DEBUG PATTERN_INCLUDE Docker/.DS_Store is included
2021-10-07 22:44:40.458 DEBUG PATTERN_INCLUDE Docker/DockerDesktop/ is included
2021-10-07 22:44:40.458 DEBUG LIST_EXCLUDE Docker/DockerDesktop/ is excluded by attribute
...

And lo and behold:

(base) Johns-iMac-Pro :: /Volumes/Data » xattr Docker/DockerDesktop
com.apple.metadata:com_apple_backup_excludeItem

I was able to clear the attribute, then set it back (using value of 1 (?)).

Okay, I feel better. It’s still a little weird that the whole volume and the directories and files I checked within say [Excluded] by tmutil, but it looks like everything else is included.

Thanks so much for your help!

[saspus]

@JohnAtl

It’s still a little weird that the whole volume and the directories and files I checked within say [Excluded] by tmutil, but it looks like everything else is included.

This is because time machine would not backup that folder for other reasons; this time not because of an attribute but because it’s an external volume. The attribute is just one mechanism TM is using to make a decision. In other words — attribute present? Skip. Attribute not present — check other things.

@JohnAtl

I was able to clear the attribute, then set it back (using value of 1 (?)).

You can use tmutil addexclusion <path>; this would be more compatible way to do it going forward. Btw I think it sets the value other than 1. I don’t know if specific value carries any meaning.