Honoring com_apple_backup_excludeItem on MacOS

[arno]

@shirazdindar

find . -xattrname "com.apple.metadata:com_apple_backup_excludeItem" -exec ls -dp {} \;

@saspus

From which directory do you run this?

The . near the start of that command refers to the current directory; wherever your shell is when you run the command. You could replace that . with whatever directory you want to search. You could, for example, use ~ to indicate your home directory or / to search your whole hard drive. You probably want to replace it with the root of your repository.

[Christoph]

I’m confused by this thread. Could someone who understands explain how I apply the -exclude-by-attribute=true setting on MacOS with the web-ui?

[arno]

You run it as a duplicacy command:
duplicacy set -exclude-by-attribute=true

[saspus]

@arno

You run it as a duplicacy command:
duplicacy set -exclude-by-attribute=true

This won’t work, as explained here:

@gchen

Unfortunately this would not work as the web GUI will recreate the preferences file every time it runs a backup job (or other jobs). In the coming web GUI version this option will be enabled by default.

Currently there is no way to set that flag when using web ui.

[arno]

Ah right. I missed the reference to the web ui. Sorry about that.

[gchen]

This is from the 1.5.0 release thread:

This is done by setting the preference key exclude_by_attribute to true in new backups created in 1.5.0, which means existing backups will not skip these files by default. If you want to change this behavior for existing backups, manually modify the preference key exclude_by_attribute in ~/.duplicacy-web/duplicacy.json to true .

[Christoph]

Thanks a lot for the clarifications.

@gchen

This is done by setting the preference key exclude_by_attribute to true in new backups created in 1.5.0, which means existing backups will not skip these files by default.

So since 1.5.0 is the first version I ever installed on my mac, I take it that this is enabled by default in my case. So this means that duplicacy will apply the same exclusions as time machine, without me doing anything, right?

[saspus]

PSA: Not all data that is excluded by Time Machine has that attribute set.

In other words, if you want to replicate Time Machine’s behavior concerning exclusions you would need to add few locations manually to the filters file.

Example folders that don’t have that attribute and yet are skipped by Time Machine:

~/Library/Logs					<-- Logs, standard locations. 
~/Library/Caches				<-- not unexpected either
~/Library/Metadata/CoreSpotlight<-- That’s interesting!
~/.Trash						<-- 

Your duplicacy filters file therefore should have at least

-Library/Logs/
-Library/Caches/
-Library/Metadata/CoreSpotlight/
-.Trash/

or

-*/Library/Logs/
-*/Library/Caches/
-*/Library/Metadata/CoreSpotlight/
-*/.Trash/

depending on whether repository root is at user’s home or elsewhere respectively

There perhaps are more instances of excluded stuff; those can be discovered with a script similar to the one below (I did not run it on the whole system though, only on my home folder, you may want to tweak it accordingly.)

#!/usr/bin/python
import os
import xattr
import subprocess

maxdepth = 5

for root, dirs, files in os.walk(os.environ["HOME"]):

    continue_traversing = True

    if "com.apple.metadata:com_apple_backup_excludeItem" in xattr.listxattr(root):
        continue_traversing = False
    else:
        if "Excluded" in subprocess.check_output(["tmutil", "isexcluded", root]):
            print(root)
            continue_traversing = False

    if continue_traversing and root.count(os.sep) >= maxdepth:
        continue_traversing = False

    if not continue_traversing and len(dirs):
        del dirs[:]

The output is a list of folders that are excluded by Time Machine but don’t have that attribute set.