Revisiting filters - no love for me

[MorganG]

Hi - I thought I understood filters, but now I am not sure.

I want to include a separate volume in my main backup of root “/”. I read somewhere here that the way to do it is make a soft link of that volume to root, and it will be included - but it isn’t. I have a link, let’s call it “Drobo”, that links from /Volumes/Drobo to /Drobo. In my include patterns I have put

 +Drobo/

It’s now the very first thing in the filters file, it used to be much lower down. I’ve also tried it like

+Drobo/* and some other variants.

It is never included.

Maybe I’m missing something, so I re-read the Filters/Include exclude patterns page here.

I got even more confused by this example:

For instance, the following pattern list doesn’t do what is intended, since the foo directory will be excluded so the foo/bar will never be visited:

+foo/bar/*
-*

Yet above this, it says:

When matching a path against a list of patterns, the path is compared with the part after “+” or “-”, one pattern at a time. Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison.

So if it truly follows this logic, I am confused.

Say I had a file

foo/bar/bot.txt

If it compares top to bottom as the text says, it would first compare

foo/bar/bot.txt 

to the pattern

foo/bar/*

This should match, right? So, accordingly, it stops comparison and never gets to

-*

and it should work as intended.

Now, I suspect that it is more complicated, because I wonder if you’re actually building an in-memory file-tree from the patterns in the file, and doing includes/excludes against the tree, not against the actual list of patterns? If that were the case, then your statement

For instance, the following pattern list doesn’t do what is intended, since the foo directory will be excluded so the foo/bar will never be visited:

Makes a lot more sense.

So, now I am wondering, what’s really going on with these, and most importantly, why can’t I get my external volume to be included?

Thanks!
Morgan

[Droolio]

I remember reading somewhere that symlinks on some systems are seen as files, not folders. You may have to include both:

+Drobo
+Drobo/

Edit: Possibly relevant thread.

[saspus]

TLDR: folder and symlink to a folder behave differently for the purposes of filtering. I think this shall be addressed.

Actual second version of this post follows

Ok. I first freaked out, wrote a lengthy bug report with reproduction, and then just tried the equivalent of:

+Drobo
+Drobo/*
-*

essentially adding that symlink explicitly and it worked:

alexmbp:test alex$ ls -alt
total 0
drwxr-xr-x   3 alex  wheel   96 Jan  2 20:27 folder/
drwxr--r--   5 alex  wheel  160 Jan  2 20:27 .duplicacy/
drwxr-xr-x   5 alex  wheel  160 Jan  2 20:27 ./
drwxrwxrwt  13 root  wheel  416 Jan  2 20:23 ../
lrwxr-xr-x   1 alex  wheel   31 Jan  2 20:14 symlink@ -> /Volumes/Duplicacy Web Edition/

Filters:

alexmbp:test alex$ cat .duplicacy/filters
+symlink
+symlink/*
+folder/*
-*

symlink is a symlink to a mount point, and folder is a folder.

alexmbp:test alex$ duplicacy -d backup --dry-run
Storage set to /tmp/storage
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
top: /tmp/test, quick: true, tag:
Downloading latest revision for snapshot test
Listing revisions for snapshot test
No previous backup found
Indexing /tmp/test
There are 0 compiled regular expressions stored
Loaded 4 include/exclude pattern(s)
Pattern: +symlink
Pattern: +symlink/*
Pattern: +folder/*
Pattern: -*
Listing
symlink is included by pattern +symlink
folder/ is included by pattern +folder/*
Listing folder/
folder/file is included by pattern +folder/*
Listing symlink/
symlink/.DS_Store is included by pattern +symlink/*
symlink/.VolumeIcon.icns is included by pattern +symlink/*
symlink/Applications is included by pattern +symlink/*
symlink/.background/ is included by pattern +symlink/*
symlink/Duplicacy Web Edition.app/ is included by pattern +symlink/*
Listing symlink/.background/
Listing symlink/Duplicacy Web Edition.app/
symlink/Duplicacy Web Edition.app/Contents/ is included by pattern +symlink/*
Listing symlink/Duplicacy Web Edition.app/Contents/
symlink/Duplicacy Web Edition.app/Contents/Info.plist is included by pattern +symlink/*
symlink/Duplicacy Web Edition.app/Contents/MacOS/ is included by pattern +symlink/*
symlink/Duplicacy Web Edition.app/Contents/Resources/ is included by pattern +symlink/*
symlink/Duplicacy Web Edition.app/Contents/_CodeSignature/ is included by pattern +symlink/*
Listing symlink/Duplicacy Web Edition.app/Contents/MacOS/
symlink/Duplicacy Web Edition.app/Contents/MacOS/duplicacy_web_osx_x64 is included by pattern +symlink/*
Listing symlink/Duplicacy Web Edition.app/Contents/Resources/
symlink/Duplicacy Web Edition.app/Contents/Resources/duplicacy.icns is included by pattern +symlink/*
Listing symlink/Duplicacy Web Edition.app/Contents/_CodeSignature/
symlink/Duplicacy Web Edition.app/Contents/_CodeSignature/CodeResources is included by pattern +symlink/*

…

So, I see that the behaviors is inconsistent here:

1. to include a contents of a folder you jus need to specify +Folder/*, the +Folder itself was not necessary.
2. But when that folder is a symlink you must specify explicitly both the symlink name and its contents.

Here is what happens if I remove the +symlink and leave only +symliink/*

alexmbp:test alex$ cat .duplicacy/filters
+symlink/*
+folder/*
-*
alexmbp:test alex$ duplicacy -d backup --dry-run
Storage set to /tmp/storage
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
top: /tmp/test, quick: true, tag:
Downloading latest revision for snapshot test
Listing revisions for snapshot test
No previous backup found
Indexing /tmp/test
There are 0 compiled regular expressions stored
Loaded 3 include/exclude pattern(s)
Pattern: +symlink/*
Pattern: +folder/*
Pattern: -*
Listing
symlink is excluded by pattern -*
folder/ is included by pattern +folder/*
Listing folder/
folder/file is included by pattern +folder/*

Question for @gchen – why is the behavior different here? (I sort of suspect why, but the actual question is whether this shall be “fixed” as to not confuse users?)

[saspus]

Ah. You posted while I was frantically testing Should have refreshed the page before submitting.

Still, perhaps it would be a good idea to process this use case separately since the point of symlinks is to pretend to be an actual folder there and different behaviour may be confusing for users.