Include patterns for symbolic links

[skidvd]

Are symbolic links processed differently relative to Include patterns? Or am I still misunderstanding something fundamental about include patterns?

My goal is to be able to include all nested contents (whether file or subdir) of either the dir or symlink to a dir.

Under the root of my repository, I have the following:

|-Data
|-test1
|-test2
|-Software

In the above, Data and Software are actual directories and test1 and test are symlinks to other areas on the file system (they are dirs there with their own hierarchy of contents under them).

My .filters file has the following contents:

+Data/
+Data/*
+test1/
+test1/*
+test2/
+test2/*
+Software/
+Software/*
-*

It appears that the contents of the non symlinks (Data and Software) and their respective nested contents are being included. However, the contents of the symlinked dirs appears to be being ignored as none of their nested content shows up in any revision of the backups when I attempt to check via something like:

duplicacy -v history “test1/suba/sub/subc/*”

nor

duplicacy -v history “test1/suba/sub/subc/filename”

[skidvd]

It appears that it I add the following to the .filters (before the other test1 and test2 entries)

+test1
+test2

then the symbolic links are followed and their contents are included. Is this the expected behavior? I did not get this from the wiki page description Filters/Include exclude patterns

FYI, this is the with the latest 2.1.1 release

** EDIT: This does NOT appear to be working. Please see below for further findings **

[TheBestPessimist]

It appears that it I add the following to the .filters (before the other test1 and test2 entries)

+test1
+test2

I think it is a bug. It should work by just having your original filters file:

+test1/
+test1/*
+test2/
+test2/*

hint: you can make something more code-like by surrounding them with backticks: `.

eg what i wrote above is:

```
+test1/
+test1/*
+test2/
+test2/*
```

[skidvd]

I am clearly still not understanding/doing something correctly with filters and symlinks.

My .filters file has the following contents (where both test1 and test2 ar top-level symlinks to another dir):

+Data/
+Data/*
+test1/
+test1/*
+test2/
+test2/*
+Software/
+Software/*
-*

If I run duplicacy with the -d option as suggested here and I see the following in the output:

test1 is excluded by pattern -*

I see this whether I have the ‘/’ terminated version or not (or even both).

It was my understanding that an earlier matching rule (presumably either +test1/ and/or + test1/) should have been accepted without further processing, so I’m not even sure how it got to the - rule in the case of the test1 symlink?

What am I missing/doing wrong please.

TIA!

[TheBestPessimist]

Try to add also

+test1
+test2

before their respective “folder” entries. (this will respond to my hypothesis that duplicacy treats symlinks like files first and as folders second.

(basically, you want to have something like

+test1
+test1/
+test1/*

[skidvd]

I see this whether I have the ‘/’ terminated version or not (or even both).

As I mentioned in my earlier reply, this does not appear to be effective in getting the contents to be included - i.e. I still see the excluded message for the symlinks.

[Christoph]

this the expected behavior?

Maybe this helps?

Improving the instructions for include/exclude patterns (filters)

Duplicacy borrows the include/exclude model from rsync. First, the indexing starts at the root of the repository and do a non-recursive listing. Every file or folder is matched against the patterns in order. If an excluded pattern is matched, the file or folder will not be backed up. If it is a folder, Duplicacy will no descend into the excluded folder to list its files/subdirectories. Therefore, it is possible to exclude an entire subdirectory tree at once without wasting time checking eve…

---

I think it is a bug.

What does @gchen say?

[gchen]

test1 is excluded by pattern -*

This means test1 is a file. Is it a symlink? The current implementation will apply the patterns before converting symlinks to directories, so this may be the issue.

[skidvd]

Yes, test1 is a symlink to another dir.

[skidvd]

For grins and giggles, as a test, - WITHOUT making any changes to filters as described above - I went ahead and mounted the dir that the test1 symbolic link refers to as follows:

mount --bind /path/to/test1 test1

Not too surprisingly, the directory is then included:

test1/ is included by pattern +test1/

It would appear that I am either doing something incorrectly with the include filters for symlinks or that there is a problem with them? Hard mounts like this are discouraged and quite problematic, so this is really not a good solution, but I just thought I would pass this info along.

@gchen, does this help to confirm that this is a bug? Is there anything else I can try with the symlink approach not already mentioned above?

[gchen]

You just need to add the `+test`` pattern as @TheBestPessimist suggested:

+test1
+test1/
+test1/*

[skidvd]

@gchen unfortunately, as mentioned above, this does NOT work - even when adding in the +test1 pattern.

[gchen]

This was from my test and it worked as expected.

Loaded 3 include/exclude pattern(s)
Pattern: +test
Pattern: +test/*
Pattern: -*
Listing
duplicacy_win.exe is excluded by pattern -*
duplicacy_win_x64_2.1.1.exe is excluded by pattern -*
file1 is excluded by pattern -*
test is included by pattern +test
Listing test/
test/file2 is included by pattern +test/*

Can you post the output from duplicacy -d backup -dry-run if you think it still doesn’t work?

[skidvd]

@gchen, just to put this all in one post, my filters file includes:

+test1       # This should not be needed, but is to follow symlink
+test1/
+test1/Qmultimedia/
+test1/Qmultimedia/*
+test2      # This should not be needed, but is to follow symlink
+test2/
-test2/home/*/tmp/
-test2/opt/google/
+test2/*
-*

Yet, as requested, the output from the -d backup command (either with or without the -dry-run option) reports the following:

...
test1 is excluded by pattern -*
...
test2 is excluded by pattern -*
...

Even more telling, is the fact that I now see (I didn’t see this in previous run with --bind version) a large number of lines in the output similar to the following:

Delete chunk 0082d8c5ca25330221ad4dc148da74bfd60446ca2cc952ab7459f4b1034d65cd from the snapshot cache
Delete chunk 00ee77e0de606dd6c8f0812b5a464f95ca9273e190a543ec9dadb4fb03f51db2 from the snapshot cache

This backup run against the symlinks has removed the dirs/files that had been included by the immediately preceding mount --bind backup run that had been completed just before the symlink version (as there were no ‘organic’ or other deletes that could otherwise explain this). This is further confirmed by checking the history of one such file with the following command:

duplicacy -v history -r 22 -r $last "test1/Qmultimedia/Photos/2018/Spring/20180516_084431.jpg"

and the associated output:

Using 16384 iterations for key derivation
     22:         2051696 2018-05-16 08:44:31 ce1a37c6188699daa9fa9f42d9a97652d3d0fbaf339a1d6c23ea4fa9e8579ed4 test1/Qmultimedia/Photos/2018/Spring/20180516_084431.jpg
     23:
current:         2051696 2018-05-16 08:44:31                                                                  test1/Qmultimedia/Photos/2018/Spring/20180516_084431.jpg

The links are as follows:

>:~$ ls -ld test1 test2
lrwxrwxrwx 1 gouldt gouldt 23 Aug 19 17:49 test1 -> /media/gouldt/test1
lrwxrwxrwx 1 gouldt gouldt 22 Aug 19 17:50 test2 -> /media/gouldt/test2

This is all with the very latest 2.1.1 version on an Ubuntu 16.04 LTS system.

Please let me know what else I can do to help you.

[skidvd]

@gchen, kindly nudged. Would really like to get this fixed or understand what is going on please.

[gchen]

+test1       # This should not be needed, but is to follow symlink

If this is the exact line in your filter file, then I can see where the issue is: the whole line including the comment is interpreted as the pattern. A # will be treated as a comment only when it starts the line; otherwise it will be part of the pattern.

Other than that this I can’t think of any other reason why it won’t work for you.

[skidvd]

@gchen, I understand what you are saying, but this does not seem to line up with my actual results. Why would it work for the --bind version but not the symlinks (same filter file used for both)?

I am wondering if the fact that the symbolic links are to mounted directories (one is Windows share via cifs and the others is an NFS mount) may be effecting this. All dirs, subdirs and files under them behave as expected via the symbolic links, so I am hoping not. However, I am attempting to discover why duplicacy is not able to deal with them appropriately in my environment?

[gchen]

Did you try removing the comment in that line? Why not give it a try? I would be very surprised if it doesn’t work.

The reason the --bind version worked was that in that case test1 became a directory and was therefore excluded by the pattern +test1/.

[TheBestPessimist]

A # will be treated as a comment only when it starts the line; otherwise it will be part of the pattern.

THIS

I’ve also been bitten by this in my filters file. Ah, why didn’t I notice it .

[skidvd]

@gchen nailed it, of course. I never would have thought that inline comments were not supported/allowable. Thanks! I suppose the only remaining issues to consider are:

1. documenting the unexpected behavior of comments (looks like @TheBestPessimist is already on this)
2. determining why (and perhaps removing the need for) a redundant +test1 entry is required fro symbolic links