Filters/Include exclude patterns

How-to
#23

The key to make sense of it is that duplicacy tries to match every line one by one and if as result a specific directory turns out to be “don’t include” it won’t go inside of it, and therefore will never check anything inside of it.

So you force it to include all the intermediate paths, for it to actually see a sub-paths matched for inclusion.

Honestly, duplciacy shall be smart enough, to figure out that if I say +a/b/c/d/e/ that means it has to revisit each sub-path and not force the user to write it out explicitly. But it is not the case today.

#24

Hey everyone, I’m using the patterns above but am experiencing unexpected results. The first few are simple folder inclusions but tailscale and syncthing are never backed up while the others are. The last inclusion of specifically named files is also not working. Does anyone know why the simple inclusions wouldn’t be processed? Any wisdom on how to include specific file naming convention? image

#25

Are they all in the root of the repository? Does duplicacy have permissions to read them?

Last one would not work because Duplicacy traverses the directory structure and matches the patterns one by one. If some [sub]directory does not match — it will not check its contents and therefore will never get to the file path that matches your pattern.

I don’t defend this approach, just describing how it works.

The only way I can think of is to include all directories before your regex file pattern.

+*/
+i:.*/[your file pattern]$

This will ensure all directory hierarchy is traversed and then your file pattern will get a chance to get matched.

#26

Thanks for the reply!

  1. Yeah, they are in the root and I’m pretty sure duplicacy has access, or at least there’s no reason it shouldn’t from what I can tell. Still can’t get them to load.
  2. Ok, I’ve tried this along with a number of other things. I don’t want all of the other files in those directories to be loaded (it’s filling up my external storage) is there a way to add them get the ones I want and then remove all the others? Here’s another try:
  • +i:/appdata_backup/[^/]+/$
  • +i:/appdata_backup/[^/]+/tower-2-flash-backup*.zip$
  • -appdata_backup/*
    But it just includes everything still. Any advice?
#27

Patterns ending with / only match directories, not files.

#28

Got it, yeah, the intent was to include the directories and then include the specific files before excluding the rest of the files in those directories. I’ve tried a number of different options but still unable to get that to work so I may need to figure out another alternative.

#29

This is what I just did as a proof of concept:

Create a bunch of files

mkdir -p in/{1,2,3}/{1,2,3}/ out
 
touch in/{1,2,3}/{1,2,3}/randomfile.txt
touch in/{1,2,3}/{1,2,3}/this_i_want.txt
touch in/{1,2,3}/{1,2,3}/this_i_want_as_well.txt
touch in/{1,2,3}/{1,2,3}/not_this.txt
touch in/{1,2,3}/randomfile.txt
touch in/{1,2,3}/this_i_want.txt
touch in/{1,2,3}/this_i_want_as_well.txt
touch in/{1,2,3}/not_this.txt

find in -type f -exec dd if=/dev/urandom of={} bs=1k count=1  \;

add a filter file like so

+*/
+*/*want*.txt

Run backup and observe that only files I want get picked up and none that I don’t, from the whole directory structure:

% dpl backup -dry-run
Storage set to /tmp/out
No previous backup found
Indexing /tmp/in
Parsing filter file /tmp/in/.duplicacy/filters
Loaded 2 include/exclude pattern(s)
Packed in/1/this_i_want.txt (1024)
Packed in/1/this_i_want_as_well.txt (1024)
Packed in/1/1/this_i_want.txt (1024)
Packed in/1/1/this_i_want_as_well.txt (1024)
Packed in/1/2/this_i_want.txt (1024)
Packed in/1/2/this_i_want_as_well.txt (1024)
Packed in/1/3/this_i_want.txt (1024)
Packed in/1/3/this_i_want_as_well.txt (1024)
Packed in/2/this_i_want.txt (1024)
Packed in/2/this_i_want_as_well.txt (1024)
Packed in/2/1/this_i_want.txt (1024)
Packed in/2/1/this_i_want_as_well.txt (1024)
Packed in/2/2/this_i_want.txt (1024)
Packed in/2/2/this_i_want_as_well.txt (1024)
Packed in/2/3/this_i_want.txt (1024)
Packed in/2/3/this_i_want_as_well.txt (1024)
Packed in/3/this_i_want.txt (1024)
Packed in/3/this_i_want_as_well.txt (1024)
Packed in/3/1/this_i_want.txt (1024)
Packed in/3/1/this_i_want_as_well.txt (1024)
Packed in/3/2/this_i_want.txt (1024)
Packed in/3/2/this_i_want_as_well.txt (1024)
Packed in/3/3/this_i_want.txt (1024)
Packed in/3/3/this_i_want_as_well.txt (1024)
Backup for /tmp/in at revision 1 completed

Note, if you only have +*/ in filters – then nothing gets backed up. I honestly have no idea why this is not added by default by duplicacy:

% cat .duplicacy/filters
+*/
% dpl backup -dry-run
Storage set to /tmp/out
No previous backup found
Indexing /tmp/in
Parsing filter file /tmp/in/.duplicacy/filters
Loaded 1 include/exclude pattern(s)
No files under the repository to be backed up
#30

HERO! I could have promised that I did this in my various attempts but how you laid it out makes a lot of sense and it WORKED! Thanks so much. For the record here’s how I adapted it to my case:

#Include all Directories (but not files)
+*/
#Include *just* the backup zip files
+*/*tower-2-flash-backup*.zip
#Exclude all other files in the structure that I don't want
-appdata_backup/*
#31

Nice!

Ive edited your post to add ``` before and after the text, otherse * are consumed by formatter and completely mangle your filters file :slight_smile:

Probably you don’t need this – if you only have include patterns, than anything else is excluded by default.

1 Like