Filters/Include exclude patterns

For the backup command, the include/exclude patterns are read from a file named filters under the .duplicacy directory. For the restore command, the include/exclude patterns are specified as the command line arguments.

Duplicacy offers two different methods for providing include/exclude filters, wildcard matching and regular expression matching. You may use one method exclusively or you may combine them as you deem necessary.

All paths are relative to the repository (the folder you execute duplicacy from), without a leading β€œ/”. As the upmost folder on Windows is a drive, this means drive letters are not part of the path of a pattern. The path separator is always a β€œ/”, even on Windows. Paths are case sensitive.

Wildcard Matching

An include pattern starts with β€œ+”, and an exclude pattern starts with β€œ-”. Patterns may contain wildcard characters β€œ*” which matches a path string of any length, and β€œ?” matches
a single character. Note that both β€œ*” and β€œ?” will match any character including the path separator β€œ/”.

When matching a path against a list of patterns, the path is compared with the part after β€œ+” or β€œ-”, one pattern at a time. Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison. If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.

Patterns ending with a β€œ/” apply to directories only, and patterns not ending with a β€œ/” apply to files only.

Patterns ending with β€œ*” and β€œ?”, however, apply to both directories and files. When a directory is excluded, all files and subdirectories under it will also be excluded. Therefore, to include a subdirectory, all parent directories must be explicitly included.

This is because duplicacy considers all matches and exclusions at each level in the file tree before descending into the next level (a β€œbreadth first” search for matches). This makes the search much more efficient, but can produce confusion about the order an interpretation of filter rules.

For instance, the following pattern list doesn’t do what is intended, since the foo directory will be excluded so the foo/bar will never be visited:

+foo/bar/*
-*

This does not work because
-*
implies
-foo/

So when duplicacy examines the first level of the file tree for matches and exclusions, it excludes foo/ and everything underneath. That means that it never goes to the second level into foo/, and therefore never sees a match for foo/bar/. It also excludes all other top-level directories, producing an empty backup.

So, we have to make sure foo/ is included first, before the wildcard excludes it. Here is the correct way to include foo as well:

+foo/bar/*
+foo/
-*

The following pattern list includes only files under the directory foo/ but not files under the subdirectory foo/bar:

-foo/bar/
+foo/*
-*

To include a directory while excluding all files under that directory, use these patterns:

+cache/
-cache/?*

To include files in a directory and exclude its subdirectories:

-folder/*/*
+folder/*
-*

Regular Expression Matching

An include pattern starts with β€œi:”, and exclude pattern starts with β€œe:”. The part of the filter after the include/exclude prefix must be a valid regular expression. The
regular expression syntax is the same general syntax used by Perl, Python, and other languages.
Full details for the supported regular expression syntax and features are available here.

When matching a path against a list of patterns, the path is compared with the part after β€œi:” or β€œe:” one pattern at a time. Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison. If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.

Some examples of regular expression filters are shown below:

# always include sqlite databases
i:\.sqlite$
# exclude sqlite temp files
e:\.sqlite-.*$
# exclude temporary file names
e:.*/?~.*$
# exclude common file types (case insensitive)
e:(?i)\.(bak|mp4|mkv|o|obj|old|tmp)$
# exclude lotus notes full text directories
e:\.ft/.*$
# exclude any cache files/directories with cache in the name (case insensitive)
e:(?i).*cache.*
# exclude lightroom previews
e:(?i).* Previews\.lrdata/.*$
# exclude Qt source
e:(?i)develop/qt[0-9]/.*$
# exclude any git stuff
e:\.git/.*$
# exclude cisco anyconnect log files: matches .cisco/log/* or .cisco/vpn/log/*, etc
e:\.cisco/.*/?log/.*$
# exclude trash bin stuff
e:\.Trash/.*$
# exclude old firefox stuff
e:Old Firefox Data/.*$
# exclude dirx stuff: excludes Documents/dir0/*, Documents/dir1/*, ...
e:Documents/dir[0-9]*/.*$
# exclude downloads
e:Downloads/.*$
# exclude duplicacy test stuff
e:DUPLICACY_TEST_ZONE/.*$
# exclude lotus notes stuff
e:Library/Application Support/IBM Notes Data/.*$
# exclude mobile backup stuff
e:Library/Application Support/MobileSync/Backup/.*$
# exclude movies
e:Movies/.*$
# exclude itunes stuff
e:Music/iTunes/iTunes Media/.*$
# include everything else
i:.*
# include Firefox profile but nothing else from Mozilla
i:(?i)/AppData/[^/]+/Mozilla/$
i:(?i)/AppData/[^/]+/Mozilla/Firefox/
e:(?i)/AppData/[^/]+/Mozilla/

Explanation of the regex above:

  • /[^/]+/: has the purpose of assuring that there is exactly 1 folder between AppData and Mozilla
  • we need to include
    • the Mozilla folder, but nothing it contains (therefore the $)
    • the Firefox folder, and everything it contains
    • exclude everything in the Mozilla folder which is not contained in the rules above
    • (important) put the $ include rule(s) for each folder we want to include up to the actual folder where we take everything, (check Google Chrome profile below). (note: someone please explain this better)
# include Google Chrome profile but nothing else from Google
# note that we include the whole profile, because we are unsure how many "users" are added beside the "Default" profile
i:(?i)/AppData/[^/]+/Google/$
i:(?i)/AppData/[^/]+/Google/Chrome/$
i:(?i)/AppData/[^/]+/Google/Chrome/User Data/
e:(?i)/AppData/[^/]+/Google/

As seen in the examples above, you may add comments to your filters file by starting the line with a β€œ#” as the first character of the line.
The entire comment line will be ignored and can be used to document the meaning of your include/exclude wildcard and regular expression filters. Completely blank lines are
also ignored and may be used to make your filters list more readable. Note that if you add # anywhere else but at the beginning of a line, it will be interpreted as part of the pattern, not as a comment.

Testing filters

Filters can be easily tested using the backup command: duplicacy -d -log backup -enum-only. This is further explained in Backup command details.

Importing patterns from other files

You can now @import other files into the filters file by using

@/the/full/path/to/the/customised-filters-file
@/the/full/path/to/the/some-other-filters-file

other filters below

See the details in Filters just got a big upgrade: @import files.

Custom filters file location

Start with version 2.3.0, you can now specify the location of the filters file rather that the default one at .duplicacy/filters. To do this, run the set command as:

set -storage second -filters <path>

The path can be an absolute path, or relative to the repository path.

You can also edit the .duplicacy/preferences file directly to add the filters key.

This option means that you can now use a different filters file for each storage.

2 Likes
Backup command details
Filters just got a big upgrade: @import files
Filters for files in subdirectories
Need to be able to select only specific folders in the repository to back up
Filters on a share
Duplicacy - exclude files video tutorial
How to run duplicacy as a cron job on linux?
Restore command details
Improving the instructions for include/exclude patterns (filters)
Backing up serval directories at once
Am I doing this right?
Patterns for exclusion/inclusion are confusing
Include patterns for symbolic links
Include patterns with wildcards
Include a folder for backup in Mac user library
Backup from specific directory without cd
Which folders in ProgramData and AppData should be backed up?
Considering Duplicacy with Wasabi
Filter pattern to include only one type of file in several subfolders
Windows Backup Failed to read the symlink
Filters and patterns - include specific folders
Help with filters - exclude directory everywhere
Restore files in a folder but not its subfolder
Filters key in preferences file
How to initiate a backup for multiple repositories?
Windows version - exclude common system folders automatically?
Restore files in a folder but not its subfolder
Revisiting filters - no love for me
Multiple uploads into same revision / merge revisions
Restore files in a folder but not its subfolder
Patterns for exclusion/inclusion are confusing
Patterns for exclusion/inclusion are confusing
Let me limit the folders that are checked during a backup
Backing on External Drive w/ Time Machine Hidden Directories
Symbolic link repository and filters
Duplicacy User Guide
Include/exclude help
Reducing memory usage deciding what to back up
Proposal: following symlinks by pattern
Files-from -- filter by mtime size
Why does it fail to list folder - when it's excluded?
Why does it fail to list folder - when it's excluded?
Edit pattern list in a text editor
Some newbie questions - backup organisation
Some newbie questions - backup organisation
How to properly do an initial backup to B2?
Exlusions to backup not working
Duplicacy using 3x storage of source for backup

Do I understand correctly that with Wildcard Matching, the entire path must be specified? For example:

/home/joe/foo/bar the directory I want to exclude
-/bar/ wrong
-*/bar/ right

But in Regular Expression Matching, the expression only needs to match part of the path? For example:

/home/joe/foo/bar the directory I want to exclude
e:/bar/$ right
e^.*/bar/$ also right

(also, is /$ at the end the regular expression the correct way to exclude a directory?)

A post was split to a new topic: Patterns for exclusion are confusing

I did some testing, and I believe that the answer is yes: with Wildcard Matching, the entire path must be specified, but in Regular Expression Matching, the expression only needs to match part of the path. If you want a regular expression to only match the end of a path, put $ at the end of the regular expression (or /$ in the case of directories).

3 Likes

This is correct, and mostly because the only regex matching function in Go, regexp.Match, behaves more like Search in other languages (I had been confused by this for a while). So if you need to match the entire path then the anchors ^ and $ are required.

And yes, /$ at the end matches directories only.

3 Likes

4 posts were split to a new topic: How to restore files after a certain date

2 posts were merged into an existing topic: How to restore files after a certain date

I like to back up my entire Linux root volume, in the hopes I can later (a) restore it fully and (b) see changes to any areas by diffing backups. This should work, right?

Has anyone made a filter file for this purpose, excluding those folders that don’t need backup up, such as /dev, /proc and log dirs?

Update

Here’s my go at it:

# This file is /.duplicacy/filters

# exclude any ".cache" folders
e:(?i)/\.cache/.*

# everything in root dir, excluding symlinks, kernel files and temporary stuff
-bin
-boot/
-dev/
-.duplicacy/
+etc/
+home/
-initrd.*
-installimage*
-lib
-lib32
-lib64
-libx32
-lost+found/
-media/
-mnt/
+opt/
-proc/
+root/
-run/
-sbin
+srv/
-sys/
-tmp/
+usr/
+var/
-vmlinuz*

I’m using Duplicacy Web and I don’t understand where the filters file goes. I don’t have a .duplicacy folder. I have .duplicacy-web in my home directory. I created ~/.duplicacy-web/filters and it’s not used. I then created ~/.duplicacy-web/repositories/localhost/0/.duplicacy/filters and that file is not used and also gets deleted somehow when I run a backup. Also tried ~/.duplicacy-web/repositories/localhost/all/.duplicacy/filters and same thing.

I also tried it through the GUI but upon saving, it says

Failed to write the pattern file: open /Users/matt/.duplicacy-web/filters/localhost/0: not a directory

Thanks for your help.