For the backup command, the include/exclude patterns are read from a file named filters under the .duplicacy directory. For the restore command, the include/exclude patterns are specified as the command line arguments.
Duplicacy offers two different methods for providing include/exclude filters, wildcard matching and regular expression matching. You may use one method exclusively or you may combine them as you deem necessary.
All paths are relative to the repository (the folder you execute duplicacy from), without a leading “/”. As the upmost folder on Windows is a drive, this means drive letters are not part of the path of a pattern. The path separator is always a “/”, even on Windows. Paths are case sensitive.
Wildcard Matching
An include pattern starts with “+”, and an exclude pattern starts with “-”. Patterns may contain wildcard characters “*” which matches a path string of any length, and “?” matches
a single character. Note that both “*” and “?” will match any character including the path separator “/”.
When matching a path against a list of patterns, the path is compared with the part after “+” or “-”, one pattern at a time. Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison. If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.
Patterns ending with a “/” apply to directories only, and patterns not ending with a “/” apply to files only.
Patterns ending with “*” and “?”, however, apply to both directories and files. When a directory is excluded, all files and subdirectories under it will also be excluded. Therefore, to include a subdirectory, all parent directories must be explicitly included.
This is because duplicacy considers all matches and exclusions at each level in the file tree before descending into the next level (a “breadth first” search for matches). This makes the search much more efficient, but can produce confusion about the order an interpretation of filter rules.
For instance, the following pattern list doesn’t do what is intended, since the foo directory will be excluded so the foo/bar will never be visited:
+foo/bar/*
-*
This does not work because
-*
implies
-foo/
So when duplicacy examines the first level of the file tree for matches and exclusions, it excludes foo/ and everything underneath. That means that it never goes to the second level into foo/, and therefore never sees a match for foo/bar/. It also excludes all other top-level directories, producing an empty backup.
So, we have to make sure foo/ is included first, before the wildcard excludes it. Here is the correct way to include foo as well:
+foo/bar/*
+foo/
-*
The following pattern list includes only files under the directory foo/ but not files under the subdirectory foo/bar:
-foo/bar/
+foo/*
-*
To include a directory while excluding all files under that directory, use these patterns:
+cache/
-cache/?*
To include files in a directory and exclude its subdirectories:
-folder/*/*
+folder/*
-*
Regular Expression Matching
An include pattern starts with “i:”, and exclude pattern starts with “e:”. The part of the filter after the include/exclude prefix must be a valid regular expression. The
regular expression syntax is the same general syntax used by Perl, Python, and other languages.
Full details for the supported regular expression syntax and features are available here.
When matching a path against a list of patterns, the path is compared with the part after “i:” or “e:” one pattern at a time. Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison. If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.
Some examples of regular expression filters are shown below:
# always include sqlite databases
i:\.sqlite$
# exclude sqlite temp files
e:\.sqlite-.*$
# exclude temporary file names
e:.*/?~.*$
# exclude common file types (case insensitive)
e:(?i)\.(bak|mp4|mkv|o|obj|old|tmp)$
# exclude lotus notes full text directories
e:\.ft/.*$
# exclude any cache files/directories with cache in the name (case insensitive)
e:(?i).*cache.*
# exclude lightroom previews
e:(?i).* Previews\.lrdata/.*$
# exclude Qt source
e:(?i)develop/qt[0-9]/.*$
# exclude any git stuff
e:\.git/.*$
# exclude cisco anyconnect log files: matches .cisco/log/* or .cisco/vpn/log/*, etc
e:\.cisco/.*/?log/.*$
# exclude trash bin stuff
e:\.Trash/.*$
# exclude old firefox stuff
e:Old Firefox Data/.*$
# exclude dirx stuff: excludes Documents/dir0/*, Documents/dir1/*, ...
e:Documents/dir[0-9]*/.*$
# exclude downloads
e:Downloads/.*$
# exclude duplicacy test stuff
e:DUPLICACY_TEST_ZONE/.*$
# exclude lotus notes stuff
e:Library/Application Support/IBM Notes Data/.*$
# exclude mobile backup stuff
e:Library/Application Support/MobileSync/Backup/.*$
# exclude movies
e:Movies/.*$
# exclude itunes stuff
e:Music/iTunes/iTunes Media/.*$
# include everything else
i:.*
# include Firefox profile but nothing else from Mozilla
i:(?i)/AppData/[^/]+/Mozilla/$
i:(?i)/AppData/[^/]+/Mozilla/Firefox/
e:(?i)/AppData/[^/]+/Mozilla/
Explanation of the regex above:
-
/[^/]+/: has the purpose of assuring that there is exactly 1 folder betweenAppDataandMozilla - we need to include
- the
Mozillafolder, but nothing it contains (therefore the$) - the
Firefoxfolder, and everything it contains - exclude everything in the
Mozillafolder which is not contained in the rules above - (important) put the
$include rule(s) for each folder we want to include up to the actual folder where we take everything, (check Google Chrome profile below). (note: someone please explain this better)
- the
# include Google Chrome profile but nothing else from Google
# note that we include the whole profile, because we are unsure how many "users" are added beside the "Default" profile
i:(?i)/AppData/[^/]+/Google/$
i:(?i)/AppData/[^/]+/Google/Chrome/$
i:(?i)/AppData/[^/]+/Google/Chrome/User Data/
e:(?i)/AppData/[^/]+/Google/
As seen in the examples above, you may add comments to your filters file by starting the line with a “#” as the first character of the line.
The entire comment line will be ignored and can be used to document the meaning of your include/exclude wildcard and regular expression filters. Completely blank lines are
also ignored and may be used to make your filters list more readable. Note that if you add # anywhere else but at the beginning of a line, it will be interpreted as part of the pattern, not as a comment.
Testing filters
Filters can be easily tested using the backup command: duplicacy -d -log backup -enum-only. This is further explained in Backup command details.
Importing patterns from other files
You can now @import other files into the filters file by using
@/the/full/path/to/the/customised-filters-file
@/the/full/path/to/the/some-other-filters-file
other filters below
See the details in Filters just got a big upgrade: @import files.
Custom filters file location
Start with version 2.3.0, you can now specify the location of the filters file rather that the default one at .duplicacy/filters. To do this, run the set command as:
set -storage second -filters <path>
The path can be an absolute path, or relative to the repository path.
You can also edit the .duplicacy/preferences file directly to add the filters key.
This option means that you can now use a different filters file for each storage.
