Welcome do the forum, @fbreve!
Yes, it is not necessary. And I also exclude all the msf files and caches.
I have no issue backing up my Thunderbird profile while in use, when using the -vss option flag.
I’m having the same problem. The documentation is sorely missing 
I have no idea where to put the filters file if you use the web ui
I’m not a paid user so I have only backed up from the command line, but in case it helps I have successfully used a filters file in Windows in that situation.
I notice that the web ui does in fact generate a .duplicacy folder for each repository. They’re in .duplicacy-web\repositories\localhost\all. I speculate that a filters file there might work, but it would cost me money to find out for sure.
Use the Web GUI to create the filters file for each backup job in the first place. It’s under the Backup tab, click the Edit link under Include/Exclude column for a backup.
You can exclusively use the GUI to add, delete and rearrange lines, so you technically don’t need a text editor, although you can edit the file after it’s been created, and it’ll be in the numbered repository directory - i.e. .duplicacy-web\repositories\localhost\0\.duplicacy\filters (where 0 is the order in which backup tasks were created).
To add to this, you can use duplicacy’s GUI editor (that is really awkward to use in the first place, compared to a plain old text editor) to create a filter file that contains just a single line: import contents of another file. And that other file can be elsewhere. I like to store mine in the Documents folder, that gets replicated everywhere, can be edited from anywhere, and is available to all duplicacy installs.
In the section about regex matching, you write
This does not match my experience, given a list of patterns as follow:
...(more patterns, some regex, some not)
i:.thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/(Archive|Drafts|Folders|INBOX|Sent|Spam|Trash).*
e:.thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/.*
...(more patterns, some regex, some not)
It does not stop at the i: pattern, but actually gives the e: pattern priority, and nothing is backed up from the 127.0.0.1 directory, even though the i: pattern appears first, and does match some files in the 127.0.0.1 directory.
2023-05-24 09:28:55.329 TRACE SNAPSHOT_PATTERN Pattern: i:.thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/(Archive|Drafts|Folders|INBOX|Sent|Spam|Trash).*
2023-05-24 09:28:55.329 TRACE SNAPSHOT_PATTERN Pattern: e:.thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/.*
2023-05-24 09:29:00.581 DEBUG PATTERN_INCLUDE .thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1.msf is included
2023-05-24 09:29:00.581 DEBUG PATTERN_EXCLUDE .thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/ is excluded by pattern e:.thunderbird/6wxt3a2w.default/ImapMail/127.0.0.1/.*
None of the examples clearly show how to do what I want to do (ie i: and e: both match a path, and the first matching pattern should win). This might require clarifying what you mean with the path is said to be included without further comparisons (in a regex context).
I want to exclude one folder, but include one subfolder within that excluded folder – is that possible?
I currently have only exclude filters, eg. one for the new case:
-e/dontbackup/
Now I thought I could include a subfolder like this:
+e/dontbackup/pleasebackupthis/
-e/dontbackup/
But that doesn’t work?
Also this doesn’t help:
+e/
+e/dontbackup/pleasebackupthis/
-e/dontbackup/
Also no luck with:
+e/dontbackup/pleasebackupthis/*
-e/dontbackup/
Ha, I finally got it:
+e/
+e/dontbackup/
+e/dontbackup/pleasebackupthis/*
-e/dontbackup/*
I found it confusing that I had to incude the folder and then later on exclude all files and subfolders – but it makes sense.
3 Likes
The key to make sense of it is that duplicacy tries to match every line one by one and if as result a specific directory turns out to be “don’t include” it won’t go inside of it, and therefore will never check anything inside of it.
So you force it to include all the intermediate paths, for it to actually see a sub-paths matched for inclusion.
Honestly, duplciacy shall be smart enough, to figure out that if I say +a/b/c/d/e/ that means it has to revisit each sub-path and not force the user to write it out explicitly. But it is not the case today.
Hey everyone, I’m using the patterns above but am experiencing unexpected results. The first few are simple folder inclusions but tailscale and syncthing are never backed up while the others are. The last inclusion of specifically named files is also not working. Does anyone know why the simple inclusions wouldn’t be processed? Any wisdom on how to include specific file naming convention? 
Are they all in the root of the repository? Does duplicacy have permissions to read them?
Last one would not work because Duplicacy traverses the directory structure and matches the patterns one by one. If some [sub]directory does not match — it will not check its contents and therefore will never get to the file path that matches your pattern.
I don’t defend this approach, just describing how it works.
The only way I can think of is to include all directories before your regex file pattern.
+*/
+i:.*/[your file pattern]$
This will ensure all directory hierarchy is traversed and then your file pattern will get a chance to get matched.
Thanks for the reply!
- Yeah, they are in the root and I’m pretty sure duplicacy has access, or at least there’s no reason it shouldn’t from what I can tell. Still can’t get them to load.
- Ok, I’ve tried this along with a number of other things. I don’t want all of the other files in those directories to be loaded (it’s filling up my external storage) is there a way to add them get the ones I want and then remove all the others? Here’s another try:
- +i:/appdata_backup/[^/]+/$
- +i:/appdata_backup/[^/]+/tower-2-flash-backup*.zip$
- -appdata_backup/*
But it just includes everything still. Any advice?
Patterns ending with / only match directories, not files.
Got it, yeah, the intent was to include the directories and then include the specific files before excluding the rest of the files in those directories. I’ve tried a number of different options but still unable to get that to work so I may need to figure out another alternative.
This is what I just did as a proof of concept:
Create a bunch of files
mkdir -p in/{1,2,3}/{1,2,3}/ out
touch in/{1,2,3}/{1,2,3}/randomfile.txt
touch in/{1,2,3}/{1,2,3}/this_i_want.txt
touch in/{1,2,3}/{1,2,3}/this_i_want_as_well.txt
touch in/{1,2,3}/{1,2,3}/not_this.txt
touch in/{1,2,3}/randomfile.txt
touch in/{1,2,3}/this_i_want.txt
touch in/{1,2,3}/this_i_want_as_well.txt
touch in/{1,2,3}/not_this.txt
find in -type f -exec dd if=/dev/urandom of={} bs=1k count=1 \;
add a filter file like so
+*/
+*/*want*.txt
Run backup and observe that only files I want get picked up and none that I don’t, from the whole directory structure:
% dpl backup -dry-run
Storage set to /tmp/out
No previous backup found
Indexing /tmp/in
Parsing filter file /tmp/in/.duplicacy/filters
Loaded 2 include/exclude pattern(s)
Packed in/1/this_i_want.txt (1024)
Packed in/1/this_i_want_as_well.txt (1024)
Packed in/1/1/this_i_want.txt (1024)
Packed in/1/1/this_i_want_as_well.txt (1024)
Packed in/1/2/this_i_want.txt (1024)
Packed in/1/2/this_i_want_as_well.txt (1024)
Packed in/1/3/this_i_want.txt (1024)
Packed in/1/3/this_i_want_as_well.txt (1024)
Packed in/2/this_i_want.txt (1024)
Packed in/2/this_i_want_as_well.txt (1024)
Packed in/2/1/this_i_want.txt (1024)
Packed in/2/1/this_i_want_as_well.txt (1024)
Packed in/2/2/this_i_want.txt (1024)
Packed in/2/2/this_i_want_as_well.txt (1024)
Packed in/2/3/this_i_want.txt (1024)
Packed in/2/3/this_i_want_as_well.txt (1024)
Packed in/3/this_i_want.txt (1024)
Packed in/3/this_i_want_as_well.txt (1024)
Packed in/3/1/this_i_want.txt (1024)
Packed in/3/1/this_i_want_as_well.txt (1024)
Packed in/3/2/this_i_want.txt (1024)
Packed in/3/2/this_i_want_as_well.txt (1024)
Packed in/3/3/this_i_want.txt (1024)
Packed in/3/3/this_i_want_as_well.txt (1024)
Backup for /tmp/in at revision 1 completed
Note, if you only have +*/ in filters – then nothing gets backed up. I honestly have no idea why this is not added by default by duplicacy:
% cat .duplicacy/filters
+*/
% dpl backup -dry-run
Storage set to /tmp/out
No previous backup found
Indexing /tmp/in
Parsing filter file /tmp/in/.duplicacy/filters
Loaded 1 include/exclude pattern(s)
No files under the repository to be backed up
HERO! I could have promised that I did this in my various attempts but how you laid it out makes a lot of sense and it WORKED! Thanks so much. For the record here’s how I adapted it to my case:
#Include all Directories (but not files)
+*/
#Include *just* the backup zip files
+*/*tower-2-flash-backup*.zip
#Exclude all other files in the structure that I don't want
-appdata_backup/*
Nice!
Ive edited your post to add ``` before and after the text, otherse * are consumed by formatter and completely mangle your filters file 
Probably you don’t need this – if you only have include patterns, than anything else is excluded by default.
1 Like
I want to exclude a specific directory existing in various locations. The name of the directory starts with “$”. From my understanding, the proper syntax is
e:/$RECYCLE.BIN/$
Do I have to escape the first “$” or does Duplicacy recognize it is part of a filename?
How do I escape? Simply with a double slash?
e://$RECYCLE.BIN/$
$ means end of line in regex. You don’t need regex for this simple case::
-*/$RECYCLE.BIN/