Help with filters again

I’m trying to set up a backup certain parts of my Windows profile, so C:\Users\ProfileName…

Trying to achieve the following but I’m getting the logic of the filters wrong:

Include a few folders within C:\Users\ProfileName
Include C:\Users\ProfileName\AppData but exclude all subfolders except a few handpicked ones

So by default, everything under AppData gets excluded apart from the individual subfolders that I include.

What would be the correct way of setting up this kind of filter using wildcards (never used regex)? I’m using the GUI.

Thanks

You need to know two things about how Duplicacy handles filters, and hopefully everything clicks in place.

  1. For each file or directory Duplicacy encounters it tries to match the filters line by line, and stops when it finds a match, be that include or exclude.
  2. If the rule ends with / it matches a directory, else — file. * matches everything, including the /.

So, for your case, assuming the repository is initialized in C:\Users\ProfileName, the filters may look something like this

# Allow traversing the AppData folder, to enumerate the contents inside. 
# This is a match for AppData folder itself only, it will allow duplicacy to proceed traversing the
#  folder hierarchy and the objects inside will get a chance to be matched. 
+AppData/

# include everything under handpicked folders
+AppData/Handpicked1/*
+AppData/Handpicked2/*
# ..

# exclude everything else from AppData. If we got here — the object in question didn’t match 
# any of the rules above, so we don’t want it. 
-AppData/*

# … other stuff 

Few notes:

  1. Yes, you can have comments in the filters file like above

  2. The UI editing the filters in the webUI is very bad. I recommend adding a reference to another filter file in there, and editing that filter file instead with your favourite text editor. For example:

    # Including content of the filter file line
    @c:/Users/ProfileName/Duplicacy.filters.txt
    

Thanks for taking the time to reply and sorry it’s taken a few days to get back.

Unfortunately I can’t get this to work and I can’t see where I’m going wrong.

So this backup has for a source folder C:\Users\Myname\

and this is the filter I’m using. Attempting to include the Desktop folder and my email data:

  • +Desktop/* - WORKS FINE
  • +AppData/Local/
  • +AppData/Local/eM Client/* NOT BACKED UP
  • -AppData/Local/*
  • -*

I’m making some mistake with the logic of this but I cannot see what.

you need to add AppData/ itself - otherwise it won’t visit it to see the AppData/Local rule you specified.

+Desktop/*

# Look into AppData folder 
+AppData/
# Look into AppData/Local folder
+AppData/Local/
# add the em client and below
+AppData/Local/eM Client/*
# reject anything else in that folder
-AppData/Local/*

# reject everything else 
-*

Alternatively, you can make it revisit all folders everywhere and simplify further

# Look into every folder but not file
+*/

# Match desktop
+Desktop/*

# match the eM client and below
+AppData/Local/eM Client/*

# reject anything else
-*

This will result in it backing up a ton of empty folders — but it will simplify filters — you won’t need to explicitly descend into every subfolder.

1 Like

Thanks for your clear explanation, really appreciated.

One more question though which will hopefully be the last - I want to exclude the contents of the Brave ‘Cache’ subfolder - so AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/Cache

I’ve added a line to the filter but it’s still backing up the contents of Cache/

+*/
+Desktop/*
+AppData/Local/BraveSoftware/*
-AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/Cache/* - STILL BACKING UP CONTENTS
+AppData/Roaming/eM Client/*
-*

This line

Will match everything under that folder, at which point duplicacy will stop and won’t check your next more specific line.

You can do this:

...
# traverse all folders under brave software. Note, rule ends with / to match directories only. 
# It will match any subdirectory of BraveSoftware
+AppData/Local/BraveSoftware/*/

# exclude contents of the cache folder. This rule matches files, and will get hit during enumeration 
-AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/Cache/*
 
# include everything else in that file -- if we got here -- were are not in Cache folder, 
# because otherwise we would hit rule above. Hence, include.
+AppData/Local/BraveSoftware/*
...

Alternativley, to have more effeicen rules, you may want todo this:

...
+AppData/Local/BraveSoftware/
+AppData/Local/BraveSoftware/Brave-Browser/
+AppData/Local/BraveSoftware/Brave-Browser/User Data/
+AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/ 
-AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/Cache/*
 
+AppData/Local/BraveSoftware/*
...

Another approach is to exclude all cache folders everywhere in the very beginning with -*/Cache/* – then you don’t have to manually descend into every one.

1 Like

Thanks for you continued help.

I haven’t tried the first two yet, but the final option would be my preferred choice as it will cover caches in other app folders. I’ve tried this:

-*/Cache/* –
+*/

+Desktop/*

+AppData/Local/BraveSoftware/*
-AppData/Local/BraveSoftware/Brave-Browser/User Data/Default/Cache/*
+AppData/Roaming/eM Client/*

-*

…but Cache and everything in it is still being backed up.

What is that trailing Unicode em-dash doing there?

I copied it in from your previous reply by accident.

Seems to be working now! Thank you.

As a side note - it’s real-world examples like this which would be of enormous benefit if included in the help files.

Thanks again for all your help.

1 Like

Yeah, it’s not user friendly at all… ideally, duplicacy web shall do the right thing based on the user wants, and then convert that into language duplicacy CLI understands. Otherwise what’s the point of the gui, if user has to learn the filters language anyway.

I have sort of figured out the filters not because I really wanted another challenge :slight_smile:

The filter file however is not the only way to exclude stuff. I’d say is the least desirable approach: metadata (information about what to exclude) lives separately from the data, so if you move data Around you have to also remember to update the metadata.

On macOS for example Time Machine uses extended attributes on a file or directory to allow developers to signal that this file or directory does not need to be backed up. This way metadata lives right with data and moves around with it. And who can know better what to exclude from a specific app than developers of said app.

Duplicacy and many other tools honor that flag, so on macOS my Duplicacy filter file is empty.

There is no such convention on windows.

There is however another way - Duplicacy can check for presence of a special hidden. .nobackup file to skip the directory. I don’t remember if that’s the name of the file exactly of it can be overridden — but it may be a viable workaround worth looking into.

Then you would remove your filters file and just drop that file into folded you want excluded. It seems to be more sane and easier to manage approach, maybe it is worth looking into.

Edit:

It looks like there is no way to specify it in the GUI, and modifying the preferences file won’t work because duplicacy web overwrites them.

@gchen, is there any way to make -no-backup-file work with webgui?

TBF this is the first hiccup I’ve had with it. Back in the day I used to use a thing called OopsBackup which was a Windows version of Time Machine and worked beautifully. But, Altaro stopped supporting it years ago and it doesn’t run well on newer version of Windows. So I tried Duplicati but had multiple issues with it and eventually found Duplicacy.

1 Like

Oh yes, duplicati is in permanent beta and is quite unstable.

I myself switched to duplicacy after trying enormous number of tools following sunset of consumer- (and enshittification of pro-) version of CrashPlan.

I’m not using the web ui, CLI version is still far ahead of any competition, it’s unbelievable.