Some newbie questions - backup organisation

[towerbr]

where I include (i.e. whitelist) a subfolder via filter, at that moment it will not include all other subfolders?
Or do I always have to explicitely include/whitelist subfolders?

Let’s say your folder structure looks something like:

folder/
├── music1/
├── folder-foo/
│   ├── file-a.txt
│   ├── file-b.txt
│   └── file-c.txt
├── music2/
├── folder-bar/
...

The filters file of this repository could be:

+music1/*
+music2/*
-*

The last line will EXclude everything else.

Looks I need to read a bit more

Take a look:

Filters/Include exclude patterns

For the backup command, the include/exclude patterns are read from a file named filters under the .duplicacy directory. For the restore command, the include/exclude patterns are specified as the command line arguments. Duplicacy offers two different methods for providing include/exclude filters, wildcard matching and regular expression matching. You may use one method exclusively or you may combine them as you deem necessary. All paths are relative to the repository (the folder you execute d…

[Tortuosit]

Ah yeah, also saw, that web-UI is quite helpful. But that is an unexpected order! I’d expect it to read top down. I.e. without any knowledge, I’d use:

-* (Exclude all at first)
+include1/*
+include2/*
…

Your example:
+music1/*
+music2/*
-*

reads like: Include music1/music2, then exclude everything. I.e., do nothing.

[towerbr]

I’d expect it to read top down.

And that’s the way it works!

Filters/Include exclude patterns

Therefore, the order of the patterns is significant. If a match with an include pattern is found, the path is said to be included without further comparisons. If a match with an exclude pattern is found, the path is said to be excluded without further comparison.

This way you propose:

I’d use:

-* (Exclude all at first)
+include1/*
+include2/*

the first line would always be found first and the other lines would never be read.

[Tortuosit]

I thought now I’d understand its way of thinking but I was proven wrong, because it does nothing. I thought this is the right way of thinking:

Duplicacy goes through all files and folders. I have to think from each individual file/folder perspective. Say “C:\folder1\Iamafile.txt”. Now it looks into filter line 1. Say “+blah”. It does not match. Next filter line. “-*”. Match. File is excluded, no further processing. Next file.

Wrong way of thinking?

So now my situation is this:

M:\ M:\Folder1\ M:\Folder2\ M:\FolderFoo\

I want to include anything. The only thing I want to exclude is M:\FolderFoo. I added via the web-ui:

-FolderFoo/ +*

Does nothing. I am disappointed. My whole thinking was wrong. Also I am too stupid to add line breaks here in code tags.

[Tortuosit]

I hope my thinking was right. After failed attempts, I always deleted Repositories via web ui, also deleted Storages, and there was a mess of filter files and other files left in .duplicacy-web\repositories\localhost…
That may have been my cause of struggling.

I’ve cleaned that up, restarted httpd, now it is working with my expected filters. Will see at restore time, if it included the folders I want.

[Tortuosit]

In restore - Are backups only listed after totally finished? It says “No previous backups for this backup ID”.

[TheBestPessimist]

So scripting + windows inbuilt cron is the way to go here?

See Scripts and utilities index.

[Tortuosit]

I suggest you set the repository to a root folder that contains the 5 folders and select the 5 folders using filters.

I find myself making more use of NTFS links (Symlinks) in order to include stuff that is not on the same file system level. I’m OK with this. Average user could not do this. A more flexible, organisation based include system would be nice.

[Christoph]

I thought now I’d understand its way of thinking but I was proven wrong,

I think I was similarly confused. Did you see this topic:

Improving the instructions for include/exclude patterns (filters)

I think the instructions for the include/exclude patterns could be improved, but I am myself too unsure of things to make any edits in the wiki. So I’m starting this thread to clarify some things first and once I (or anyone else) has a clear view of things, I/they can edit the wiki. So, let’s start with the first example under “1. Wildcard matching”: +foo/bar/* -* It is introduced by saying the following pattern list doesn’t do what is intended, since the foo directory will be excluded so t…

[towerbr]

It’s because @Tortuosit was reading like this:

Your example:
+music1/*
+music2/*
-*

reads like: Include music1/music2, then exclude everything. I.e., do nothing.

While the correct logic / reading is:

I (Duplicacy ) am backing up a file whose path matches music1/*. OK! Stop reading the filtering pattern and back up.
I’m backing up a file whose path matches music2/*. OK! Stop reading the filtering pattern and back up.
I’m backing up a file whose path does not match either music1/* or music2/*. Ah! But there is a third filter that says “all” (*), and it says to exclude, so I won’t do anything. Next file!

And there is an important detail that I took a while to understand when I started using , but it is essential to consider when you are creating your filtering logic:

Improving the instructions for include/exclude patterns (filters)

First, the indexing starts at the root of the repository and do a non-recursive listing. (…). If it is a folder, Duplicacy will no descend into the excluded folder to list its files/subdirectories. Therefore, it is possible to exclude an entire subdirectory tree at once without wasting time checking ever file/folder under it.

This is a good example:

Improving the instructions for include/exclude patterns (filters)

Assuming foo/bar/file.txt is the only file you want to back up, the correct patterns are:

+foo/bar/file.txt
+foo/bar/
+foo/
-*

Duplicacy lists the root of the repository. The foo/ folder matches the third pattern, while all others will match the fourth one and thus get excluded. In the next step Duplicacy lists the foo/ folder and only foo/bar/ will be included. Finally, Duplicacy lists foo/bar/ and finds foo/bar/file.txt which is the only file with a matching include pattern while all others will be excluded by -* . Overall, Duplicacy only needs to list 3 directories to locate the only file to be included without iterating through the entire directory tree.

[towerbr]

Also I am too stupid to add line breaks here in code tags.

Sorry, I hadn’t noticed this doubt before.

Use four spaces

https://www.markdownguide.org/basic-syntax/#code-blocks

or fenced code blocks

https://www.markdownguide.org/extended-syntax/#fenced-code-blocks

[Tortuosit]

Thanks for clarification. Yes, I meanwhile understood I need to think from the individual file perspective exactly as you wrote. Filtering order makes sense then. But I think it is unuasual. But easy as well.

I came from a result list based thinking. Like:

• At first exclude/blacklist all -> So we have an empty list as a start
• Then do whitelisting, i.e., add all matching files to the list
• Then we have the finished list: Start backing up what is in that list

[Tortuosit]

Will read it, thx. Meanwhile I have understood the way filters work here.

[towerbr]

Then do whitelisting, i.e., add all matching files to the list

Ok, just remember that you will have to place the include/whitelisting patterns before the “exclude/blacklist all” pattern you created in the previous step you mentioned.

[Tortuosit]

Ok, just remember that you will have to place the include/whitelisting patterns before the “exclude/blacklist all” pattern you created in the previous step you mentioned.

So now it looks like I have not understood. sigh

Filters:
-folder1/
+*

This will not include anything if my testing is not broken. But from my file perspective:

“I am file \bleh.txt”
“Do I match -folder1/ ? No - Continue”
“Do I match +* ? Yes. - Include me”

[Tortuosit]

This does backup nothing, no matter if +* is on top or at the bottom. Will study later. The god of pain is arising…

-/ntuser.dat
-/appdata/local/
+*

No included files.

+*
-/ntuser.dat
-/appdata/local/

No included files.

[towerbr]

So now it looks like I have not understood.

Yes, I know it’s confusing…

But it’s because the first example was with full EXclusion at the end

+music1/*
+music2/*
-*

and the last example was with full INclusion at the end:

-folder1/
+*

But from my file perspective:

“I am file \bleh.txt”
“Do I match -folder1/ ? No - Continue”
“Do I match +* ? Yes. - Include me”

Your understanding is completely correct.

Ok, just remember that you will have to place the include/whitelisting patterns before the “exclude/blacklist all” pattern you created in the previous step you mentioned.

Let me put it in a wider perspective:

Include or exclude parameters for specific folders/files must be placed before wildcard parameters.

[Christoph]

• /ntuser.dat
• /appdata/local/
  +*

No included files.

+*

• /ntuser.dat
• /appdata/local/

No included files.

You might want to remove the / at the beginning of the paths, but that doesn’t explain “no included files”. Are you sure that no files are included? Anf if so, that there are actually files to be included?

[Tortuosit]

Thanks for your reply, I wrote in some other thread: Cause was, I had symlinks to the real backup target NOT on my backup root level.

[Christoph]

I wrote in some other thread

It is very easy to link or even quote other posts. To link, use the URL you get from the icon under the respective post. To quote, select the text you want to quote and hit the “quote” button.

Doing so makes it a lot easier for other users to navigate the forum and find relevant information.