Exclude List not updated

Hi

I’m testing duplicacy for our 3rd Level Backup @backblaze,
It runs good with my initial setup (app. 2TB of data)
now i want to include my main storage (30TB), which was excluded beforhand.

But duplicacy seems to ignore the updated exclude-list.

How can i force duplicacy to update the list?

cheers Guido

The include/exclude patterns are stored in a file named filters under the .duplicacy directory (which is under the repository directory). Does the content match the list shown in the Repository Setup dialog?

Hi

Thank you for your comment, i checked the filters file, it reflects the settings in the gui.

I made some tests adding and removing the line in the gui.
the file .duplicacy/filters reflects the entry all the time.

But i am not able to duplicate the (formerly excluded) folder
+Kunden/

I also tried to include it (+folder/) with no better result.

Would it help to delete the whole .duplicacy - folder and start from scratch?

Can you post here the content of the filters file? I suspect that there is another pattern that excludes the Kunden directory.

Ok,

I deleted the content of the folder, an rebuild the Filters but “unden” Folder still excluded
Here is the listing, the filter and the log:
(The Folder Kunden, has the name “XyZ Kunden” with a Space, but “services Kunden” has also a Space, so I thought it doesn’t matter )

listing

drwxrwxr-x+    26 Mär 23:20 .
drwxrwxrwt@    26 Mär 18:43 ..
-rw-r--r--@    22 Mär 10:03 .DS_Store
drwx------     15 Mär 16:29 .Spotlight-V100
drwxrwxrwt@    18 Jul  2014 .TemporaryItems
d-wx-wx-wt@    24 Mär 09:08 .Trashes
-rw-r--r--@    24 Jun  2014 .com.apple.timemachine.donotpresent
drwxr-xr-x+    27 Mär 10:58 .duplicacy
drwx------     26 Mär 04:05 .fseventsd
drwxr-x---+     3 Jul  2014 HR.Scan
-rwxrw----@     4 Jun  2002 ICON?
drwxrwx---@    21 Aug  2016 IT
drwxr-x---@    16 Jan 14:54 Agentur
drwxrwx---+    22 Mär 09:42 Kunden
drwxrwx---+     1 Feb 15:26 services Kunden 
drwxr-xr-x+     8 Mär 17:07 Library
drwxr-xr-x+     6 Jan  2016 OD_BU
drwxr-x---@    14 Feb  2013 PIS
drwxrwx---+    24 Mär 14:39 Transport
drwxr-xr-x     27 Jun  2014 _CCC Archives
drwxr-xr-x+    25 Mär 02:13 _CCC SafetyNet

Filters:

+Kunden/
-_CCC SafetyNet/
-_CCC Archives/
-IT/
-Transport/
-.Spotlight-V100/
-.fseventsd/
-.Trashes/
-.TemporaryItems/
-PIS/
-OD_BU/
-Library/
-HR.Scan/
filters (END)

LOG:

15:34:32.906    Last backup at revision 2 found
15:34:32.906    Indexing /Volumes/BackUp
15:34:33.055    Loaded 13 include/exclude pattern(s)
15:34:33.055    Pattern: +Kunden/
15:34:33.055    Pattern: -_CCC SafetyNet/
15:34:33.055    Pattern: -_CCC Archives/
15:34:33.055    Pattern: -IT/
15:34:33.055    Pattern: -Transport/
15:34:33.055    Pattern: -.Spotlight-V100/
15:34:33.055    Pattern: -.fseventsd/
15:34:33.055    Pattern: -.Trashes/
15:34:33.055    Pattern: -.TemporaryItems/
15:34:33.055    Pattern: -PIS/
15:34:33.055    Pattern: -OD_BU/
15:34:33.055    Pattern: -Library/
15:34:33.055    Pattern: -HR.Scan/

What did you mean by ‘The Folder Kunden, has the name “XyZ Kunden” with a Space’?

It doesn’t look like a permission issue, but was there a log message saying it complained about not being able to list the Kunden folder?

What did you mean by ‘The Folder Kunden, has the name “XyZ Kunden” with a Space’?

Ignore it, the name listed is different, then what we use in reality, due to privacy reasons.

And no there was no complain. in the log

The complet verbose log:

22:22:13.000    Command: duplicacy -background -log -d backup -stats -threads 64 -limit-rate 7500
22:22:13.289    Storage set to b2://xxxxx
22:22:13.289    Reading the environment variable DUPLICACY_B2_ID
22:22:13.616    Reading the environment variable DUPLICACY_B2_KEY
22:22:15.470    Reading the environment variable DUPLICACY_PASSWORD
22:22:16.380    Compression level: 100
22:22:16.380    Average chunk size: 4194304
22:22:16.380    Maximum chunk size: 16777216
22:22:16.380    Minimum chunk size: 1048576
22:22:16.380    Chunk seed: 6b990f0cxx3f2222a959f75d006fdxxbcc1448985b28d4bbe7a9782cf66fbf
22:22:16.380    top: /Volumes/BackUp, quick: true, tag: 
22:22:16.381    Downloading latest revision for snapshot lt-serverbackup-root-BackUp
22:22:16.381    Listing revisions for snapshot lt-serverbackup-root-BackUp
22:22:16.748    Loaded file snapshots/lt-serverbackup-root-BackUp/2 from the snapshot cache
22:22:16.886    Chunk 80762390e77e9c85d7a84149d295c28f8472dd76b93af40a0727b82f81548004 has been loaded from the snapshot cache

… snip all the Chunks

22:33:33.533    Chunk 06f593fd31ca06487b211ccb8ca1690a17dbc71d93c99f0b221bf47aafb76b46 has been loaded from the snapshot cache
22:34:00.713    Last backup at revision 2 found
22:34:00.713    Indexing /Volumes/BackUp
22:34:00.768    Loaded 12 include/exclude pattern(s)
22:34:00.768    Pattern: -_CCC Archives/
22:34:00.768    Pattern: -_CCC SafetyNet/
22:34:00.768    Pattern: -HR.Scan/
22:34:00.768    Pattern: -IT/
22:34:00.768    Pattern: -Library/
22:34:00.768    Pattern: -OD_BU/
22:34:00.768    Pattern: -Transport/
22:34:00.768    Pattern: -PIS/
22:34:00.768    Pattern: -.Spotlight-V100/
22:34:00.768    Pattern: -.fseventsd/
22:34:00.768    Pattern: -.Trashes/
22:34:00.768    Pattern: -.TemporaryItems/

(I re-formatted your post for better reading.)

I uploaded a temporary build 1.2.2 with more verbose logging. Specifically, there will be a log message when a directory is being excluded, or listed. Hopefully with this version you’ll know what happens to that directory.

Hi gchen,

Thank you for this bild. Here the complete Output, I’m still missing a hint, why the “XyZ Kunden” is excluded.

Sorry for the mis-format, of the text, I havent fount the code-Tag !? Another Try :wink:

2017-03-29 15:58:12.953 INFO STORAGE_SET Storage set to b2://bu-server-XyZ
2017-03-29 15:58:12.953 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_ID
2017-03-29 15:58:13.064 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_KEY
2017-03-29 15:58:14.795 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_PASSWORD
2017-03-29 15:58:16.316 INFO CONFIG_INFO Compression level: 100
2017-03-29 15:58:16.316 INFO CONFIG_INFO Average chunk size: 4194304
2017-03-29 15:58:16.316 INFO CONFIG_INFO Maximum chunk size: 16777216
2017-03-29 15:58:16.316 INFO CONFIG_INFO Minimum chunk size: 1048576
2017-03-29 15:58:16.316 INFO CONFIG_INFO Chunk seed: 6b990f0caa3f2222a959f75d006fdfa8fbcc1448985b28d4bbe7a9782cf66fbf
2017-03-29 15:58:16.317 DEBUG BACKUP_PARAMETERS top: /Volumes/BackUp, quick: true, tag: 
2017-03-29 15:58:16.317 TRACE SNAPSHOT_DOWNLOAD_LATEST Downloading latest revision for snapshot XyZ-serverbackup-root-BackUp
2017-03-29 15:58:16.317 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot XyZ-serverbackup-root-BackUp
2017-03-29 15:58:16.697 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/XyZ-serverbackup-root-BackUp/2 from the snapshot cache
2017-03-29 15:58:16.847 DEBUG CHUNK_CACHE Chunk 80762390e77e9c85d7a84149d295c28f8472dd76b93af40a0727b82f81548004 has been loaded from the snapshot cache

SNIP 1120 lines of DEBUG CHUNK_CACHE Chunk

2017-03-29 16:09:17.108 DEBUG CHUNK_CACHE Chunk 06f593fd31ca06487b211ccb8ca1690a17dbc71d93c99f0b221bf47aafb76b46 has been loaded from the snapshot cache
2017-03-29 16:09:40.530 INFO BACKUP_START Last backup at revision 2 found
2017-03-29 16:09:40.530 INFO BACKUP_INDEXING Indexing /Volumes/BackUp
2017-03-29 16:09:40.619 INFO SNAPSHOT_FILTER Loaded 12 include/exclude pattern(s)
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -_CCC Archives/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -_CCC SafetyNet/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -HR.Scan/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -IT/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -Library/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -OD_BU/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -Transport/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -PIS/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -.Spotlight-V100/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -.fseventsd/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -.Trashes/
2017-03-29 16:09:40.620 TRACE SNAPSHOT_PATTERN Pattern: -.TemporaryItems/
2017-03-29 16:09:40.620 DEBUG LIST_ENTRIES Listing 
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE .Spotlight-V100/ is excluded
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE .TemporaryItems/ is excluded
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE .Trashes/ is excluded
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE .fseventsd/ is excluded
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE HR.Scan/ is excluded
2017-03-29 16:09:41.633 DEBUG LIST_EXCLUDE IT/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE Library/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE OD_BU/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE PIS/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE Transport/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE _CCC Archives/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_EXCLUDE _CCC SafetyNet/ is excluded
2017-03-29 16:09:41.676 DEBUG LIST_ENTRIES Listing XyZ Agentur/
2017-03-29 16:09:41.961 DEBUG LIST_ENTRIES Listing XyZ Agentur/.TemporaryItems/
2017-03-29 16:09:42.034 DEBUG LIST_ENTRIES Listing XyZ Agentur/.TemporaryItems/folders.1001/
2017-03-29 16:09:42.057 DEBUG LIST_ENTRIES Listing XyZ Agentur/.TemporaryItems/folders.1005/

SNIP 1000000 Lines of  DEBUG LIST_ENTRIES Listing 

2017-03-29 18:46:07.718 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/City_Selection/03_Online
2017-03-29 18:46:07.902 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/City_Selection/POS
2017-03-29 18:46:08.278 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/Kids_LowRes
2017-03-29 18:49:18.531 INFO STORAGE_SET Storage set to b2://bu-server-XyZ
2017-03-29 18:49:18.531 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_ID
2017-03-29 18:49:18.731 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_KEY
2017-03-29 18:49:20.549 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_PASSWORD
2017-03-29 18:49:21.460 INFO CONFIG_INFO Compression level: 100
2017-03-29 18:49:21.460 INFO CONFIG_INFO Average chunk size: 4194304
2017-03-29 18:49:21.460 INFO CONFIG_INFO Maximum chunk size: 16777216
2017-03-29 18:49:21.460 INFO CONFIG_INFO Minimum chunk size: 1048576
2017-03-29 18:49:21.460 INFO CONFIG_INFO Chunk seed: 6b990f0caa3f2222a959f75d006fdfa8fbcc1448985b28d4bbe7a9782cf66fbf
2017-03-29 18:49:21.596 DEBUG DELETE_PARAMETERS id: , revisions: [], tags: [], retentions: [0:180 7:30 1:3], exhaustive: false, exclusive: false, dryrun: false, deleteOnly: false, collectOnly: false
2017-03-29 18:49:21.669 INFO RETENTION_POLICY Keep no snapshots older than 180 days
2017-03-29 18:49:21.669 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 30 day(s)
2017-03-29 18:49:21.669 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 3 day(s)
2017-03-29 18:49:21.669 TRACE SNAPSHOT_LIST_IDS Listing all snapshot ids
2017-03-29 18:49:21.827 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot XyZ-serverbackup-admin-BackUp
2017-03-29 18:49:22.216 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/XyZ-serverbackup-admin-BackUp/1 from the snapshot cache
2017-03-29 18:49:22.216 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot XyZ-serverbackup-root-BackUp
2017-03-29 18:49:22.582 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/XyZ-serverbackup-root-BackUp/1 from the snapshot cache
2017-03-29 18:49:22.782 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/XyZ-serverbackup-root-BackUp/2 from the snapshot cache
2017-03-29 18:49:22.785 INFO SNAPSHOT_NONE No snapshot to delete

Subfolders of XyZ Kunden are included:

2017-03-29 18:46:07.718 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/City_Selection/03_Online
2017-03-29 18:46:07.902 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/City_Selection/POS
2017-03-29 18:46:08.278 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/Kids_LowRes

So I guess there must be a line saying Listing XyZ Kunden otherwise I can’t see how it can get to any of these subfolders.

This forum supports markdown so to include preformatted text you can use <pre> or ```

2017-03-29 16:22:18.545 DEBUG LIST_ENTRIES Listing XyZ Kunden/

Yes this debug line is there, it’s one of the omitted 1’000’000 lines.

So the directory is listed but did you mean any files under it didn’t get backed up?

Any other lines between these two lines?

2017-03-29 18:46:08.278 DEBUG LIST_ENTRIES Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_SuÃàdafrika/XyZ_TomoBrejc_SA_2012/XyZ_LowRes/Kids_LowRes
2017-03-29 18:49:18.531 INFO STORAGE_SET Storage set to b2://bu-server-XyZ

Yes nothing Backed up from this directory “XyZ Kunden/” (There is app. 30TB of Data, so it should be obviuos, if uploaded)

And there was nothing between those Lines in the log.

The Job started again Yesterday, the log-Tail seems a bit different this Morning:


18:45:37.944    Listing XyZ Kunden/VOS/01 Jobs/2013/Allgemein/1HJ/1HJ_Shooting_Südafrika/Voegele_TomoBrejc_SA_2012/Voegele_LowRes/Kids_LowRes/
18:48:08.000    Command: duplicacy -background -log -d prune -a -keep 0:180 -keep 7:30 -keep 1:3
18:48:09.631    Storage set to b2://bu-server-XyZ
18:48:09.631    Reading the environment variable DUPLICACY_B2_ID
18:48:09.860    Reading the environment variable DUPLICACY_B2_KEY
18:48:11.670    Reading the environment variable DUPLICACY_PASSWORD
18:48:12.985    Compression level: 100
18:48:12.985    Average chunk size: 4194304
18:48:12.985    Maximum chunk size: 16777216
18:48:12.985    Minimum chunk size: 1048576
18:48:12.985    Chunk seed: 6b990f0caa3f2222a959f75d006fdfa8fbcc1448985b28d4bbe7a9782cf66fbf
18:48:13.054    id: , revisions: [], tags: [], retentions: [0:180 7:30 1:3], exhaustive: false, exclusive: false, dryrun: false, deleteOnly: false, collectOnly: false
18:48:13.160    Keep no snapshots older than 180 days
18:48:13.160    Keep 1 snapshot every 7 day(s) if older than 30 day(s)
18:48:13.160    Keep 1 snapshot every 1 day(s) if older than 3 day(s)
18:48:13.160    Listing all snapshot ids
18:48:13.318    Listing revisions for snapshot XyZ-serverbackup-admin-BackUp
18:48:13.648    Loaded file snapshots/XyZ-serverbackup-admin-BackUp/1 from the snapshot cache
18:48:13.648    Listing revisions for snapshot XyZ-serverbackup-root-BackUp
18:48:13.998    Loaded file snapshots/XyZ-serverbackup-root-BackUp/1 from the snapshot cache
18:48:14.206    Loaded file snapshots/XyZ-serverbackup-root-BackUp/2 from the snapshot cache
18:48:14.209    No snapshot to delete

The Log changed from DEBUG LIST ENTRIES to Listing!

The backup command crashed however the GUI didn’t catch the error. Can you run the Duplicacy command line executable in Terminal to see what the error is? The duplicacy command line executable is usually under /Applications/Duplicacy.app/Contents/Resources, unless you installed Duplicacy in a different directory.

Change to your repository directory and then run the command as follows:

cd /Volumes/BackUp
/Applications/Duplicacy.app/Contents/Resources/duplicacy_osx_x64_1.2.0 --log -v backup 
XyZ-serverbackup:BackUp root# /Applications/Duplicacy.app/Contents/Resources/duplicacy_osx_x64_1.2.0 --log -v backup 
2017-03-31 13:46:26.624 INFO STORAGE_SET Storage set to b2://bu-server-XyZ
2017-03-31 13:46:31.017 TRACE SNAPSHOT_DOWNLOAD_LATEST Downloading latest revision for snapshot XyZ-serverbackup-root-BackUp
2017-03-31 13:46:31.017 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot XyZ-serverbackup-root-BackUp
2017-03-31 13:58:13.470 INFO BACKUP_START Last backup at revision 2 found
2017-03-31 13:58:13.472 INFO BACKUP_INDEXING Indexing /Volumes/BackUp
2017-03-31 13:58:13.606 INFO SNAPSHOT_FILTER Loaded 12 include/exclude pattern(s)
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -_CCC Archives/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -_CCC SafetyNet/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -HR.Scan/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -IT/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -Library/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -OD_BU/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -Transport/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -PIS/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -.Spotlight-V100/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -.fseventsd/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -.Trashes/
2017-03-31 13:58:13.606 TRACE SNAPSHOT_PATTERN Pattern: -.TemporaryItems/

Killed: 9
XyZ-serverbackup:BackUp root# 
XyZ-serverbackup:BackUp root# 

Killed 9?

Not being able to catch the kill signal is a bug in the GUI version, but I guess the reason why the CLI version was killed was that it ran out of memory. How many files do you know are under /Volumes/BackUp?

4’500’000 items. 40 TB
ist this size to big?

The total size of the files doesn’t matter as much as the number of files, because the entire repository tree has to be loaded into memory for processing. A repository containing 4.5 millions isn’t enormously large, but the current version has a flaw that it also loads all extended attributes when indexing the repository, and this may cause it to run out of memory.

I uploaded a new build 1.2.2 which doesn’t load all extended attributes into memory. The bug where a killed process isn’t reported has also been fixed in this new build. Please try it to see if this one works better.

Thanks for your effort.

Unfortunately, the backup does not finish his task:

09:38:12.000    Command: duplicacy -background -log backup -stats -threads 64 -limit-rate 7500
09:38:12.717    Storage set to b2://bu-server-XyZ
09:52:40.199    Last backup at revision 2 found
09:52:40.219    Indexing /Volumes/BackUp
09:52:40.258    Loaded 12 include/exclude pattern(s)
12:25:53.000    Command: duplicacy -background -log prune -a -keep 0:180 -keep 7:30 -keep 1:3
12:25:54.454    Storage set to b2://bu-server-XyZ
12:25:59.614    Keep no snapshots older than 180 days
12:25:59.614    Keep 1 snapshot every 7 day(s) if older than 30 day(s)
12:25:59.614    Keep 1 snapshot every 1 day(s) if older than 3 day(s)
12:26:00.932    No snapshot to delete