Prune terminology

Support
#1

I am having some problems understanding the terminology.

I have written a script to handle our daily backup. At the end of the backup of all ZFS pools on the FreeNAS box I run a prune command.

I am not sure if I should run for each “storage” or by using the “-all” flag.

The confusing part is that I specify ‘-storage b2-support’ - this corresponds to a block in the JSON preferences file called “name”. But when I do this the command displays “Storage set to b2://flypelican-freenas” which is the storage in the preferences file ““storage”: “b2://flypelican-freenas””

image
image792×184 24.4 KB

The line in my script that I run daily is:
$DUPLICACY_CMD -log prune -all -exhaustive -keep 0:30 -keep 7:7 -threads $NUM_THREADS >> "${LOGDIR}/$ZFSSNAPSHOTNAME/$RUNNAME.log"

Should I be running “-all”? I assume this runs the prune against all “id’s” and looks after any un-referenced chunks? I haven’t yet tested the “-keep” params but I’m hoping that what I have above will keep me 0 snapshot after 30 days and 1 each week ie. a 4 x weekly snapshots?

#2

The storage is referred to by the storage name (b2-support) but it is uniquely identified by the storage url (b2://flypelican-freenas). If you have multiple directories to back up to the same storage, each directory can name the storage differently but the storage url will be the same.

You’ll just need one prune operation for each storage, with the -all option. This is more efficient than running one prune from each directory without the -all option.

The -exhaustive option should not be included in a script running regularly, because if there is another backup in progress it will collect all chunks that have been uploaded by that backup and turn them into fossils which takes time. It won’t corrupt the backup though because these fossils will later be turned back into normal chunks.

#3

Thanks for the reply. So where you say, “You’ll just need one prune operation for each storage”, does this mean one prune operation per “Storage URL” or “Storage Name”? See how the terminology is confusing?
I changed my script so that I run the prune command once a week:

+ /mnt/pool0/support/duplicacy/duplicacy_freebsd_x64_2.4.0 -log prune -all -exclusive -keep 0:30 -threads 15

But then I get confused again. I specify “-all” then in the output I see:

2020-03-14 15:04:37.631 INFO REPOSITORY_SET Repository set to /mnt/duplicacy/trainingandchecking

Screen Shot 2020-03-14 at 3.05.18 pm
Screen Shot 2020-03-14 at 3.05.18 pm1780×562 178 KB

If I specify “-all” why is it defaulting to the first listed storage ID in the preferences file? Note: I have one preferences file outside of the storage being backed-up that contains the info for each “ID”

[
    {
        "name": "b2-trainingandchecking",
        "id": "trainingandchecking",
        "repository": "/mnt/duplicacy/trainingandchecking",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    },
    {
        "name": "b2-support",
        "id": "support",
        "repository": "/mnt/duplicacy/support",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    },
    {
        "name": "b2-camo",
        "id": "camo",
        "repository": "/mnt/duplicacy/camo",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    },
    {
        "name": "b2-pilots",
        "id": "pilots",
        "repository": "/mnt/duplicacy/pilots",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    },
    {
        "name": "b2-aocdocuments",
        "id": "aocdocuments",
        "repository": "/mnt/duplicacy/aocdocuments",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    },
    {
        "name": "b2-office",
        "id": "office",
        "repository": "/mnt/duplicacy/office",
        "storage": "b2://flypelican-freenas",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "b2_id": "xxxxxxxxxxxxxxx",
            "b2_key": "xxxxxxxxxxxxxx",
            "password": "xxxxxxxxxxxxxxxxxxx"
        },
        "filters": "/mnt/pool0/support/duplicacy/b2/filters"
    }
]
#4

-all doesn’t mean to prune all the storages in the preferences file. All duplicacy command (except the copy command) only work with one storage at a time. -all actually means the prune command applies to all ids on that storage.

In your case the storage named b2-trainingandchecking is selected because it is the first one. You can select the second one by adding -storage b2-support.

#5

How can I have different IDs on one storage? What would that preferences file look like?
As it stands you’re saying I would need to run the prune command multiple times as in my preferences file I have one storage ‘name’ for each ‘id’

#6

This would be different backup “clients” (not a duplicacy term, but my own) that back-up to that storage.

My mapping of duplicacy terms that makes sense to me from other backup solutions is:

Repository = Backup Source
Snapshot ID = client name
Storage = Backup Destination

The following post covers this in more details: About Duplicacy nomenclature

#7

Thanks - I’ve read that nomenclature page over and over during the past few months trying to implement these Duplicacy backups but I’m still falling over my feet with it all.

I have numerous FreeNAS “shares” that I have been backing up to individual “Snapshot ID’s” - are you saying that these should all backup using the one “Snapshot ID”?

The way I do backups is as follows - for each SMB share off the FreeNAS server (each is a ZFS dataset)

  • perform a ZFS snapshot of the dataset (SMB share)
  • mount the snapshot on the FreeNAS server
  • perform the Duplicacy backup to B2 storage
    • I ‘cd’ into the preferences dir that hold the ‘preferences’ file (above) and run the duplicacy binary from there
    • Snapshot ID matches the dataset name
  • rename the ZFS snapshot and destroy and ZFS snapshots older than 1d (so I can locally rollback and change without pulling back from B2)

I am not sure if I have implement this correctly, I couldn’t think of any other way of doing it.

Screen Shot 2020-03-16 at 12.13.12 am
Screen Shot 2020-03-16 at 12.13.12 am1416×306 54.4 KB

This part of the prune forum help page implies that I shouldn’t want to run a prune from each repository. I simply do not understand the last line:

The design of :d: however was based on the assumption that only one instance would run the prune command (using -all ). This can greatly simplify the implementation.

Only run from “one instance” - what instance?

Screen Shot 2020-03-16 at 12.24.31 am
Screen Shot 2020-03-16 at 12.24.31 am1006×150 6.26 KB

Take these two preference file entries:

[
    {
            "name": "b2-ntlstorage01-accounts",
            "id": "ntlstorage01-accounts",
            "repository": "/mnt/duplicacy/accounts",
            "storage": "b2://flypelican-duplicacy",
            "encrypted": true,
            "no_backup": false,
            "no_restore": false,
            "no_save_password": false,
            "nobackup_file": "",
            "keys": {
                "b2_id": "xxx",
                "b2_key": "xxx",
                "password": "xxx"
            },
            "filters": "/mnt/pool0/support/duplicacy/config/filters"
        },
        {
            "name": "b2-ntlstorage01-aocdocuments",
            "id": "ntlstorage01-aocdocuments",
            "repository": "/mnt/duplicacy/aocdocuments",
            "storage": "b2://flypelican-duplicacy",
            "encrypted": true,
            "no_backup": false,
            "no_restore": false,
            "no_save_password": false,
            "nobackup_file": "",
            "keys": {
                "b2_id": "xxx",
                "b2_key": "xxx",
                "password": "xxx"
            },
            "filters": "/mnt/pool0/support/duplicacy/config/filters"
        }
    ]

What is the “selected storage”??? Is it “storage”: “b2://flypelican-duplicacy” or “name”: “b2-ntlstorage01-aocdocuments”

Could I run the following to prune BOTH b2-ntlstorage01-accounts and b2-ntlstorage01-aocdocuments:
duplicacy -prune -storage b2-ntlstorage01-accounts -all

Or would I have to run:
duplicacy -prune -storage b2-ntlstorage01-accounts -all
duplicacy -prune -storage b2-ntlstorage01-aocdocuments -all

The thing that is doing my head in is that both the above example “Storage Names” ( “b2-ntlstorage01-aocdocuments” & “b2-ntlstorage01-accounts” ) backup to the same “Storage” ( b2://flypelican-duplicacy ) and will only ever have one “Storage ID” each ( “ntlstorage01-aocdocuments” and “ntlstorage01-accounts” ).

I ran a “dry-run” prune on an explicitly set storage ID " b2-ntlstorage01-support" but the debug output shows it ran the prune on all ID’s across all repositories in storage b2://flypelican-duplicacy

Behavior is the same if I remove “-all”

 /mnt/pool0/support/duplicacy/duplicacy_freebsd_x64_2.4.0 -debug prune -dry-run -all -storage b2-ntlstorage01-support

Repository set to /mnt/duplicacy/support
Storage set to b2://flypelican-duplicacy
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_ID
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_ID
Reading b2_id from preferences
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_KEY
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_KEY
Reading b2_key from preferences
download URL is: https://f000.backblazeb2.com
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_ID
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_ID
Reading b2_id from preferences
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_KEY
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_KEY
Reading b2_key from preferences
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_PASSWORD
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_PASSWORD
Reading password from preferences
Using 16384 iterations for key derivation
[0] URL request 'HEAD https://f000.backblazeb2.com/file/flypelican-duplicacy/nesting' returned status code 404
https://f000.backblazeb2.com/file/flypelican-duplicacy/nesting did not return headers
Chunk read levels: [1], write level: 1
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: xxxxxxx
Hash key: xxxxxxx
ID key: xxxxxxx
File chunks are encrypted
Metadata chunks are encrypted
Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_PASSWORD
Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_PASSWORD
Reading password from preferences
id: , revisions: [], tags: [], retentions: [], exhaustive: false, exclusive: false, dryrun: true, deleteOnly: false, collectOnly: false
Listing all snapshot ids
Listing revisions for snapshot ntlstorage01-calm
Downloaded file snapshots/ntlstorage01-calm/1
Downloaded file snapshots/ntlstorage01-calm/2
Listing revisions for snapshot ntlstorage01-flightoperations
Downloaded file snapshots/ntlstorage01-flightoperations/1
Listing revisions for snapshot ntlstorage01-support
Loaded file snapshots/ntlstorage01-support/1 from the snapshot cache
Loaded file snapshots/ntlstorage01-support/2 from the snapshot cache
No snapshot to delete

I find the terminology used profoundly confusing.

1 Like
#8

Normally, if you only have one physical storage (URL) then you don’t need several names for it. I mean, you can, but your life will be simplified if you use the one name. It’s really just a client-side friendly name for using a different storage other than the default (first in the list; usually called ‘default’).

However, in your case, you seem to be using a slightly more advanced configuration where you’re using one preferences file and using references to the repository location. Off the top of my head, I don’t think you can specify multiple snapshot IDs using the same storage name, as the storage name is the ‘sector’ for choosing the backup job (and snapshot ID) to run. So yea, you may have to use unique names.

Otherwise, you can use separate .duplicacy\preferences for each snapshot ID (which is usually the standard configuration and therefore the terminology changes slightly.

#9

This is essentially what I have, each ZFS dateset has a .duplicacy file that points to the one preferences file so that the sensitive password / key data is nowhere near user shares.

I am trying to follow what the nomenclature says I should be doing.

  • Storage Name = Unique identifier for the JSON section in preferences file
  • Repository ID = Unique ID for each series of snapshots
  • Storage URL = same for all my B2 backups
#10

So I have done more experimenting today. I have run two different ‘dry-run’ prune commands.

This first one, I specify “-storage” to be one of my preference file storage names “b2-ntlstorage01-support” but using the “-all” parameter:

 /mnt/pool0/support/duplicacy/duplicacy_freebsd_x64_2.4.0 -debug -log prune -dry-run -storage b2-ntlstorage01-support -all -keep 0:90 -keep 7:30 -threads 10

This one is the same but instead of specifying “-all” I specify a single snapshot ID “ntlstorage01-support”:

/mnt/pool0/support/duplicacy/duplicacy_freebsd_x64_2.4.0 -debug -log prune -dry-run -storage b2-ntlstorage01-support -id ntlstorage01-support -keep 0:90 -keep 7:30 -threads 10

Both commands result in the same log output - the prune is touching ALL snapshot ID’ s - not just ntlstorage01-support. You can see in the output that its hitting other ID’s such as ntlstorage01-calm, ntlstorage01-pilots etc

> 2020-03-16 11:44:28 ######## START PRUNE ########
> 2020-03-16 11:44:28.845 INFO REPOSITORY_SET Repository set to /mnt/duplicacy/support
> 2020-03-16 11:44:28.845 INFO STORAGE_SET Storage set to b2://flypelican-duplicacy
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_ID
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_ID
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_PREFERENCE Reading b2_id from preferences
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_KEY
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_KEY
> 2020-03-16 11:44:28.845 DEBUG PASSWORD_PREFERENCE Reading b2_key from preferences
> 2020-03-16 11:44:29.616 INFO BACKBLAZE_URL download URL is: https://f000.backblazeb2.com
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_ID
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_ID
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_PREFERENCE Reading b2_id from preferences
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_B2_KEY
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_B2_KEY
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_PREFERENCE Reading b2_key from preferences
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_PASSWORD
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_PASSWORD
> 2020-03-16 11:44:30.311 DEBUG PASSWORD_PREFERENCE Reading password from preferences
> 2020-03-16 11:44:31.202 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
> 2020-03-16 11:44:31.417 TRACE BACKBLAZE_CALL [0] URL request 'HEAD https://f000.backblazeb2.com/file/flypelican-duplicacy/nesting' returned status code 404
> 2020-03-16 11:44:31.417 DEBUG BACKBLAZE_LIST https://f000.backblazeb2.com/file/flypelican-duplicacy/nesting did not return headers
> 2020-03-16 11:44:31.417 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
> 2020-03-16 11:44:31.436 INFO CONFIG_INFO Compression level: 100
> 2020-03-16 11:44:31.436 INFO CONFIG_INFO Average chunk size: 4194304
> 2020-03-16 11:44:31.436 INFO CONFIG_INFO Maximum chunk size: 16777216
> 2020-03-16 11:44:31.436 INFO CONFIG_INFO Minimum chunk size: 1048576
> 2020-03-16 11:44:31.436 INFO CONFIG_INFO Chunk seed: xxxxxxxxx
> 2020-03-16 11:44:31.436 TRACE CONFIG_INFO Hash key: xxxxxxxxx
> 2020-03-16 11:44:31.436 TRACE CONFIG_INFO ID key: xxxxxxxxx
> 2020-03-16 11:44:31.436 TRACE CONFIG_INFO File chunks are encrypted
> 2020-03-16 11:44:31.436 TRACE CONFIG_INFO Metadata chunks are encrypted
> 2020-03-16 11:44:31.436 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2-NTLSTORAGE01-SUPPORT_PASSWORD
> 2020-03-16 11:44:31.436 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_B2_NTLSTORAGE01_SUPPORT_PASSWORD
> 2020-03-16 11:44:31.436 DEBUG PASSWORD_PREFERENCE Reading password from preferences
> 2020-03-16 11:44:31.436 DEBUG DELETE_PARAMETERS id: ntlstorage01-support, revisions: [], tags: [], retentions: [0:90 7:30], exhaustive: false, exclusive: false, dryrun: true, deleteOnly: false, collectOnly: false
> 2020-03-16 11:44:31.437 INFO RETENTION_POLICY Keep no snapshots older than 90 days
> 2020-03-16 11:44:31.437 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 30 day(s)
> 2020-03-16 11:44:31.437 TRACE SNAPSHOT_LIST_IDS Listing all snapshot ids
> 2020-03-16 11:44:31.610 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-operations
> 2020-03-16 11:44:31.948 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-operations/1 from the snapshot cache
> 2020-03-16 11:44:31.949 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-safety
> 2020-03-16 11:44:32.295 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-safety/1 from the snapshot cache
> 2020-03-16 11:44:32.295 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-support
> 2020-03-16 11:44:32.633 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-support/1 from the snapshot cache
> 2020-03-16 11:44:32.800 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-support/2 from the snapshot cache
> 2020-03-16 11:44:32.969 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-support/3 from the snapshot cache
> 2020-03-16 11:44:33.140 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-support/4 from the snapshot cache
> 2020-03-16 11:44:33.141 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-itscratch
> 2020-03-16 11:44:33.478 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-itscratch/1 from the snapshot cache
> 2020-03-16 11:44:33.478 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-calm
> 2020-03-16 11:44:33.822 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-calm/1 from the snapshot cache
> 2020-03-16 11:44:33.991 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-calm/2 from the snapshot cache
> 2020-03-16 11:44:34.159 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-calm/3 from the snapshot cache
> 2020-03-16 11:44:34.327 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-calm/4 from the snapshot cache
> 2020-03-16 11:44:34.496 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-calm/5 from the snapshot cache
> 2020-03-16 11:44:34.496 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-customerservice
> 2020-03-16 11:44:34.844 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-customerservice/1 from the snapshot cache
> 2020-03-16 11:44:34.844 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-pelicanit
> 2020-03-16 11:44:35.184 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-pelicanit/1 from the snapshot cache
> 2020-03-16 11:44:35.184 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-pilots
> 2020-03-16 11:44:35.520 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-pilots/1 from the snapshot cache
> 2020-03-16 11:44:35.757 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-pilots/2 from the snapshot cache
> 2020-03-16 11:44:35.757 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-aocdocuments
> 2020-03-16 11:44:36.108 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-aocdocuments/1 from the snapshot cache
> 2020-03-16 11:44:36.277 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-aocdocuments/2 from the snapshot cache
> 2020-03-16 11:44:36.278 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-administration
> 2020-03-16 11:44:36.616 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-administration/1 from the snapshot cache
> 2020-03-16 11:44:36.617 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-camo
> 2020-03-16 11:44:36.955 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-camo/1 from the snapshot cache
> 2020-03-16 11:44:36.955 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-flightoperations
> 2020-03-16 11:44:37.296 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-flightoperations/1 from the snapshot cache
> 2020-03-16 11:44:37.468 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-flightoperations/2 from the snapshot cache
> 2020-03-16 11:44:37.468 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-management
> 2020-03-16 11:44:37.814 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-management/1 from the snapshot cache
> 2020-03-16 11:44:37.814 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-accounts
> 2020-03-16 11:44:38.153 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-accounts/1 from the snapshot cache
> 2020-03-16 11:44:38.320 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-accounts/2 from the snapshot cache
> 2020-03-16 11:44:38.488 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-accounts/3 from the snapshot cache
> 2020-03-16 11:44:38.658 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-accounts/4 from the snapshot cache
> 2020-03-16 11:44:38.658 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-trainingandchecking
> 2020-03-16 11:44:39.053 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-trainingandchecking/1 from the snapshot cache
> 2020-03-16 11:44:39.053 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot ntlstorage01-office
> 2020-03-16 11:44:39.390 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/ntlstorage01-office/1 from the snapshot cache
> 2020-03-16 11:44:39.391 INFO SNAPSHOT_NONE No snapshot to delete
> 2020-03-16 11:44:39 ######## END PRUNE ########

In the cache dir for storage ID “b2-ntlstorage01-support” I now have snapshot data downloaded for all the other snapshot ID’s, not just its own.

Screen Shot 2020-03-16 at 12.04.42 pm
Screen Shot 2020-03-16 at 12.04.42 pm2320×750 163 KB

Please don’t tell me I am the only one having a hard time understanding how this prune command works??!!

#11

I think I may now understand it a little better. Below is another log file that I produced running the two commands side by side.

I assume now that the prune needs to look at ALL chunks for ALL ID’s in the same storage to work out what it should fossil / delete, but if I run with a specific ID it will only remove chunks / snapshot revisions for that one ID.

Is this correct?

2020-03-16 12:17:09 ######## START PRUNE ########

2020-03-16 12:17:09 ######## USING -all  option ########

2020-03-16 12:17:09.850 INFO REPOSITORY_SET Repository set to /mnt/duplicacy/support
2020-03-16 12:17:09.850 INFO STORAGE_SET Storage set to b2://flypelican-duplicacy
2020-03-16 12:17:10.749 INFO BACKBLAZE_URL download URL is: https://f000.backblazeb2.com
2020-03-16 12:17:13.328 INFO RETENTION_POLICY Keep no snapshots older than 1 days
2020-03-16 12:17:21.638 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-support at revision 1
2020-03-16 12:17:21.657 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-support at revision 2
2020-03-16 12:17:21.657 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-accounts at revision 1
2020-03-16 12:17:21.686 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-flightoperations at revision 1
2020-03-16 12:17:21.687 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-calm at revision 1
2020-03-16 12:17:21.687 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-calm at revision 2
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk a2b689bf17a1c165480e090a48299344b043ca573fd515a9b7ae6cf9257b3750
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 2b8fec15b490103959c726abc98a529ec6088460392c299dc62a442e20ae8d63
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 1c9edf114e9051c4ef47297259a5788014f4247635db9315fdda4f023a7fbbba
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk d4d2d3a98721c720882763b508b59451cc6193da894153db38dbf0fd5a9cf662
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 218868608393db8d32e5de0e99971b3246e1d419f6addb2ca42e07387508bb65
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 49782fb1744e054297d71e1df5430fb98fc48b1142e9fe57ce99c52d2fba7d17
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 3051e6f08acc50f50b176f14fe3abb191dca8982c90463b8db980d15e2bbf1f6
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk a4e3e0039dcf51f81df7ef828a062854149a271ee0c9fa6244770d883ecc6d82
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 1fb4aee138286bd5a249cfd1adc87727fda4966186f0eee4d6608a0a01521844
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk ff9beda895019111e94605e27fde328481bf8ed6d0ba75882df56470e22425e9
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 56cfe3a7b0c2f569724671b908652f63cbed133eccac646e210fb44bb5766041
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 4257eb1242442916c6b4ea51b4e34ddff526a5af078f145da1b0c0ef099c1385
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk 8a4d019c196d5b65e3135426151d137dfc785f4442ea67a792aacbb98bcb2248
2020-03-16 12:17:21.890 INFO CHUNK_UNREFERENCED Found unreferenced chunk b3f8cc03762161595c9b83c50a8fb0fa62cdd384f6ca0e5e981326a1383bee46

2020-03-16 12:17:21 ######## USING -id ntlstorage01-support option ########

2020-03-16 12:17:21.923 INFO REPOSITORY_SET Repository set to /mnt/duplicacy/support
2020-03-16 12:17:21.923 INFO STORAGE_SET Storage set to b2://flypelican-duplicacy
2020-03-16 12:17:22.763 INFO BACKBLAZE_URL download URL is: https://f000.backblazeb2.com
2020-03-16 12:17:24.678 INFO RETENTION_POLICY Keep no snapshots older than 1 days
2020-03-16 12:17:34.123 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-support at revision 1
2020-03-16 12:17:34.125 INFO SNAPSHOT_DELETE Deleting snapshot ntlstorage01-support at revision 2
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk 2b8fec15b490103959c726abc98a529ec6088460392c299dc62a442e20ae8d63
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk d4d2d3a98721c720882763b508b59451cc6193da894153db38dbf0fd5a9cf662
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk a4e3e0039dcf51f81df7ef828a062854149a271ee0c9fa6244770d883ecc6d82
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk 218868608393db8d32e5de0e99971b3246e1d419f6addb2ca42e07387508bb65
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk 8a4d019c196d5b65e3135426151d137dfc785f4442ea67a792aacbb98bcb2248
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk a2b689bf17a1c165480e090a48299344b043ca573fd515a9b7ae6cf9257b3750
2020-03-16 12:17:34.354 INFO CHUNK_UNREFERENCED Found unreferenced chunk 49782fb1744e054297d71e1df5430fb98fc48b1142e9fe57ce99c52d2fba7d17
2020-03-16 12:17:34 ######## END PRUNE ########
#12

This is correct.

Your preferences files shows that there is only one storage (b2://flypelican-duplicacy) and two snapshot ids (ntlstorage01-accounts and ntlstorage01-aocdocuments). If you run the prune command once for each id (via -storage b2-ntlstorage01-accounts and -storage b2-ntlstorage01-aocdocuments) then the cache will full of duplicate snapshot chunks because the cache is organized by the storage names (b2-ntlstorage01-accounts and b2-ntlstorage01-aocdocuments).

[
    {
            "name": "b2-ntlstorage01-accounts",
            "id": "ntlstorage01-accounts",
            "repository": "/mnt/duplicacy/accounts",
            "storage": "b2://flypelican-duplicacy",
            "encrypted": true,
            "no_backup": false,
            "no_restore": false,
            "no_save_password": false,
            "nobackup_file": "",
            "keys": {
                "b2_id": "xxx",
                "b2_key": "xxx",
                "password": "xxx"
            },
            "filters": "/mnt/pool0/support/duplicacy/config/filters"
        },
        {
            "name": "b2-ntlstorage01-aocdocuments",
            "id": "ntlstorage01-aocdocuments",
            "repository": "/mnt/duplicacy/aocdocuments",
            "storage": "b2://flypelican-duplicacy",
            "encrypted": true,
            "no_backup": false,
            "no_restore": false,
            "no_save_password": false,
            "nobackup_file": "",
            "keys": {
                "b2_id": "xxx",
                "b2_key": "xxx",
                "password": "xxx"
            },
            "filters": "/mnt/pool0/support/duplicacy/config/filters"
        }
    ]
1 Like
#13

So I am OK to run a prune -all from one of the snapshot ID’s against the shared storage and I will successfully prune all my ID’s chunks?

#14

Yep.

Remember that pruning happens in two steps - collection of fossils then deletion of fossils - and between these two phases, a successful backup has to be run in order to determine that the fossilised chunks are safe to delete.

2 Likes