B2 pruning not deleting a lot of old revisions

Support
#1

I haven’t been able to figure this one out. I have 2 B2 buckets, one for each of my backup plans. both are configured the same including the prune options. In one, if I look I see some chunks having 13 revisions, which is what I expect. Storage on that one in Duplicacy is 458GB and the bucket on B2 says 591GB. (edit: forgot to mention, the second bucket there are many chunks with 113 revisions, daily for the last 2 months or so)

The main issue is the other one. Duplicacy says the storage is 25GB and B2 shows 358GB. Lifecycle on B2 is set to keep all.

Prune on both is

-keep 0:183 -keep 30:35 -keep 7:7 -keep 1:2 -a

I have already tried deleting the cache and ran prune with -exhaustive -exclusive

I cannot figure out why on B2 for that second bucket there are still daily revisions for roughly 60 days.

Running prune command from /root/.duplicacy-web/repositories/localhost/all
Options: [-log prune -storage b2 -keep 0:183 -keep 30:35 -keep 7:7 -keep 1:2 -a -d -threads 4]
2024-03-13 18:04:20.640 INFO STORAGE_SET Storage set to b2://backup
2024-03-13 18:04:20.855 INFO BACKBLAZE_URL Download URL is: https://f999.backblazeb2.com
2024-03-13 18:04:21.451 INFO RETENTION_POLICY Keep no snapshots older than 183 days
2024-03-13 18:04:21.451 INFO RETENTION_POLICY Keep 1 snapshot every 30 day(s) if older than 35 day(s)
2024-03-13 18:04:21.451 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 7 day(s)
2024-03-13 18:04:21.451 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 2 day(s)
2024-03-13 18:04:23.308 INFO SNAPSHOT_NONE No snapshot to delete

Any ideas how to remove those extra 100ish revisions?

Thanks

ps. Thanks for the great software.

#2

How many revisions are created per day?

Not sure what can cause that. Can you post the log from a check job?

#3

Scheduled is once per night. Occasionally, or when testing changes, I’ll manually run the backup. The vast majority of the last two months would have had only single revision per day.

Running check command from /root/.duplicacy-web/repositories/localhost/all
Options: [-log check -storage b2-backup -a -tabular]
2024-03-15 09:00:39.556 INFO STORAGE_SET Storage set to b2://bucket
2024-03-15 09:00:39.706 INFO BACKBLAZE_URL Download URL is: https://f999.backblazeb2.com
2024-03-15 09:00:40.512 INFO SNAPSHOT_CHECK Listing all chunks
2024-03-15 09:01:09.772 INFO SNAPSHOT_CHECK 2 snapshots and 34 revisions
2024-03-15 09:01:09.773 INFO SNAPSHOT_CHECK Total chunk size is 23,925M in 4930 chunks
2024-03-15 09:01:09.785 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 389 exist
2024-03-15 09:01:09.791 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 435 exist
2024-03-15 09:01:09.796 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 470 exist
2024-03-15 09:01:09.802 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 507 exist
2024-03-15 09:01:09.808 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 541 exist
2024-03-15 09:01:09.813 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 546 exist
2024-03-15 09:01:09.819 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 553 exist
2024-03-15 09:01:09.824 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 560 exist
2024-03-15 09:01:09.829 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 567 exist
2024-03-15 09:01:09.835 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 572 exist
2024-03-15 09:01:09.841 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 573 exist
2024-03-15 09:01:09.846 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 574 exist
2024-03-15 09:01:09.851 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 575 exist
2024-03-15 09:01:09.858 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 578 exist
2024-03-15 09:01:09.863 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 582 exist
2024-03-15 09:01:09.869 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 583 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back02 at revision 584 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 389 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 435 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 470 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 507 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 541 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 546 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 553 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 560 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 567 exist
2024-03-15 09:01:10.504 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 572 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 573 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 574 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 575 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 578 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 582 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 583 exist
2024-03-15 09:01:10.505 INFO SNAPSHOT_CHECK All chunks referenced by snapshot back01 at revision 584 exist
2024-03-15 09:01:10.691 INFO SNAPSHOT_CHECK 
    snap | rev |                          | files | bytes | chunks | bytes | uniq | bytes | new | bytes |
 back01 | 389 | @ 2023-09-19 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   5 |    4K |
 back01 | 435 | @ 2023-10-24 08:44       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 470 | @ 2023-11-28 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 507 | @ 2024-01-02 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 541 | @ 2024-02-06 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 546 | @ 2024-02-11 08:45       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 553 | @ 2024-02-18 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 560 | @ 2024-02-25 08:40       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 567 | @ 2024-03-03 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 572 | @ 2024-03-08 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 573 | @ 2024-03-09 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 574 | @ 2024-03-10 08:46       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 575 | @ 2024-03-11 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 578 | @ 2024-03-12 03:36       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 582 | @ 2024-03-13 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 583 | @ 2024-03-14 08:41       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | 584 | @ 2024-03-15 08:42       |     5 |    8K |      5 |    4K |    0 |     0 |   0 |     0 |
 back01 | all |                          |       |       |      5 |    4K |    5 |    4K |     |       |

     snap | rev |                          |  files |   bytes | chunks |   bytes | uniq |    bytes |  new |    bytes |
 back02 | 389 | @ 2023-09-19 08:41       | 159086 | 17,677M |   3523 | 16,504M | 1062 |   5,342M | 3523 |  16,504M |
 back02 | 435 | @ 2023-10-24 08:43       | 159089 | 16,653M |   3343 | 15,483M |   73 | 440,285K |  882 |   4,321M |
 back02 | 470 | @ 2023-11-28 08:41       | 159095 | 16,656M |   3352 | 15,491M |   35 | 213,470K |   82 | 448,096K |
 back02 | 507 | @ 2024-01-02 08:41       | 159095 | 16,656M |   3349 | 15,490M |   29 | 207,290K |   41 | 278,282K |
 back02 | 541 | @ 2024-02-06 08:41       | 159095 | 16,656M |   3355 | 15,490M |   25 | 177,434K |   38 | 226,728K |
 back02 | 546 | @ 2024-02-11 08:43       | 159095 | 16,656M |   3354 | 15,490M |   22 | 162,095K |   22 | 162,095K |
 back02 | 553 | @ 2024-02-18 08:41       | 159095 | 16,656M |   3354 | 15,491M |   26 | 183,088K |   28 | 199,096K |
 back02 | 560 | @ 2024-02-25 08:40       | 159095 | 16,656M |   3354 | 15,490M |   23 | 162,095K |   27 | 191,133K |
 back02 | 567 | @ 2024-03-03 08:41       | 159095 | 16,656M |   3348 | 15,490M |   50 | 283,285K |   52 | 295,510K |
 back02 | 572 | @ 2024-03-08 08:41       | 159095 | 16,656M |   3358 | 15,490M |   29 | 172,214K |   62 | 299,476K |
 back02 | 573 | @ 2024-03-09 08:41       | 159095 | 16,656M |   3359 | 15,490M |   31 | 179,871K |   31 | 179,871K |
 back02 | 574 | @ 2024-03-10 08:44       | 159095 | 16,656M |   3353 | 15,490M |   29 | 185,655K |   29 | 185,655K |
 back02 | 575 | @ 2024-03-11 08:41       | 159095 | 16,656M |   3352 | 15,490M |    0 |        0 |   32 | 201,976K |
 back02 | 578 | @ 2024-03-12 03:36       | 159095 | 16,656M |   3352 | 15,490M |    0 |        0 |    0 |        0 |
 back02 | 582 | @ 2024-03-13 08:41       | 159095 | 16,656M |   3351 | 15,490M |   23 | 161,725K |   23 | 161,725K |
 back02 | 583 | @ 2024-03-14 08:41       | 159095 | 16,656M |   3349 | 15,490M |   24 | 174,569K |   24 | 174,569K |
 back02 | 584 | @ 2024-03-15 08:41       | 159095 | 16,656M |   3356 | 15,490M |   29 | 169,383K |   29 | 169,383K |
 back02 | all |                          |        |         |   4925 | 23,925M | 4925 |  23,925M |      |          |
#4

I’m going to take the opportunity to start over by wiping my Duplicacy configuration and going from scratch using docker. I’ll let you know if I see this happen in the future. I do know I screwed up earlier and until recently had prune keeps accidentally in wrong order. I’ll wipe my storage including backblaze.

#5

Could this be the problem? When I used B2 with Duplicacy I had the lifecycle set to only keep the current version and I didn’t have bucket size problems.

#6

It has to be set to “keep all versions” for duplicacy to work correctly. You may get away without it if you only backup from one machine.

#7

I have started a fresh backup setting using docker now (Thanks @saspus for having that available).
It took a few days but now I have the two B2 buckets with a combined ~500GB instead of ~2.7TB. This is what I expect. Doing a quick browse of those files in B2, none of the files in B2 have revisions, unlike the buckets before I started from scratch. I believe this should be the correct.

Past: each file had multiple revisions in B2 and each file (chunk) had a 0 byte revision.
Current: each file/chunk so far only has the 1 copy (no revisions) in B2.

I have only the 1 backup going from my NAS. Local computers copy to the nas if I want data backed up, and Duplicacy handles backing up from there. I believe Duplicacy creates a chunk. If something changes (file’s data for instance), a new chunk is created (assuming it’s unique) and written to the drive (then copied to B2 in my case).

There should not ever be revisions of the file in B2. This is correct? This is why the keep prior versions can be set to older than 7 or more in the bucket settings?

Thanks

#8

Incorrect. Prune requires renaming/hiding/fossilizing chunks and relies on versioning on B2, due to the API limitations.

To support correct prune behaviour. Prune is a two step process – it fossilizes chunks before deleting them. On b2 fossilization creates a new version of the file.

Relevant discussion: Restore is very slow · Issue #362 · gilbertchen/duplicacy · GitHub

#9

Thanks for the clarification and the help. I’m considering this closed for now and will keep a closer eye on B2 for a bit to make sure it’s doing what I am expecting.

#10

I’m also struggling with this. I’ve been backing up unraid appdata files to B2 and it was getting too large, about 850GB. I adjusted the prune settings to not keep as far back and after the prune Duplicacy showed the size as 380GB but B2 is still showing 850GB. I’m trying more prunes with exhaustive and exclusive flags, but I’m not sure what to do here. Wondering if I disable pruning etc and just have B2 delete any file older than 20 days or so (I don’t need too much of this type of data). I’ll still need a way to delete the data in the bucket - maybe deleting the entire bucket and creating it again, as it should be smaller.

#11

Either b2 takes time to update the stats on the bucket, or you need to run prune again, and this is a two-step process.

Of this still does not help, maybe you have some orphaned chunks, run exhaustive prune.

This is guaranteed to corrupt your backup data store

It makes no sense to do so until you figure out what happened with this one. And once you do — there would be no need to start over.

The value of backup is in its long version history, so you can go back decades in time and restore rotted or corrupted file. If ion start over you lose all version history.

#12

I ran prune multiple times but just now it finished with exhaustive and has pruned down to 383GB. I take it that I should not have exhaustive always set though, right? I had to modify the scheduled task to run it but can remove that part. I ran it with “-keep 0:20 -a -exhaustive -exclusive -threads 10” in the options.

To be clear about these files, they are mostly compressed and locally I make weekly automated backups and keep the last 28 days of backups. I don’t need to go back too far with these items as I would know of an issue pretty soon and really old backup files aren’t going to be useful for this type of data. I just want to be able to have the last 28 days of backup available - to me it would seem that setting this to a short timeframe on b2 would work - revisions to the files don’t get made: new ones get made every week. Not sure why it has to be set to keep forever.

#13

Nice.

Normally you should not need to run with exhaustive — it’s significantly slower, and results in a lot of api calls — it needs so enumerate all files, to delete those not referenced by any revisions.

There should be no such orphaned chunks during normal operation but there are a few scenarios when they can stay behind.