Prune job running for over 24 hours

Support
#1

Hi there.

Having a strange issue I’ve never encountered before. Prune jobs tend to last minutes on my backup, but the last one has been going for over 24 hours.
The log looks like this:

    Options: [-log prune -storage NASBackup -keep 0:90 -keep 7:30 -keep 1:7 -a]
    2020-11-16 22:58:26.297 INFO STORAGE_SET Storage set to gcd://xxxx
    2020-11-16 22:58:29.220 INFO RETENTION_POLICY Keep no snapshots older than 90 days
    2020-11-16 22:58:29.220 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 30 day(s)
    2020-11-16 22:58:29.220 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 7 day(s)
    2020-11-16 22:58:33.581 INFO SNAPSHOT_DELETE Deleting snapshot 1 at revision 8
    2020-11-16 23:02:39.126 WARN CHUNK_FOSSILIZE Chunk 5cc520ee4b62865147e513bcac9a91f181dfc03591b1f178d79228ea91837ae1 is already a fossil

The log is then full of the ‘WARN CHUNK_FOSSILIZE’ message, which is repeated for different chunks every few seconds and is currently at 7000 lines. The progress bar in the WebUI is blank.

Activity is shown on the GCD ‘activity’ tab, which indicates that the application is moving files around. I’m a little bit confused by this, mostly due to the lack of indication as to what is going on. Is there a problem with the prune process or the backup overall?

Thanks

Edit: Should also mention that my backup dataset is about 16TB

#2

That can happen if revision 8 was supposed to be deleted by the previous prune run but for some reason it didn’t but still managed to mark all referenced chunks into fossils. Can you check the last prune log before this one to see if there was anything there?

You can also add a -threads 8 option to speed it up.

#3

The last run I had showed this:

Options: [-log prune -storage NASBackup -keep 0:90 -keep 7:30 -keep 1:7 -a]
2020-11-16 07:20:14.679 INFO STORAGE_SET Storage set to gcd://backup
2020-11-16 07:20:17.580 INFO RETENTION_POLICY Keep no snapshots older than 90 days
2020-11-16 07:20:17.580 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 30 day(s)
2020-11-16 07:20:17.580 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 7 day(s)
2020-11-16 07:20:21.497 INFO FOSSIL_COLLECT Fossil collection 1 found
2020-11-16 07:20:21.497 INFO FOSSIL_DELETABLE Fossils from collection 1 is eligible for deletion
2020-11-16 07:20:21.497 INFO PRUNE_NEWSNAPSHOT Snapshot 1 revision 18 was created after collection 1
2020-11-16 07:20:33.169 INFO SNAPSHOT_DELETE Deleting snapshot 1 at revision 8
2020-11-16 07:20:34.315 INFO CHUNK_DELETE The chunk 5b60faab025154fcddde8cffedd3f341278e7a2d1cea69a03d82cd830f3bc889 has been permanently removed
2020-11-16 07:20:35.413 INFO CHUNK_DELETE The chunk 3aa329f9ba3952e75281dea774ea226289ac155b910f646868c82011014d0763 has been permanently removed
2020-11-16 07:20:36.435 INFO CHUNK_DELETE The chunk 104ad9f445de19a438ba101d6376de5f7cbb74692242767f614d2614a096f195 has been permanently removed
2020-11-16 07:20:37.488 INFO CHUNK_DELETE The chunk b8347e83a48be31fa0e8831d3e23c0126e92abb7fc95f374bca38eb852d03447 has been permanently removed

This also ran like this for a few hours but nothing else was being added to the log. I will add the -threads option and see if that helps.

Cheers

#4

So I’ve left it running with the - threads 8 option and much more is getting done, the log is now almost 40mb! The entries are alternating between chunks being marked as fossils and fossils already flagged as fossils.
I saw that there is a command to check for rogue leftover chunks which may be left as a result of incomplete backups. How would I run this through the webui? I’m running in a docker so not too sure how I’d accomplish this. Feel like it might be an idea to run this after this prune has completed?

#5

So this line appeared in both logs.

I think when the second prune job of that day started, the first prune job was still running, at which time the revision 8 file hadn’t been deleted.

You can avoid this problem in the feature by running the prune job less frequently and together with -threads 8. There is really no need to run multiple prune jobs in one day.

#6

The only reason I ran it twice in one day was to add the -threads 8 option, otherwise it runs once a week following my backup.

#7

So the process completed thankfully! Cheers for your help.

In terms of running an -exhaustive check, how would I accomplish this through webui/docker? Duplicacy has the backup at 15.48TB but the drive is showing that 16.2TB is used, feels like there is some left over stuff somewhere.
Thanks

#8

[quote=“Fromnack, post:7, topic:4500”]
In terms of running an -exhaustive check, how would I accomplish this through webui/docker? [/quote]

If this is a one-time procedure, I’d create a new schedule, disable all days on that schedule (so it never runs automatically) and add the prune step. I’d ignore the options on the creation popup and jus save the step, then copy the options from the original prune job-step, click on the current options in the new step, paste the copied options in and add the “-exhaustive” option.

I always keep an empty job (that never runs automatically) for such maintenance steps. Once the maintenance step has completed to my satisfaction, I’ll just delete it from that job.