Prune/Copy Groundhogs Day

Support
,
#1

I’ve noticed what appears to be an infinite prune/copy loop with my cloud storage.

First, the job prunes both the local and cloud storages.

Output from pruning the local storage:
(duplicacy prune -all -keep 0:180 -keep 30:90 -keep 7:30 -keep 1:3)

Storage set to /bkup2/duplicacy
Keep no snapshots older than 180 days
Keep 1 snapshot every 30 day(s) if older than 90 day(s)
Keep 1 snapshot every 7 day(s) if older than 30 day(s)
Keep 1 snapshot every 1 day(s) if older than 3 day(s)
Fossil collection 42 found
Ignore snapshot CHARYBDIS-David-Users whose last revision was created 7 days ago
Ignore snapshot J742845-W10-J742845-J742845 whose last revision was created 7 days ago
Fossils from collection 42 is eligible for deletion
Snapshot scylla-archive1 revision 336 was created after collection 42
Snapshot CIRCE-dgray-D revision 1845 was created after collection 42
Snapshot Calypso-davidgray-davidgray revision 288 was created after collection 42
Deleting snapshot CIRCE-dgray-D at revision 1670
Deleting snapshot CIRCE-dgray-D at revision 1833
Deleting snapshot CIRCE-dgray-D at revision 1834
Deleting snapshot CIRCE-dgray-D at revision 1835
Deleting snapshot CIRCE-dgray-D at revision 1836
Deleting snapshot CIRCE-dgray-D at revision 1837
Deleting snapshot Calypso-davidgray-davidgray at revision 124
Deleting snapshot Calypso-davidgray-davidgray at revision 276
Deleting snapshot Calypso-davidgray-davidgray at revision 277
Deleting snapshot Calypso-davidgray-davidgray at revision 278
Deleting snapshot Calypso-davidgray-davidgray at revision 279
Deleting snapshot Calypso-davidgray-davidgray at revision 280
Deleting snapshot J742845-W10-J742845-J742845 at revision 2981
Deleting snapshot scylla-archive1 at revision 306
Fossil collection 43 saved
The snapshot CIRCE-dgray-D at revision 1670 has been removed
The snapshot CIRCE-dgray-D at revision 1833 has been removed
The snapshot CIRCE-dgray-D at revision 1834 has been removed
The snapshot CIRCE-dgray-D at revision 1835 has been removed
The snapshot CIRCE-dgray-D at revision 1836 has been removed
The snapshot CIRCE-dgray-D at revision 1837 has been removed
The snapshot Calypso-davidgray-davidgray at revision 124 has been removed
The snapshot Calypso-davidgray-davidgray at revision 276 has been removed
The snapshot Calypso-davidgray-davidgray at revision 277 has been removed
The snapshot Calypso-davidgray-davidgray at revision 278 has been removed
The snapshot Calypso-davidgray-davidgray at revision 279 has been removed
The snapshot Calypso-davidgray-davidgray at revision 280 has been removed
The snapshot J742845-W10-J742845-J742845 at revision 2981 has been removed
The snapshot scylla-archive1 at revision 306 has been removed

Output from pruning the cloud storage. Note the bolded line where revision 4441 of J742845-W10-J742845-J742845 is removed:
(duplicacy prune -exclusive -all -keep 0:180 -keep 30:90 -keep 7:30 -keep 1:3 -storage wasabi)

Storage set to s3://us-east-1@s3.wasabisys.com/duplicacy-prod
Keep no snapshots older than 180 days
Keep 1 snapshot every 30 day(s) if older than 90 day(s)
Keep 1 snapshot every 7 day(s) if older than 30 day(s)
Keep 1 snapshot every 1 day(s) if older than 3 day(s)
Deleting snapshot CIRCE-dgray-D at revision 1670
Deleting snapshot CIRCE-dgray-D at revision 1833
Deleting snapshot CIRCE-dgray-D at revision 1834
Deleting snapshot CIRCE-dgray-D at revision 1835
Deleting snapshot CIRCE-dgray-D at revision 1836
Deleting snapshot CIRCE-dgray-D at revision 1837
Deleting snapshot Calypso-davidgray-davidgray at revision 124
Deleting snapshot Calypso-davidgray-davidgray at revision 276
Deleting snapshot Calypso-davidgray-davidgray at revision 277
Deleting snapshot Calypso-davidgray-davidgray at revision 278
Deleting snapshot Calypso-davidgray-davidgray at revision 279
Deleting snapshot Calypso-davidgray-davidgray at revision 280
Deleting snapshot J742845-W10-J742845-J742845 at revision 2981
Deleting snapshot J742845-W10-J742845-J742845 at revision 4441
Deleting snapshot scylla-archive1 at revision 306
The snapshot CIRCE-dgray-D at revision 1670 has been removed
The snapshot CIRCE-dgray-D at revision 1833 has been removed
The snapshot CIRCE-dgray-D at revision 1834 has been removed
The snapshot CIRCE-dgray-D at revision 1835 has been removed
The snapshot CIRCE-dgray-D at revision 1836 has been removed
The snapshot CIRCE-dgray-D at revision 1837 has been removed
The snapshot Calypso-davidgray-davidgray at revision 124 has been removed
The snapshot Calypso-davidgray-davidgray at revision 276 has been removed
The snapshot Calypso-davidgray-davidgray at revision 277 has been removed
The snapshot Calypso-davidgray-davidgray at revision 278 has been removed
The snapshot Calypso-davidgray-davidgray at revision 279 has been removed
The snapshot Calypso-davidgray-davidgray at revision 280 has been removed
The snapshot J742845-W10-J742845-J742845 at revision 2981 has been removed
**The snapshot J742845-W10-J742845-J742845 at revision 4441 has been removed**
The snapshot scylla-archive1 at revision 306 has been removed

Next a local-to-cloud copy is performed. And revision 4441 is copied back to the cloud even though it was just pruned:
(duplicacy copy -from default -to wasabi -threads 2)

[cut/pasted from end of copy job]

Chunk 9829b8d23d7dbe185353448aff86859f795407b7cdbbf27117e60f307792753e (215120/215254) copied to the destination
Copy complete, 215254 total chunks, 1836 chunks copied, 213418 skipped
Copied snapshot CIRCE-dgray-D at revision 1845
Copied snapshot CIRCE-dgray-D at revision 1846
Copied snapshot CIRCE-dgray-D at revision 1847
Copied snapshot CIRCE-dgray-D at revision 1848
Copied snapshot CIRCE-dgray-D at revision 1849
Copied snapshot CIRCE-dgray-D at revision 1850
Copied snapshot Calypso-davidgray-davidgray at revision 288
Copied snapshot Calypso-davidgray-davidgray at revision 289
Copied snapshot Calypso-davidgray-davidgray at revision 290
Copied snapshot Calypso-davidgray-davidgray at revision 291
Copied snapshot Calypso-davidgray-davidgray at revision 292
Copied snapshot Calypso-davidgray-davidgray at revision 293
**Copied snapshot J742845-W10-J742845-J742845 at revision 4441**
Copied snapshot scylla-archive1 at revision 336

I suspect this may have something to do with the “-exclusive” option on the cloud prune. What’s happening and how can I correct it?

#2

Do you still have revision 4441 in local storage?
If you still have it, it makes sense to copy it to wasabi, since that’s what you ask duplicacy to do.

Side question/request: if you aren’t upgraded to the latest version of duplicacy, you should do it as there is a fix to how prune selects what revisions to prune! (Ref: Duplicacy CLI 2.1.2 has been released)

#3

I see a related thread about this subject started by you: Impact of Prune on storage Copy operations

#4

Yep. Revision 4441 is in local storage and I believe that it should be eligible for deletion based the retention criteria. From what I can tell, the cloud prune is working correctly, but the local prune doesn’t want to remove rev 4441.

I’m using v2.1.2 for Linux. I believe that’s the latest, correct?

1 Like
#5

Yes. I believe that resulted in a tweak to the prune function.

1 Like
#6

How about you try to run the prune with debug info, to see why 4441 is ignored?

duplicacy -d -log prune -keepxxxxx

The output should be something like

2019-02-04 15:28:26.652 DEBUG SNAPSHOT_DELETE Snapshot testId at revision 79 to be deleted - older than x days, less than y days from previous

#7

Okay, I first performed another copy to cloud to make sure the snapshot revision was in both storages. The local storage prune completely ignored rev 4441. The cloud prune deleted it.

The key difference appears to be the use of fossil collection on the local storage while I prune the cloud storage with the -exclusive option. That shouldn’t make a difference in what is ultimately removed, but that’s all I can find. Attaching the log output for reference.

duplicacy_prune2_2019-02-04T135612.log.zip (61.7 KB)

#8

The -exclusive option made a difference here. Since 4441 is the last snapshot from its repository id, so it can’t be deleted because there may be an ongoing backup that relies on 4441. With the -exclusive option, you’re telling Duplicacy that no other backups are in progress so it will delete 4441.

4 Likes
#9

That makes sense, So to stop the endless prune/copy of the same snapshot, just remove the -exclusive option?

1 Like
#10

Right, without -exclusive it should keep 4441 in both places.

1 Like