Prune is not completing

Support
,
#1

Running web 1.1 and a daily scheduled backup runs backup, prune, check.
Prune job has been running for more than a day, looking like it will not complete.

Prune log:

Running prune command from /cache/localhost/all
Options: [-log prune -storage B2-Backup -keep 0:365 -keep 30:30 -keep 7:7 -keep 1:1 -a]
2019-10-31 07:25:31.067 INFO STORAGE_SET Storage set to b2://ptr727-Duplicacy-Server-1-Backup
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep no snapshots older than 365 days
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 30 day(s) if older than 30 day(s)
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 7 day(s)
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 1 day(s)
2019-10-31 07:25:33.299 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 13
2019-10-31 07:25:34.111 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 20
2019-10-31 07:25:34.849 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 26
2019-10-31 07:25:35.599 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 29
2019-10-31 07:25:36.314 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 30
2019-10-31 07:25:37.071 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 31
2019-10-31 07:25:37.812 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 32
2019-10-31 07:25:38.551 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 33
2019-10-31 07:25:39.283 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 34

Web log:

2019/10/30 18:34:44 Starting schedule Daily
2019/10/30 18:34:44 Created log file /logs/backup-20191030-183444.log
2019/10/30 18:34:44 Running /home/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_2.2.3 [-log backup -storage B2-Backup -stats]
2019/10/30 18:34:44 Set current working directory to /cache/localhost/0
2019/10/31 07:22:46 Created log file /logs/backup-20191031-072246.log
2019/10/31 07:22:46 Running /home/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_2.2.3 [-log backup -storage B2-Media -stats]
2019/10/31 07:22:46 Set current working directory to /cache/localhost/1
2019/10/31 07:25:31 Created log file /logs/prune-20191031-072531.log
2019/10/31 07:25:31 Running /home/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_2.2.3 [-log prune -storage B2-Backup -keep 0:365 -keep 30:30 -keep 7:7 -keep 1:1 -a]
2019/10/31 07:25:31 Set current working directory to /cache/localhost/all

Any ideas?

#2

It finally completed, took more than two days.

2019-10-31 07:25:39.283 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 34
2019-11-02 12:28:48.321 INFO FOSSIL_COLLECT Fossil collection 1 saved
#3

How big is your storage (e.g., number of chunks, total size)? That seems like the kind of performance I’d expect from Google Drive (probably even slow for a reasonable amount of data on Google Drive) or some other consumer product rather than more business-oriented object storage like B2.

#4

Depending on the number of chunks to delete, it may take a very long time. You can use the -threads option to specify multiple deleting threads (Prune command details)

#5

Prune:

Running prune command from /cache/localhost/all
Options: [-log prune -storage B2-Backup -keep 0:365 -keep 30:30 -keep 7:7 -keep 1:1 -a]
2019-10-31 07:25:31.067 INFO STORAGE_SET Storage set to b2://ptr727-Duplicacy-Server-1-Backup
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep no snapshots older than 365 days
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 30 day(s) if older than 30 day(s)
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 7 day(s) if older than 7 day(s)
2019-10-31 07:25:32.836 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 1 day(s)
2019-10-31 07:25:33.299 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 13
2019-10-31 07:25:34.111 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 20
2019-10-31 07:25:34.849 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 26
2019-10-31 07:25:35.599 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 29
2019-10-31 07:25:36.314 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 30
2019-10-31 07:25:37.071 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 31
2019-10-31 07:25:37.812 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 32
2019-10-31 07:25:38.551 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 33
2019-10-31 07:25:39.283 INFO SNAPSHOT_DELETE Deleting snapshot B2-Backup-Backup at revision 34
2019-11-02 12:28:48.321 INFO FOSSIL_COLLECT Fossil collection 1 saved
2019-11-02 12:28:48.383 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 13 has been removed
2019-11-02 12:28:48.543 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 20 has been removed
2019-11-02 12:28:48.734 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 26 has been removed
2019-11-02 12:28:48.878 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 29 has been removed
2019-11-02 12:28:49.790 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 30 has been removed
2019-11-02 12:28:49.935 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 31 has been removed
2019-11-02 12:28:50.087 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 32 has been removed
2019-11-02 12:28:50.222 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 33 has been removed
2019-11-02 12:28:50.357 INFO SNAPSHOT_DELETE The snapshot B2-Backup-Backup at revision 34 has been removed

Check:

Running check command from /cache/localhost/all
Options: [-log check -storage B2-Backup -a -tabular]
2019-11-02 12:29:29.650 INFO STORAGE_SET Storage set to b2://ptr727-Duplicacy-Server-1-Backup
2019-11-02 12:29:31.545 INFO SNAPSHOT_CHECK Listing all chunks
2019-11-02 12:30:41.468 INFO SNAPSHOT_CHECK 1 snapshots and 4 revisions
2019-11-02 12:30:41.523 INFO SNAPSHOT_CHECK Total chunk size is 2020G in 436020 chunks
2019-11-02 12:30:42.696 INFO SNAPSHOT_CHECK All chunks referenced by snapshot B2-Backup-Backup at revision 1 exist
2019-11-02 12:30:43.604 INFO SNAPSHOT_CHECK All chunks referenced by snapshot B2-Backup-Backup at revision 28 exist
2019-11-02 12:30:44.566 INFO SNAPSHOT_CHECK All chunks referenced by snapshot B2-Backup-Backup at revision 35 exist
2019-11-02 12:30:45.542 INFO SNAPSHOT_CHECK All chunks referenced by snapshot B2-Backup-Backup at revision 36 exist
2019-11-02 12:30:53.184 INFO SNAPSHOT_CHECK 
             snap | rev |                          |   files | bytes | chunks | bytes |   uniq |    bytes |    new |    bytes |
 B2-Backup-Backup |   1 | @ 2019-09-02 11:26 -hash | 1636627 | 1488G | 289677 | 1316G |  98319 | 483,877M | 289677 |    1316G |
 B2-Backup-Backup |  28 | @ 2019-10-07 00:00       | 1627102 | 1250G | 240174 | 1077G |  48665 | 239,501M |  48816 | 239,693M |
 B2-Backup-Backup |  35 | @ 2019-10-14 00:00       | 1627102 | 1250G | 240155 | 1078G |  48677 | 240,091M |  48677 | 240,091M |
 B2-Backup-Backup |  36 | @ 2019-10-30 18:35       | 1627102 | 1251G | 240328 | 1079G |  48850 | 241,147M |  48850 | 241,147M |
 B2-Backup-Backup | all |                          |         |       | 436020 | 2020G | 436020 |    2020G |        |          |
#6

If I’m reading that check output correctly, it looks like each revision is adding 48000+ unique chunks to the storage (?). If so, then I’m guessing that a lot of chunks had to be fossilized when removing 9 snapshot revisions.

I’d be curious how much adding more threads helps.

It might be worth bumping up the threads to something like 24 to chew through all of the unique chunks in each revision. FWIW, the prune command documentation doesn’t suggest using more than 30 threads.

1 Like
#7

I added -threads 8 to the prune schedule option, will see how long it takes next time.