2020-06-17 18:46:55.515 TRACE LIST_FILES Listing chunks/
2020-06-17 18:47:39.364 TRACE SNAPSHOT_LIST_IDS Listing all snapshot ids
2020-06-17 19:18:40.589 INFO SNAPSHOT_CHECK 13 snapshots and 16301 revisions
2020-06-17 19:18:40.592 INFO SNAPSHOT_CHECK Total chunk size is 427,554M in 105613 chunks
Maybe it’s the number of revisions.
But that would only affects the second stage and listing still takes 31 minutes.
Yea, listing snapshots, not chunks. The optimisation above, for Google Drive, S3 et al, is for listing chunks - which was very time-consuming.
As far as I can see, that doesn’t appear to be your problem here, as yours is done in ~44 seconds, before going onto listing snapshots - that seems to be the time-consuming part, which is rather weird.
Between these 2 lines, Duplicacy loaded all revisions one by one and made sure that every referenced chunk was in the list of existing chunks. Because you have such a large number of revisions (16302 of them), 31 minutes is very reasonable.
Fair enough.
The truth is I have never pruned or deleted any snapshots. 
I think I can delete lots snapshots
So it’s time to prune a bit.
Thank you all.
My problem is that the prune command is a bit limited in the web-ui
I have posted a new topic with a web-ui UX request for custom cli commands.
Actually it may seem limited, but it’s really not. You can just ignore the proposed retention options and paste the full options into the “Options” field.
And when editing a prune job, you anyway have only the option field to deal with.
Interesting, so it would be something like:
-id test-id -keep 30:360 -keep 7:180 -keep 1:30 -threads 8
again -does the parameter id work for example?.
Are there any restrictions on cli options?
If so we should really change the descriptions or the web ui guide.
Will try. But I didn’t wanna guess with prune. 
Thank you
Yes, something like that. The options are just passed to the CLI, so I don’t see why -id wouldn’t work.
Well, you can always use the -dry-run option. Assuming that one works, of course 
Exactly, that is precisely the question. 
Yes, -dry-run will work as an option.
I’ll release a new CLI version next week.
I can confirm this latest version fixes my original issue. Listing all chunks now runs in 15 mins on 15.5TB of data on GCD with 10 threads. Thank you!
2020-07-06 01:00:01.698 INFO STORAGE_SET Storage set to gcd://backup
2020-07-06 01:00:04.038 INFO SNAPSHOT_CHECK Listing all chunks
2020-07-06 01:15:08.902 INFO SNAPSHOT_CHECK 6 snapshots and 114 revisions
2020-07-06 01:15:09.034 INFO SNAPSHOT_CHECK Total chunk size is 15548G in 3243443 chunks
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.