Confusion Over Copy Progress

I’ve recently started using the Web Edition and I’m a little confused about the progress of my copy schedule.

I currently back up my data to a local drive, which works fine, and I’ve created a second schedule that copies that backup to B2. I’m still in the initial upload of that copy.

A few days ago, the progress bar had reached about 50000/816000. For reasons unrelated to Duplicacy, I had to reboot my machine at one point. When the copy resumed, the count started completely over. After a few more days, it’s now reached about 100000/816000, which is about 12% along. However, checking the size of my bucket on B2, it’s at about 1,530GB, which is almost 40% of the size of 3.8TB backup it’s copying.

So, what I’m wondering is:
a) What is the progress bar counting specifically?
b) Why did the count start completely over when it resumed after the reboot?
c) Why does the progress bar not seem to reflect the size of the upload completed?

1 Like

I can’t answer a) or c) as these are related to the web edition and I haven’t really paid attention to what the copy command does there (I use CLI + script for copy from local to cloud)…

But regarding b) - that is normal to start over. However, it does skip blocks that already exist on the destination.

When you start a copy, it first inventories all the chunks on the storage. Then it goes through your 816,000 chunks and skips the ones that already exist and uploads the missing ones.

What you may notice, is that it doesn’t necessarily process them in the same order as before, so the first 50,000 won’t all be skipped. Proportionally though, it’ll skip 6% of chunks and upload the rest, and the minimal amount of overhead in skipping chunks can be mitigated when you use more -threads.

1 Like

OK, I checked the log and it looks like a little over 25% of the chunks have been skipped so far, so that makes sense.

I have my number of threads set to 8, since that’s about where it seemed to top out my upload speed limit. Would I benefit overhead-wise if I increased that? Any suggestion as to how many threads would be ideal?

Thanks for the help!

I think only for the particular case of skipping many many many chunk files, it would be helpful to use even more threads.
While skipping the chunks almost no upload will be used, so having more threads will make the skipping faster.
On the other hand, when skipping is finished and uploading resumes (skipping and uploading may happen at the same time) the multiple threads won’t make anything faster (i also hope they won’t make everything slower).

1 Like

AHH :sweat_smile: I was going to raise a bug or support request regarding the copy to B2. It sure looks like the copy starts from the beginning each time (I too have had to restart my copy for various reasons)

Thanks for the clarification there, that’s great to know it isn’t really starting from the beginning each time - I’ve got about 2TB and about 10MBit/s upstream. Starting from scratch would turn out to be a deal breaker for me and duplicacy :sob: