The "check" command is so slow that it can never finish. Is there anything that can be done to improve the speed of "check"?

[Droolio]

@saspus

Google Drive is designed for an entirely different purpose and does not perform well when [ab]used as a replacement for S2/GCS/Azure. There is some small wiggle room to improve batching, similar to --fast-list rclone implements, but it’s not magic, ultimately fetching metadata for millions of files via goodle drive API will still be taking ages.

To be clear, the reason @TyTro check is taking ages has little to do with Google. It can take hours even with local storage - with 6 million chunks, and 38k snapshots - the ListAllFiles() process can take eons.

This is too much overhead for Duplicacy to deal with in a linear manner, and is closer to an exponential curve. It’ll only get worse the longer OP omits to prune their hourly snapshots. This, to me, is the main purpose of prune - not just to reduce the total amount of data, but to make the number of snapshots more manageable - for operations like prune, check, and copy.

@TyTro Even a ‘light’ prune schedule of -keep 1:14 will help you out immensely. It’ll get rid of the hourly snapshots beyond 2 weeks, keeping only 1 a day forever. (Do you really need hourly snapshots going back that far? If so, I’d recommend git to manage your project, or use different retentions for different backup IDs.) Either way, would strongly suggest you consider doing this. Not only will it reduce the number of snapshots and chunks - if you run it regularly, the runtime from then on will remain relatively fixed, and linear.

Incidentally, Duplicacy could indeed support batch list (implemented by Rclone’s --fast-list) and probably speed up the listing process x20, which isn’t insignificant. Already supported by Google’s API, no more excuses to keep knocking it.

[TyTro]

Thanks!

@TyTro Even a ‘light’ prune schedule of -keep 1:14 will help you out immensely. It’ll get rid of the hourly snapshots beyond 2 weeks, keeping only 1 a day forever. (Do you really need hourly snapshots going back that far? If so, I’d recommend git to manage your project, or use different retentions for different backup IDs.) Either way, would strongly suggest you consider doing this. Not only will it reduce the number of snapshots and chunks - if you run it regularly, the runtime from then on will remain relatively fixed, and linear.

I don’t need hourly snapshots, I just set it to hourly thinking “more doesn’t hurt, I have a fast PC and infinite storage, so why not”. I didn’t know that Duplicacy scales badly with many revisions. Once per day or once per week would be totally fine for me.

I have changed my schedules to once per day now, that should help in the future then. I am also trying to run prune as you suggested, but unfortunately I’m not having much success with that, prune seems to take forever and doesn’t even show me any indicator how long it will take. I’ve created a new thread about the issue with prune:

Prune is very slow and doesn't have any progress bar, how long does it actually take and what is limiting the speed?

Based on the discussion in the other thread (The "check" command is so slow that it can never finish. Is there anything that can be done to improve the speed of "check"? - #10 by Droolio) where I was talking about check being slow, I am now trying to follow the suggestion of running prune to make check faster. But prune seems even slower than check, and while check at least has a progress bar, prune just shows me this, no matter how long I wait: [image] At first I tried running it with -all,…

[TyTro]

I’ve been using service account with a unreleased version of duplicacy that honors few additional flags in the token file, and described the process here: Duplicacy backup to Google Drive with Service Account | Trinkets, Odds, and Ends

@saspus I am trying to follow that nice guide you wrote now that my CLI version is 3.0.1. But I have an issue at step 18.

In there you write to enable the checkbox “Enable G Suite Domain-wide Delegation”:

But that checkbox does not exist for me. Probably because I am using Google Workspace, not G Suite. Do I need that checkbox, or can I just ignore that it does not exist?

This is how it looks like for me:

Edit:

I do think it works fine, even without being able to check that checkbox. Duplicacy seems to be able to successfully access the storage.

Now I just have the issue that for doing this:

delete the storage from duplicacy_web and then add it back with the same name.

I seem to need to know the encryption password, right? And it seems I don’t know that any more. Or more exactly, the password I remember it clearly being is not recognized as the correct password if I try entering it in the “Settings” of Duplicacy Web.

Actually, I’m a bit confused, why is there an “encryption password” setting in the settings of Duplicacy Web, that seems to globally apply to all storages, and then there’s another encryption password I need to enter when adding a new storage? Are those different passwords or the same one?

[saspus]

@TyTro

This is how it looks like for me:

There is a link about delegation, no? What’s written there? Maybe they moved the checkbox elsewhere.

Without enabling delegation service account won’t be able to access user’s account. It will only see its own 35GB account.

That was on google workspace btw, they still referred to g-suite in many places. I don’t have google workspace account anymore, so I can’t look it up for you, sorry.

@TyTro

I do think it works fine, even without being able to check that checkbox.

Ensure data is actually goes to the intended account and not service account itself.

@TyTro

encryption password” setting in the settings of Duplicacy Web,

This protects the web UI itself and encrypts other passwords. It is not storage encryption.

If you don’t have your storage encryption password anymore, and cannot extract it from a keychain — start over, as this backup is useless for you.

[TyTro]

Ensure data is actually goes to the intended account and not service account itself.

I have tested adding a new storage, and Duplicacy is correctly adding a new folder with the name I enter on the correct google drive in the correct path. So it seems to work correctly.

This protects the web UI itself and encrypts other passwords. It is not storage encryption.
If you don’t have your storage encryption password anymore, and cannot extract it from a keychain — start over, as this backup is useless for you.

I assume that means the password I remember is the password for the storage, and not the password for the web UI then. Can I reset the password for the Web UI somehow? I don’t remember at all that I would have created two different passwords when I set up Duplicacy.

So just to be sure I understand it correctly, there are three different passwords in total? Because in the Settings of the Web UI there’s also the “Administrative Password”, which is blank for me as I don’t use it. So there is the “Encryption Password” of the WebUI, then there’s the “Administrative Password” of the WebUI, and then there’s a per-storage “Encryption Password”?

Is there any way how I test if I still remember the password for the storage? Without deleting my current storage, as I’d like to keep it if I don’t actually remember the password.

[TyTro]

I managed to remember the password, so all is working now I think. I have deleted the storage, added it back with the same name and my custom .json file, and as far as I can see, everything is working now. So now I need to try running check or prune with 50 threads and hopefully it will be faster

Edit:

I’m running a check and a prune simultaneously with each 50 threads now, and I can see in the Google Drive metrics that Duplicacy is only doing ~4 API calls per second. The CPU usage of Duplicacy is ~9%, the RAM usage is 2.4 GB (of 128 GB available), and disk usage is 0.1 MB/s. check does not seem to be any faster now. So I’m still not sure what actually slows it down?

I would really like to either get my CPU to 100%, get my RAM to 100%, get my SSD to 100%, or get my Google Drive API Limit to 100%, which is 20,000 API calls per 100 seconds. So what exactly do I need to do to achieve this, maxing out at least one of these things?

Or is that impossible because the problem is actually that the slow check code in Duplicacy is single-threaded and doesn’t benefit from good hardware or a good cloud service?

I can see in task manager that the duplicacy_win_x64_3.0.1.exe is only having a total of 38 threads. Why? I am clearly running it with -threads 50, and I’m sure windows isn’t lying, so why is it not spawning at least 50 threads?

[saspus]

@TyTro

So now I need to try running check or prune with 50 threads and hopefully it will be faster

It won’t be faster and I would not exceed 4 threads. If you want crazy parallelism use google cloud storage or amazons3. Google drive was not designed for the usecase duplicacy users force on it and either due to design decisions or anti abuse limitations latency will be quite large.

This however does not matter at all. Backup is a background process, if prune takes 4 weeks — there is nowhere to hurry.

[TyTro]

It won’t be faster and I would not exceed 4 threads. If you want crazy parallelism use google cloud storage or amazons3. Google drive was not designed for the usecase duplicacy users force on it and either due to design decisions or anti abuse limitations latency will be quite large.

That does not make sense to me based on what I’m seeing. Why do you think that I am slowed down in any way by Google Drive? I am far from my API limits (20000 queries per 100 seconds). Latency should be completely irrelevant when stuff can happen in parallel. Even if asking “does a chunk exist” would take 1 minute to get a reply, it could still be fast to check if 5 million chunks exist by just doing 10000 things in parallel. Which is what the threads option is doing, right?

So I don’t think you’re right here, this does not seem to be related to any limit from Google Drive. I’m sure this would be exactly same fast on S3. Because check does not even use any network stuff at all for the slow part of the work. 0 API calls for hours. It’s only working on local data. The limit here seems to be the code that Duplicacy is running not being able to make full use of the available resources.

This however does not matter at all. Backup is a background process, if prune takes 4 weeks — there is nowhere to hurry.

It’s normal to reboot a PC at least one a day - so if something takes 4 weeks, and can’t resume after a restart, it can never finish for an average user, which is a big problem.

[saspus]

@TyTro

Because check does not even use any network stuff at all for the slow part of the work.

Are you passing any arguments to check? If not, it’s just a list request(s). They are very slow with google drive. Having thousands of files in one “folder” is also not a usecase google drive is optimized for. There is a way to fetch list in bulk (rclone uses it, see —fast-list option, it’s marginally faster) but duplicacy does not do that. From my experience, prune on a 2.5TB google drive duplicacy backup took about 14 days. Now I’m using tools and services designed for the purpose and don’t have these issues, and recommend everyone do the same. Few bucks savings a month comes at a huge cost of time investment, and is a false economy. My advice — don’t use *-drive services as backup targets.

@TyTro

It’s normal to reboot a PC at least one a day

It’s horrific and absolutely not normal. Why would you do that so often? If you are having issues causing you to restart — address those issues first. I, and everyone I know, only reboot on major OS updates. Even portables — sleep/wake/sleep/wake… I’ve just checked, uptime on my laptop is 45 days. Windows gaming rig — 81 days (I disabled updates)

[saspus]

Also, check your network latency — maybe bufferbloat play a role here amplifying the effect

[Droolio]

Did you manage to bring the number of revisions down to a sensible number?

[TyTro]

Are you passing any arguments to check?

I am running it with -d | -threads 50 -id X at the moment.

I can really tell you that the slow part of check is completely working on local cached data, it’s not doing anything with the network. So whether the storage is on S3 or Google Drive is completely irrelevant.
The first part of check does talk with the storage, that’s the “Listing chunks” part. That pretty much runs 5 times faster when running it with -threads 50 compared to running it with -threads 10, and I see on the Google Drive API monitor that its doing a lot of queries. So that’s perfect, but that is not where check spends the most time, especially with -threads 50 that is super quick. Check spends the most time in what comes after that, which is this what originally would have taken 35 hours for me:

INFO SNAPSHOT_CHECK All chunks referenced by snapshot X at revision Y exist
DEBUG CHUNK_CACHE Chunk X has been loaded from the snapshot cache

and that is doing 0 network requests. I see that both in task manager, and in my Google Drive API monitor. It’s all reading from the local cache. And it’s not affected by the threads option it seems. This just seems to be some badly optimized mostly single-threaded code, that’s the whole issue, and nothing else.

[TyTro]

I am making progress with that. Not finished yet, but should be in a few days. 3 IDs of my 4 IDs are finished running a prune with 1:14. The 4th one had a lot of stuff that needs to be deleted so that one I had to slowly move down from 1:600 to 1:570 to 1:540 etc, and that took each around 15 hours, so I could scale that quite well to finish before I shut down my PC each day. But that slow part of prune actually benefits from more network threads, so I can do that way quicker now with 50 threads instead of just 10 threads so I’ll have it done in 1-2 days.

[saspus]

@TyTro

I can really tell you that the slow part of check is completely working on local cached data, it’s not doing anything with the network.

This does not sound right. Duplicacy explicitly checks the presence of chunks on the storage. Cache is only used when the chunk is confirmed to exist on the target, and instead of downloading the file again – it will use the one already cached. The only chunks that are fetched during check are those needed to unpack the snapshot file. Those will be grabbed from cache, if exist. I’m surprised to see that it makes no network request, as the only job check is doing – checking for presence of chunk files on the target storage.

There would be no reason to check for them locally.

Try enabling profiler and look at the few stackshots during the lengthy operation, to find out were does it actually spend time at:

here: Deadlock during backup - #3 by gchen

[TyTro]

I’m surprised to see that it makes no network request, as the only job check is doing – checking for presence of chunk files on the target storage .

Isn’t that what’s happening in the first part of what check is doing? When it’s “Listing chunks”. This stuff:
Listing chunks/d2; 20487 items returned

That to me sounds like what’s checking if the chunks do exist on the storage. That is doing network requests, and finishing relatively quickly.

And the slow stuff that happens later is checking what chunks a specific revision references, and seeing if it’s in the list of chunks that were downloaded at the beginning. And that’s what’s all happening locally.

[saspus]

@TyTro

And the slow stuff that happens later is checking what chunks a specific revision references, and seeing if it’s in the list of chunks that were downloaded at the beginning. And that’s what’s all happening locally.

It does make sense. So, it’s seeking in files that takes time? is your cache folder on an HDD? Since CPU utilization is low – the only remaining thing is disk IO

From what I remember with having about 2TB on google drive, check was pretty fast, but prune would take weeks.

Do try the profiler, It’s really interesting to see what is it doing. There woudl be either wait on disk io, or somethign else strange.

[TyTro]

My cache folder is on my fastest SSD, a PCIe4 NVMe SSD. To be exact a Samsung 980 Pro 2 TB.

I’m quite sure that what’s limiting it is simply that the code that Duplicacy runs is badly optimized regarding CPU usage. For example, if I run 4 checks simultaneously with each a different ID, it’s using 4 times as much CPU and is 4 times faster compared to running 1 check that checks all IDs. Because then it’s 4 times as parallel as before, Duplicacy is simply not able to parallelize the local checks correctly. And it also feels like it’s using some inefficient data structure that makes it way slower than it would need to be. Like doing a lookup in an array instead of using a map/set. Something that’s O(n) instead of O(1) when it locally checks if a chunk exists in the list of downloaded chunks.

[TyTro]

@saspus here’s the profile:

goroutine 8149 [running]:
runtime/pprof.writeGoroutineStacks(0x1148b60, 0xc00229e540, 0x30, 0xe506c0)
	/usr/local/go/src/runtime/pprof/pprof.go:693 +0xa5
runtime/pprof.writeGoroutine(0x1148b60, 0xc00229e540, 0x2, 0xc000101000, 0x0)
	/usr/local/go/src/runtime/pprof/pprof.go:682 +0x4b
runtime/pprof.(*Profile).WriteTo(0x1840780, 0x1148b60, 0xc00229e540, 0x2, 0xc00229e540, 0xc0000b2ea0)
	/usr/local/go/src/runtime/pprof/pprof.go:331 +0x3f8
net/http/pprof.handler.ServeHTTP(0xc00053a071, 0x9, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/pprof/pprof.go:253 +0x385
net/http/pprof.Index(0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/pprof/pprof.go:371 +0x8e9
net/http.HandlerFunc.ServeHTTP(0xfb1458, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2050 +0x4b
net/http.(*ServeMux).ServeHTTP(0x1859060, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2429 +0x1b7
net/http.serverHandler.ServeHTTP(0xc0001cc2a0, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2868 +0xaa
net/http.(*conn).serve(0xc049e6e000, 0x1161fa0, 0xc0006dc000)
	/usr/local/go/src/net/http/server.go:1933 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2994 +0x3b8

goroutine 1 [runnable]:
reflect.Value.NumMethod(0xd87c20, 0xc048dc8210, 0x194, 0xc0693ed6a5)
	/usr/local/go/src/reflect/value.go:1337 +0xb7
encoding/json.(*decodeState).literalStore(0xc0372a0000, 0xc0693ed6a4, 0x42, 0x25d495c, 0xd87c20, 0xc048dc8210, 0x194, 0x1cb1600, 0x3d6cc85, 0xc0372a0028)
	/usr/local/go/src/encoding/json/decode.go:957 +0x24e8
encoding/json.(*decodeState).value(0xc0372a0000, 0xd87c20, 0xc048dc8210, 0x194, 0xd87c20, 0xc048dc8210)
	/usr/local/go/src/encoding/json/decode.go:384 +0x1fa
encoding/json.(*decodeState).array(0xc0372a0000, 0xcf70a0, 0xc000810090, 0x16, 0xc0372a0028, 0xc0005e615b)
	/usr/local/go/src/encoding/json/decode.go:558 +0x1b4
encoding/json.(*decodeState).value(0xc0372a0000, 0xcf70a0, 0xc000810090, 0x16, 0xc0005e6120, 0x152e2e)
	/usr/local/go/src/encoding/json/decode.go:360 +0x10c
encoding/json.(*decodeState).unmarshal(0xc0372a0000, 0xcf70a0, 0xc000810090, 0xc0372a0028, 0x0)
	/usr/local/go/src/encoding/json/decode.go:180 +0x1f4
encoding/json.Unmarshal(0xc06773c000, 0x3d6cc85, 0x4286000, 0xcf70a0, 0xc000810090, 0xc000391e60, 0xc0005e6270)
	/usr/local/go/src/encoding/json/decode.go:107 +0x11f
github.com/gilbertchen/duplicacy/src.(*Snapshot).LoadChunks(0xc00012a620, 0xc06773c000, 0x3d6cc85, 0x4286000, 0xc06773c000, 0x3d6cc85)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshot.go:402 +0x8f
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).GetSnapshotChunkHashes(0xc000164750, 0xc00012a620, 0x0, 0xc0005e67b0)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshotmanager.go:632 +0x485
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).CheckSnapshots(0xc000164750, 0x17f4140, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x64, ...)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshotmanager.go:876 +0xf6a
main.checkSnapshots(0xc0003e6480)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:987 +0x83a
github.com/gilbertchen/cli.Command.Run(0xf56493, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf7acc6, 0x20, 0x0, ...)
	/Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/command.go:160 +0x7a4
github.com/gilbertchen/cli.(*App).Run(0xc0006ebd40, 0xc000040100, 0xc, 0x10, 0x0, 0x0)
	/Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/app.go:179 +0x6c8
main.main()
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2231 +0x6aa5

goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000b0d70)
	/Users/gchen/zincbox/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0xd4
created by go.opencensus.io/stats/view.init.0
	/Users/gchen/zincbox/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x5e

goroutine 53 [syscall, 7 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:168 +0xaf
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x4b

goroutine 54 [chan receive, 7 minutes]:
main.main.func2(0xc00010d7a0)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2225 +0x67
created by main.main
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2224 +0x6a70

goroutine 55 [IO wait]:
internal/poll.runtime_pollWait(0x19ad1be4e88, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0000eee38, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0000eec98, 0xc000529ba8, 0xf0, 0xc0020e0000, 0xc000529bc8)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).acceptOne(0xc0000eec80, 0x74c, 0xc0020e0000, 0x2, 0x2, 0xc0000eec98, 0x29d93, 0x19aac88bc98, 0x0, 0x400)
	/usr/local/go/src/internal/poll/fd_windows.go:810 +0x9c
internal/poll.(*FD).Accept(0xc0000eec80, 0xc000529d58, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/internal/poll/fd_windows.go:844 +0x168
net.(*netFD).accept(0xc0000eec80, 0xb68f039861b14801, 0xc0000b00f0, 0xb68f039861b14823)
	/usr/local/go/src/net/fd_windows.go:139 +0x85
net.(*TCPListener).accept(0xc000810420, 0xc049e6e000, 0xc0000b0138, 0xfd14d)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x39
net.(*TCPListener).Accept(0xc000810420, 0xc000529e40, 0x18, 0xc000582480, 0x315bd8)
	/usr/local/go/src/net/tcpsock.go:261 +0x6b
net/http.(*Server).Serve(0xc0001cc2a0, 0x115b340, 0xc000810420, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2962 +0x29c
net/http.(*Server).ListenAndServe(0xc0001cc2a0, 0xc0001cc2a0, 0x0)
	/usr/local/go/src/net/http/server.go:2891 +0xc5
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3145
main.setGlobalOptions.func1(0xc0000420d0, 0xe)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:162 +0x65
created by main.setGlobalOptions
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:161 +0x28e

goroutine 72 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 73 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 41 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x19ad1be4cb8, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0001ae1b8, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0001ae018, 0xfb0da0, 0xc00059d601, 0x56625, 0xc0006c4900)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).Read(0xc0001ae000, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x2f6
net.(*netFD).Read(0xc0001ae000, 0xc002a4a000, 0x115eb, 0x115eb, 0x7, 0x50c, 0xc0001ae128)
	/usr/local/go/src/net/fd_posix.go:55 +0x56
net.(*conn).Read(0xc0006d61a8, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:183 +0x98
crypto/tls.(*atLeastReader).Read(0xc000811b60, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x185a100, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:776 +0x6a
bytes.(*Buffer).ReadFrom(0xc0004e9078, 0x1147000, 0xc000811b60, 0x1c22c, 0xdcc1a0, 0xf1b360)
	/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0004e8e00, 0x19ad2370008, 0xc0006d61a8, 0x5, 0xc0006d61a8, 0x1e8)
	/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0004e8e00, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:605 +0x12d
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0004e8e00, 0xc0005c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1276 +0x173
bufio.(*Reader).Read(0xc000593020, 0xc0005c4038, 0x9, 0x9, 0x11, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:227 +0x238
io.ReadAtLeast(0x1146de0, 0xc000593020, 0xc0005c4038, 0x9, 0x9, 0x9, 0xc00059dd10, 0x28b05c, 0xc0005ac0d8)
	/usr/local/go/src/io/io.go:328 +0x8e
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader(0xc0005c4038, 0x9, 0x9, 0x1146de0, 0xc000593020, 0x0, 0x0, 0xc0006c4ba0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1554 +0x8e
net/http.(*http2Framer).ReadFrame(0xc0005c4000, 0xc0043655d8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1812 +0xa8
net/http.(*http2clientConnReadLoop).run(0xc00059dfb0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:8603 +0xbc
net/http.(*http2ClientConn).readLoop(0xc0006c4a80)
	/usr/local/go/src/net/http/h2_bundle.go:8526 +0x6c
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7320 +0x705

goroutine 71 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x0)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 74 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 75 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 76 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 77 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x6)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 78 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x7)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 79 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x8)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 80 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x9)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 81 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xa)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 98 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xb)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 99 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xc)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 100 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xd)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 101 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xe)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 102 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xf)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 103 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x10)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 104 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x11)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 105 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x12)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 106 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x13)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 107 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x14)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 108 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x15)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 109 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x16)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 110 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x17)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 111 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x18)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 112 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x19)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 113 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 114 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 115 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 116 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 117 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 118 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 119 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x20)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 120 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x21)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 121 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x22)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 122 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x23)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 123 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x24)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 124 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x25)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 125 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x26)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 126 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x27)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 127 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x28)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 128 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x29)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 129 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 130 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 131 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 132 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 133 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 134 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 135 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x30)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 136 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x31)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 137 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x32)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 138 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x33)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 139 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x34)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 140 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x35)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 141 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x36)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 142 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x37)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 143 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x38)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 144 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x39)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 145 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 146 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

[TyTro]

goroutine 147 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 148 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 149 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 150 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 151 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x40)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 152 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x41)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 153 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x42)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 154 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x43)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 155 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x44)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 156 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x45)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 157 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x46)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 158 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x47)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 159 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x48)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 160 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x49)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 161 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 162 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 163 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 164 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 165 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 166 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 167 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x50)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 168 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x51)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 169 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x52)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 170 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x53)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 171 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x54)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 172 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x55)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 173 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x56)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 174 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x57)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 175 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x58)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 176 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x59)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 177 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 178 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 179 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 180 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 181 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 182 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 183 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x60)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 184 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x61)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 185 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x62)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 186 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x63)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 8280 [IO wait]:
internal/poll.runtime_pollWait(0x19ad1be4da0, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0001ae438, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0001ae298, 0xfb0da0, 0xc039a89e01, 0x1e749, 0xc000594000)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).Read(0xc0001ae280, 0xc0046a2041, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x2f6
net.(*netFD).Read(0xc0001ae280, 0xc0046a2041, 0x1, 0x1, 0xc0166b84e0, 0x0, 0xc0006dc098)
	/usr/local/go/src/net/fd_posix.go:55 +0x56
net.(*conn).Read(0xc0d2b1c008, 0xc0046a2041, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:183 +0x98
net/http.(*connReader).backgroundRead(0xc0046a2030)
	/usr/local/go/src/net/http/server.go:672 +0x5f
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:668 +0xdb

(didn’t all fit in one message, is above the 31000 character limit)

[Droolio]

@TyTro

I’m quite sure that what’s limiting it is simply that the code that Duplicacy runs is badly optimized regarding CPU usage.

Said this from the beginning, but until you get the number of revisions down to a reasonable level, this is totally expected. (I know you said it’ll take more time, yet this is what you have to do…)

I haven’t looked at the check and prune code, but I can tell you from experience, that if you have that many revisions - processing will be progressively slower - regardless of RAM, CPU, API, and bandwidth utilisation.

Under normal circumstances (regular pruning) you wouldn’t need to threadify it, but you’d have to specifically code with parallelisation in mind, assuming O(n) or less complexity. The -threads is just the number of downloads and not necessarily any attempt to parallelise the main algo. Some algorithms can be O(n2) complex and parallelisation wouldn’t have much use there. Doubt Duplicacy is as bad as that, but it does get non-linearly worse. Gchen will know more about this. Perhaps it could well be optimised, but the sheer number of revisions is your stumbling block here IMO and bumping -threads isn’t gonna help.

I know the above might be stating the obvious, but check and prune do a pretty good job when the revision count is normal, so I’d urge to evaluate again when you’ve reached there. If things markedly improve, then we have an answer, and perhaps there’s indeed scope to optimise the code. However, expecting it to scale well i.e. O(n) might not be possible - processing all revisions, from all snapshots (regardless of if you do them one by one; they all have to be evaluated), is necessary; particularly for prune, so you still want to keep the number of revisions in the hundreds, low thousands, but not the tens or hundreds of thousands.