The "check" command is so slow that it can never finish. Is there anything that can be done to improve the speed of "check"?

[TyTro]

Are you passing any arguments to check?

I am running it with -d | -threads 50 -id X at the moment.

I can really tell you that the slow part of check is completely working on local cached data, it’s not doing anything with the network. So whether the storage is on S3 or Google Drive is completely irrelevant.
The first part of check does talk with the storage, that’s the “Listing chunks” part. That pretty much runs 5 times faster when running it with -threads 50 compared to running it with -threads 10, and I see on the Google Drive API monitor that its doing a lot of queries. So that’s perfect, but that is not where check spends the most time, especially with -threads 50 that is super quick. Check spends the most time in what comes after that, which is this what originally would have taken 35 hours for me:

INFO SNAPSHOT_CHECK All chunks referenced by snapshot X at revision Y exist
DEBUG CHUNK_CACHE Chunk X has been loaded from the snapshot cache

and that is doing 0 network requests. I see that both in task manager, and in my Google Drive API monitor. It’s all reading from the local cache. And it’s not affected by the threads option it seems. This just seems to be some badly optimized mostly single-threaded code, that’s the whole issue, and nothing else.

[TyTro]

I am making progress with that. Not finished yet, but should be in a few days. 3 IDs of my 4 IDs are finished running a prune with 1:14. The 4th one had a lot of stuff that needs to be deleted so that one I had to slowly move down from 1:600 to 1:570 to 1:540 etc, and that took each around 15 hours, so I could scale that quite well to finish before I shut down my PC each day. But that slow part of prune actually benefits from more network threads, so I can do that way quicker now with 50 threads instead of just 10 threads so I’ll have it done in 1-2 days.

[saspus]

@TyTro

I can really tell you that the slow part of check is completely working on local cached data, it’s not doing anything with the network.

This does not sound right. Duplicacy explicitly checks the presence of chunks on the storage. Cache is only used when the chunk is confirmed to exist on the target, and instead of downloading the file again – it will use the one already cached. The only chunks that are fetched during check are those needed to unpack the snapshot file. Those will be grabbed from cache, if exist. I’m surprised to see that it makes no network request, as the only job check is doing – checking for presence of chunk files on the target storage.

There would be no reason to check for them locally.

Try enabling profiler and look at the few stackshots during the lengthy operation, to find out were does it actually spend time at:

here: Deadlock during backup - #3 by gchen

[TyTro]

I’m surprised to see that it makes no network request, as the only job check is doing – checking for presence of chunk files on the target storage .

Isn’t that what’s happening in the first part of what check is doing? When it’s “Listing chunks”. This stuff:
Listing chunks/d2; 20487 items returned

That to me sounds like what’s checking if the chunks do exist on the storage. That is doing network requests, and finishing relatively quickly.

And the slow stuff that happens later is checking what chunks a specific revision references, and seeing if it’s in the list of chunks that were downloaded at the beginning. And that’s what’s all happening locally.

[saspus]

@TyTro

And the slow stuff that happens later is checking what chunks a specific revision references, and seeing if it’s in the list of chunks that were downloaded at the beginning. And that’s what’s all happening locally.

It does make sense. So, it’s seeking in files that takes time? is your cache folder on an HDD? Since CPU utilization is low – the only remaining thing is disk IO

From what I remember with having about 2TB on google drive, check was pretty fast, but prune would take weeks.

Do try the profiler, It’s really interesting to see what is it doing. There woudl be either wait on disk io, or somethign else strange.

[TyTro]

My cache folder is on my fastest SSD, a PCIe4 NVMe SSD. To be exact a Samsung 980 Pro 2 TB.

I’m quite sure that what’s limiting it is simply that the code that Duplicacy runs is badly optimized regarding CPU usage. For example, if I run 4 checks simultaneously with each a different ID, it’s using 4 times as much CPU and is 4 times faster compared to running 1 check that checks all IDs. Because then it’s 4 times as parallel as before, Duplicacy is simply not able to parallelize the local checks correctly. And it also feels like it’s using some inefficient data structure that makes it way slower than it would need to be. Like doing a lookup in an array instead of using a map/set. Something that’s O(n) instead of O(1) when it locally checks if a chunk exists in the list of downloaded chunks.

[TyTro]

@saspus here’s the profile:

goroutine 8149 [running]:
runtime/pprof.writeGoroutineStacks(0x1148b60, 0xc00229e540, 0x30, 0xe506c0)
	/usr/local/go/src/runtime/pprof/pprof.go:693 +0xa5
runtime/pprof.writeGoroutine(0x1148b60, 0xc00229e540, 0x2, 0xc000101000, 0x0)
	/usr/local/go/src/runtime/pprof/pprof.go:682 +0x4b
runtime/pprof.(*Profile).WriteTo(0x1840780, 0x1148b60, 0xc00229e540, 0x2, 0xc00229e540, 0xc0000b2ea0)
	/usr/local/go/src/runtime/pprof/pprof.go:331 +0x3f8
net/http/pprof.handler.ServeHTTP(0xc00053a071, 0x9, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/pprof/pprof.go:253 +0x385
net/http/pprof.Index(0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/pprof/pprof.go:371 +0x8e9
net/http.HandlerFunc.ServeHTTP(0xfb1458, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2050 +0x4b
net/http.(*ServeMux).ServeHTTP(0x1859060, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2429 +0x1b7
net/http.serverHandler.ServeHTTP(0xc0001cc2a0, 0x115b520, 0xc00229e540, 0xc000040200)
	/usr/local/go/src/net/http/server.go:2868 +0xaa
net/http.(*conn).serve(0xc049e6e000, 0x1161fa0, 0xc0006dc000)
	/usr/local/go/src/net/http/server.go:1933 +0x8cd
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2994 +0x3b8

goroutine 1 [runnable]:
reflect.Value.NumMethod(0xd87c20, 0xc048dc8210, 0x194, 0xc0693ed6a5)
	/usr/local/go/src/reflect/value.go:1337 +0xb7
encoding/json.(*decodeState).literalStore(0xc0372a0000, 0xc0693ed6a4, 0x42, 0x25d495c, 0xd87c20, 0xc048dc8210, 0x194, 0x1cb1600, 0x3d6cc85, 0xc0372a0028)
	/usr/local/go/src/encoding/json/decode.go:957 +0x24e8
encoding/json.(*decodeState).value(0xc0372a0000, 0xd87c20, 0xc048dc8210, 0x194, 0xd87c20, 0xc048dc8210)
	/usr/local/go/src/encoding/json/decode.go:384 +0x1fa
encoding/json.(*decodeState).array(0xc0372a0000, 0xcf70a0, 0xc000810090, 0x16, 0xc0372a0028, 0xc0005e615b)
	/usr/local/go/src/encoding/json/decode.go:558 +0x1b4
encoding/json.(*decodeState).value(0xc0372a0000, 0xcf70a0, 0xc000810090, 0x16, 0xc0005e6120, 0x152e2e)
	/usr/local/go/src/encoding/json/decode.go:360 +0x10c
encoding/json.(*decodeState).unmarshal(0xc0372a0000, 0xcf70a0, 0xc000810090, 0xc0372a0028, 0x0)
	/usr/local/go/src/encoding/json/decode.go:180 +0x1f4
encoding/json.Unmarshal(0xc06773c000, 0x3d6cc85, 0x4286000, 0xcf70a0, 0xc000810090, 0xc000391e60, 0xc0005e6270)
	/usr/local/go/src/encoding/json/decode.go:107 +0x11f
github.com/gilbertchen/duplicacy/src.(*Snapshot).LoadChunks(0xc00012a620, 0xc06773c000, 0x3d6cc85, 0x4286000, 0xc06773c000, 0x3d6cc85)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshot.go:402 +0x8f
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).GetSnapshotChunkHashes(0xc000164750, 0xc00012a620, 0x0, 0xc0005e67b0)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshotmanager.go:632 +0x485
github.com/gilbertchen/duplicacy/src.(*SnapshotManager).CheckSnapshots(0xc000164750, 0x17f4140, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x64, ...)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_snapshotmanager.go:876 +0xf6a
main.checkSnapshots(0xc0003e6480)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:987 +0x83a
github.com/gilbertchen/cli.Command.Run(0xf56493, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf7acc6, 0x20, 0x0, ...)
	/Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/command.go:160 +0x7a4
github.com/gilbertchen/cli.(*App).Run(0xc0006ebd40, 0xc000040100, 0xc, 0x10, 0x0, 0x0)
	/Users/gchen/zincbox/go/pkg/mod/github.com/gilbertchen/cli@v1.2.1-0.20160223210219-1de0a1836ce9/app.go:179 +0x6c8
main.main()
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2231 +0x6aa5

goroutine 6 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0000b0d70)
	/Users/gchen/zincbox/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:154 +0xd4
created by go.opencensus.io/stats/view.init.0
	/Users/gchen/zincbox/go/pkg/mod/go.opencensus.io@v0.22.3/stats/view/worker.go:32 +0x5e

goroutine 53 [syscall, 7 minutes]:
os/signal.signal_recv(0x0)
	/usr/local/go/src/runtime/sigqueue.go:168 +0xaf
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.Notify.func1.1
	/usr/local/go/src/os/signal/signal.go:151 +0x4b

goroutine 54 [chan receive, 7 minutes]:
main.main.func2(0xc00010d7a0)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2225 +0x67
created by main.main
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:2224 +0x6a70

goroutine 55 [IO wait]:
internal/poll.runtime_pollWait(0x19ad1be4e88, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0000eee38, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0000eec98, 0xc000529ba8, 0xf0, 0xc0020e0000, 0xc000529bc8)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).acceptOne(0xc0000eec80, 0x74c, 0xc0020e0000, 0x2, 0x2, 0xc0000eec98, 0x29d93, 0x19aac88bc98, 0x0, 0x400)
	/usr/local/go/src/internal/poll/fd_windows.go:810 +0x9c
internal/poll.(*FD).Accept(0xc0000eec80, 0xc000529d58, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/internal/poll/fd_windows.go:844 +0x168
net.(*netFD).accept(0xc0000eec80, 0xb68f039861b14801, 0xc0000b00f0, 0xb68f039861b14823)
	/usr/local/go/src/net/fd_windows.go:139 +0x85
net.(*TCPListener).accept(0xc000810420, 0xc049e6e000, 0xc0000b0138, 0xfd14d)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x39
net.(*TCPListener).Accept(0xc000810420, 0xc000529e40, 0x18, 0xc000582480, 0x315bd8)
	/usr/local/go/src/net/tcpsock.go:261 +0x6b
net/http.(*Server).Serve(0xc0001cc2a0, 0x115b340, 0xc000810420, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2962 +0x29c
net/http.(*Server).ListenAndServe(0xc0001cc2a0, 0xc0001cc2a0, 0x0)
	/usr/local/go/src/net/http/server.go:2891 +0xc5
net/http.ListenAndServe(...)
	/usr/local/go/src/net/http/server.go:3145
main.setGlobalOptions.func1(0xc0000420d0, 0xe)
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:162 +0x65
created by main.setGlobalOptions
	/Users/gchen/zincbox/duplicacy/duplicacy/duplicacy_main.go:161 +0x28e

goroutine 72 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 73 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 41 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x19ad1be4cb8, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0001ae1b8, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0001ae018, 0xfb0da0, 0xc00059d601, 0x56625, 0xc0006c4900)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).Read(0xc0001ae000, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x2f6
net.(*netFD).Read(0xc0001ae000, 0xc002a4a000, 0x115eb, 0x115eb, 0x7, 0x50c, 0xc0001ae128)
	/usr/local/go/src/net/fd_posix.go:55 +0x56
net.(*conn).Read(0xc0006d61a8, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:183 +0x98
crypto/tls.(*atLeastReader).Read(0xc000811b60, 0xc002a4a000, 0x115eb, 0x115eb, 0x0, 0x185a100, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:776 +0x6a
bytes.(*Buffer).ReadFrom(0xc0004e9078, 0x1147000, 0xc000811b60, 0x1c22c, 0xdcc1a0, 0xf1b360)
	/usr/local/go/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0004e8e00, 0x19ad2370008, 0xc0006d61a8, 0x5, 0xc0006d61a8, 0x1e8)
	/usr/local/go/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0004e8e00, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:605 +0x12d
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0004e8e00, 0xc0005c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:1276 +0x173
bufio.(*Reader).Read(0xc000593020, 0xc0005c4038, 0x9, 0x9, 0x11, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:227 +0x238
io.ReadAtLeast(0x1146de0, 0xc000593020, 0xc0005c4038, 0x9, 0x9, 0x9, 0xc00059dd10, 0x28b05c, 0xc0005ac0d8)
	/usr/local/go/src/io/io.go:328 +0x8e
io.ReadFull(...)
	/usr/local/go/src/io/io.go:347
net/http.http2readFrameHeader(0xc0005c4038, 0x9, 0x9, 0x1146de0, 0xc000593020, 0x0, 0x0, 0xc0006c4ba0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1554 +0x8e
net/http.(*http2Framer).ReadFrame(0xc0005c4000, 0xc0043655d8, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:1812 +0xa8
net/http.(*http2clientConnReadLoop).run(0xc00059dfb0, 0x0, 0x0)
	/usr/local/go/src/net/http/h2_bundle.go:8603 +0xbc
net/http.(*http2ClientConn).readLoop(0xc0006c4a80)
	/usr/local/go/src/net/http/h2_bundle.go:8526 +0x6c
created by net/http.(*http2Transport).newClientConn
	/usr/local/go/src/net/http/h2_bundle.go:7320 +0x705

goroutine 71 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x0)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 74 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 75 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 76 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 77 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x6)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 78 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x7)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 79 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x8)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 80 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x9)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 81 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xa)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 98 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xb)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 99 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xc)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 100 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xd)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 101 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xe)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 102 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0xf)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 103 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x10)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 104 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x11)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 105 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x12)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 106 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x13)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 107 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x14)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 108 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x15)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 109 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x16)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 110 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x17)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 111 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x18)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 112 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x19)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 113 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 114 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 115 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 116 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 117 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 118 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x1f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 119 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x20)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 120 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x21)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 121 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x22)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 122 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x23)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 123 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x24)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 124 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x25)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 125 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x26)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 126 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x27)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 127 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x28)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 128 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x29)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 129 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 130 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 131 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 132 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 133 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 134 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x2f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 135 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x30)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 136 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x31)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 137 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x32)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 138 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x33)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 139 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x34)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 140 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x35)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 141 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x36)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 142 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x37)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 143 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x38)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 144 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x39)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 145 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 146 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

[TyTro]

goroutine 147 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 148 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 149 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 150 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x3f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 151 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x40)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 152 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x41)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 153 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x42)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 154 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x43)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 155 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x44)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 156 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x45)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 157 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x46)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 158 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x47)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 159 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x48)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 160 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x49)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 161 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 162 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 163 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 164 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 165 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 166 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x4f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 167 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x50)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 168 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x51)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 169 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x52)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 170 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x53)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 171 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x54)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 172 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x55)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 173 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x56)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 174 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x57)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 175 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x58)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 176 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x59)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 177 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5a)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 178 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5b)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 179 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5c)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 180 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5d)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 181 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5e)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 182 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x5f)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 183 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x60)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 184 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x61)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 185 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x62)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 186 [select]:
github.com/gilbertchen/duplicacy/src.CreateChunkOperator.func1(0xc0001cf4a0, 0x63)
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:86 +0x191
created by github.com/gilbertchen/duplicacy/src.CreateChunkOperator
	/Users/gchen/zincbox/duplicacy/src/duplicacy_chunkoperator.go:83 +0x1ab

goroutine 8280 [IO wait]:
internal/poll.runtime_pollWait(0x19ad1be4da0, 0x72, 0x114ab20)
	/usr/local/go/src/runtime/netpoll.go:227 +0x65
internal/poll.(*pollDesc).wait(0xc0001ae438, 0x72, 0x17ce700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x4c
internal/poll.execIO(0xc0001ae298, 0xfb0da0, 0xc039a89e01, 0x1e749, 0xc000594000)
	/usr/local/go/src/internal/poll/fd_windows.go:175 +0x113
internal/poll.(*FD).Read(0xc0001ae280, 0xc0046a2041, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_windows.go:441 +0x2f6
net.(*netFD).Read(0xc0001ae280, 0xc0046a2041, 0x1, 0x1, 0xc0166b84e0, 0x0, 0xc0006dc098)
	/usr/local/go/src/net/fd_posix.go:55 +0x56
net.(*conn).Read(0xc0d2b1c008, 0xc0046a2041, 0x1, 0x1, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:183 +0x98
net/http.(*connReader).backgroundRead(0xc0046a2030)
	/usr/local/go/src/net/http/server.go:672 +0x5f
created by net/http.(*connReader).startBackgroundRead
	/usr/local/go/src/net/http/server.go:668 +0xdb

(didn’t all fit in one message, is above the 31000 character limit)

[Droolio]

@TyTro

I’m quite sure that what’s limiting it is simply that the code that Duplicacy runs is badly optimized regarding CPU usage.

Said this from the beginning, but until you get the number of revisions down to a reasonable level, this is totally expected. (I know you said it’ll take more time, yet this is what you have to do…)

I haven’t looked at the check and prune code, but I can tell you from experience, that if you have that many revisions - processing will be progressively slower - regardless of RAM, CPU, API, and bandwidth utilisation.

Under normal circumstances (regular pruning) you wouldn’t need to threadify it, but you’d have to specifically code with parallelisation in mind, assuming O(n) or less complexity. The -threads is just the number of downloads and not necessarily any attempt to parallelise the main algo. Some algorithms can be O(n2) complex and parallelisation wouldn’t have much use there. Doubt Duplicacy is as bad as that, but it does get non-linearly worse. Gchen will know more about this. Perhaps it could well be optimised, but the sheer number of revisions is your stumbling block here IMO and bumping -threads isn’t gonna help.

I know the above might be stating the obvious, but check and prune do a pretty good job when the revision count is normal, so I’d urge to evaluate again when you’ve reached there. If things markedly improve, then we have an answer, and perhaps there’s indeed scope to optimise the code. However, expecting it to scale well i.e. O(n) might not be possible - processing all revisions, from all snapshots (regardless of if you do them one by one; they all have to be evaluated), is necessary; particularly for prune, so you still want to keep the number of revisions in the hundreds, low thousands, but not the tens or hundreds of thousands.

[TyTro]

Since three of my four IDs have finished pruning, I can run check on them already and see how the speed is in a “normal” scenario. That is what I have been testing check with recently. It’s of course way faster than before, check finishes in a few hours, so no actual “problem” any more. But It’s 2 minutes of listing chunks from the network, and then a few hours of local work. And I’m just very sure by now that it’s totally unnecessary that it takes hours.

I’m a C/C++ programmer myself, primarily working on writing fast well optimized code. I don’t do network stuff, so I can’t judge that, but I can judge that some completely locally running code that just checks if a list of integers is contained in another list of integers should not take 35 hours, even in an extreme case of many revisons. If it takes so long, it’s simply badly written code.

Bascally, the code is doing this I assume, some very rough guesses:

1. There’s a list of all chunks. So let’s say a list of 10 million chunks. How many bytes is a chunk hash? I don’t know, my first guess would be it’s a 256 bit hash. So 32 bytes. The whole list of chunks is 320 MB then. Very manageable, fits nicely in RAM. Now we don’t want to store this list as an array of course, that would be a bad data structure, we want to store it as a set. So there’s some memory overhead with that, so let’s say it’s 1 GB then. Still fits nicely in RAM.

2. There’s a list of revisions. Every revision references a certain number of chunks. Let’s say every revision references 5 million chunks. Which would mean this list of chunks a revision references is 160 MB. And let’s say there are a total of 50000 revisions, to have an extreme case. That’s in total 8 TB of data, so I guess I don’t quite understand how this works. The code is never doing more than 0.1 MB/s disk read, so I guess it’s stored more efficiently somehow.

3. So what the code is doing is having to check 50000 times if 5 million chunks are contained in a list of 10 million chunks. Definitely a lot of checks.

4. So first the code downloads the list of existing chunks from the storage. This is very fast with threads- 50, so doesn’t really matter how it’s doing this.

5. Then it needs to write the list of 10 million hashes into a set. That’s also very fast.

6. Then it starts looking at revisions. So for every revision, it needs to load a list of 5 million entries (160 MB of data) into RAM, loop over this list, and run a AllChunksList.contains() check for every entry. For every revision, that is 5 million lookups. Based on my experience, I know that a lookup in an efficient set is roughly ~20 x86 instructions (I spent a lot of time profiling set performance in my own C/C++ code). But our 1 GB set does not fit in the CPU cache. So this will be limited by how fast the RAM is. So let’s assume 200 clock cycles per lookup to be very conservative (in practice it will be faster). So our 5 million lookups take 1 billion clock cycles. That’s equivalent to 0.25 seconds of time on one CPU thread on my CPU.

7. So we run code that takes 0.25 seconds per revision. Now we run that 50000 times. That’s in total 12500 seconds, or 208 minutes, or 3.4 hours. That’s with one thread. But we can multithreaded this code nicely. So on my CPU with 32 threads, let’s divide this time by 32. Then we’re at 6.5 minutes.

So 6.5 minutes. In practice, the above scenario probably takes at least 50 hours or so with the current code in Duplicacy. 50 hours vs 6.5 minutes.

[Droolio]

All that sounds great in theory but it’s based on a lot of assumptions IMO.

The reality is, your one remaining ID still has thousands of revisions to process versus the 30k+ you had in the beginning. Have those chunks merely been fossilised or actually deleted (-exclusive / -collect-only)?

TBH you’d probably have saved a lot of time just doing all IDs and starting with a higher -keep 1:X threshold (even if it appeared to run quicker otherwise), because Duplicacy still has to load all those IDs into memory anyway, plus the chunk list. Perhaps it doesn’t matter which way around it is? Though if those chunks haven’t been deleted yet, it’s likely keeping the runtime high.

Either way, had you been pruning from the beginning, you’d probably never have noticed any issues. That was my main point.

Besides the fact that multithreading the code has to be concious design implementation - the compiler isn’t gonna multithread it for you - parellelisation won’t reduce the order of complexity. Clearly it can’t deal with a lot of revisions, but is it the algo at fault or inefficient data structures, or maybe just golang itself (there’s been many memory performance bumps over the years, which Duplicacy might not be taking advantage of).

Anyway, you’re very likely right there’s numerous optimisations that can be made, but under normal circumstances none of it would be necessary - with a sane amount of revisions. The degree of optimisation required to utilised 100% bandwidth and CPU would like be overkill for most use cases, as bandwidth bottlenecks etc. will hit first. For other reasons, too, most people should want to keep revisions low - and generally with backups you have a staggered retention period (hourly vs daily vs weekly vs monthly).

Not saying it shouldn’t be looked at btw. The source code is open to all. Personally, I’d like to see ListAllFiles() improved for both local drives and Google Drive, without having to use unhealthy -thread count (which is detrimental for local and very probably GCD too).

[sevimo]

@TyTro

I’m a C/C++ programmer myself, primarily working on writing fast well optimized code. I don’t do network stuff, so I can’t judge that, but I can judge that some completely locally running code that just checks if a list of integers is contained in another list of integers should not take 35 hours, even in an extreme case of many revisons. If it takes so long, it’s simply badly written code.

If that’s the case, I’d urge you to take a look at the actual code, it’s on github. Even if you’ve never seen Go in your life, it’s not hard to understand what’s going on if you have a lot of experience with C/C++. This way you don’t need to speculate on how it might be done, and see how it is actually done. More so, it would be great if you can recommend some optimizations, perhaps even submit a PR.

Keeping number of revisions low is a way to combat performance hits, and may not fit all use cases naturally. I’d rather see a performance increase on operations if this is indeed possible. But it needs to be comprehensive, e.g. you’ve made a lot of assumptions on size of the storage, and I would hate to see faster performance on small/medium storages, and complete inability to support large storages due to memory blow up, I’ve seen this story before.

[TyTro]

@sevimo

If that’s the case, I’d urge you to take a look at the actual code, it’s on github. Even if you’ve never seen Go in your life, it’s not hard to understand what’s going on if you have a lot of experience with C/C++. This way you don’t need to speculate on how it might be done, and see how it is actually done. More so, it would be great if you can recommend some optimizations, perhaps even submit a PR.

I am sure that @gchen very likely knows exactly how this slow code of check could be made faster. It’s not rocket science. I guess the code is still in the “First implementation that’s working, but not optimized yet” state and there was just never the time or motivation to go back to it and make it faster. As someone who has never written a line of Go code, I’m definitely not the right person to try to judge Go code or even try to fix it myself. A PR by someone like me who has never written a line of Go before can’t be good, even if it would be “working” it would be very ugly Go code that no one would want to merge most likely.

[sevimo]

@TyTro

It’s not rocket science. I guess the code is still in the “First implementation that’s working, but not optimized yet” state and there was just never the time or motivation to go back to it and make it faster.

It’s just that you have a use case and skills to make it happen. Most other people would likely either not care or won’t be able to fix that, and a single developer can only do so much, he has to prioritize issues, and this one is unlikely to be on top of the priority list.

[TyTro]

I’d also find it a bit weird to spend time trying to optimize a software myself that I paid for and that I want to simply work ideally out of the box…

But I have looked a few minutes at the code now, it looks nicely structured and is easier to read than I would have expected. I’d need to spend a lot more time to be able to judge the overall program flow, but one thing I can immediately see is that it’s using a map[string]bool to store the list of chunks.

So first, it’s a waste of resources to use a map that maps a key for a bool, that’s a set, a set would need less RAM and be faster. But Google tells me that this is just a stupid design decision from Go to not have an actual set container, and instead map a key to a bool. That’s clearly not designed with performance in mind. But I’m sure there are third-party set libraries for Go that can be used. Here is a great C++ hashmap benchmark, most of those maps also have sets, and maybe some of those also have a Go implementation.

But the other thing about that is of course that it’s mapping a string to a bool. So all the chunk hashes are stored as strings. A string is super inefficient compared to an integer value, especially when using it in a map or set, because with a string you need to calculate the hash of that string every time you do a lookup, and that’s slow. It also uses way more memory to use a string instead of an integer value. A SHA256 hash that is 32 byte of memory as an integer needs at least 64 bytes of memory when stored as a string. And if you actually have a high quality 32 byte hash, which we have here, then using it in a set (or a map, if you have no other option…) is super efficient because it means you don’t need to hash it at all when doing a lookup, because it already is a hash.

So much of the slowness might simply come from the code using a string to represent a chunk hash. And this is likely the case in the whole of the duplicacy code then, chunk hashes are surely used everywhere in the code. So a fix for this would need to basically fix this everywhere, using integers instead of strings, and make all code of Duplicacy much faster and use less RAM, not just this check operation.

[Droolio]

@TyTro

So much of the slowness might simply come from the code using a string to represent a chunk hash.

Not just the chunk hashes I believe, but the chunk IDs (the file names for the chunks) - since they are, in fact, strings - there’s really no getting around that.

Question is, is it more efficient to convert those strings back to integers on the fly, during ListAllFiles(), before doing lookups? Possibly even throw away the json metadata structure storing chunk hashes, since it has to be parsed whether hashes are stored as strings or integers in json. Basically trading RAM for CPU…

@TyTro

But I’m sure there are third-party set libraries for Go that can be used.

Edit: Can’t remember where and when I found this, but this looks promising.

GitHub - alphadose/haxmap: Fastest and most memory efficient golang...