Trouble with v2.3 CLI and SFTP Storage

Support
#1

Please describe what you are doing to trigger the bug:

Backing up to two to storage types with the CLI version 2.3 - first via SFTP to a Mac Pro on the local network, secondly to a Wasabi account.

Please describe what you expect to happen (but doesn’t):

Backing up to the local SFTP storage should happen quite quickly (it always has until recently with v2.2 and v2.3).

Please describe what actually happens (the wrong behaviour):

Backup to the Mac Pro with SFTP is showing (with -stats enabled) 20-30 KB (kilobytes) per second. Backing up to Wasabi over my 1.5 mbps outbound runs faster, at the moment, than the local network-based SFTP storage. I transferred a large file to the Mac Pro via rsync and that transferred at a rate of 100+ MB/sec over my local wired gigabit network.

The output from the profile is included below:

Blockquote
goroutine 81 [running]:
runtime/pprof.writeGoroutineStacks(0xfdefa0, 0xc00035a000, 0x0, 0x0)
/usr/local/go/src/runtime/pprof/pprof.go:679 +0x9d
runtime/pprof.writeGoroutine(0xfdefa0, 0xc00035a000, 0x2, 0x40bc49, 0xc0159e4160)
/usr/local/go/src/runtime/pprof/pprof.go:668 +0x44
runtime/pprof.(*Profile).WriteTo(0x16b7e20, 0xfdefa0, 0xc00035a000, 0x2, 0xc00035a000, 0xc000152360)
/usr/local/go/src/runtime/pprof/pprof.go:329 +0x390
net/http/pprof.handler.ServeHTTP(0xc00c048071, 0x9, 0xff0840, 0xc00035a000, 0xc000140300)
/usr/local/go/src/net/http/pprof/pprof.go:245 +0x356
net/http/pprof.Index(0xff0840, 0xc00035a000, 0xc000140300)
/usr/local/go/src/net/http/pprof/pprof.go:268 +0x6f7
net/http.HandlerFunc.ServeHTTP(0xe888c0, 0xff0840, 0xc00035a000, 0xc000140300)
/usr/local/go/src/net/http/server.go:1995 +0x44
net/http.(*ServeMux).ServeHTTP(0x16ca1a0, 0xff0840, 0xc00035a000, 0xc000140300)
/usr/local/go/src/net/http/server.go:2375 +0x1d6
net/http.serverHandler.ServeHTTP(0xc000265860, 0xff0840, 0xc00035a000, 0xc000140300)
/usr/local/go/src/net/http/server.go:2774 +0xa8
net/http.(*conn).serve(0xc0000e41e0, 0xff4b00, 0xc017b00040)
/usr/local/go/src/net/http/server.go:1878 +0x851
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2884 +0x2f4

goroutine 1 [chan send, 5 minutes]:
github.com/gilbertchen/duplicacy/src.(*ChunkUploader).StartChunk(…)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:72
github.com/gilbertchen/duplicacy/src.(*BackupManager).Backup.func3(0xc0013fc300, 0xc02b4df700)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_backupmanager.go:534 +0x4cf
github.com/gilbertchen/duplicacy/src.(*ChunkMaker).ForEachChunk(0xc0000f6900, 0xfdf0a0, 0xc00000e0d8, 0xc0002fd2f8, 0xc0002fd118)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkmaker.go:286 +0x90c
github.com/gilbertchen/duplicacy/src.(*BackupManager).Backup(0xc0002ae900, 0xc000034180, 0x8, 0x1, 0x1, 0x0, 0x0, 0x1, 0x0, 0x0, …)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_backupmanager.go:510 +0x3820
main.backupRepository(0xc0002bc120)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:745 +0x66f
github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli.Command.Run(0xe4b6c7, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe73f2c, 0x30, 0x0, …)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli/command.go:160 +0x74d
github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli.(*App).Run(0xc000185c20, 0xc0000e2000, 0x5, 0x5, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/gilbertchen/cli/app.go:179 +0x64f
main.main()
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:2074 +0x5b58

goroutine 18 [syscall, 16 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 4 [chan receive, 16 minutes]:
main.main.func1(0xc000151f80)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:2068 +0x60
created by main.main
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:2067 +0x5b26

goroutine 5 [IO wait]:
internal/poll.runtime_pollWait(0x7f4e08e3ef08, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc00015e498, 0x72, 0x0, 0x0, 0xe4c886)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc00015e480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba
net.(*netFD).accept(0xc00015e480, 0xc000060f70, 0xc000060f00, 0x40bc49)
/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0001580d8, 0xc0000cce08, 0xda3bc0, 0xde61671cb2832b01)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).AcceptTCP(0xc0001580d8, 0xc0000cce30, 0x4c5416, 0x5de68007)
/usr/local/go/src/net/tcpsock.go:247 +0x48
net/http.tcpKeepAliveListener.Accept(0xc0001580d8, 0xc0000cce80, 0x18, 0xc000001080, 0x6e50f4)
/usr/local/go/src/net/http/server.go:3264 +0x2f
net/http.(*Server).Serve(0xc000265860, 0xff0f00, 0xc0001580d8, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2859 +0x22d
net/http.(*Server).ListenAndServe(0xc000265860, 0xc000265860, 0x0)
/usr/local/go/src/net/http/server.go:2797 +0xe4
net/http.ListenAndServe(…)
/usr/local/go/src/net/http/server.go:3037
main.setGlobalOptions.func1(0x7fff432688ad, 0xe)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:158 +0x73
created by main.setGlobalOptions
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go:157 +0x1b5

goroutine 6 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f4e08e3ee38, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc0002c0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002c0180, 0xc0002f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0002c0180, 0xc0002f7000, 0x1000, 0x1000, 0xc0000cdcb0, 0x4172bf, 0x7f4e086aba90)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e078, 0xc0002f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
bufio.(*Reader).Read(0xc00009e360, 0xc000142221, 0x5, 0x5, 0xc0000cdda8, 0x40bc49, 0xc00dec2000)
/usr/local/go/src/bufio/bufio.go:223 +0x23e
io.ReadAtLeast(0xfddc20, 0xc00009e360, 0xc000142221, 0x5, 0x5, 0x5, 0x7f4e0b28fd98, 0x0, 0x0)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(…)
/usr/local/go/src/io/io.go:329
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*streamPacketCipher).readPacket(0xc000142200, 0xc0000004cb, 0xfddc20, 0xc00009e360, 0xc0002ce000, 0x25, 0xc2f, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/cipher.go:150 +0x84
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*connectionState).readPacket(0xc0002bc5a0, 0xc00009e360, 0x4, 0xc0000cdfb0, 0xc000146a80, 0xc0000cdf30, 0x4054cc)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/transport.go:129 +0x68
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*transport).readPacket(0xc0002bc5a0, 0xc0000cdf20, 0x3, 0x0, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/transport.go:113 +0x3b
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket(0xc000304000, 0xc0000cdf00, 0xc00dec2000, 0x25, 0x25, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:374 +0x53
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).readLoop(0xc000304000)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:197 +0x4d
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.newClientTransport
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:134 +0x181

goroutine 7 [select, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc000304000)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:268 +0xfa
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.newClientTransport
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:135 +0x1a3

goroutine 34 [chan receive, 2 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*handshakeTransport).readPacket(0xc000304000, 0xc00dec2000, 0x25, 0x25, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/handshake.go:187 +0x4e
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*mux).onePacket(0xc00026f0a0, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/mux.go:215 +0x48
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*mux).loop(0xc00026f0a0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/mux.go:190 +0x35
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.newMux
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/mux.go:128 +0x155

goroutine 35 [chan receive, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*Client).handleGlobalRequests(0xc000163b40, 0xc0002ae1e0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:138 +0x6d
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:57 +0x97

goroutine 36 [chan receive, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*Client).handleChannelOpens(0xc000163b40, 0xc0002ae120)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:147 +0x52
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:58 +0xc3

goroutine 37 [sync.Cond.Wait, 16 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc000163b10, 0x0)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc000163b00)
/usr/local/go/src/sync/cond.go:56 +0x9e
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*mux).Wait(0xc00026f0a0, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/mux.go:110 +0x82
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient.func1(0xc000163b40)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:60 +0x33
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:59 +0xe5

goroutine 38 [chan receive, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*forwardList).handleChannels(0xc000163b50, 0xc0002ae240)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/tcpip.go:186 +0x58
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:63 +0x133

goroutine 39 [chan receive, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*forwardList).handleChannels(0xc000163b50, 0xc0002ae2a0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/tcpip.go:186 +0x58
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.NewClient
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/client.go:64 +0x178

goroutine 40 [chan receive, 16 minutes]:
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*Session).wait(0xc00029a1b0, 0xc0002ae300, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/session.go:423 +0x7a
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.newSession.func1(0xc00029a1b0, 0xc0002ae300)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/session.go:593 +0x35
created by github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.newSession
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/session.go:592 +0xb4

goroutine 41 [sync.Cond.Wait, 2 minutes]:
runtime.goparkunlock(…)
/usr/local/go/src/runtime/proc.go:307
sync.runtime_notifyListWait(0xc000163bd0, 0xf8)
/usr/local/go/src/runtime/sema.go:510 +0xf9
sync.(*Cond).Wait(0xc000163bc0)
/usr/local/go/src/sync/cond.go:56 +0x9e
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*buffer).Read(0xc0001f79e0, 0xc00db6a038, 0x4, 0x4, 0x0, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/buffer.go:94 +0x1ca
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*channel).ReadExtended(0xc0001780c0, 0xc00db6a038, 0x4, 0x4, 0x0, 0x18, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/channel.go:351 +0x23f
github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh.(*channel).Read(0xc0001780c0, 0xc00db6a038, 0x4, 0x4, 0xc0013d0c00, 0x4, 0xc0013ce380)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/golang.org/x/crypto/ssh/channel.go:528 +0x5b
io.ReadAtLeast(0xfde3a0, 0xc000142300, 0xc00db6a038, 0x4, 0x4, 0x4, 0xc000150600, 0xc0000ceee0, 0x404fa5)
/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(…)
/usr/local/go/src/io/io.go:329
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.recvPacket(0xfde3a0, 0xc000142300, 0xc000000391, 0xc0001ed9e8, 0x1, 0x17, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/packet.go:142 +0x82
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.(*conn).recvPacket(…)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/conn.go:22
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.(*clientConn).recv(0xc000142300, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/conn.go:75 +0xf2
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.(*clientConn).loop(0xc000142300)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/conn.go:60 +0x5b
created by github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.NewClientPipe
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/client.go:143 +0x253

goroutine 65 [select, 16 minutes]:
github.com/gilbertchen/duplicacy/src.CreateChunkDownloader.func1(0xc0000e4820, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:80 +0x162
created by github.com/gilbertchen/duplicacy/src.CreateChunkDownloader
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkdownloader.go:77 +0x1d4

goroutine 12 [chan receive, 2 minutes]:
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.(*clientConn).sendPacket(0xc000142300, 0xfea780, 0xc00023fbf0, 0xc00023fbf0, 0xe88e20, 0xc00d0e19e0, 0x42c2ef, 0xc000000008, 0xe88e20)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/conn.go:109 +0xac
github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp.(*Client).Rename(0xc000142300, 0xc018892000, 0x74, 0xc020e42150, 0x67, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/pkg/sftp/client.go:610 +0xd8
github.com/gilbertchen/duplicacy/src.(*SFTPStorage).UploadFile.func3(0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_sftpstorage.go:330 +0x32d
github.com/gilbertchen/duplicacy/src.(*SFTPStorage).retry(0xc00031a240, 0xc00d0e1d10, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_sftpstorage.go:125 +0x70
github.com/gilbertchen/duplicacy/src.(*SFTPStorage).UploadFile(0xc00031a240, 0x0, 0xc01c1000f0, 0x48, 0xc02e000000, 0x806d6c, 0x1000000, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_sftpstorage.go:307 +0x3b1
github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Upload(0xc0000301e0, 0x0, 0xc0013fc180, 0x92, 0xc0000dc201)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:138 +0x246
github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Start.func1(0xc0000301e0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:60 +0x83
created by github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Start
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:55 +0x48

goroutine 13 [IO wait]:
internal/poll.runtime_pollWait(0x7f4e08e3ed68, 0x72, 0xffffffffffffffff)
/usr/local/go/src/runtime/netpoll.go:182 +0x56
internal/poll.(*pollDesc).wait(0xc020e36518, 0x72, 0x0, 0x1, 0xffffffffffffffff)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b
internal/poll.(*pollDesc).waitRead(…)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc020e36500, 0xc00023ff61, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc020e36500, 0xc00023ff61, 0x1, 0x1, 0xc003e6f780, 0xc003e6f800, 0xc003e6f880)
/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e008, 0xc00023ff61, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:177 +0x69
net/http.(*connReader).backgroundRead(0xc00023ff50)
/usr/local/go/src/net/http/server.go:677 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:673 +0xca

#2

Can you run the benchmark command and post the results here?

#4

Here are the benchmark results for the SFTP storage:

Blockquote
Storage set to sftp://******@10...//Volumes/Data/Backup_Duplicacy
Generating 256.00M byte random data in memory
Writing random data to local disk
Wrote 256.00M bytes in 1.22s: 209.32M/s
Reading the random data from local disk
Read 256.00M bytes in 0.05s: 5325.04M/s
Split 256.00M bytes into 53 chunks without compression/encryption in 2.05s: 125.06M/s
Split 256.00M bytes into 53 chunks with compression but without encryption in 2.72s: 94.21M/s
Split 256.00M bytes into 53 chunks with compression and encryption in 2.94s: 87.15M/s
Deleting 37 temporary files from previous benchmark runs
Generating 64 chunks
Uploaded 256.00M bytes in 29678.95s: 9K/s
Chunk 0 has mismatched hashes: 2fd471e5291520cf2cbd1262840e2f78f5329001efefaf9b79e511a4e6187db5 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 1 has mismatched hashes: e6db06e7ef7095366171bc75810a31853aa681d3bd4191d6c2d931090501b4ad != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 2 has mismatched hashes: f8058965c62bdc48e0ffa58bc9f72d2e076dd005e1d97238caa4add19910b0bd != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 3 has mismatched hashes: dcfdc5e1c5d10d55e102caa33ebe0ff4abdedecae0127b7d97616d5a8c6a3e25 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 4 has mismatched hashes: 4f0f35a5c498be4d2589a57f9818b4afaec7e83c24b37131edbfe7539dfc12c2 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 5 has mismatched hashes: 80f93eaf5bcb54f57a32b66e9295b2301310a30e181a7b329682676189a315b8 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 6 has mismatched hashes: 7ceaf1ea0d0e2c266cb0a623e6cdd72aba92c4ba683c4e57bc6ed733b66911d9 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 7 has mismatched hashes: 3520ce0126667c1725b4e82f0f53d5ed9b2fa73b2ffdf471270d669c80cb6fe5 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 8 has mismatched hashes: c35793d9241227b98b7949158d5c9139032ad5e2ac35cfc22f1a292832c96fec != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 9 has mismatched hashes: a92d09aaaa820f3946a07cd6918cfcb9b31aa4a39a4b487173c0ce29fc592a6f != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 10 has mismatched hashes: c4958bd6fe5bc8e121755f999c06701f434c1d3e2adc4766f352e91c1bcdea9f != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 11 has mismatched hashes: f20143a011198b7cb1341c6ba92db11224e3b89931c0f8d26c43efaead2626d9 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 12 has mismatched hashes: aeaf132a4aa08d0a3d7a8ba0755ee8a4ae3a88dbc899de8df2f642295c05f447 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 13 has mismatched hashes: ff7faabd54debacf42841eab39bac3a435ef5cf8e0edc7aabecd2c20ea78f70f != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 14 has mismatched hashes: 80bf0f48152931bc2aa3150a2d460e9f1ba667e81f23af065ef68fa4cf44aac6 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 15 has mismatched hashes: 62e67ca2a59e57b9447cca33757488872390e9b3e7380c62f56715aaaec87224 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 16 has mismatched hashes: c66ac39d253e761b0f0cb7f0955b600cdbabb326a2c99acc22f5db44324c4bc5 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 17 has mismatched hashes: e03f1a51b1ff8a3fef262aa20147329ea1e0d39a540a1fe468a0dad2cd3c260b != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 18 has mismatched hashes: 22f7b37bec88cac3939e3f4548451786e4274cf3d42c5dd1798bc2ba5a52b054 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 19 has mismatched hashes: d40407b363ab9d44905058b5b16c9ca355dc26d794530541cccacb9258005866 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 20 has mismatched hashes: 6bbf815c039b8f4885c5ba0bd6d4e2920fc53d7eb363d782584bae60b928b1f0 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 21 has mismatched hashes: 179b282bcf24698bd89ad42abbe7aadf37e978d56cf485279153d462aa263c7a != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 22 has mismatched hashes: 27118863b46c0b1314c2c3eb9c053e9428a6f14fa88de3a30cd71e36085390df != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 23 has mismatched hashes: ecf41c2f489beba576bb5a6cb224764c407fab168274428766c37a703949121d != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 24 has mismatched hashes: 162aef1bd5802a5699ddc1e8d94d613b25ea809fd93093529591b1b0c1a9f5e2 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 25 has mismatched hashes: 4baac29a1e052c1b6ebc6d900ab05f58abf00b744ec7ff7a1f7f0a76f41c47b5 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 26 has mismatched hashes: 177e74d467f82ba76ad9451a421fc730ba087ab4cae94b56a7d91d91a71470c5 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 27 has mismatched hashes: c7992510028aa05ad267170cb7339eb8cd194994bd84c8944f8c4c6b690550b6 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 28 has mismatched hashes: b565a11735f99a215b1e17361a844deec039ca7748b685c3dfcd317900d8ab1d != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 29 has mismatched hashes: 850bdcfcfa6daadfa0066995b474e3f65688c56cfd4327a91f00ab0e36973edf != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 30 has mismatched hashes: e2fd59c2b655af252011cc403b1f751463372b909ed083a38e2e7b6158a3b672 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 31 has mismatched hashes: 392214973d3ba9d5aa66cb1f0857c6dc608b63730da873aefef9ea6283c01649 != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 32 has mismatched hashes: 025fe20492905de5cf41d1d9b1bef294d5734236e44a7a79e0f69c99dc9fa986 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 33 has mismatched hashes: c8b6cb0347340405850031eb31b97b70a69912f96d6b013322f337fb635fb4a0 != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 34 has mismatched hashes: 99ecfdd0fa1acc9ade43d9044ead6a13f388109021131a0223e39734809775a9 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 35 has mismatched hashes: 9bc4cd350efa9e02f14a53e04a86108e47208016d5f9f0ae48115cb68c1344d2 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Chunk 36 has mismatched hashes: d13d9606dc3bb6ea8dac3dce8f1596d90b4d7a56385b71c1c780ddc6d39e7cbe != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 37 has mismatched hashes: 078f2599bad6b79a17a729e028614c40f6b3b57ea10d6f94ad8f83a46bbc4f09 != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 38 has mismatched hashes: 3430bbd06ddb4278d575f6c0ebb84d857af7eae36ffd39c5c616faeec54d1ea9 != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 39 has mismatched hashes: dc6948ee5595cc7b3b5ff136fead07524099843a002b43401c7001d5c926a75c != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Chunk 40 has mismatched hashes: ed8f8f0e70c8b9b5bdeedc799c62a1e9e9c08d0d7f410f42d6353f515c2efcdd != bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
Encountered an error (failed to send packet: EOF); retry after 1 second(s)
Failed to download the chunk: file does not exist

#5

The mismatched hashes messages means the downloaded file is different from the uploaded one. It looks like the hard disk on your sftp server is failing.

1 Like
#6

Thanks gchen, I believe the problem is with the drive.

#7

For anyone: Feel free to use the :heart: button on the posts that you found useful.

For the OP of any #support topic: you can mark the post that solved your issue by ticking the :checked: under the post. That of course may include your own post :slight_smile: