SFTP: Runaway sshd processes on retries/reconnects

Bug
#1

Please describe what you are doing to trigger the bug:
My connection isn’t the best and certain ssh connections will drop from time to time (but not all ssh connections always drop at the same time). When this happens, duplicacy doesn’t clean up old ssh threads before spawning new ones.

This is with duplicacy 3.2.5 running on Debian Linux.

Repro steps:
On client, run duplicacy backup with threads=4.
On server, manually kill one of the 4 sshd processes.

Please describe what you expect to happen (but doesn’t):
Expect duplicacy to close the remaining 3 ssh connections before starting 4 new ones.

Please describe what actually happens (the wrong behaviour):
The 3 existing sshd processes remain on the server and 4 new ones are started, resulting in a total of 7 processes. If you kill one of the 4 new sshd processes, then 6 existing ones remain and 4 new ones are started, resulting in 10 sshd processes. And so on and so on…

For a long-running initial backup with several ssh connection drops, this results in an ever-growing number of sshd processes eventually exhausting all the resources on my sever.

#2

Is duplicacy still being maintained? Reported this 22 days ago with no response.

#3

Pinging @gchen for a response.

#4

I couldn’t reproduce the issue. I set up a Debian 12 VM as the storage server and ran a backup from a client using 4 threads. In both cases — whether the SFTP subsystem was internal-sftp or sftp-server — there was only a single SSH/SFTP server process.

When you killed one of the sshd processes, did the Duplicacy CLI print any warning or error messages?

#6

Oh yes. On the first run, there is only 1 sshd process. If you kill that one while files are transferring, then more will pop up without closing previous ones. Though in my most recent repro, it’s not as consistent in terms of how many pop up. But it does grow unboundedly as single sshd connections die off, since existing ones are not getting cleaned up and/or reused.

root@spirit:~# /usr/bin/pgrep -u pbsbackup sshd
165286
root@spirit:~# kill 165286
root@spirit:~# /usr/bin/pgrep -u pbsbackup sshd
165314
165325
165336
165372
165383
165394
165405
root@spirit:~# kill 165372
root@spirit:~# /usr/bin/pgrep -u pbsbackup sshd
165314
165325
165336
165383
165394
165442
165453
165464
165475

Logs from duplicacy:

2025-09-05 13:31:19.614 TRACE PACK_START Packing backup/.chunks/0008/00082ef78cb11d9178fa7fe3b1594883b835ee190606b354e0ac1f3b63560253
2025-09-05 13:31:19.718 INFO UPLOAD_PROGRESS Uploaded chunk 51 size 1568101, 8.94MB/s 1 day 15:54:50 0.0%
2025-09-05 13:31:19.718 INFO PACK_END Packed backup/.chunks/0008/00082ef78cb11d9178fa7fe3b1594883b835ee190606b354e0ac1f3b63560253 (6222363)
2025-09-05 13:31:19.718 TRACE PACK_START Packing backup/.chunks/0008/00083a63c7709f8ae6ce8868f6a24287486c6eae092f2a9100bc57aef59e3eaf
2025-09-05 13:31:20.237 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:31:20.385 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:31:20.420 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:31:20.585 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
...
2025-09-05 13:32:35.033 INFO UPLOAD_PROGRESS Uploaded chunk 264 size 802732, 8.28MB/s 1 day 19:04:50 0.0%
2025-09-05 13:32:35.033 INFO PACK_END Packed backup/.chunks/002a/002a40f7a8b6e810c9cd40146d68aeac2155881bc24ef6de1c7f42df13192d53 (5104826)
2025-09-05 13:32:35.033 TRACE PACK_START Packing backup/.chunks/002a/002a907da6b9618cb5a425506ec1d5ee223e689bdccc7fbf461952fd15f6411b
2025-09-05 13:32:35.122 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:32:35.123 WARN SFTP_RETRY Encountered an error (Uploaded file but failed to store it at /home/pbsbackup/backup/chunks/13/cca156aeb9bd006ca74a17c7cd0a8bd804f1ad957a94ade42a35ceacad06a4: failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:32:35.450 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:32:35.526 WARN SFTP_RETRY Encountered an error (failed to send packet: EOF); retry after 1 second(s)
2025-09-05 13:32:39.206 INFO UPLOAD_PROGRESS Uploaded chunk 263 size 1230966, 7.94MB/s 1 day 20:53:04 0.0%
2025-09-05 13:32:39.207 INFO PACK_END Packed backup/.chunks/002a/002a907da6b9618cb5a425506ec1d5ee223e689bdccc7fbf461952fd15f6411b (5910415)
2025-09-05 13:32:39.207 TRACE PACK_START Packing backup/.chunks/002a/002ab2e78c40594d9656507ff7f9720d541903421284a5987d74c8d32ba4c79f
2025-09-05 13:32:39.292 INFO UPLOAD_PROGRESS Uploaded chunk 265 size 3047287, 7.97MB/s 1 day 20:42:50 0.0%
#7

There isn’t too much that can be done on the client side. Duplicacy uses the go sftp library which doesn’t seem to provide a function to clear unused ssh connections.

You can try adding a line UnusedConnectionTimeout 5m to /etc/ssh/sshd_config. In my test, the number of sshd processes started to drop after setting this parameter.