Ssh: handshake failed when attempting to -copy to sftp destination storage

Support
#1

I am attempting to copy from my Synology to my MacOS desktop (hostname is bethel) as a secondary backup destination. The following -copy command prompts “Enter SSH password”, even though .duplicacy/preferences is configured with an ssh keyfile (the private keyfile has no password set).

Command:

tom@synology:/volume1/zz_backups$ ~/bin/duplicacy -d -log copy -from synology -to bethel

Command output:

2023-05-27 09:12:29.735 INFO STORAGE_SET Source storage set to /volume1/zz_duplicacy-backups
2023-05-27 09:12:29.735 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_SYNOLOGY_PASSWORD
2023-05-27 09:12:29.735 DEBUG PASSWORD_PREFERENCE Reading password from preferences
2023-05-27 09:12:29.928 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2023-05-27 09:12:29.953 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
2023-05-27 09:12:29.965 INFO CONFIG_INFO Compression level: 100
2023-05-27 09:12:29.966 INFO CONFIG_INFO Average chunk size: 4194304
2023-05-27 09:12:29.966 INFO CONFIG_INFO Maximum chunk size: 16777216
2023-05-27 09:12:29.966 INFO CONFIG_INFO Minimum chunk size: 1048576
2023-05-27 09:12:29.966 INFO CONFIG_INFO Chunk seed: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:12:29.966 TRACE CONFIG_INFO Hash key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:12:29.966 TRACE CONFIG_INFO ID key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:12:29.966 TRACE CONFIG_INFO File chunks are encrypted
2023-05-27 09:12:29.966 TRACE CONFIG_INFO Metadata chunks are encrypted
2023-05-27 09:12:30.133 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_SYNOLOGY_PASSWORD
2023-05-27 09:12:30.133 DEBUG PASSWORD_PREFERENCE Reading password from preferences
2023-05-27 09:12:30.133 INFO STORAGE_SET Destination storage set to sftp://tom@bethel//Volumes/exFAT/duplicacy
2023-05-27 09:12:30.133 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_BETHEL_SSH_KEY_FILE
2023-05-27 09:12:30.133 DEBUG PASSWORD_PREFERENCE Reading ssh_key_file from preferences
2023-05-27 09:12:30.200 DEBUG SSH_PUBLICKEY Attempting public key authentication
2023-05-27 09:12:30.200 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_BETHEL_SSH_KEY_FILE
2023-05-27 09:12:30.201 DEBUG PASSWORD_PREFERENCE Reading ssh_key_file from preferences
2023-05-27 09:12:30.247 DEBUG SSH_PASSWORD Attempting password login
2023-05-27 09:12:30.247 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_BETHEL_SSH_PASSWORD
2023-05-27 09:12:30.247 DEBUG KEYRING_GET Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: exec: "dbus-launch": executable file not found in $PATH
Enter SSH password:

Adding -background to the command supresses the password prompt; however, it fails with a ssh handshake error:

Command:

tom@synology:/volume1/zz_backups$ ~/bin/duplicacy -d -background -log copy -from synology -to bethel

Command output:

2023-05-27 09:24:13.871 INFO STORAGE_SET Source storage set to /volume1/zz_duplicacy-backups
2023-05-27 09:24:13.871 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_SYNOLOGY_PASSWORD
2023-05-27 09:24:13.871 DEBUG PASSWORD_PREFERENCE Reading password from preferences
2023-05-27 09:24:14.013 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2023-05-27 09:24:14.031 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
2023-05-27 09:24:14.032 INFO CONFIG_INFO Compression level: 100
2023-05-27 09:24:14.032 INFO CONFIG_INFO Average chunk size: 4194304
2023-05-27 09:24:14.032 INFO CONFIG_INFO Maximum chunk size: 16777216
2023-05-27 09:24:14.032 INFO CONFIG_INFO Minimum chunk size: 1048576
2023-05-27 09:24:14.032 INFO CONFIG_INFO Chunk seed: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:24:14.032 TRACE CONFIG_INFO Hash key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:24:14.032 TRACE CONFIG_INFO ID key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-05-27 09:24:14.032 TRACE CONFIG_INFO File chunks are encrypted
2023-05-27 09:24:14.032 TRACE CONFIG_INFO Metadata chunks are encrypted
2023-05-27 09:24:14.033 INFO STORAGE_SET Destination storage set to sftp://tom@bethel//Volumes/exFAT/duplicacy
2023-05-27 09:24:14.034 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_BETHEL_SSH_KEY_FILE
2023-05-27 09:24:14.034 DEBUG PASSWORD_PREFERENCE Reading ssh_key_file from preferences
2023-05-27 09:24:14.034 DEBUG KEYRING_GET Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: exec: "dbus-launch": executable file not found in $PATH
2023-05-27 09:24:14.034 DEBUG KEYRING_GET Failed to get the value from the keyring: keyring/dbus: Error connecting to dbus session, not registering SecretService provider: exec: "dbus-launch": executable file not found in $PATH
2023-05-27 09:24:14.098 ERROR STORAGE_CREATE Failed to load the SFTP storage at sftp://tom@bethel//Volumes/exFAT/duplicacy: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain

Both ssh & sftp work from the command line:

tom@synology:/volume1/zz_backups$ ssh -i /var/services/homes/tom/.ssh/id_rsa tom@bethel
Last login: Sat May 27 09:07:37 2023 from 10.10.10.15
[@bethel:~] %

tom@synology:/volume1/zz_backups$ sftp -i /var/services/homes/tom/.ssh/id_rsa tom@bethel
Connected to bethel.
sftp>

.duplicacy/preferences:

[
    {
        "name": "synology",
        "id": "synology-zz_backups--synology",
        "repository": "",
        "storage": "/volume1/zz_duplicacy-backups",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        "filters": "",
        "exclude_by_attribute": false
    },
    {
        "name": "bethel",
        "id": "synology-zz_backups--bethel",
        "repository": "",
        "storage": "sftp://tom@bethel//Volumes/exFAT/duplicacy",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "nobackup_file": "",
        "keys": {
            "password": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "ssh_key_file": "/var/services/homes/tom/.ssh/id_rsa"
        },
        "filters": "",
        "exclude_by_attribute": false
    }
]
#2

Which version of macOS are you connecting to, and which duplicacy version is this?

I’m wondering if the difference in supported cyphers in macOS, Synology, and go library duplicacy is using. Try sftpc:// instead of sftp:// in duplicacy configuration file.

#3

MacOS Ventura 13.4 (22F66)
Duplicacy 3.1.0 (27FF3E)

Try sftpc:// instead of sftp:// in duplicacy configuration file.

Sorry I forgot to mention it before, but I had also tried that after seeing a mention in one of the other forum posts.

I also forgot to mention that if I manually enter the ssh password (when run without the -background option), it works (although that defeats the purpose of password-less use in a script).

Thanks!

#4

@saspus, I just setup a test sftp “copy” from the synology to one of my linux servers, so the problem definitely appears to be with some incompatibility between “incoming sftp” on macOS Ventura, and the synology/duplicacy/sftp combo. (Note: I’ve never seen a problem with “outgoing” sftp backups from macOS using duplicacy to the synology, nor to any linux server.)

As I’ve always setup my storages to be bit-identical, I think I’ll just use rsync to copy the synology storage to the external drive on the mac desktop (bethel). Are you aware of any gotchas with this method to provide a 2nd local storage location?

#5

I was going to research it a bit in the evening to have some proof before replying, but also did not want to keep you waiting, so here is what I got so far:

I suspect this may have something to do with the version of OpenSSH that is included in macOS Ventura, that disables RSA by default. For example, I know for a fact that in order to connect to vast majority of Linux servers from macOS I had to cave in and put the following into my ~/.ssh/config (because keeping adding those as command like options got old too quickly):

PubkeyAcceptedKeyTypes=+ssh-rsa
HostKeyAlgorithms=+ssh-rsa

Perhaps macOS OpenSSH server rejecting your clients RSA key for exact same reason.

So you can either turn the support back in OpenSSH server on macOS or generate a more secure key, that is by default supported. You can use ssh -Q key and ssh -Q cipher, etc to find out.

But again, I can only test it in the evening, so while I’m 99% sure that that’s what is happening, there is still 1% of other shenanigans.

#6

@saspus, I’ll look forward to seeing the results of your testing. My understanding is that while OpenSSH 9 (shipped with Ventura) disabled RSA/SHA1, RSA/SHA-256/512, can still be used. As I migrated away from SHA1 awhile back, I don’t think this is the issue I’m seeing. To further support that hypothesis, I can access the macOS machine from the synology using ssh, sftp, and rsync (all of which as using the same ssh private/public keys) without any problems–only the combination of sftp via duplicacy, from the synology to the mac, has an issue. Additionally, I’ve never had an issue using ssh from macOS to linux; however, most of my servers are running newer versions of Debian which support 256/512 keys.

As I mentioned in my previous message, I think I can just fall back to using rsync (if necessary), but I’d like to figure out this puzzle as well. :grinning:

1 Like
#7

Ah yes, good point! That’s why I usually refrain from commenting until I test myself :slight_smile:. then it sounds like ssh implementation in the go library duplicacy is using does/expects something else.

Answering your other question, yes, you can rsync bit-identical storages; that’s the whole point of that flag existing.

#8

Ok, the problem is outdated crypto and sys modules baked into duplicacy. @gchen needs to update them.

This is with the current version (BTW I don’t have linux, so I reproduced your issue on FreeBSD):

truenas% export DUPLICACY_SSH_KEY_FILE=~/.ssh/id_rsa

truenas% ssh-keygen -lf  $DUPLICACY_SSH_KEY_FILE
3072 SHA256:7IDUvg0XvYjq+ZpsUoer00E8JfqBiLezkXNgzJYw7y0 alex@truenas.home.saspus.com (RSA)

truenas% ./duplicacy -d init test sftpc://alex@obsidian//tmp/test
Reading the environment variable DUPLICACY_SSH_KEY_FILE
Attempting public key authentication
Reading the environment variable DUPLICACY_SSH_KEY_FILE
Attempting password login
Reading the environment variable DUPLICACY_SSH_PASSWORD
Failed to store the value to the keyring: keyring: No suitable keyring provider found (check your build flags)
Enter SSH password: 
Attempting keyboard interactive login
Reading the environment variable DUPLICACY_SSH_PASSWORD
Failed to store the value to the keyring: keyring: No suitable keyring provider found (check your build flags)
Enter SSH password:
Failed to load the SFTP storage at sftpc://alex@obsidian//tmp/test: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey password keyboard-interactive], no supported methods remain

Then I tried to build from source:

git clone https://github.com/gilbertchen/duplicacy
cd duplicacy/
go build duplicacy/duplicacy_main.go

and got the same result.

Then, credit to go - Golang SSH client error "unable to authenticate, attempted methods [none publickey], no supported methods remain" - Stack Overflow, I’ve updated crypto, sys, and other dependencies that broke, according to compiler suggestions:

go get golang.org/x/crypto
go get golang.org/x/sys
go get storj.io/uplink@v1.9.0
go get golang.org/x/net/idna@v0.10.0

go build duplicacy/duplicacy_main.go

and Wham!:

truenas% ./duplicacy_main -d init test sftpc://alex@obsidian//tmp/test
Reading the environment variable DUPLICACY_SSH_KEY_FILE
Attempting public key authentication
Reading the environment variable DUPLICACY_SSH_KEY_FILE
Reading the environment variable DUPLICACY_SSH_KEY_FILE
Compression level: 100
Average chunk size: 4194304
Maximum chunk size: 16777216
Minimum chunk size: 1048576
Chunk seed: 6475706c6963616379
Hash key: 6475706c6963616379
ID key: 6475706c6963616379
/mnt/pool1/homes/alex will be backed up to sftpc://alex@obsidian//tmp/test with id test

So, before duplicacy is updated, you can build one from source and update crypto libraries yourself, but you’ll need to do all the testing yourself then…

For reference, FreeBSD 13.1-RELEASE-p7, macOS 22G5027e (current 13.5 beta)

1 Like
#9

Thanks for the thorough post–at least I now know I’m not going crazy! :slight_smile:

I’ve never messed with go, but this will give me an excuse to read up on it a bit and compile myself. Thanks again! I’ll go ahead and mark your post as a solution.

#10

I couldn’t find an easy way to build it on the synology, so after a bit of searching figured out how to cross-compile on my Macbook (also running Ventura):

install go

brew install go

also add the following line to the appropriate startup file (e.g. .profile):

export GOPATH=$HOME

download duplicacy source

mkdir -p ~/src/github.com/gilbertchen
cd ~/src/github.com/gilbertchen
git clone https://github.com/gilbertchen/duplicacy.git
cd duplicacy

install updated dependencies (per your response above)

go get golang.org/x/crypto
go get golang.org/x/sys
go get storj.io/uplink@v1.9.0
go get golang.org/x/net/idna@v0.10.0

add additional dependency suggested during first build attempt

go get golang.org/x/crypto/ssh/terminal@v0.9.0

build

GOOS=linux GOARCH=amd64 go build -o duplicacy_synology duplicacy/duplicacy_main.go
1 Like
#11

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.