Struggling with CLI and how to use multiple destinations

Bug
,
#1

Like many, I’d love to get something to replace Crashplan. I’m tinkering with the CLI demo on a windows machine. Is it possible to setup the .duplicacy/preferences file to use more than one storage? Or… can we simply run a batch file with options to run the same source to two separate destinations?

Ideally, I’d like to send, as a practical example, a users profile folder (excluding several locations, like AppData in general) to two locations: 1. An SFTP host using password auth. and 2. a cloud storage. (I use Google Cloud)

Also, the ability to backup multiple backup sources to multiple destinations would be good to have.

Aside from above issues, I have not successfully setup a backup using CLI only. I could using GUI though, but GUI doesn’t have features I need. I tried to run init, with an sftp storage option, and it asks for an SSH pass, then a storage pass, after than it errors with : cipher: message authentication failed. But if I setup the same options using GUI only, it works.

Oh and how do you setup emails using CLI?

It would be really handy to simply have a .json file we could edit all the job options, much easier than trying to setup command line over and over I think.

This is a really cool program, I’m excited to see how it evolves.

#2

cipher: message authentication failed usually means that a wrong storage password is entered.

When you set up the repository and storage using GUI, did you choose a storage password. If so, then when you run init for a different repository then a -e option is required.

Here is the set of commands to do what you want:

cd repository
duplicacy init repository_id sftp://user@server/path -e
duplicacy add google repository_id gcs://bucket
duplicacy backup  # create revision 1
duplicacy copy -r 1 -from default -to google # copy revision 1 from sftp to google cloud

This approach is better than running the backup twice, because it gives the same backup on two different storages.

No, the CLI version doesn’t support email notifications.

The .duplicacy/preferences is the json file containing all configuration stuff. It should be fairly easy to modify or create it on your own.

#3

Wow, awesome! That short cli example answered several other questions too! Ok, thanks so much, I’ll do some playing around and see how it goes.

Also, just occurrred to me that I’m sending to the same SFTP destination I started my $HOME\document folder, but now am trying just the $HOME folder. I’ll setup all from scratch on both ends, I bet that’s it.

Thanks again!

#4

Is there an option to specify the last revision only (-r last/tail)?
If I run those commands as a script and copy the last revision only, I will still get the full backup on the second storage with all the revision history, right?

Thanks.

#5

There is no way to specify the last revision unfortunately (I should have added one). No, the second storage will only have the revisions that you copied from the primary storage, but each copied revision is a full backup.

1 Like
How to check only the latest revision?
#6

Just an fyi, I got this working but I couldn’t do the copy portion. It said SFTP and Google sere incompatible. That’s ok, I just ran two jobs, and for most of my clients that will be good enough.

I agree with Moshnic, having that “last revision” would be very nice if running copy right after another profile run. (even though I can use it now, I can see using it elsewhere)

Also, even though your duplicacy CLI is very nice and easy, I’m setting up some batch file and php script wrappers to ease some common tasks and job settings on systems, as well as to get CLI output and email status info when done.

Couple questions:
Is there a way to copy the stored ssh password and storage password json file (i saw the file, can’t remember which one)? I would be nice to just copy that to another system, rather than needing to re-init and type all the passwords over and over.

#7

If you provide the -copy option to the add command then the two storages will be compatible. Chunks can be copied without encryption/decryption between two compatible storages, and that is the only mode supported so that is why it won’t copy if storages aren’t compatible.

The .duplicacy/preferences file stores all the preferences as well as passwords. This file can be freely copied to other computers to avoid the need to initialize other repositories.

#8

Very cool! Thanks again!

#9

Not sure if i should start a separate thread or not, but will the GUI support multiple destinations any time soon? Is it a planned feature?

#10

Is the “copy” command smart enough to only copy over the chunks needed?

I’ve been playing with 1 Source and 2 Destinations on my local PC and my intent was to perform backups to one Destination and then keep the 2nd Destination in Sync with the “copy” command. Basically what you described above.

It works great, but it ‘seems’ like every single chunk is re-copied to the 2nd Destination each time the “copy” command is used, even if the chunk already exists there.

I don’t have a good way to observe which files are actually being added/deleted so don’t know for sure if everything is being re-written, so figured I’d ask.

I experimented by using “copy” to the 2nd Destination without specifying a revision (so everything was copied over: revisions 1-3) then made a change and backed it up to make revision 4, then another change and backup to make 5. I then ran the “copy” command to only copy over revision 5 to the 2nd Destination. I then listed the 2nd Destination and sure enough it showed 1,2,3 and 5. So this made me think it must only be sending what it needs… OR its sending all the chunks (as the CLI seems to show) and only the snapshot meta for what is needed.

I’m trying to figure out if its better to just do a Backup to each Destination to more efficiently use my bandwidth when the destination isn’t local, or if the Copy command is the right way forward.

#11

May have been able to answer my own question… found a SFTP server for Windows that was able to log the activity to screen. “Rebex Tiny SFTP Server”

I put a ~200MB file in my Source Folder and Backed it Up to my 1st Destination, it created 53 chunks. Then did a “copy” to the 2nd Destination, and watched the log as it created 53 Chunks.

I then put a small 1KB file in the Source, Backed it Up to the 1st Destination (which reported there were no 54 chunks) and then ran “copy” to the 2nd Destination. The CLI claimed all 54 chunks we copied, but looking at the SFTP log I can see that only 4 were copied plus a snapshot file.

I presume this is therefore showing it is copying only what is needed each time.

#12

Jonnie, which version are you running? The PR by Jeff should fix the issue and it is included in version 2.0.9.

#13

Hey gchen, I was on 2.0.7. Thanks for the heads up about 2.0.9. I did just try it out but ran into trouble authenticating the existing SFTP setup.

“Failed to download the configuration file from the storage: Failed to retrieve the config file: cipher: message authentication failed”

If I run the 207 binary instead, it works fine. I also tried to delete my keychain and known_hosts files and tried again, but no joy, bug?

This is the output from the SFTP log for 2.0.7:

01:01:03.280 Debug Server: Accepted connection from 127.0.0.1:42843.
01:01:03.284 Info Server: Session 4: Started on connection from 127.0.0.1:42843.
01:01:03.285 Debug SSH: Session 4: Local SSH version: SSH-2.0-RebexSSH_1.0.6461.0
01:01:03.286 Debug SSH: Session 4: Remote SSH version: SSH-2.0-Go
01:01:03.288 Debug SSH: Session 4: Performing algorithm negotiation and key exchange.
01:01:03.289 Debug SSH: Session 4: Performing key exchange using diffie-hellman-group14-sha1 with ssh-rsa.
01:01:03.320 Debug SSH: Session 4: Current encryptor is aes128-ctr/hmac-sha2-256.
01:01:03.327 Debug SSH: Session 4: Current decryptor is aes128-ctr/hmac-sha2-256.
01:01:03.329 Debug SSH: Session 4: Key exchange finished.
01:01:03.330 Debug SSH: Session 4: Performing authentication.
01:01:03.331 Debug SSH: Session 4: Starting authentication as 'tester' for 'ssh-connection'.
01:01:03.332 Info Server: Session 4: Authentication for 'tester' succeeded.
01:01:03.333 Debug SSH: Session 4: Authenticated as 'tester' for 'ssh-connection'.
01:01:03.335 Debug Server: Session 4: Starting SftpModule(6) subsystem.
01:01:03.336 Debug SFTP: Getting item info on '/': success.
01:01:03.338 Debug SFTP: Getting item info on '/Backups': success.
01:01:03.343 Debug SFTP: Getting item info on '/Backups/config': success.
01:01:03.345 Debug SFTP: Opening file '/Backups/config' (Open, Read): success.
01:01:03.346 Debug SFTP: Getting item info on '/Backups/config': success.
01:01:03.348 Debug SFTP: Closing file '/Backups/config': success.
01:01:03.374 Debug SFTP: Getting item info on '/Backups/snapshots/Source/1': success.
01:01:03.376 Debug SFTP: Getting item info on '/Backups/snapshots/Source/2': success.
01:01:03.378 Debug SFTP: Getting item info on '/Backups/snapshots/Source/3': success.
01:01:03.380 Debug SFTP: Getting item info on '/Backups/snapshots/Source/4': success.
01:01:03.383 Debug SFTP: Getting item info on '/Backups/snapshots/Source/5': success.
01:01:03.385 Debug SFTP: Getting item info on '/Backups/snapshots/Source/6': success.
01:01:03.388 Debug SFTP: Getting item info on '/Backups/snapshots/Source/7': success.
01:01:03.392 Info SSH: Session 4: Connection reset by peer.
01:01:03.393 Info Server: Session 4: Closed connection from 127.0.0.1:42843.

This is the output from the SFTP log for 2.0.9:

01:04:34.181 Debug Server: Accepted connection from 127.0.0.1:43033.
01:04:34.184 Info Server: Session 5: Started on connection from 127.0.0.1:43033.
01:04:34.185 Debug SSH: Session 5: Local SSH version: SSH-2.0-RebexSSH_1.0.6461.0
01:04:34.186 Debug SSH: Session 5: Remote SSH version: SSH-2.0-Go
01:04:34.187 Debug SSH: Session 5: Performing algorithm negotiation and key exchange.
01:04:34.193 Debug SSH: Session 5: Performing key exchange using diffie-hellman-group14-sha1 with ssh-rsa.
01:04:34.223 Debug SSH: Session 5: Current encryptor is aes128-ctr/hmac-sha2-256.
01:04:34.232 Debug SSH: Session 5: Current decryptor is aes128-ctr/hmac-sha2-256.
01:04:34.233 Debug SSH: Session 5: Key exchange finished.
01:04:34.234 Debug SSH: Session 5: Performing authentication.
01:04:34.236 Debug SSH: Session 5: Starting authentication as 'tester' for 'ssh-connection'.
01:04:38.511 Info Server: Session 5: Authentication for 'tester' succeeded.
01:04:38.514 Debug SSH: Session 5: Authenticated as 'tester' for 'ssh-connection'.
01:04:38.516 Debug Server: Session 5: Starting SftpModule(7) subsystem.
01:04:38.517 Debug SFTP: Getting item info on '/': success.
01:04:38.519 Debug SFTP: Getting item info on '/Backups': success.
01:04:38.524 Debug SFTP: Getting item info on '/Backups/config': success.
01:04:38.526 Debug SFTP: Opening file '/Backups/config' (Open, Read): success.
01:04:38.527 Debug SFTP: Getting item info on '/Backups/config': success.
01:04:38.529 Debug SFTP: Closing file '/Backups/config': success.
01:04:38.548 Info SSH: Session 5: Connection reset by peer.
01:04:38.549 Info Server: Session 5: Closed connection from 127.0.0.1:43033.

So it looks like it connects fine to the SFTP server, but the CLI claims it fails. Could it be trying to use the SSH/SFTP password as the Encryption password?

#14

I had a further poke about, and it seems that I can overcome the issue if I hardcode the passwords into the preference file:

"keys": {
    "ssh_password"  : "********",
    "password"      : "*************"
}

I hadn’t done this before, nor had I tried the environmental variables route. I had just entered them manually and let Duplicacy save them to the keychain file.

As I mentioned before, if I wipe out the keychain file and don’t have the keys hardcoded into the preferences, the 2.0.9 binary will create the keychain file as expected but then have that error. However the 2.0.7 binary is able to use the keychain info and work fine, so I guess 2.0.9 is making the keychain file just fine, its just not using it right.

#15

I couldn’t reproduce the bug – version 2.0.9 stores storage password in the keychain correctly and also has no problem using it when needed. Can you try removing those two passwords from the preference file and run duplicacy -d list which will show where passwords are retrieved from.

I did find a different bug there though. The init command doesn’t ask for the storage password if DUPLICACY_PASSWORD is set in the environment or the preference file. This is troublesome because you don’t get to enter the password twice and you may forget what is set by the environment variable. However, since you said you didn’t set the passwords using the environment variables or the preference file so this shouldn’t be the cause for your case.

#16

ok, so I think I have found the issue and can reproduce it. Basically if the keyring has an entry called “password” in it, it causes the SFTP login to fail. When I blanked it out before, I would first List from “default” which was my local destination and it would create the “password” entry in the keyring, then I would try the “SFTP” destination and it would fail (but 207 was fine).

If I wipe out the keyring file and then first try the SFTP Destination, it successfully logs in and creates the two expected entries in the keyring file: “SFTP_password” and “SFTP_ssh_password”. If I then try the “Local” destination it successfully creates the “password” entry. Now if I try the “SFTP” one again it fails.

My set up was as follows:

  • Source: C:\Temp\Duplicacy\Source
  • Local Destination (name=default): C:\Temp\Duplicacy\Local
  • SFTP Destination (name=SFTP): sftp://tester@localhost:2200/Backups

Both Destinations are Encrypted, and use different Passwords. The SSH password is different to the encryption one too.

  • Local Destination Encryption Password: duplicacy1234
  • SFTP Destination Encryption Password: duplicacy5678
  • SFTP Destination SSH Password: password

I’ve put together a test from scratch on my Windows 10 machine to show howto reproduce the error. Will post the steps below.

#17

Initialize my Source and define a Local Destination as Default (with encryption, password=duplicacy1234):

duplicacy207 init -encrypt Source "C:\Temp\Duplicacy\Local"
    Enter storage password for C:\Temp\Duplicacy\Local:*************
    Re-enter storage password:*************
    C:\Temp\Duplicacy\Source will be backed up to C:\Temp\Duplicacy\Local with id Source

Add my SFTP Destination as name SFTP (with encryption, password=duplicacy5678):

duplicacy207 add -encrypt -copy default SFTP Source "sftp://tester@localhost:2200/Backups"
    Enter SSH password:********
    Enter storage password for sftp://tester@localhost:2200/Backups:*************
    Re-enter storage password:*************
    Enter storage password for C:\Temp\Duplicacy\Local:*************
    C:\Temp\Duplicacy\Source will be backed up to sftp://tester@localhost:2200/Backups with id Source

Using 207, LIST default first and enter the encryption password:

duplicacy207 -d list -storage default
    Storage set to C:\Temp\Duplicacy\Local
    Reading the environment variable DUPLICACY_PASSWORD
    Enter storage password:*************
    Compression level: 100
    Average chunk size: 4194304
    Maximum chunk size: 16777216
    Minimum chunk size: 1048576
    Chunk seed: bc1665d442edf121fdcbfe36e947443d10268ec6a2d6b83e4e472057f573471c
    id: Source, revisions: [], tag: , showFiles: false, showChunks: false
    Listing revisions for snapshot Source

Still using 207, LIST the SFTP and enter the encryption password (ssh password already cached from ADD command):

duplicacy207 -d list -storage SFTP
    Storage set to sftp://tester@localhost:2200/Backups
    Attempting password login
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Reading the environment variable DUPLICACY_SFTP_PASSWORD
    Enter storage password:*************
    Compression level: 100
    Average chunk size: 4194304
    Maximum chunk size: 16777216
    Minimum chunk size: 1048576
    Chunk seed: bc1665d442edf121fdcbfe36e947443d10268ec6a2d6b83e4e472057f573471c
    id: Source, revisions: [], tag: , showFiles: false, showChunks: false
    Listing revisions for snapshot Source

SUCCESS
Looking at the Keyring file I can see 3 entries: [ SFTP_password, SFTP_ssh_password, password ]

Let’s Wipe out the Keyring and try with 209:

rm .duplicacy/keyring

Using 209, LIST default first and enter the encryption password:

duplicacy209 -d list -storage default
    Storage set to C:\Temp\Duplicacy\Local
    Reading the environment variable DUPLICACY_PASSWORD
    Keyring file not read: open C:\Temp\Duplicacy\Source/.duplicacy/keyring: The system cannot find the file specified.
    Enter storage password:*************
    Compression level: 100
    Average chunk size: 4194304
    Maximum chunk size: 16777216
    Minimum chunk size: 1048576
    Chunk seed: bc1665d442edf121fdcbfe36e947443d10268ec6a2d6b83e4e472057f573471c
    Reading the environment variable DUPLICACY_PASSWORD
    id: Source, revisions: [], tag: , showFiles: false, showChunks: false
    Listing revisions for snapshot Source

Still using 209, LIST the SFTP and enter the ssh password (because no longer cached):

duplicacy209 -d list -storage SFTP
    Storage set to sftp://tester@localhost:2200/Backups
    Reading the environment variable DUPLICACY_SFTP_SSH_KEY_FILE
    Attempting password login
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Enter SSH password:********
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Reading the environment variable DUPLICACY_SFTP_PASSWORD
    Failed to download the configuration file from the storage: Failed to retrieve the config file: cipher: message authentication failed

FAILURE
Looking at the Keyring file I can see 2 entries: [ SFTP_ssh_password, password ]

Let’s Wipe out the Keyring and try listing the SFTP first:

rm .duplicacy/keyring

Using 209, LIST SFTP first and enter the ssh password and the encryption password:

duplicacy209 -d list -storage SFTP
    Storage set to sftp://tester@localhost:2200/Backups
    Reading the environment variable DUPLICACY_SFTP_SSH_KEY_FILE
    Attempting password login
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Keyring file not read: open C:\Temp\Duplicacy\Source/.duplicacy/keyring: The system cannot find the file specified.
    Enter SSH password:********
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Reading the environment variable DUPLICACY_SFTP_PASSWORD
    Enter storage password:*************
    Compression level: 100
    Average chunk size: 4194304
    Maximum chunk size: 16777216
    Minimum chunk size: 1048576
    Chunk seed: bc1665d442edf121fdcbfe36e947443d10268ec6a2d6b83e4e472057f573471c
    Reading the environment variable DUPLICACY_SFTP_PASSWORD
    id: Source, revisions: [], tag: , showFiles: false, showChunks: false
    Listing revisions for snapshot Source

Still using 209, LIST the default now and enter the encryption password:

duplicacy209 -d list -storage default
    Storage set to C:\Temp\Duplicacy\Local
    Reading the environment variable DUPLICACY_PASSWORD
    Enter storage password:*************
    Compression level: 100
    Average chunk size: 4194304
    Maximum chunk size: 16777216
    Minimum chunk size: 1048576
    Chunk seed: bc1665d442edf121fdcbfe36e947443d10268ec6a2d6b83e4e472057f573471c
    Reading the environment variable DUPLICACY_PASSWORD
    id: Source, revisions: [], tag: , showFiles: false, showChunks: false
    Listing revisions for snapshot Source

SUCCESS
Looking at the Keyring file I can see 3 entries again: [ SFTP_password, SFTP_ssh_password, password ]

However… If I now try SFTP again with 209…

duplicacy209 -d list -storage SFTP
    Storage set to sftp://tester@localhost:2200/Backups
    Reading the environment variable DUPLICACY_SFTP_SSH_KEY_FILE
    Attempting password login
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Enter SSH password:********
    Reading the environment variable DUPLICACY_SFTP_SSH_PASSWORD
    Reading the environment variable DUPLICACY_SFTP_PASSWORD
    Failed to download the configuration file from the storage: Failed to retrieve the config file: cipher: message authentication failed

If I delete the entry from the keyring file for “password” then it works again, but it always prompts for both ssh password and encryption key, everytime. And appears to rewrite the contents keyring entries each time too.

If I list default, then it starts failing again.

#18

Its worth noting that I just tried backing up to Hubic instead of SFTP and it works fine with 207, but when I move to 209 I am reprompted for the Token Path and then the Auth Error happens:
Failed to download the configuration file from the storage: Failed to retrieve the config file: cipher: message authentication failed

#19

This is a regression bug introduced in 2.0.9. It has been fixed by this commit. Please let me know if you need a Windows binary.

However, the same issue may still occur if you name the storage ssh, because both the storage password for this storage and the ssh password for the default storage would both be ssh_password. I think a simple solution is to refuse to take ssh as the storage name.