SFTP Error - failed to configure storage

Support
#1

Just trying to start with duplicacy and hit a roadblock with WEB as well as CLI versions.
When trying to initialize SFTP destination - here’s what i getting:

C:\1\4>duplicacy init mywork sftp://ge@192.168.0.16:22/X:1/
Enter SSH password:**********
Failed to configure the storage: Uploaded file but failed to store it at X:1/config: file does not exist

(similar error getting with WEB UI)

Password is correct. In fact I do see that some config file (config.fshyemmp.tmp) is indeed uploaded.
Any tips?

Thanks,

Gleb

#2

Are you running on synology?
Then you’re probably having the following issue: How to install and setup Duplicacy CLI on Synology?

#3

No. Destination is a volume created by StableBit software.
Apparently everything works with /C:/ as destination but not with StableBit /X:/ volume.
X: drive otherwise works just fine with anything else.

Not sure about reasons.
PS: source and destination are Windows 10 Pro desktops.

Gleb

#4

CloudDrive? I’m not aware of any StableBit software that hosts a sftp server…

#5

StableBit software creates “DrivePool” - local drive that is a logical aggregate of multiple local disks.
Open SSH is hosted by Windows 10 Pro.

Apparently SFTP-ing to this drive doesn’t work which is kind of show stopper. It works to any other drive on that machine.

Any other way to fast transfer files within local network?

#6

I run DrivePool as well, but my Duplicacy storage is hosted via a network share, not sftp. I suspect your issue may be related to OpenSSH, since DrivePool presents as a proper kernel level drive.

The syntax of your sftp URL looks a little odd - what does the :1 in X:1 signify? Normally I’d expect it to be a full path, but I’m not familiar with Windows 10’s ssh implementation.

#7

Are you guys backing up a DrivePool as well, or just using it as a destination?

I may have to create a thread (don’t want to hijack this one really) regarding backing up DrivePool ‘disks’ as it’s a challenge I am facing. CrashPlan refugee here. Perhaps there’s quite a few of us here using Stablebit software :slight_smile:

#8

I’m doing both…

For local storage, workstation backups (Duplicacy Web Edition) to a Windows Server network share, which resides on DrivePool. The same Windows Server backs up (Duplicacy CLI) various other DrivePool folders directly to Google Drive and then copies the local Duplicacy storage to the same cloud storage.

No issues with DrivePool at all (their full suite of software is ace :slight_smile: ). Just point Duplicacy at the DrivePool, not the individual disks.

#9

To add context - the “1” in “sftp://ge@192.168.0.16:22/X:1/” is actually directory name i testing with. It could be anything else.
SCP / WinSCP fails during “rename” operation from “foo.filepart” to actual “foo” file.
This thing fails and I suspect due to the way DrivePool filesystem looks like to SSH server or something. (To note: there are no problems to transfer small files - no .filepart and renaming takes place).

This is relevant thread I’ve opened at StableBit:

#10

Hey Droolio,

  1. How to you send data to DrivePool - via UNC mapped path or how?
  2. By any chance, could you try sending data to DrivePool via SCP or WinSCP just to test if that works for you? Make it 10MB+ or so file - because small files works (SCP doesn’t create .filepart for small files)

Thanks,

Gleb

#11

I’m afraid I can’t really test this as I don’t have OpenSSH installed on my server.

In your X:1 example, do you use a leading forward slash after the drive letter and before the path? e.g. sftp://ge@192.168.0.16:22/X:/Backups? Also, if you have a trailing slash at the end, try omit that.

#12

OK I decided to do a bit of experimenting in a virtual machine…

Win 10 guest, installed trials of Duplicacy and DrivePool and installed the OpenSSH server feature.

I can confirm seeing the same issue as you. When files (like the config file) are being uploaded to the server, it’s unable to rename the tmp file into place.

This does seem to be an issue with DrivePool, but I don’t think Duplicacy is part of the problem here, as it occurs with plain old FileZilla - I’m just not able to rename a file on a DrivePool, but am able to do so on a normal drive. Regardless of file size, it seems.

Then I installed Cygwin’s implementation of OpenSSH, which I’ve been using for a long time and am quite familiar with it. Again, with FileZilla, I’m unable to rename files hosted on DrivePool. (And Duplicacy fails to configure the storage due to this.)

So there does seem to be an issue with at least two implementations of OpenSSH working with DrivePool!

2 Likes
#13

Thanks for taking time and confirming! So in a quest to push files to drive pool I have to avoid WinSCP / scp-over-ssh approach. I suspect actually that configuring scp to avoid resumable transfers will fix it - but not sure how to do it with duplicacy?

2 solutions that works in Windows 10 (source) -> Windows 10 (destination with DrivePool) scenario:

  1. Resilio Sync is pretty smart to find a way to connect 2 boxes and push data. However it is slow.
  2. rsync from WSL layer in Windows.

Resilio, once configured just monitors changes in file system and auto-propagates / auto-mirrors changes. So being unattended is plus.
Resilio on Linux (Linux -> Windows 10 DrivePool) is 20 times slower than rsync doing the same.

Rsync fom WSL layer takes just a little over a minute to scan 95,000 files over 10,000 folders (around 50GB of data) to do delta sync with destination box over ZeroTier IP layer - which is pretty amazing.

Gleb

#14

Is there a reason you’re not using plain old network share? I assume by the alternatives you’ve gone over, the two boxes aren’t on the same network?..

Resilio or rsync of course won’t allow you to directly backup, without an intermediary local storage, but if that’s the plan, have you tried Syncthing instead?

Another alternative is to backup to a non-DrivePool location - even one that is designated to a DrivePool.

I know that seems hacky, but even on my DrivePool-over-network-share, I’ve set a rule to place all backups on one physical disk. As 1) I don’t want backups to be split over multiple disks, and 2) I use the extra space on a 10TB SnapRAID parity partition, since my biggest data drive is only 5TB. There’d be no harm in placing data on a disk used for DrivePool, but outside the actual pool - it will just balance based on free disk space anyway.

Hopefully the DrivePool devs can put their finger on why both OpenSSH implementations don’t play ball. It is a bit of a limitation that you can’t host sftp directly on it!

#15

There’s mention of ZeroTier (Funny how we’re all using the same software here :slight_smile: ) so technically the machines should be considered to be on the same network segment.

1 Like
#16

Indeed.
ResilioSync/Duplicacy -> StableBit -> CrashPlan -> Off site cloud
All above made possible by ZeroTier :slight_smile:

ZeroTier is so good it should be illegal.
I can backup in real time my static Linux box (certain app trees) to Windows Laptop which i carry all around. Regardless of location, firewalls, ports, etc…

1 Like