Chunk uploading error with Synology

saspus · 17 December 2018 10:06

No, it starts at some other locations - not the root of the filesystem, nor in my home directory. The server is Synology DiskStation. It seems to start on a root of a volume the shares reside on, except the home folder which is apparently symlinked.

Press + to add storage and fill in as depicted

Screen Shot 2018-12-17 at 1.51.22 AM.png1266×1104 57.3 KB
Screen Shot 2018-12-17 at 1.51.22 AM.png
Click green button to browse for a directory; this appears (marked my home):

Screen Shot 2018-12-17 at 1.52.06 AM.png1248×1108 45.2 KB
Screen Shot 2018-12-17 at 1.52.06 AM.png
I’m browsing to under Backups (which on the server is /volume1/Backups) and highlight the folder with my repository

Screen Shot 2018-12-17 at 1.52.36 AM.png1270×1128 47.6 KB
Screen Shot 2018-12-17 at 1.52.36 AM.png
Then I press select button and see what gets put to storage config dialog (no leading slash):

Screen Shot 2018-12-17 at 1.52.59 AM.png1228×1094 59.9 KB
Screen Shot 2018-12-17 at 1.52.59 AM.png
Press continue, specify password and storage name, and here what gets created – note single /, instead of an expected double //

Screen Shot 2018-12-17 at 2.01.42 AM.png998×552 25.2 KB
Screen Shot 2018-12-17 at 2.01.42 AM.png

gchen · 18 December 2018 02:10

This is a Synology issue. They run a modified version of ssh server so its behavior is kind of non-stardard/weird.

My proof is that if sftp://server/Backups/duplicacy were the wrong path, you would have gotten an error when trying to download the config file before getting to the chunk uploading step. The chunk uploading error you got indicated that Duplicacy had already successfully downloaded the config file and the last snapshot, so the path must be right. The error itself I suspect was caused by the chunk path being too long. If you change it to a shorted path like /Backups/dup it might work.

saspus · 18 December 2018 05:25

This would make sense, except adding / also fixed the issue. I’m not sure I understand what is going there.

I replicated the same behavior with the current released command line duplicacy – I changed the storage URL for my main repository in .duplicacy/preferences replacing double slash with single slash and this is how backup attempt went:

mymbp:~ alex$ duplicacy backup
Storage set to sftp://alex@tuchka/Backups/duplicacy
Last backup at revision 393 found
Indexing /Users/alex
Loaded 49 include/exclude pattern(s)
Packed .viminfo (39384)
Packed Library/Calendars/B1440438-A29A-48E3-A1B8-6EF3D7AE60EC.caldav/Info.plist (3992)
Packed Library/Calendars/EBB2504D-7C87-41BD-B7A8-D04268D18574.caldav/Info.plist (158526)
Packed Library/Keychains/login.keychain-db (2081988)
Failed to upload the chunk 9ba53db1737aac7e01b3b8053f5c484bc27ad5e1ec2072cf6f1c5cf1b848f0c7: EOF

Indeed, as you said, config file download succeeded but upload failed.

Putting back double slash made it work:

mymbp:~ alex$ duplicacy backup
Storage set to sftp://alex@tuchka//Backups/duplicacy
Last backup at revision 393 found
Indexing /Users/alex
Loaded 49 include/exclude pattern(s)
Packed .viminfo (37864)
Packed Library/Calendars/B1440438-A29A-48E3-A1B8-6EF3D7AE60EC.caldav/Info.plist (3992)
Packed Library/Calendars/EBB2504D-7C87-41BD-B7A8-D04268D18574.caldav/Info.plist (158526)
Packed Library/Keychains/login.keychain-db (2081988)
Packed Library/Keychains/D5B96A85-D7B3-5951-AD88-7C9760BC6424/keychain-2.db (14999552)
Backup for /Users/alex at revision 394 completed

Hence, two question:

Why reading the config file succeeds but uploading chunks fails? Path is a path in both cases, where does the difference come from?
Synology DSM is pretty popular storage; and it is unobvious that the issue is synology specific (maybe it is not limited to only synology). If we leave it as-is the forums will be flooded with “I added SFTP storage but it does not work” type of posts… And it is not obvious for the user how to fix it either – you can’t edit storage and hunting for config files is not what GUI users want to do.

gchen · 18 December 2018 13:09

A chunk path in your case is Backups/duplicacy/chunks/64-byte-hash while the path of the config file is Backups/duplicacy/config.

I first encountered this issue when developing Vertical Backup which is written in Python so I’m pretty sure this is limited to Synology only. Besides, rclone users ran into this as well: https://github.com/ncw/rclone/issues/1474 (and there it was a Duplicacy user who first found out the trick of using a single slash).

Sadly I can’t think of a better solution than manually editing the storage url.

TheBestPessimist · 18 December 2018 17:09

@gchen i agree with @saspus here: even if it’s synology’s fault i think you should make duplicacy handle this situation.

Of course this means that you should also add a message just below (or a tooltip) asking the user if he uses synology and direct him to a “synology doesn’t fix his shyte” page, to make the user aware of the incompatibility stuff. Who knows maybe some users will actually contact synology and tell them to fix their stuff.

leerspace · 18 December 2018 18:56

FWIW restic, Duplicati, and I’m sure other applications (besides rclone and others, mentioned above) also have this issue with synology’s non-standard implementation. The solution seems to be just document in an FAQ that Synology’s non-standard behavior breaks stuff and requires manual intervention to get it to work.