Web GUI restoring paths starting with @

Bug
#1

Is this still an issue as I also cannot restore single folders/files?

The root of my backup is prefixed with a “@” symbol.

Entry from the log file

root@myhomestorage:/tmp/duplicacy/logs# cat restore-20230604-103635.log

2023-06-04 10:36:35.044 INFO REPOSITORY_SET Repository set to /volume1/datarestore

2023-06-04 10:36:35.044 INFO STORAGE_SET Storage set to storj://12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777/backups/data/

2023-06-04 10:36:35.196 INFO SNAPSHOT_FILTER Parsing filter file /tmp/duplicacy/repositories/localhost/restore/snapshot-homes/phillipmcmahon/scripts/*

2023-06-04 10:36:35.196 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)

2023-06-04 10:36:35.281 INFO RESTORE_INPLACE Forcing in-place mode with a non-default preference path

2023-06-04 10:36:35.281 INFO RESTORE_INDEXING Indexing /volume1/datarestore

2023-06-04 10:36:35.281 INFO SNAPSHOT_FILTER Parsing filter file /tmp/duplicacy/repositories/localhost/restore/.duplicacy/filters

2023-06-04 10:36:35.281 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)

2023-06-04 10:36:35.334 INFO RESTORE_START Restoring /volume1/datarestore to revision 2

2023-06-04 10:36:35.708 INFO DOWNLOAD_PROGRESS Downloaded chunk 1 size 1450329, 1.38MB/s 02:24:35 0.0%

2023-06-04 10:36:35.709 ERROR RESTORE_CHOWN Failed to change uid or gid: lchown /volume1/datarestore/@snapshot-homes/phillipmcmahon/.ssh/authorized_keys: operation not permitted

I gave the duplicacy account “Full Control” on the datarestore shared folder.

If I include the -ignore-owner flag, then instead of the single file it wants to pull down the entire backup set for that ID.

Any help would be appreciated.

Web-ui version restoring full backup instead of single file
#2

I don’t think @ caused that. It was a permission issue, so it means that you’ll likely need to run Duplicacy as root in order to change the uid or gid of that file.

Can you check the log ~/.duplicacy-web/logs/duplicacy_web.log to see the arguments passed to the CLI with -ignore-owner enabled? Like this line:

2023/06/04 23:10:46 Running /Users/gchen/.duplicacy-web/bin/duplicacy_osx_arm64_3.1.0 [-log restore -r 6 -storage local -overwrite -stats -ignore-owner -- file1]
#3

I ran it just now and grabbed the log entrie(s)

2023/06/05 09:18:45 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log -d info -repository /tmp/duplicacy/repositories/localhost/all storj://12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777/backups/data/]

2023/06/05 09:18:48 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log list -storage storj -id data]

2023/06/05 09:18:52 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log list -id data -r 2 -storage storj -files]

2023/06/05 09:19:15 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log restore -r 2 -storage storj -overwrite -stats – @snapshot-homes/phillipmcmahon/scripts/*]

The error I get while attempting to restore only my scripts folder is

The restore command encountered an error:
Failed to change uid or gid: lchown /volume1/datarestore/@snapshot-homes/phillipmcmahon/.ssh/authorized_keys: operation not permitted

Exit code: 100

The log entry from that job is also below

2023-06-05 09:19:15.399 INFO REPOSITORY_SET Repository set to /volume1/datarestore
2023-06-05 09:19:15.399 INFO STORAGE_SET Storage set to storj://12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777/backups/data/
2023-06-05 09:19:15.686 INFO SNAPSHOT_FILTER Parsing filter file /tmp/duplicacy/repositories/localhost/restore/snapshot-homes/phillipmcmahon/scripts/*
2023-06-05 09:19:15.686 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2023-06-05 09:19:15.751 INFO RESTORE_INPLACE Forcing in-place mode with a non-default preference path
2023-06-05 09:19:15.769 INFO RESTORE_INDEXING Indexing /volume1/datarestore
2023-06-05 09:19:15.769 INFO SNAPSHOT_FILTER Parsing filter file /tmp/duplicacy/repositories/localhost/restore/.duplicacy/filters
2023-06-05 09:19:15.769 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2023-06-05 09:19:17.439 INFO RESTORE_START Restoring /volume1/datarestore to revision 2
2023-06-05 09:19:17.462 ERROR RESTORE_CHOWN Failed to change uid or gid: lchown /volume1/datarestore/@snapshot-homes/phillipmcmahon/.ssh/authorized_keys: operation not permitted

Interestingly the file it is complaining about isn’t in the path it is supposed to be restoring.

Adding the --ignore-owner starts to pull the entire backup ID which isn’t what I want.

The log when adding that flag is

2023/06/05 09:51:42 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log restore -r 2 -storage storj -overwrite -stats --ignore-owner – @snapshot-homes/phillipmcmahon/scripts/*]

#4

Sorry I forgot @ is an include directive in include/exclude patterns. If you select a file starting with @ then Duplicacy will try to load patterns from that file. This usually fails so there are no include/exclude patterns causing all files to be restored.

A workaround is to manually enter the file path with a preceding + in the Options box, such as:

+@snapshot-homes/phillipmcmahon/scripts/*
#5

Adding that results in the following call

2023/06/06 08:18:03 Running /volume1/@appstore/duplicacy/.duplicacy-web/bin/duplicacy_linux_x64_3.1.0 [-log restore -r 2 -storage storj -overwrite -stats +@snapshot-homes/phillipmcmahon/scripts/* --]

The top-level files and subdirectories are restored, but then it fails (silently) when it attempts to change the ownership of the first file.

2023/06/06 08:18:04 ERROR RESTORE_CHOWN Failed to change uid or gid: lchown /volume1/datarestore/@snapshot-homes/phillipmcmahon/scripts/remote-backup.sh: operation not permitted
2023/06/06 08:18:04 Failed to restore files for backup data revision 2 in the storage storj: Failed to change uid or gid: lchown /volume1/datarestore/@snapshot-homes/phillipmcmahon/scripts/remote-backup.sh: operation not permitted

If I add --ignore-owner as the first option (–ignore-owner +@snapshot-homes/phillipmcmahon/scripts/*) the restore works, I need to update permissions manually.

  1. With the way the Synology package is deployed (non-root), is this the state of things as expected and that it is a mandatory requirement to add --ignore-owner, and, if required, manually change permissions after the restore operation?

  2. Things work as expected if set up a copy of the duplicacy storage backend in the datarestore directory and run the following command directly under my own user. My expectation is that the web UI can do the same. It’s the reason I bought a 5-year license to avoid having to open the terminal on my NAS to get files back.

duplicacy restore -r 2 -storage storj -overwrite -stats +@snapshot-homes/phillipmcmahon/scripts/* --

#6

@gchen

Could you take a look at my last response, keen to get your input on how the web app operates.

#7

Not gchen, but I’ll try.

  1. It’s not duplicacy’s limitation — it’s who can change file owners on Linux. By default it’s only root. There are ways around that weakening security to a varying degree. One is to set “setid” attribute on duplicacy executable (I.e. chmod u+s path/to/duplicacy-CLI. See man chmod for further guidance). Then it will always run as super user regardless of who launched it, and therefore will always be able to change the attributes. With SELinux you can have finer control, but Synology does not use SELinux.

  2. Command line:

    +@snapshot-homes/phillipmcmahon/scripts/*

    Always quote the arguments: you want to pass this to duplicacy as-is, and not have shell expand it.

     '+@snapshot-homes/phillipmcmahon/scripts/*'

  3. It worked because you ran it as yourself, so all files were created owned by your user.

    If that is acceptable — i.e. if you only backup and restore files that this specific user owns — you can run duplicacy as that user. Under what user does Synology run packages and how to change that — I have no idea. (Last time I looked it was most backwardly and horribly designed contraption. But if you run duplicacy via systemd — it’s trivial to change)

#8

Thanks!

  1. Understood, I assumed as the user had full control over the restore directory it could also change permissions. I can live with the extra step to set permissions.

  2. Thanks.

  3. Yep, knew this was the case. Was hoping that the WebUI could somehow replicate this. Interestingly from what of your previous posts you referenced the following link Duplicacy Web on Synology Diskstation without Docker | Trinkets, Odds, and Ends that looks to get the WebUI running under a manually created user. Although I still think this might hit the same limitation as point 1. I shall stick with the current setup, for the “blue moon” situation where I have to restore files I’ll just reset permissions.

Thanks for your input.

#9

Quick one - do you have the steps to run duplicacy via systemd as an arbitrary user?

#10

Yes, but you have control of who is that user. It can be root, or someone else with the right entitelements.

You can simply add the setid flag on duplicacy CLI engine: it already has access to all of your data anyway, so you don’t increase disclosure by making it root. On the upside – it will be able to backup anything and restore anything, without you needing to worry about users and permissions.

#11

Starting with DSM 7.0, Synology creates a special user account for each installed app and only run the app as that user. This app-specific user doesn’t have root privileges.

But, there is a way to run the Duplicacy app as root:

First, stop the Duplicacy app on the Synology web page. Then log in the box via ssh. Run this command:

sudo env HOME=/volume1/@appstore/duplicacy /volume1/@appstore/duplicacy/usr/bin/duplicacy_web

You should now be able to access the web GUI and run a restore job with root permission.

1 Like
#12

Thanks a lot for the tip. I decided to use the docker image, which makes it super simple to select the UID and GUID of the run-time account.

Appreciate your help!

#13

I used that command successfully, but once I close Terminal on my MacBook and shut off SSH access, it shuts down the Duplicacy application running on the Synology and I need to start it via the package centre and it loses root access.

Is there any way to get the root access to be persistent/permanent on a synology without leaving SSH on and Terminal open on my MacBook?

#14

You can use tmux or screen to keep apps running after your disconnect ssh tunnel.

I.e. start tmux, or screen, and then run the commands.

#15

I would honestly recommend running the excellent docker image also provided by @saspus on your Synology. You can run as root or any other user quite easily via the docker-config.yml file parameters.

#16

Thank you for kind words!

I’ll take it a step further, and suggest running it on the diskstation natively. Docker does not really add any value here.

This is the guide for dsm6 and adaptation for dsm7 in the comments. (You’ll need to click a link to load legacy disqus comments there. I don’t load them by default to avoid subjecting my visitors to third party tracking. New commenting system is based on GitHub and does not track)

1 Like
#17

Thanks @saspus I’ll check out that link in detail.

I have it currently natively installed by using downloads from here and directly importing into package centre. Is the method linked in your post a better alternative?

#18

No, no packages involved. DSM is a Linux based OS, so you can run apps just like on any other Linux — using service manager the platform provides. Of course, software that expects a lot of dependencies may not work on a cut down purpose built Linux like DSM and docker containers provide a way to packages said dependencies without messing and polluting host OS.

Duplicacy does not have this requirement; it’s a monolithic self-container single executable. It does not need separating dependencies and therefore can run directly. Making it also available on platforms that don’t support docker

#19

Perfect, so you recommend your method instead of installing gchen’s package directly in package centre?

#20

When I did that duplicacy official Synology packages did not exist. After they were created I had seen no reason to start using them because the native way just worked.

Now that packages are available it’s reasonable to expect that this would be the best way to run duplicacy, the most platform conforming.

But reading how many flaws Synology packaging system has (not duplicacy’s fault, it’s literally how Synology packages work — even their documentation does not make any sense) — id say if you are ok with terminal and command line — avoid packages on Synology.

Like everything on DSM is eye candy for simple usecases, but as soon as you need to do something a tiny bit outside of a straight line things fall apart.