CLI and WebUI - why don't they know each other?

Hi Duplicacy experts,

I have a question on CLI/WebUI interoperability.

I am currently evaluating Duplicacy as an alternative to the Relica/restic combo. I am super impressed so far from the backup speed and the well-designed Web UI.

Now when I tried to use the CLI, I discovered that the CLI command knows nothing about the central WebUI config location and hence has not the faintest clue where the repositories are, let alone that they exist at all. Several forum threads confirm this current status; however, I found no answers for the following two questions:

  • Is there any way (hidden config, env var,ā€¦) to make the CLI tool use the WebUI config instead of searching for directory-local config files?
  • Are there plans to unify CLI and WebUI to use the same config by default?

I think this is especially important for newcomers. They will expect (just like I did) that WebUI and CLI can be used interchangeably without any further effort.

Thanks

2 Likes

This is all by design - CLI tool is just that - tool to perform backups, not to manage backups created somewhere else. It is actually Web UI which uses CLI to to perform backup.
I havenā€™t tried myself (I just donā€™t use UI), but if you start CLI tool from the folder with repository, it will work with it without an issue.

The web GUI stores preferences files under ~/.duplicacy-web/repositories/localhost, and you can run the CLI from there. For instance, to run the first backup:

cd ~/.duplicacy-web/repositories/localhost/1
~/.duplicacy-web/bin/duplicacy_osx_x64_2.3.0 backup

It is also perfectly fine to initialize the repository again on the directory to be backed up and run backup from there.

cd /path/to/repository
~/.duplicacy-web/bin/duplicacy_osx_x64_2.3.0 init repository_id storage_url
~/.duplicacy-web/bin/duplicacy_osx_x64_2.3.0 backup
2 Likes

Thank you for your feedback @dgcom.

This is precisely what does not work for me. I have created a repo for, say /User/myname/mydata in WebUI, but when I cd there and type duplicacy list, it says there is no repo. I expected that the CLI tool grabs into the Web config to find out that there actually is something in this directory.

Thank you @gchen for the help. I tried your steps and they work. However, I would have expected this to work in a more intuitive way out of the box, without the need for intimate knowledge about the internal workings of Web UI and CLI (and the CLI inside Web UI).

In other words, I would like to open a shell, type duplicacy list and get a list of all my backups back that were created via Web UI or CLI.

Am I having some wrong expectations about the intended use cases of the cli tool here?

1 Like

This works. But you need to run the CLI after changing the current working directory to the appropriate location (which has the initialised .duplicacy configuration directory at the root). Thereā€™s no getting around that.

IMO you probably want to use duplicacy list -all and run the command from ~/.duplicacy-web/repositories/localhost/all. (Paths 0, 1 etc. are staging areas for specific repository related tasks like a backup job - all is generally used for operations on the storage i.e. check, prune, so list -all makes sense to run it from there.)

1 Like

I feel that we are talking past each other. (And I am referring to all of the answers in this thread so far.)

This is exactly NOT what I want to do. I do not want to read through loads of forum articles to understand the low-level technical details of the backup tool I intend to use, just to find out that there is some dot directory with Paths 0, 1, or all inside, in order to run very basic commands like listing my repositories and backups. I hope I could make the problem clear now.

My suggestion: Since there is one single, central location of the Web UI config files, make the cli tool aware of that location, have it grab the data from there and make it a bit more user friendly.

So when I cd into a directory that the WebUI backs up, the cli tool should be able to list my backups there, instead of insisting that there is nothing at this location.

Is the intent of my post clearer now?


Edited to add: Please donā€™t get me wrong. I highly appreciate all of your help and support. Not every forum is as active and welcoming as this one, where I posted a noob question and got helpful answers within a day. Thanks so much for your help.

What I tried to explain is that your view on Duplicacy is so much from a high expert level that it seems the most normal thing to you to cd into a dotsomething/wuggah/blah/blah/all path to make the cli command work. My point is that a Duplicacy newbie, even one with a CS background and years of IT experience, might not (yet) be ready to dive into the deep details of the tools for doing basic backup stuff.

2 Likes

I am not the developer but, you asked if you have some wrong expectations about the intended use case of the CLI toolā€¦ IMO, yes.

Very few users will want or need to use the CLI with the Web UI - the fact that we can is only down to development history of both components. The CLI tool was designed to operate standalone, which operates on disparate repositories. The Web UI - which is not (yet) feature-complete - a wrapper around it. The fact that we can borrow the Web UIs configuration directory for running the CLI is also a happy quirk of its design, not necessarily its design intent.

Consequently, it is not a requirement to employ the Web UIs existing configuration in order to perform operations on your backup storage with the CLI. i.e. You can initialise a dummy repository elsewhere and continue to use the CLI as standalone (or on a different machine).

The choice is ultimately yours, but you now have that choice. My prior post was merely to inform you how to achieve what you want to do with the current implementation.

In my view, the fact that some people may still occasionally need to use the CLI alongside the Web UI speaks to the need of further development on the Web UI, which would eventually obviate the need to ever run the CLI with the Web UI (although the option will always still be there).

As the UI source code is closed, this is obviously a decision for the developerā€¦

This is certainly one idea. Rclone does something similar with its CLI and config, although it only allows for one config, and Duplicacy can have many storages and more complex relationships with backup IDs.

Alternatively, the same outcome could be achieved, simply, by showing - inside the UI somewhere - a copy-pasteable path where you can handily run the CLI from (without prior initialisation) - via additional elements in the form of tooltips, a text box and whatnot. Furthermore, there may be a case that some of this may need to be better documented.

Personally, I think itā€™d be far easier and less messy to do this than to make the CLI ā€˜UI-awareā€™, just to save the user from performing one extra command. It doesnā€™t need to be more complicated than that for ad-hoc CLI usage, and CLI scripts can always perform the extra step on their own.

I mean, if cd-ing to the appropriate location was a stumbling block for you, I have news. To use the CLI, youā€™ll still need to rtfm. To use the CLI standalone, the initialisation and cd phase is part of its design, necessarily, and is adequately documented at least. So yes, you may well need to have more intimate knowledge about the internal workings. At the end of the day, donā€™t expect the CLI and Web UI to have the same level of intuitiveness - if you want ā€œintuitiveā€, stick with the Web UI.

Otherwise, the CLI is open source, and anyone is free to create a PR. Bare in mind, any fundamentally big changes should be backwards compatible.

5 Likes

i completely understand your point. I could have written that feature request. The reason I didnā€™t is that I started with duplicacy when the web-ui didnā€™t exist so I had to learn the CLI and once you have done that, then the web-ui is just another instance of the CLI running behind a webserver. From that perspective, things make sense as they are and some of your suggestions would seem to introduce some unnecessary confusions between the two versions and create false expectations about the integration between the two. If duplicacy list just have me all backups that would create the impression that duplicacy is integrated with the web-ui (which it isnā€™t).

With that said, though, I still like the idea of a duplicacy command that spits out all repositories on a machineā€¦ But then again, locate duplicacy or something like that would probably do the job, at least on Linux.

So I guess the question here is whether duplicacy should be developed further towards a centralized config location (but see What if we remove the feature `-pref-dir`).

1 Like

Valid points by several. I believe more tools will be developed to make the entry easier, also for personal free users.

The best user experience will be available in the GUI version and so you have alternatives.

Thank you @Droolio for the extensive answer. All of what you write makes sense. Indeed, I expected CLI and WebUI to work hand-in-hand, but I completely see the point of having just one UI for all possible tasks.

Maybe I reached out for the CLI exactly because the WebUI is not yet feature-complete. And personally, I can live with the given workaround (cdā€™ing into .duplicacy/ā€¦ etc). This trick should get a place in the CLI doc to help other newbies.

About the options of (not) making the CLI tool aware of existing configs, I keep a PR in mind, although I most certainly will have no time in the foreseeable future for digging into the code and creating that option. (I am not even sure atm if I underestimate the necessary amount of changes for that.)

3 Likes

Thank you @Christoph for your helpful comment.

Based on

and

I realize that I have not taken the impacts of historical development into account. I somehow expected that CLI and WebUI are the two sides of a coin.

I can live with that, now that I know whatā€™s going on and why the CLI behaves this way. Maybe someone will want to add a hint to the CLI docs to help other newbies undertanding the role of the CLI command.

2 Likes

New user here, but I thought Iā€™d provide some feedback too on this.

I was expecting the WebGUI to configure all of its scheduled tasks in cron calling CLI commands. This would mean I donā€™t have to run the webgui 24x7 and have yet another service exposed. In my view this architecture would have improved security and performance impact to the host.

1 Like

This is actually a great idea.

Duplicacy web should take advantage of OS scheduler (systemd on Linux, task scheduler on windows, launchd on macOS, service on FreeBSD) and not need to reimplement the existing functionality.

This also will force Duplicacy to fully adopt keychain/keyring on each os to pass credentials to the CLI ā€” today it does so via environment variables which is not ideal (super user can read them in plaintext)

1 Like