Renaming: Repository, Storage, and Storage Path | WebUI vs CLI

Support
,
#1

I have read many entries in this forum regarding the preferences file and what you can and cannot change in it for the sake of troubleshooting and just for OCD. However, it is only referenced in brief, implicit terms (eg. “just rename the prefs file!”) and my testing has turned out completely backwards results.

What I would like to know, ideally in the form of formal documentation and not just an offhanded comment, is what I can and cannot change after the creation of a storage, repository, etc… My first and only assumption is that the only way to accomplish any of this is through the editing of the preferences file (though a formally supported and UI-enabled feature would obviously be preferred).

Things that are currently tripping me up:

  1. It appears CLI vs WebUI write the contents of the .duplicacy folder differently and store them in different places (CLI in the root of the repository vs the centralized location with the WebUI).

  2. The contents of each preferences file contain many entries (one for each storage, it appears) so any one repository or storage is actually mentioned multiple times in multiple places.

  3. There is a preferences file both at the repository level and in “all” (which look to contain different information)
    /.duplicacy-web/repositories/localhost/0(1,2,3…)/.duplicacy/preferences
    /.duplicacy-web/repositories/localhost/all/.duplicacy/preferences

  4. (the big one) No matter what I change in the prefs file, after I run a command from the WebUI, it reverts all changes in the prefs file back to the original values.

Why I’d like to do any of this is for the sake of troubleshooting (there are many mentions of this by gchen (developer) as being the preferred method to get yourself out of trouble in various scenarios), moving location of a repository or storage, and just for the sake of organization as my usage of Duplicacy grows and original plans, naming conventions, etc. no longer make sense.

My shallow understanding of lock-free tells me this shouldn’t technically cause issues as long as you’re being smart about it. I also see this as a (possibly) simple addition to the actual WebUI, and definitely something that if formally implemented in the UI would be very easy for users to understand and take advantage of.

Any information or suggestions on this would be very helpful. And aside from my specific needs, given how much I see subtle references to this in the forum, it feels like it should be formally documented in some way. …And apologies if this is explained somewhere. I have been on these forums for months and have read nearly every entry with the tag “preferences” “rename” etc.

3 Likes
#2

I agree some formal documentation plus an ability to change things in the UI (and even some UI hints to show what you can’t change and why) is ultimately warranted, and I’m sure these will eventually make its way into the Web UI eventually.

To answer some of your points…

  1. This is because the Web UI leverages an optional feature of the CLI to do with referencing the location of the repository. If, with CLI, you init a repository, by default it sets the repository root at the current working directory, and places that .duplicacy folder there. Backup commands are then run after cding to the repository root.
    However, if you use -repository during init, the mentioned path is stored in the preferences file (otherwise it’s just an empty setting i.e. "repository": "",), and you can run the backup command from cding elsewhere - wherever the .duplicacy is located, or rather, its parent. It can be anywhere, and AFAICT the Web UI uses these temporary staging areas 0,1,2,3… etc. for that purpose. A place to run each backup command (CLI) from.

  2. Yup, repositories can have multiple storages.

  3. I could be wrong, but I believe 0,1,2,3… etc. are where repository-centric operations - such as backup - are run, for each repository. The all folder is likely operations which run for all repositories, such as check(?).

  4. From what I can tell, the staging areas 0,1,2,3… etc. are recreated from settings stored in the .duplicacy-web\duplicacy.json file.

Regarding what you can manually change and what you can’t…

It’s probably not the best idea to tinker with the duplicacy.json, unless you have a good idea where all the fields are (e.g. storage names, indexes and repository names are referenced under .duplicacy\stats\ as well - not updating those may break something).

One thing you can’t change with encrypted storages is the repository ids, because those are encoded in the snapshot files and encrypted with no way to edit them.

I mean, you can change a backup/repository id but then all future backups will start from scratch (albiet re-use many of the chunks already uploaded for the old id). Picking a good id name and sticking with it, is therefore advised.

You can change storage names - they’re just used for the UI and for referencing different storage but, if it were me, I’d just use the Web UI to re-add it with the proper name. Same with schedule names.

1 Like
#3

Thanks for that. This reaffirms some assumptions I had made and definitely helps me understand a few things better. There are a few complaints, a few feature requests, and some additional questions below. If anyone feels the same way about any of this, I’m happy to formalize specific items in other places (eg. feature/documentation requests)

  1. got it!

  2. In my case, each repository has exactly 1 storage. However, every time I add a storage, local or remote, copy-compatible or not, and even without pointing a repository at it, that new storage creates a new entry in each repository’s preferences file. This really threw me for a loop.

  3. I think you’re right about that, at least in that each numbered folder represents a repository. That number is used as a repository index # in the .json as well. The “all” folder’s preferences file appears to just reference the storage names + storage locations (url or local path). The id’s are all “duplicacyweb” and the “repository” fields are blank.

  4. You just hhaaaaaadd to tell me about the .json file?! OK, 1 hour later, and it looks like that is where all the action is. Agreed that it is very unwise to tinker with such a thing, but given that I’m still playing with test data… I successfully* changed the storage name and storage path. New stats files were generated with the new storage names, I’m guessing after I ran checks, though a simple file rename might have worked fine. All repository preference files were updated after restarting the Duplicacy service.

(*ran a successful backup, check, prune, and restore)

I am making massive leaps of faith and could be doing major damage in some unforeseen ways, but on the surface, it appears this is totally viable for the time being. I am the first to admit I’m in way over my head here so please - somebody with a better understanding step in and stop me.

So. Here are my findings/new assumptions:

  1. JSON is the master data source.

  2. Adding formal support to edit these names, paths, etc. in the UI shouldn’t be too difficult (or at least it isn’t an impossibility).

  3. I’m now assuming that all these references in the forum to people changing parameters in the preferences files are solely referring to repositories created with the CLI. CLI = self-contained. You make a change and it will stick. Those managed by the WebUI follow a different set of rules with the data stored in the .json file and propagated to the preference files for the sake of running the actual commands triggered by the user in the WebUI.

  4. Assuming #7 is correct, this can create some strange and unwanted issues if, for example, one were to initialize a repository using the CLI, then make the WebUI aware of it (or any other mixing of WebUI and CLI for that matter). Given that the WebUI is constantly referred to as a front-end for the CLI, this assumption could get people in hot water quick. As a “normal” user, I’m not supposed to need to know that the preferences file even exists, right?

In an ideal world I’d be able to change these parameters in the WebUI, AND I would have the option to keep the whole “.duplicacy” folder in the repository, regardless of it being managed through the WebUI or not, or at least achieve the functional equivalent of that in one way or another. This would minimize confusion (including here in the forum as people intermix discussions of WebUI and CLI).

Are there other ways in which under the hood the CLI & WebUI differ? I feel like having that knowledge up front would have helped me make an educated decision with how I want to configure my setup. Clean explanations for how these things work would also go a long way toward a more enjoyable user experience, and help hook new users that are testing and trying to figure things out, but who might hit an issue like this, get frustrated and go try another backup solution.

2 Likes
#4

I would disagree with that assessment, as someone who started off with the CLI and moved existing backups to the legacy GUI and now the Web Edition. With both the CLI and the Web UI, most of the preferences are easily recreated, since Duplicacy is designed largely to be database-less.

The .duplicacy folder merely holds an association with your repository and the destination storage(s), and maybe also the filters file, but that too can be recreated/copied. Plus it holds a temporary cache, which can always be nuked between operations.

If somehow the entire .duplicacy folder got nuked, it’s very easy to reassociate a repository with an existing storage. This goes with the CLI and Web UI, and is why moving from CLI to Web UI is relatively straightforward. (The old .duplicacy folder sitting in the repository can be deleted.)

With the CLI you just init again with the same repository; repository id against the storage URL. The storage name, if you use one, doesn’t functionally matter - it’s just used as a reference when distinguishing between other storages, and the storage name isn’t stored anywhere on the actual storage.

With the Web UI it’ll show existing repository ids in a storage when adding a backup job - making it very easy to start from scratch if need be, but to be able to resume incremental backups without destroying the association of a repository and existing snapshots in the backup storage - all that matters is the backup/repository id.

Basically, the only things a user needs to know in order to recreate their backups, is the list of repository paths, their corresponding ids (which can be retrieved from the storage), the storage URL and storage credentials, encryption password, and any filters. Then they can set new schedules.

However, I’m sure Duplicacy could make rebuilding a backup configuration easier in future. Personally, I’d like to see the filters file get stored as metadata with each snapshot. An init could then retrieve the filters file from the latest snapshot, when doing a new configuration for example.

Maybe it could do the same with the duplicacy.json and store one for each host, but then a whole lot of logic would need to be added regarding host names and adopting configurations of other hosts etc… Then the matter of pointing repositories at paths that may now be slightly different. Instead, I would favour a Summary page where the Web Edition details all the important settings in a human-readable format, which can be copied and pasted. Similar to how Veeam Agent details its backup jobs.

Anyway, I’m rambling a bit but this topic is interesting. :slight_smile:

3 Likes
#5

More clarity thanks to you - much appreciated.

I am 100% on board with everything you’ve said here, though there is still one gotcha in my mind, likely because it is exactly what tripped me up and led me down this rabbit hole: in moving between the CLI and WebUI, you are assuming one knows the differences in how things are managed, specifically where and how the configuration information is stored and the implications of those differences.

(again, assuming that what I’ve gathered from forums and my own testing is correct…)
If you start with the CLI, set everything up just the way you like, then move to the WebUI, you basically have to re-create that configuration, which will then be written to a .json file in a completely new place (AFAICT), leaving the old config in the dust. If you go back to the CLI, even just to do a quick task (and there are still things I’m finding which you can only do in the CLI), if you are referencing any filters, preferences, etc., it will be pulling from your old preferences as they were back when you switched to the WebUI.

I completely understand (and love) how you can jump on any machine, point duplicacy at a storage, and have it list all the snapshot IDs it finds and you can pick up right where you left off. But, there is another layer to this that is being ignored (generally speaking, not by you specifically, Droolio) that I still think could use some decent documentation and improvement to how it is managed. I see lots of helpful general information in tutorial form (eg. step 1: initialize repository. step 2: backup repository), but I don’t see anything that gives an overarching explanation of how all of these things are working, managed, etc… The Nomenclature ‘Article’ is much more in line with what I would imagine to be most helpful for this kind of topic.

I think something along these lines makes a lot of sense and even in a CLI environment could be totally viable - when you initialize, if it finds preexisting preferences, filters, etc. it could prompt Y/N to import those, or even display them for the user to have the opportunity to make an educated decision based on the actual contents.

Well, like you, I’m less frustrated and more intrigued at this point, and therefore rambling. I still feel a bit green to make a formal feature request, but here’s what I’ve boiled all this down to:

  1. Feature Request: Let me edit storage name and path, and repository path in the WebUI, instead of having to wipe and recreate.

  2. Documentation Request: Article/Wiki-style write up on how preferences and filters are stored and managed in CLI vs WebUI. Also (maybe separate bite-sized doc), current gotcha’s / need-to-know’s for intermixing management of repositories between CLI and WebUI.

  3. Feature Request: A re-thinking of how and where configuration information is stored, ideally allowing that information to travel with the backed-up data to be later recalled, reused, and even imported. At a minimum, a summary page as Droolio mentioned (a la Veeam Agent) would at least give users a decent starting point and allow Duplicacy to dodge the responsibility of things like re-linking paths or re-creating schedules, since the previous configuration info is simply displayed for the user to do with what he or she wishes.

Thanks for the info and ideas. Mostly loving Duplicacy and excited to keep testing (and breaking) it.

3 Likes
#6

Just reinforcing and supporting this idea (which has already been discussed here in the forum a few times), because I think it’s the last feature that would make preferences files completely “free” from separate configuration folders.

IMHO, this feature, plus RSA encryption, would make the new CLI version fantastic.

2 Likes