"repository" init and add options

[TheBestPessimist]

I see no problem with going ahead and making the change. Gilbert can revert if he disagrees.

But if he (yet) doesn’t want to -> we would’ve both made useless work.
Let’s just wait for a little more time to speak his mind now that he was announced about our intention .

[towerbr]

here is one way you could make restoring to a temporary folder from any of your repositories using this alternative scheme relatively simple

Liked it! Is really a simple scheme and makes the settings even more centralized, without the use of .duplicacy files. I think I’ll implement it …

[towerbr]

Well, or I’m missing something very obvious or is not so simple …

• I edited the preferences file ("repository": "",) and added the path to repository;
• Removed the .duplicacy file from repository;
• Changed the cd command in my script, replacing the repository with the folder where the preferences file is.

and when I execute a backup…

ERROR REPOSITORY_PATH Repository has not been initialized

If I return the .duplicacy file to repository and also return the cd command (pointing to repository), even without removing the repository path from the preferences file, backup performs perfectly.

Any ideas?

[ben-ptl]

• Changed the cd command in my script, replacing the repository with the folder where the preferences file is.

That’s odd. Did you cd to the .duplicacy folder? Because the preferences file should be inside the .duplicacy folder, and you should actually be changing to the parent directory (one level above) the .duplicacy folder that contains the preferences file.

I would also suggest double checking the repository path is valid, properly quoted, etc. Maybe test this out by using the init command to create a new temporary backup that uses the -repository option to refer to the actual root folder you’re backing up, but have the storage be a local temporary directory. Then inspect the formatting of the resulting preferences file to verify your manual edits look the same. Of course then at least start a backup to the temporary storage to make sure it’s working

I don’t see why this wouldn’t work the same for you as it is for me. I’ve tested on Linux and Windows so far, and it all looks good.

[Nelvin]

Nelvin:

If I don’t miss anything here, completely removing any .duplicacy file or folder from repositories would enable us to have “multiple repositories” sharing the same root folder?

If that is how it works, that would indeed be ingenious. Have you tried it?

I just gave this a try and it seems to work as expected

All you need to do is create a dedicated folder for your preferences, then create a subfolder for each of your, let’s call it “virtual repository”, change the current workdir into one of these and then just init a repository as usual just with an added -repository pointing to the rootfolder of wherever you want to base your virtual repository at.

This way, no .duplicacy file will be created in the actual folder of your repository, deduplication works great as usual and you can have a different set of filters for each of these virtual repositories.

Obviously each virtual repository needs it’s own snapshot-id.

I wonder how reliable this is as it’s a new option, but in my quick local test, everything went fine.

[Christoph]

Excellent! Feel free to enrich the wiki with this information. In order to keep the wiki as concise as possible, I’d suggest to just write a few sentences into the wiki and then add more details to a reply to the wiki and refer to it from the wiki.

Hm, as I write this, I’m thinking: maybe this is a bit confusing? Maybe it’s easier to just refer to the post above?

[ben-ptl]

I may need a minute to muster up the nerve to edit the wiki without feeling like I’m not qualified to make that kind of contribution…

That’s fine. But you’ll see that the wiki has absolutely no infirmation yet, about how the option works (except for the basic -help description), so it’s easy to improve it
And if you’re unsure, you can always flag the post for staff attention after editing.

I gave it a shot:

The -repository option specifies how the repository root directory is defined in the preferences file. This may be specified as either an absolute or relative path. Relative paths are relative to the current working directory of Duplicacy at the time it is executed (when the preferences file is being parsed). This option allows for the possibility of the repository configuration files and the repository itself being maintained in separate file system locations. When not specified, an empty repository path is written to the preferences file, causing Duplicacy to treat its current working directory as the repository root.

[Christoph]

Looks good to me. But @gchen might want to double check.

Duplicacy at the time it is executed

There is a possibility for misunderstanding here, isn’t there? Does this refer to the execution of the init command or any command?

[Nelvin]

@ben-ptl Thanks for adding this to the wiki.

With regards to the current working dir - I guess this should always be the folder where one wants the .duplicacy folder to be created. If you init the repository from a different folder (using the -prefs-dir option), you’ll get the .duplicacy file with the path to the prefs in your repository?

Running the backup is not possible from a different CDW anyway, as there’s no option to define the path to the prefs, if I didn’t miss anything?

[towerbr]

I’m back (short business trip). Direct to the point: problem solved.

• Changed the cd command in my script, replacing the repository with the folder where the preferences file is.

That’s odd. Did you cd to the .duplicacy folder?

I said I wasn’t realizing something obvious: then the folder where the preferences file is (let’s call it “settings folder”) has to be initialized … Well, that’s weird … but obviously It worked.

change the current workdir into one of these and then just init a repository

I’ve adopted another solution: instead of initialize, create the .duplicacy folder. etc, I simply moved the .duplicacy file from the repository to the settings folder.

Now I make a cd command to the settings folder, which finds the .duplicacy file, which points to … the setting folder …

Would not it be much simpler if Duplicacy identified the presence of the preferences file and adopted it?

Probably doesn’t make sense to use the two options (-pref-dir and -repository) as I’m doing. It may be better to re-adopt the .duplicacy folder + the -repository option, and abandon the use of the .duplicacy file. It no longer makes sense with the configuration completely centralized as it is now.

Anyway, it’s working … Thanks for the hints!

[ben-ptl]

@towerbr looking at the Duplicacy source code, I can see why this didn’t work for you originally. Duplicacy won’t read a preferences file in the current working directory. It will either 1.) look for a preferences file inside a .duplicacy directory in the current working directory, or 2.) if .duplicacy is actually a file in the current working directory, it will read its contents for the path to a directory (with any name), and then look for the preferences file there.

By putting a .duplicacy file inside your arbitrarily named preferences folder with that folder’s path as its contents, you’re triggering code path 2.) described above. The fact that the preferences file is in the same directory is immaterial. Duplicacy is just loading the preferences file from the path you pointed it to.

[ben-ptl]

Duplicacy at the time it is executed

There is a possibility for misunderstanding here, isn’t there? Does this refer to the execution of the init command or any command?

I agree this wording is a little unclear. From what I can see, the repository path is retrieved from the getRepositoryPreference function defined here. From skimming the source code and from what I’ve seen in practice, the repository path is primarily (only?) really used by the backup and restore commands. I’m not sure if I should clarify with something like the following:

Relative paths are relative to the current working directory of Duplicacy at the time it is executed with the backup or restore command.

… or if it’s even a good idea to mention this. Personally, I’m not sure if using a relative path is such a good idea for this option. In any case, it’s definitely not relative to the path at the time of running the init command. Whatever you specify as the path is entered directly in the preferences file (relative paths aren’t resolved prior to writing the repository key).

@ben-ptl Thanks for adding this to the wiki.

With regards to the current working dir - I guess this should always be the folder where one wants the .duplicacy folder to be created. If you init the repository from a different folder (using the -prefs-dir option), you’ll get the .duplicacy file with the path to the prefs in your repository?

Yes, I believe either the .duplicacy folder or file will always be created in the current working directory when you run the .init command. That doesn’t mean you can’t manually relocate/create/modify them after the fact, though…

Running the backup is not possible from a different CDW anyway, as there’s no option to define the path to the prefs, if I didn’t miss anything?

I think we’re saying the same thing. I believe you need to execute most commands (not init) from either 1.) A directory with a .duplicacy subfolder containing the preferences file or 2.) A directory with a .duplicacy file containing the absolute path to a folder containing the preferences file.

[gchen]

This -repository option was added to provide a simple way to separate the repository root from the directory hosting the .duplicacy/preferences file. So instead of adding a global -pref-dir option as implemented by this PR, we just need this -repository option for the init and add commands. Now you can cd into the directory where the .duplicacy/preferences file is, and run a command there, but the actual directory being backed up or restored will be the one that is stored in the .duplicacy/preferences file. As a result, the -pref-dir option for the init command won’t be necessary any more.

You can pass relative and absolute paths to this option, but I would suggest using absolute paths just to be safe.

[towerbr]

Just a feedback, I modified the whole configuration of my centralized settings folder to the format "-repository in preferences + cd to the settings folder" (and deleted the .duplicacy files from the repositories).

All the jobs are working perfectly and I think the configuration got “cleaner”.

[ben-ptl]

So now that we can separate the preferences from the repository itself, here’s an observation:

It seems you can have completely different snapshot ids, storage urls, repository paths, no_backup files, etc in each of the entries in the same preferences file, and select them with the -storage option and everything works fine. So we could be using the same preferences file, but be backing up completely different source paths to completely different destinations, all from the same preferences file.

When you use duplicacy -storage <my-storage-name> it seems more accurate to say you’re selecting a specific Duplicacy configuration in the preferences file by the value of its “name” key.

[Nelvin]

It seems you can have completely different snapshot ids, storage urls, repository paths, no_backup files, etc in each of the entries in the same preferences file, and select them with the -storage option and everything works fine. So we could be using the same preferences file, but be backing up completely different source paths to completely different destinations, all from the same preferences file.

Almost … what’s missing is an option to define the filters for each of these.

Hehe and just a few seconds ago I wrote on my feature request over at github that this is not needed anymore due to the -repository option

But it’s a great flexibility improvement already and to me it doesn’t matter whether I create a subfolder or another entry in a preferences file.

[Droolio]

It seems you can have completely different snapshot ids, storage urls, repository paths, no_backup files, etc in each of the entries in the same preferences file

THIS was the exact idea that crossed my mind while following this thread. IMO, it would be a very useful thing to be able to unify the configuration file…

The only thing I can see that’s required is to support a -filters option as proposed here. However, the irony is, as mentioned in the last post here, the -repository setting kinda makes different filters redundant (IF using separate .duplicacy directories).

However, I really do favour the ability to choose between unified, standalone, or repository-based .duplicacy configs. Because it isn’t far at all from being able to do it all. It just needs a way to specify filters.

Additionally, the proposed -filters option could instead take the form as an entry in the preferences file. OR preferably both: preferences or command line.

[Droolio]

Almost … what’s missing is an option to define the filters for each of these.

Heh, you beat me to it while I was typing out the above.

[towerbr]

It seems we all thought the same:

• an unified preferences file (it’s already possible);
• just missing an entry for specific filters (the more practical would be one more row in the preferences file).

[ben-ptl]

Almost … what’s missing is an option to define the filters for each of these.

Yep, you’re right, I overlooked that.

It seems we all thought the same:

• an unified preferences file (it’s already possible);
• just missing an entry for specific filters (the more practical would be one more row in the preferences file).

I agree. A “filters” key in the storage object could point to the path of a filters file. If the key doesn’t exist, Duplicacy could continue to look for a filters file in the .duplicacy folder.