"repository" init and add options

Support
#1

I’ve been doing some basic cli testing with Duplicacy. I came across the recent addition of the -repository option to the init and add commands. It seems to me that on a machine with multiple repositories, this would allow you to keep your repository preferences better organized. For example, keeping each preferences file in a subfolder of a single parent (e.g., ~/duplicacy/central-preferences/backup-one, ~/duplicacy/central-preferences/backup-two, etc). The actual repositories those preferences files represent could be in completely different file system locations.

This keeps the repository itself free of any duplicacy configuration files. For many use cases wouldn’t it eliminate the need for the init -pref-dir option? It’s kindof the opposite. The -pref-dir option leaves a .duplicacy file with the absolute path to the preferences directory in the repository, while the -repository option points back from the preferences file itself to the repository.

I’m trying to figure out if my understanding of this is correct and if there is any downside to using this option.

Thoughts?

References:

Added a -repository option to the init and add command to specify an alternate repository path

https://forum.duplicacy.com/t/version-2-1-1-has-been-released/1031

https://github.com/gilbertchen/duplicacy/commit/72239a31c42dd6e01521be19f45d13ed9caf635f

Add command details
#2

Another observation: I noticed that while a repository configured this way doesn’t normally contain any duplicacy related configuration files, when anything is restored to it an empty .duplicacy folder is created in the repository root.

I don’t believe there is any mention of this option in the wiki. @gchen @TheBestPessimist does my understanding of this option seem right? Is there any reason I wouldn’t want to use this type of configuration?

#3

If the intention is to keep all settings in a centralized location, I use this strategy with the -pref-dir option, and I don’t use the -repository option.

All my settings are in a centralized folder with a sub folder for each repository:

\ duplicacy \ preferences \ rep-name-1
\ duplicacy \ preferences \ rep-name-2
etc

Of course with that strategy there is a .duplicacy file in each repository pointing to the folder (above) where the settings are, but I think this has an interesting flexibility.

For example, to do a partial restore (some folders or files) in a temporary folder, just copy (or create) the .duplicacy file there and execute the restore.

This scheme also makes it easy to back up your settings. I use Rclone to backup the above folder (\ duplicacy \ preferences \), filtering out the cache folders.

1 Like
#4

@towerbr thanks for sharing. I like the idea of being able to back up your settings like that. I think you’d have the same advantage using just the -repository option.

Unless I’m mistaken, with your method you have to remember where the actual repository root is and execute duplicacy from that directory. There’s no reference to it in the preferences folder other than a naming convention you may choose for the folders themselves (e.g., “rep-name-1” in your case).

Using the -repository option you could use the same preferences folder scheme you mentioned, but execute duplicacy from each of the preferences subfolders.

@towerbr For example, to do a partial restore (some folders or files) in a temporary folder, just copy (or create) the .duplicacy file there and execute the restore.

I guess one minor downside to not using -pref-dir is losing the simplicity of copying or creating just the .duplicacy file for doing restores to a temporary folder.

#5

Thanks for pointing that out. I added the new options to the wikis on this forum (and change the links in the OP to point there as we are moving the documentation from github to this forum.

@TheBestPessimist shouldn’t the wiki texts on Github be replaced with a short notice and a link to the respective forum wiki?

I don’t think so. My understanding is that the -repository option just lets you create new repositories without having to navigate to the respective folder. It’s a kind of extended arm. Nothing more. After using it, you can’t tell that the repository was created using it. The -pref-dir option initiates the repository in a different way, i.e. with the config stuff in a different location. That is a difference that will persist.

#6

Hmm, it does seem like there’s a larger difference to me. It is true that you don’t have to navigate to the repository directory before creating it, but also that it creates the .duplicacy preferences directory in the current working directory instead of inside the repository directory. In addition, there is no .duplicacy file inside the repository directory that points back to the preferences folder. Inside the preferences file you’ll find that the storage object has a “repository” key that points to the full path of the repository root. Without -repository that key is just an empty string. For example:

{
“name”: “my-custom-store”,
“id”: “second-backup”,
“repository”: “/var/some/deep/path-in-the-file-system”,
“storage”: “/var/my-custom-store”,
“encrypted”: false,
“no_backup”: false,
“no_restore”: false,
“no_save_password”: false,
“nobackup_file”: “”,
“keys”: null
}

#7

With respect to backup settings, yes, it will work for both options.

I agree. But in my case (small number of repositories) this is not a problem. On the other hand you just gave me the idea of adding the line"repository": in my preferences files, even though I do not use it immediately. :wink:
EDIT: Nope, I can’t do this, or I will have problems with .duplicacy files.

As I use scripts, it’s going to be the same thing. Today I make a cd command to the repository. I would still have the cd command, but it would be to the preferences folder.

Well, IMHO it’s not a minor downside. Or, speaking by the other side, it’s a big advantage the flexibility that the .duplicay file gives to you. You can do a lot of things just copying / creating a tiny txt file…

#8

If I don’t miss anything here, completely removing any .duplicacy file or folder from repositories would enable us to have “multiple repositories” sharing the same root folder?
This would be great as it’d already solve f.i. my request to have an option for multiple filters for a repository to use in different situations/frequencies. I’d simply create multiple snapshots pointing to the same repository path but each using it’s own filter.

2 Likes
#9

Thanks for pointing that out. I should probably try things out instead of guessing how they work. :zipper_mouth_face: Woukd you mind adding a short explanation to the init-wiki?

If that is how it works, that would indeed be ingenious. Have you tried it?

Though see also this:

What kind of problems?

#10

Having the .duplicacy file in the “official” repository and the same repository indicated in the preferences file, I’m not sure what t will happen if I create a copy of the .duplicacy file in another folder, to restore some files, eg.

Will it “obey” the location of the copy of the .duplicacy file and restore the files to the temporary folder or will it restore to the original repository (indicated in the preferences file) and overwrite the files there? I have to do some tests …

1 Like
#11

I can appreciate the simplicity of that method. On the other hand, here is one way you could make restoring to a temporary folder from any of your repositories using this alternative scheme relatively simple:

Edit the preferences file for the repository you want to restore . Copy any of the existing storage objects (likely “default” if there is only one) to a new entry in the array. Change the storage name to something unique like “temporary-restore”. Change the repository to point to the temporary folder you want to restore to, let’s say “/var/temporary-restore-dir”. Then issue the restore command, specifying the name of this new storage. (e.g., duplicacy restore -r 1 -storage temporary-restore). Done.

You could precreate this “temporary-store” storage object in your preferences file for each of your repositories as standard practice and then simply change the “repository” path to point to whatever folder you want when it comes time for a restore.

The preferences file might look something like this:

{
“name”: “default”,
“id”: “my-obscure-deeply-nested-folder-backup”,
“repository”: “/var/some/deep/path-in-the-file-system”,
“storage”: “/mnt/big-external-hdd”,
“encrypted”: false,
“no_backup”: false,
“no_restore”: false,
“no_save_password”: false,
“nobackup_file”: “”,
“keys”: null
},
{
“name”: “temporary-restore”,
“id”: “my-obscure-deeply-nested-folder-backup”,
“repository”: “/var/temporary-restore-dir”,
“storage”: “/mnt/big-external-hdd”,
“encrypted”: false,
“no_backup”: false,
“no_restore”: false,
“no_save_password”: false,
“nobackup_file”: “”,
“keys”: null
}

2 Likes
Backup from specific directory without cd
#12

:grinning: Yeah, it was an interesting find to see this mostly undocumented feature when playing around with the cli. I’ve been using this application for less than a week, so I may need a minute to muster up the nerve to edit the wiki without feeling like I’m not qualified to make that kind of contribution…

1 Like
#13

That’s fine. But you’ll see that the wiki has absolutely no infirmation yet, about how the option works (except for the basic -help description), so it’s easy to improve it :wink:
And if you’re unsure, you can always flag the post for staff attention after editing.

#14

I have no idea, never knew about it. Although it seems the discussion here is pretty long and insightful.

First we have to receive @gchen’s blessing for this and afterwards i’ll do it with pleasure!

#15

He seems to be quite busy with other things these days. If the github wikis have an edit history that makes it easy to undo edits, I see no problem with going ahead and making the change. Gilbert can revert if he disagrees. Still having the github wiki just creates additional work for us if people link to it and we want those links to point to the forum Wiki…

#16

But if he (yet) doesn’t want to -> we would’ve both made useless work.
Let’s just wait for a little more time to speak his mind now that he was announced about our intention :slight_smile: .

#17

Liked it! Is really a simple scheme and makes the settings even more centralized, without the use of .duplicacy files. I think I’ll implement it … :wink:

#18

Well, or I’m missing something very obvious or is not so simple …

  • I edited the preferences file ("repository": "",) and added the path to repository;
  • Removed the .duplicacy file from repository;
  • Changed the cd command in my script, replacing the repository with the folder where the preferences file is.

and when I execute a backup…

ERROR REPOSITORY_PATH Repository has not been initialized

If I return the .duplicacy file to repository and also return the cd command (pointing to repository), even without removing the repository path from the preferences file, backup performs perfectly.

Any ideas?

#19

That’s odd. Did you cd to the .duplicacy folder? Because the preferences file should be inside the .duplicacy folder, and you should actually be changing to the parent directory (one level above) the .duplicacy folder that contains the preferences file.

I would also suggest double checking the repository path is valid, properly quoted, etc. Maybe test this out by using the init command to create a new temporary backup that uses the -repository option to refer to the actual root folder you’re backing up, but have the storage be a local temporary directory. Then inspect the formatting of the resulting preferences file to verify your manual edits look the same. Of course then at least start a backup to the temporary storage to make sure it’s working :smile:

I don’t see why this wouldn’t work the same for you as it is for me. I’ve tested on Linux and Windows so far, and it all looks good.

#20

I just gave this a try and it seems to work as expected :slight_smile:

All you need to do is create a dedicated folder for your preferences, then create a subfolder for each of your, let’s call it “virtual repository”, change the current workdir into one of these and then just init a repository as usual just with an added -repository pointing to the rootfolder of wherever you want to base your virtual repository at.

This way, no .duplicacy file will be created in the actual folder of your repository, deduplication works great as usual and you can have a different set of filters for each of these virtual repositories.

Obviously each virtual repository needs it’s own snapshot-id.

I wonder how reliable this is as it’s a new option, but in my quick local test, everything went fine.

1 Like