.duplicacy file (-pref-dir) should override repository setting

[akvarius]

( Edit: I now think this is a bug, and changed the category and title again, sorry!
.duplicacy file (pointer file) in data folder to be backed up is used to point to a preferences folder. This works well, except when the repository setting in the preferences file is used. I think the .duplicacy file should have pesedence
This affects all commands, not only backup.)

Hi!
(This is about CLI )
Thanks for an already great backup utility!

I know this has been discussed before and I believe developer has some ideas already in pipeline! But I tested a method with the current feature set.
I tested restoring to a custom folder (on same computer), using a pointer to the existing preferences path.
(If new or different computer, you need to init a repository/storage first)

A) Assuming you are not using the -repository option, this seems to work well:

1. Create the custom folder you want to restore to, and create a file ‘.Duplicacy’ with the path to the preferences folder (As if you used -pref-dir, including the ‘.duplicacy’ subfolder in the preference folder) No need to init again. (limited testing though…)
2. With the custom folder as working directory, issue the Duplicacy commands for list, restore (add pattern to specify what you want) etc.
3. When done, remove the file ‘.duplicacy’ from the custom folder

B) But let’s say the preferences folder was init’ed with option -repository pointing to the ‘real’ folder to be backed up. (I am using this as my preferred setup). Now, even if you issue the restore command from a custom folder with a .duplicacy pointer to preferences, the repository settings in the ‘preferences’ file overrides and will cause the restore to write to the original backup folder instead of the custom you want.

To work around this, I can either

• Temporarily remove the repository line from the preferences section for the storage, and add it back after restore
  CONs:
  If a backup job is scheduled to run at this time, that backup will now use the (empty) preferences folder.
  Also, forgetting to re-apply the setting (problem in users script? or forgetfulness) will cause later problems

Or I can:

• Copy the preferences folder to a temporary folder, and remove the repository line from the ‘preferences’ files.
  CONs: The cache folder might contain a large amount of chunks, which now takes double space on your disk.
  If you instead copy without the cache, downloads might incure delays/cost. (not a problem with local backup storage)
  But at least this does not disturb the original preferences file and backup jobs

May I suggest that

1. .duplicacy -pref-dir file get priority over the repository setting in the preferences file.
   This would be enough to solve my problem without having to duplicate or manipulate preferences.

2. For completeness, and ease of use, the restore command could also get the -repository option to override configured repository (backup) folder.
   (-restore-to has been suggested but I feel -repository is more consistent, corresponds to the similar option in init and add. But the name of the option is not very important to me )

3. I then suggest this priority if several config options are present:

• Commandline: restore -repository path (highest presedence)
• .duplicacy file with preference path in current folder
• Repository setting in the preferences file

(Perhaps existing files in the (real) repository could still be used as chunk sources!)

Thanks again!

[towerbr]

@akvarius

B) But let’s say the preferences folder was init’ed with option -repository pointing to the ‘real’ folder to be backed up. (I am using this as my preferred setup).

I also use this configuration, because that way I have all the settings in a single folder, (with several subfolders, one for each repository). Very easy to manage and backup (with Rclone).

After going through basically the same steps that you described, I decided to adopt the simplest solution. I simply start the repository with

duplicacy init^
 -encrypt^
 -storage-name STORAGE_WHERE_THE_FILES_ARE^
 -repository CUSTOM_FOLDER^
 _ID_TO_RESTORE^
 PATH_TO_STORAGE_WHERE_THE_FILES_ARE

(which is basically your option A, but using the -repository option)

and run the restore normally.

And then:

@akvarius

When done, remove the file ‘.duplicacy’ from the custom folder

[akvarius]

Hi @towerbr, yes, you probably have the same scenario I have (I probably learned from you!), one main “settings” folder with “backup set definitions” below it, each with a .duplicacy folder with preferences and cache etc.

Settings\
  <backupset1>\
    .duplicacy\
      keyring
      preferences
      cache\
        <storage name>\
          cunks
          snapshots
  <backupset2>\
    .duplicacy\
  (...etc)

But I think maybe I didn’t explain it well…
With option A) you don’t have to init a new preferences folder/file! Just put a pointer to your existing .duplicacy preferences folder in the restore target folder (like with -pref-dir), This target folder is not the preferences folder, it is the actual target. (only it does not work since repository setting overrides the pointer file current folder as target)

If you make a new repository:

• You must enter ID, encryption passwords and storage details again (of course we must know these details but that’s not the point! We already configured this)
• You will not benefit from existing cached chunks in .duplicacy\cache\chunks except if you include these in the copy. (The cache folder might be very large - my photo backup cache is almost one GB for natural reasons)

Your method (new preferences folder) is actually more similar to my option B/2, only that I just copied instead of init. Copy preferences folder to a new definition folder with preferences file, then delete the preferences ‘repository’ setting. (or in your case, modify it to point to restore target instead of a .duplicacy pointer file)

So - I already know how to work around this with the current limitation, but it just occurs to me that since a -pref-dir pointer file from a custom restore folder is already supported, maybe it isn’t very difficult to at least give it priority over the preference ‘repository’ setting, and maybe also add -repository option to the restore command.

This will make it easier to restore to custom folder without having to manage a new repository/settings. A bonus is that this would benefit from existing cached chunks in the preference folder without making extra copies!

I think it would be a useful improvement to the tool!
(Since GUI uses CLI it might have benefits there too)

[akvarius]

This bug report is primarily for the priority of conflicting settings.

See this feature request for a -repository option:

Feature request: "-repo" option for backup command

For automating unattended backups with Duplicacy CLI I feel it would be convenient to pass the “repo to be backed up” as a parameter to the backup command, rather than requiring the working directory to be the repo itself. Obviously I would not ask that the current working directory functionality be removed, only that something like duplicacy backup -repo /path/to/repo be an option that is supported in addition. Giving it a second thought, I’m not sure how exactly this would be superior to the …

[gchen]

-pref-dir is going to be deprecated. -repository is recommended.

[akvarius]

Thanks for the reply and heads up on the -pref-dir functionality being deprecated!
I just considered it for evaluating methods for restore, I’m happy to mark this -pref-dir precedense issue as solved or similar (cannot see how to do that for bug #site-feedback)

@gchen

-repository is recommended.

Not sure if mean that you are indeed considering -repository option as command line to override the stored repository setting for restore?

I’m happy to make a feature post for it!
(also voting for -dry-run for restore!)

[gchen]

I meant the -repository option for the init command. I think a -repository option for the restore command is not safe – restore should always only work on the directory specified in the init command otherwise it is very easy to make unrecoverable mistakes.

[akvarius]

OK,
would a command line option to redirect restore to an empty folder within the repository be OK? (or even outside repository, as far as it’s empty)
This would reduce risk of destroying something. Perhaps -export is a better name for what I try to do.

With the current options, Init implies a new “pseudo folder”.
Will using add with -repository continue to be supported?
This allows restore to make use of existing cached chunks. (I would probably set “no_backup” for that storage definition)

I feel bad for pestering you with restore, I respect your repository-oriented design and will respect whatever you think but I feel there is a feature gap here…

A restore to the repository (the folder we want to protect!) is an inherently dangerous operation. Requires -overwrite to replace existing files of course, but still I would think many would feel it’s safer to restore (export?) a few files to a separate “restore folder” instead of directly to the live data.

Especially since restore -dry-run is not an option yet. You risk loosing all changes since last backup if you get pattern wrong. I’m quite familiar with regular expressions and love your use of it, but it’s still a potential for user errors, and I have to admit I might need a couple of tries before I get it right!