Restore on a new computer in new folder is failing with error "File already exists. Please specify the -overwrite option to overwrite"

[hiavi]

Please describe what you are doing to trigger the bug:
I have a backup of a repository - only revision #1.
I am trying to restore it to a new HDD using the command

sudo duplicacy restore -r 1

I am using sudo because I am getting the following failure which as per gchen will happen because that file was created by a different user account and your current user account does not have the writable permission

Failed to change uid or gid: lchown .nomedia: operation not permitted

Please describe what you expect to happen (but doesn’t):
Clean restoration of while repository on new HDD

Please describe what actually happens (the wrong behaviour):
Restoration starts fine - but it is always failing on a given file. I deleting all files in the folder and rerunning the command but it is failing on same file consistently.

[towerbr]

“deleted all files in the folder” and “file already exists” are two strange statements applied in the same situation.

Maybe some cache issue?

Related topic?

Restore: skip existing files instead of overwrite

duplicacy 2.5 I do duplicacy -restore -r 1 "path/to/folder/*". Expected behaviour: Restore all files, skip the ones that already exist Actual behaviour: I get a message file.txt already exists. Please specify the -overwrite option to continue. Is there any way to restore and skipping existing files?

[hiavi]

Thanks @towerbr for replying on the thread.

@towerbr

“deleted all files in the folder” and “file already exists” are two strange statements applied in the same situation.

I even retried it by deleting the folder and trying the same command on a new folder - running into same failure saying that the file already exists - not sure how can a “file already exist” when the folder is completely new and I am trying to restore from a backup?

About being related to “Restore: skip existing files instead of overwrite” - I initially thought so - hence my ping on that thread

But on second thought, feel it is a different issue because @dreamflasher noted that he is trying to restore an existing folder with files to an older restore version # and hence it is understandable that he is hitting a “file already exist” issue - but I cannot fathom how it can happen when the folder is brand new.

[hiavi]

Note: I am running into the errors when using Duplicacy on Ubuntu running in Win10 WSL2 environment.

I tried to see if I can restore using GUI trial version running on parent Windows environment and I am getting the following error

Looks like the mystery is deepening

[hiavi]

I tried using Windows CLI to restore the same repository and I am getting the error of “The config file has an invalid banner”

When i did the same using Mac CLI (courtesy Paragon NTFS for Mac drivers allowing NTFS disc to be used fine on a Mac), I am hitting this error “Failed to download the configuration file from the storage: The config file has an invalid header”

Is the Storage folder OS specific? and hence cannot be used to restore from a different OS?

Nevertheless I am still having the issue of restore operation failing on Linux too with the weird “file already exists” error.

[gchen]

This invalid banner is usually caused not specifying the encryption option -e correctly when running the init command. If the storage is already encrypted you’ll need to specify the -e option.

[hiavi]

I did specify the -e option

Here is the output from my MAC

MyShell% duplicacy init -e My32GBDrive /Volumes/My\ Book\ Duo/My-WD16TB-AllBackupSafe
Enter storage password for /Volumes/My Book Duo/My-WD16TB-AllBackupSafe:***********
Failed to download the configuration file from the storage: The config file has an invalid header

Does the same error show up if the password is incorrect?

[kgorlen]

I’m getting the same error. Backup was done with the x86_64 QNAP qpkg. Restore to new folder on different, NTFS-format drive using duplicacy_win_x64_2.7.1.exe failed. Storage is not encrypted.

restore already exists.txt (5.1 KB)

[gchen]

This error actually means the storage was not encrypted but you specified the -e option. It should work if you remove the -e option.

[towerbr]

I didn’t understand … by the log he posted he managed to restore several files, and the restoration stops in the middle. If the cause was storage encryption, he wouldn’t be able to restore the initial files, right?

[kgorlen]

The -e option was not specified:

X:\Restored_Repositories\Public>duplicacy restore -r 19

[gchen]

Sorry for the confusion. I meant to reply to @hiavi and thought @kgorlen had the same problem.

@kgorlen you need to specify the -overwrite option. By default Duplicacy doesn’t overwrite existing files by default to avoid accidentally overwriting files that may not be backed up. However, if you have an incomplete file from an aborted restore operation Duplicacy can’t distinguish such a file from a normal file so you must provide -overwrite explicitly to overwrite it.

[kgorlen]

I reproduced the problem. The attached listing shows that the target directory is empty prior to the restore, so -overwrite should not be necessary, correct?

Or is something left over in .duplicacy/cache causing this problem?

restore already exists.txt (5.4 KB)

[kgorlen]

I was able to restore this file from the snapshot successfully using the qpkg web UI, without the -overwrite option, to the QNAP NAS, from which the snapshot was backed up and which is an ext4 filesystem. The failure occurs when restoring to an NTFS filesystem using the Windows CLI.

[gchen]

This is a bug and should be fixed by this commit: Fixed a bug that caused a fresh restore to fail without the -overwrit… · gilbertchen/duplicacy@bec3a0e · GitHub

This only happens to files larger than 100M bytes, for which Duplicacy will attempt to create a sparse file. But then it forgets about it and thinks this sparse file is an existing file and hence requires the -overwrite option.