Duplicacy Web UI OSX won't restore files

tateburns · 13 October 2020 13:38

Greetings, I’ve tried searching the forum for potential causes and solutions. I’ve performed a multi-terabyte backup over SFTP to and am unable to restore any files to a new location locally.

OS: Mac OS 10.15.6

The file list load without any issues from the backup. I’m attempting to restore to an alternate location. Whether I choose to restore a whole directory or individual file I get the same message:

All files to be restored are up-to-date

Here is the log output. I verified that Duplicacy Web has full disk permissions arleady.

2020-10-12 20:50:09.259 INFO REPOSITORY_SET Repository set to /Volumes/Test/Restore
2020-10-12 20:50:09.259 INFO STORAGE_SET Storage set to sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage
2020-10-12 20:50:09.293 INFO SNAPSHOT_FILTER Loaded 1 include/exclude pattern(s)
2020-10-12 20:50:09.297 INFO RESTORE_INPLACE Forcing in-place mode with a non-default preference path
2020-10-12 20:50:17.205 INFO SNAPSHOT_FILTER Parsing filter file /Users/user/.duplicacy-web/repositories/localhost/restore/.duplicacy/filters
2020-10-12 20:50:17.205 INFO SNAPSHOT_FILTER Loaded 0 include/exclude pattern(s)
2020-10-12 20:50:17.206 INFO RESTORE_START Restoring /Volumes/Test/Restore to revision 576
2020-10-12 20:50:17.255 INFO RESTORE_END Restored /Volumes/Test/Restore to revision 576
2020-10-12 20:50:17.255 INFO RESTORE_STATS Files: 0 total, 0 bytes
2020-10-12 20:50:17.255 INFO RESTORE_STATS Downloaded 0 file, 0 bytes, 0 chunks
2020-10-12 20:50:17.255 INFO RESTORE_STATS Total running time: 00:00:08

gchen · 13 October 2020 14:38

Can you find the relevant logs in ~/.duplicacy-web/logs/duplicacy_web.log and post them here? Here is an example:

2020/10/13 10:32:30 Running /Users/gchen/.duplicacy-web/bin/duplicacy_osx_x64_2.7.1 [-log restore -r 11 -storage local-test -overwrite -stats -- dir1/file1]
2020/10/13 10:32:30 Set current working directory to /tmp/duplicacy/repositories/localhost/restore
2020/10/13 10:32:30 Restored 1 files for backup test revision 11 in the storage local-test
2020/10/13 10:32:30 closing log file restore-20201013-103230.log

tateburns · 13 October 2020 15:10

Here is the relevant snippet from the duplicacy_web.log for the restore. It seems like it’s not honoring my selection for an alternate directory among other possible issues.

2020/10/13 10:01:00 Created log file /Users/user/.duplicacy-web/logs/restore-20201013-100100.log
2020/10/13 10:01:00 Created restore session ka5sn
2020/10/13 10:01:00 Running /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 [-log restore -r 590 -storage Server -overwrite -stats --  Directory/Movie.mov]
2020/10/13 10:01:00 Set current working directory to /Users/user/.duplicacy-web/repositories/localhost/restore
2020/10/13 10:01:08 Restored 0 files for backup Storage revision 590 in the storage Server
2020/10/13 10:01:08 closing log file restore-20201013-100100.log
2020/10/13 10:01:12 127.0.0.1:56817 GET /show_log?name=restore-20201013-100100.log
2020/10/13 10:01:12 127.0.0.1:56817 GET /favicon.ico
2020/10/13 10:01:28 Deleted restore session ka5sn

gchen · 13 October 2020 16:51

The restore destination directory isn’t shown in duplicacy_web.log. You can confirm this by running these commands in Terminal:

cd /Users/user/.duplicacy-web/repositories/localhost/restore
/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -log restore -r 590 -storage Server -overwrite -stats -- Directory/Movie.mov

Does the file name or path of Directory/Movie.mov contain any special characters? Some special characters may cause the pattern matching to fail.

tateburns · 13 October 2020 17:01

Unfortunately no special characters. Regardless of any file or folder from any backup revision I try to restore to any alternate location it consistently restores no files.

gchen · 13 October 2020 17:23

Can you run the list command as follows to see if that file is contained in the backup?

cd /Users/user/.duplicacy-web/repositories/localhost/restore
/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 list -r 590 -storage Server -files | grep Movie.mov

tateburns · 13 October 2020 17:47

after I cd into the directory specified and run the list command, I don’t get any results for any files in any revision. I know things are backing up because the Revision in the web UI lists all my files and my server has terabytes worth of chunks stored on it.

I tried stepping back through a few revisions and grepping for files I see listed in web UI and get no result. I feel like I’m doing something catastrohically wrong or missing some sort of basic concept at this point.

Should I be able to restore any file in a revision regardless if it has changed since my first backup?

EDIT: The answer to that is yes. I’m testing an identical setup on a Windows 10 host backing up files via SFTP and restoring works flawlessly. Can select any file in any revision and restore to alternate location.

gchen · 13 October 2020 18:04

If the web UI lists that file but you can find it with the list command, it is possible that a different backup id was used. Open the file /Users/user/.duplicacy-web/repositories/localhost/restore/.duplicacy/preferences and check the value of id under "name": "Server". This id should match the backup id you selected on the restore page. If not, you can manually edit this file to change the id to the correct one. Then list --files should be able to show the file. Once the file is found, run this restore command to restore file (with -d for more information):

cd /Users/user/.duplicacy-web/repositories/localhost/restore
/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 590 -storage Server -- Directory/Movie.mov

tateburns · 13 October 2020 20:23

Before I get in the weeds manually editing the preferences files, I’ve made a discovery. I have two backup ID’s backing up to two totally separate backup sets to different SFTP storage locations. I just tried a few files in revisions in my other backup set and they all restore flawlessly.

I went back to the one I’ve been struggling with and actually found that a very small amount of the files will actually restore without issue. For reference, though, this one I’m having trouble with is an 8TB+ backup set that took about 2 days to run the first time I ran it. I didn’t see any errors at the time reported in the Web GUI. It almost seems like there might be some corruption somewhere? It’s consistent across all the revisions that some files (very few) will restore and most will not.

EDIT: Even more the inability to restore appears to have some sort of alphabetical sorting preference for what will and won’t restore. For example, the folder structure I’m backing up goes something like this:

ASSETS
DOCMENTS
MASTERS
PROJECTS
Project Backups
Purchased

Everything from Project Backups down the tree will restore perfectly fine. Everything above that in ASSETS, DOCUMENTS, etc. won’t

gchen · 14 October 2020 00:11

To narrow down the issue, I think you should initialize /Volumes/Test/Restore as a repository and work there directly:

cd /Volumes/Test/Restore
/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 init backup_id sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage

backup_id should match what is used in the web GUI.

Then try to restore a file under, say, ASSETS:

/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 restore -r revision -- path/to/file

Note that if the file path contains spaces you’ll need to enclose it with double quotes:

/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 restore -r revision -- "path with spaces/file"

If restore fails with an error, please copy and paste the output here. If it completes without an error and without restoring the file, run the list --files command to see if the file is in the backup.

tateburns · 14 October 2020 00:32

Here is the output from the above. It first let me know the storage was already initialized, ran the restore without error but didn’t restore anything, then does not list the file. I tried grepping for the full path and just the filename. For reference, this file is available in the GUI for revision 598.

user@Mac Restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 init Storage sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage

Enter SSH password:**************

The storage 'sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage' has already been initialized

Compression level: 100

Average chunk size: 4194304

Maximum chunk size: 16777216

Minimum chunk size: 1048576

Chunk seed: 6475706c6963616379

/Volumes/Test/Restore will be backed up to sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage with id Storage

user@Mac Restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 restore -r 598 -- "ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg"

Storage set to sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage

Loaded 1 include/exclude pattern(s)

Parsing filter file /Volumes/Test/Restore/.duplicacy/filters

Loaded 0 include/exclude pattern(s)

Restoring /Volumes/Test/Restore to revision 598

Restored /Volumes/Test/Restore to revision 598

Total running time: 00:00:16

user@Mac Restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 list -r 598 -storage Storage -file | grep "ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg"

user@Mac Restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 list -r 598 -storage Storage -file | grep "texture-2060917_960_720.jpg" 

user@Mac Restore %

gchen · 14 October 2020 00:55

It should be list -files instead of list -file. I had list --files in previous posts which is also wrong.

tateburns · 14 October 2020 14:26

Thanks that did list the file succesfully. Without grepping the list I can pretty easily scan with my eyes and see that this revision contains all the files I would expect. Same with the Web GUI of course like we’ve already confirmed. I’m quite stumped where to go now! We know the file is in the revision, other files from that revision restore successfully through the GUI and CLI.

user@Mac restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 list -r 598 -storage Storage -files | grep texture-2060917_960_720.jpg

  251208 2018-12-22 04:13:04 d6351589fa5822a57395d707fb7193a8a5e33df84f3d76f19372a023b6ab9383  ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg

gchen · 14 October 2020 14:46

Can you add the -d option to the restore command and try again:

/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg"

tateburns · 14 October 2020 15:01

Does not restore the file as expected. It rips through all the files stating they’re all excluded. Including the one I’m trying to restore:

ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is excluded

Interesting it seems like the filter won’t work on these files?

gchen · 14 October 2020 15:25

Can you post the exact restore command that you used? ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is excluded means that file already exists but somehow you didn’t specify it in the command line.

tateburns · 14 October 2020 15:39

This is the command. It still excludes the file.

/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg"

This identical command, just specifying a different file, doesn’t exclude and it restores succesfully.

/Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "Purchased/Training/Lynda.com.Premiere.Pro.Commercial.Editing.Techniques/Exercise/Media/05_07_TrackCM1_4B1_C_INS.mp4

tateburns · 14 October 2020 15:54

Reviewing the output (it’s quite large, 45MB of text) I see this in the beginnging that appears to load the filter.

user@Mac Restore % /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "ASSETS/Smith Slideshow/MEDIA/Random/2018-10-18 22.35.25.jpg"\
Storage set to sftp://root@X.X.X.X//mnt/user/Duplicacy/Storage\
Reading the environment variable DUPLICACY_SSH_KEY_FILE\
Attempting password login\
Reading the environment variable DUPLICACY_SSH_PASSWORD\
Reading ssh_password from keychain/keyring\
Reading the environment variable DUPLICACY_SSH_PASSWORD\
There are 0 compiled regular expressions stored\
Loaded 1 include/exclude pattern(s)\
Chunk read levels: [1], write level: 1\
Compression level: 100\
Average chunk size: 4194304\
Maximum chunk size: 16777216\
Minimum chunk size: 1048576\
Chunk seed: 6475706c6963616379\
Hash key: 6475706c6963616379\
ID key: 6475706c6963616379\
top: /Volumes/Test/Restore, revision: 598, in-place: true, quick: true, delete: false\
+ASSETS/Smith Slideshow/MEDIA/Random/2018-10-18 22.35.25.jpg\

Then later on it appears to not load any filters:

Parsing filter file /Volumes/Test/Restore/.duplicacy/filters\
There are 0 compiled regular expressions stored\
Loaded 0 include/exclude pattern(s)\

towerbr · 14 October 2020 16:26

I think these -- are not necessary, and a + is missing:

... restore -r 598 "+ASSETS/Smith ...

gchen · 14 October 2020 16:35

The specified file should be included if it is entered correctly in the command line:

> /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg" | grep 2060917
+ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg
ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is included by pattern +ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg
ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is included by pattern +ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg
File ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg unchanged (by size and timestamp)

But if you have a typo in the file name (e.g., the trailing zero is missing) then the file will be excluded (which is likely what you’re getting):

> /Users/user/.duplicacy-web/bin/duplicacy_osx_x64_2.6.2 -d restore -r 598 -- "ASSETS/Smith Slideshow/GFX/texture-2060917_960_72.jpg" | grep 2060917
+ASSETS/Smith Slideshow/GFX/texture-2060917_960_72.jpg
ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is excluded
ASSETS/Smith Slideshow/GFX/texture-2060917_960_720.jpg is excluded