Restoring encrypted files

[adam]

Say I have a 1.6TB backup on B2. And it’s encrypted. So, from the B2 website, all I see (when I try to browse the files) are the chunks.

And let’s say that my home server got fried, and the ONLY backup is B2. But I didn’t lose my application ID or account ID or password. How can I:

1. List all the files that are on the B2 backup, and/or list only those files matching wildcard criteria (e.g., “2018taxreturn*”).
2. Pick out a specific file and restore just that.

I tried searching for the answers in the forums but failed

[Droolio]

Duplicacy doesn’t yet have a pattern search but what you would do, if using the CLI, is list the snapshots of a particular repository ID then list -files -r <revision> of (typically) the latest revision to get a complete file list.

From that you can search/grep your pattern and then run a restore. The restore command does have a pattern option but you can only specify one and you probably want to make sure the file/folder exists beforehand. With linux you can do this in one or two commands.

[adam]

Thanks for the quick response, Droolio. So does this mean that if I do a list -files -r <revision> that it would reach out to B2 to see what was there, and return the results? Because I’d likely be doing that command if my server got wiped out, and I’d be starting from scratch. And would I have to remember how I set up my repository and re-create the folder structure before doing this?

[Droolio]

In a bit more detail, what you’d do is first create a new empty folder for the repository. Or, if you’re restoring files to a fresh but existing file structure, use that instead.

(Although be mindful that, restoring - say - a Windows user profile directly into a new Windows install, very probably wouldn’t work cleanly or as expected. That’s hardly a limitation of Duplicacy though, so be prepared for a bit of manual intervention after a restore, and I’d advise restore to a clean folder structure anyway.)

Then you’d:

cd "D:\Restore"
duplicacy init -encrypt <snapshot id> b2://bucketname
duplicacy set -no-backup
duplicacy list

Instead of list, using list -all will help you identify what revision numbers, but more importantly, what other snapshot id’s are available. Thus if you got the <snapshot id> wrong, you can correct it by editing the .duplicacy\preferences file with a text editor.

For a full restore:

duplicacy restore -stats -r <last_revision>

For a partial restore:

duplicacy list -files -id <snapshot id> -r <revision> > out.txt

Look through out.txt to find the full pathname or pattern for what you want to restore, e.g.:

duplicacy restore -stats -r <revision> "AppData/Roaming/Thunderbird/Profiles/zyxwvuts.default/*"

You can do all this armed with only the B2 url and storage password.

Give it a try - create an empty folder structure and test the process out for a restore. (Careful not to restore over your existing repositories. )

[adam]

Excellent - thanks, and I will try it with a spare Ubuntu VM I’ve got kicking around on my Windows desktop. THANKS!!

[adam]

First, duplicacy list -all returns nothing.

Second, how long should the list -files command take? On a B2 archive of 29k files and about 50G, it’s been running for over 3 hours…

[Droolio]

When you init'd repository2, did you add another storage after or specified a -storage-name? That’s the only reason I can think of that list -all wouldn’t return anything.

Re list -files - that shouldn’t take up much bandwidth at all. All it’s doing is downloading the revision 1 snapshot file and outputting the file list to a file. If you have access to another terminal, try tail -f out.txt to see what it’s up to. (Maybe it hung on an error, such as asking for credentials? Do a plain list first.).

[towerbr]

how long should the list -files command take? On a B2 archive of 29k files and about 50G, it’s been running for over 3 hours…

A 75 GB B2 bucket, with 2 snapshot ids, 42 revisions related to one snapshot e 18 related to the other, ~9k files in the first and ~5k files in the other.

Little more than a minute to run:

EXECUTION: 	 DUPLICACY	 -log -verbose	 list	 -storage [redacted]	 -files	    

2019-04-18 14:54:22.006 INFO REPOSITORY_SET Repository set to [redacted]
(...)
2019-04-18 14:55:52.897 INFO SNAPSHOT_STATS Files: 9991, total size: 75231613504, file chunks: 66707, metadata chunks: 7

[adam]

It was asking for credentials Which is odd, because I set them, using the duplicacy set command.

/home/adam/Downloads/repository - this is my test repository. The duplicacy executable is also in that folder (is that bad?).
There is a folder my repository there called .duplicacy and in it is a preferences file with DUPLICACY_B2_KEY and DUPLICACY_B2_ID set.
Running the command without the > out.txt shows that duplicacy is still asking for the ID and KEY. I thought it was default behavior to always seek that stuff out… Also, for my encryption key/passphrase, is the correct key name for that BACKUP_ENCRYPTION_KEY?

Also, I rm -rf'd my test repository, recreated, and re-init’d it. Same issues. Command was:
./duplicacy init -encrypt 1 b2://SLAVESERVER followed by
./duplicacy set -no-backup

Sooo confused

[Droolio]

Hmm, your confusion is entirely understandable. The relevant docs seem to have a formatting issue that’s hidden the last column in the table of env variables. Scroll right.

You need to use b2_id, b2_key, and password (all lowercase).

[adam]

Works However, ./duplicacy list and ./duplicacy list -all still produce nothing…

I also tried setting the password, but I’m not sure that duplicacy likes it when there are characters like ! . ^ \ or /, which is why you see me putting it in manually in the screenshot.

[Droolio]

Aye… because the preferences file is in json format, the strings need to be properly escaped. Double quotes and backslash needs to be prefixed with backslash - i.e. \" and \\. Not sure if there are any others. If in doubt, you can use a converter.

Puzzling why you don’t have any revisions listed. What is “name” set to in your preferences file? Does your B2 actually have any backups in there?

[adam]

Converter works thx. “name” is set to the value “default” for both the machine doing the backing up and my test VM. In fact, the preferences files are identical, except that the no_backup flag is true on my test VM.

I feel like I’m missing something so basic…… is there anything else I should be doing on the B2 side?

[gchen]

Do you have any files under the snapshots directory in the B2 bucket? It looks like you don’t have a completed backup yet.

[adam]

You’re right, I didn’t create one. So I went ahead and created a snapshot. But still can’t get list -files to work… Is there something I should be setting differently in the preferences with name and id (which are set to default and 1, currently)?

[Droolio]

You need to specify the revision number i.e. list -files -r 1.

If you have at least one snapshot now, doing a list -all should work and show you what revisions you got.

Your repository naming convention is a little confusion btw, since it says “Listing revisions for snapshot 1”, which might make you think that’s revision 1 when in fact that’s the name of repository id.

[adam]

I really appreciate your patience… I did the entire process from scratch… still no dice, and I feel a bit stupid Here’s the result:

[gchen]

What I meant was the directory named snapshots in your b2 bucket. You should see two directories there, chunks and snapshots, and a file named config. The chunks directory is where chunks are stored, while snapshot files are stored under the snapshots directory (which is unrelated to the snapshots created by b2). If your first backup didn’t complete, you may end up with a lot of files under chunks, but nothing under snapshots. I suspect this was what happened here.

[adam]

Like this?

[towerbr]

Nope, inside the “slaveserver” bucket must exist two directories: chunks and snapshots.

Like this: