Restore from second Storage (copied)

cVillabrille · 23 March 2018 11:35

Hi,

I’m an Unix enthusiastic That makes bit difficult finding a good backup solution that could work in the variety of Linux world (Intel/ARM, Debian/RedHat, etc). Thanks for developping such a nice tool for achieving a backup (with encryption option).

I’m still under the testing phase hence, here it comes the question I would like to know whether it’s possible restoring directly from a secondary storage (configured as copy from the first) as I’m trying to do it but doesn’t matter the atetmpts, I always receive the error (i.e) “Snapshot 1 at revision 3 does not exist”.

i guess the error comes due to the second storage copies the data from the source with same Snapshot IDs (in this case Snapshot 0). But restoring, I cannot find a way to explicity specify the Snapshot ID on the revision.

Now some tech stuff…

Here it is how I have created the Storage.

/mnt/OpenDrive is a webdav through rclone mounted FS for OpenDrive
hubic, well, I guess is clear

Initial respository
# duplicacy init -e -storage-name openDrive_rclone 0 /mnt/OpenDrive/Duplicacy/Odroid/
 
Add second respository as copy from the first
# duplicacy add  -e -copy openDrive_rclone hubic 1 hubic://Duplicacy/oDroid/

Add the token file
# duplicacy set -key hubic_token -storage hubic -value /data/Downloads/hubic-token.json

Set password 
# duplicacy set -key hubic_password -storage hubic -value "super_secret_password”


# duplicacy set -key password -storage openDrive_rclone  -value "another_super_secret_password"

Run the backup 
# duplicacy backup -stats -threads 10

Copy the backup to the second storage
# duplicacy  copy -from openDrive_rclone -to hubic_rclone -threads 5

Now I have removed some files and tried to restore from the second storage (hubic). You can ask, why??.. well, Imagine tomorrow OpenDrive stops offering their service (like Amazon did) and I have to cancel the service. I want to be able to restore from the secondary Storage as I can from the primary.

However…

Here is the output when I list the secondary storage:

root@dataDroid:/data/duplicacy_tests # duplicacy list -a -storage hubic
Storage set to hubic://Duplicacy/oDroid/music/
Snapshot 0 revision 1 created at 2018-03-23 09:37 -hash
Snapshot 0 revision 2 created at 2018-03-23 09:55
Snapshot 0 revision 3 created at 2018-03-23 10:28
Snapshot 0 revision 1 created at 2018-03-23 09:37 -hash
Snapshot 0 revision 2 created at 2018-03-23 09:55
Snapshot 0 revision 3 created at 2018-03-23 10:28

Tried to restore a folder with 2 mp3 of "The Killers"
root@dataDroid:/data/duplicacy_tests # duplicacy restore -r 3  -stats -storage hubic  -- *Killers*
Storage set to hubic://Duplicacy/oDroid/music/
Snapshot 1 at revision 3 does not exist

I cannot specify the Snapshot ID, and, as this second storage is a copy and it has the Snapshot ID 0.

Is there a way to specify the Snapshot ID I’m missing?.

How I can restore from this second Storage as I can from the primary?

gchen · 23 March 2018 18:35

There is something wrong with your hubic storage. The id of the second set of revisions should be 1 instead of 0.

root@dataDroid:/data/duplicacy_tests # duplicacy list -a -storage hubic
Storage set to hubic://Duplicacy/oDroid/music/
Snapshot 0 revision 1 created at 2018-03-23 09:37 -hash
Snapshot 0 revision 2 created at 2018-03-23 09:55
Snapshot 0 revision 3 created at 2018-03-23 10:28
Snapshot 0 revision 1 created at 2018-03-23 09:37 -hash
Snapshot 0 revision 2 created at 2018-03-23 09:55
Snapshot 0 revision 3 created at 2018-03-23 10:28

How many directories are under Duplicacy/ODroid/music/snapshots if you access it from the hubic website?

cVillabrille · 25 March 2018 18:53

HI Gchen,

thanks for your quick reply.

Touché, there are two folders, 0 and 1 (saw over hubic website). However, folder 1 is empty while 0 is full of data.

I understand this is due to it is a copy hence it should have same Snapshot 0 as I’m copying form one to another (not doing a new backup in the second).

To be honest, If I follow the recommendations, Duplicacy recommend runing an initial backup in both respositories (independently. I mean, one on each on the first time) and then just run copy from one to another ( I just copy directly from one to another) to avoid download and upload. As I do not have penalty on any download/upload, I tested first this option of copying.

Is that a limitation on this copy command?. I understand that when copying it should number it with 1 (instead of 0).

I will test deleting all data from Hubic and performing a full initial backup, then just copy. (Shall I understand this is the initial way of doing two repositories backup keeping different IDs?)

Bellow you can find the preferences file to see the different IDs in both Storages:

# cat .duplicacy/preferences
[
    {
        "name": "openDrive_rclone",
        "id": "0",
        "storage": "/mnt/OpenDrive/Duplicacy/Odroid/music/",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "keys": {
            "password": "XYXYXY"
        }
    },
    {
        "name": "hubic",
        "id": "1",
        "storage": "hubic://Duplicacy/oDroid/music/",
        "encrypted": true,
        "no_backup": false,
        "no_restore": false,
        "no_save_password": false,
        "keys": {
            "hubic_password": "xxxxx",
            "hubic_token": "/data/Downloads/hubic-token.json"
        }
    }

cVillabrille · 25 March 2018 21:11

i will perform some more test and post here the results

cVillabrille · 26 March 2018 00:19

I guess I have found the issue.

Enabling the repositories, I have created as repository_id 0 for OpenDrive while 1 for hubic.

Well, that causes of course an issue with the storage naming and the respositories.

In order to have a copied repository, you must keep both respository_id as the same, otherwise the backup will be different (Note the repository_id is odroid_music in both storages).

duplicacy init -e -storage-name openDrive_rclone odroid_music /mnt/OpenDrive/Duplicacy/Odroid/music2/
duplicacy add  -e -copy openDrive_rclone hubic odroid_music hubic://Duplicacy/oDroid/music2/

Initial backup to Primary storage

duplicacy backup -storage openDrive_rclone -stats -threads 10

Initial backup to Secondary storage

 duplicacy backup -storage hubic -stats -threads 10

copy a new file and backup

cp ../01-Mr.Brightside.mp3 .
duplicacy backup -storage openDrive_rclone -stats -threads 10

copy to hubic

 duplicacy  copy -from openDrive_rclone -to hubic -threads 10

Check and remove a file

# ls
01-Mr.Brightside.mp3  11-Everything_Will_Be_Alright.mp3
# rm 01-Mr.Brightside.mp3

List storage:

# duplicacy list -a -storage openDrive_rclone
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music2/
Snapshot odroid_music revision 1 created at 2018-03-26 01:33 -hash
Snapshot odroid_music revision 2 created at 2018-03-26 01:39
# duplicacy list -a -storage hubic
Storage set to hubic://Duplicacy/oDroid/music2/
Snapshot odroid_music revision 1 created at 2018-03-26 01:36 -hash
Snapshot odroid_music revision 2 created at 2018-03-26 01:39
Snapshot odroid_music revision 1 created at 2018-03-26 01:36 -hash
Snapshot odroid_music revision 2 created at 2018-03-26 01:39

Try to restore:

# duplicacy restore -r 2  -stats -storage hubic  -- *Brightside*
Storage set to hubic://Duplicacy/oDroid/music2/
Restoring /data/Downloads/Torrents.2/duplicacy/Music to revision 2
Downloaded chunk 1 size 3445940, 481KB/s 00:00:12 38.5%
Downloaded chunk 2 size 4626718, 657KB/s 00:00:02 90.3%
Downloaded chunk 3 size 863794, 582KB/s 00:00:01 100.0%
Downloaded 01-Mr.Brightside.mp3 (8936452)
Restored /data/Downloads/Torrents.2/duplicacy/Music to revision 2
Files: 1 total, 8.52M bytes
Downloaded 1 file, 8.52M bytes, 3 chunks
Total running time: 00:01:05

It works. Remember to keep same respository_id once creating the configuration.

Still do not understand why listing the second storage appear twice the revisions… but this is another story:

# duplicacy list -a -storage hubic
Storage set to hubic://Duplicacy/oDroid/music2/
Snapshot odroid_music revision 1 created at 2018-03-26 01:36 -hash
Snapshot odroid_music revision 2 created at 2018-03-26 01:39
Snapshot odroid_music revision 1 created at 2018-03-26 01:36 -hash
Snapshot odroid_music revision 2 created at 2018-03-26 01:39

gchen · 26 March 2018 03:17

It is ok to have the same or different repository ids for different storages; either way should work.

I think the problem is with the hubic storage. It is very slow so Duplicacy may upload another snapshot file even if another with the same file name already exists.

How much files are under Duplicacy/oDroid/music2/snapshots/odroid_music? Are there two files named 1 and two named 2?

cVillabrille · 26 March 2018 07:49

Hi Gchen,

yes, through hubic website, I can see there are two folders in music/snapshots, 0 & 1:

hubiC/Duplicacy/oDroid/music/snapshots/0 --> two files, called 1 & 2.
hubiC/Duplicacy/oDroid/music/snapshots/1 --> one file, called 1.
in OpenDrive/music/snapshots/0 --> two files, called 1 & 2.

I have performed many tests and when the naming is different, I cannot restore from Secondary.

Well, If we check carefully the official documentation, it is using “my-backups” as respository_id on both respotories (Back up to multiple storages):

If you’re backing up to two cloud storage providers, issuing a copy command will cause all data to be downloaded from one cloud storage, and uploaded to the other. This can be slow, and may incur extra download costs. To avoid this - while maintaining duplicate backups - you can issue a “fake” backup to the destination cloud storage which will hopefully have many duplicate chunks with the “real” backup. The copy operation after will then copy the minimal amount from the source cloud storage.

duplicacy init my-backups --storage-name backblaze b2://bucket
duplicacy add -copy backblaze wasabi my-backups wasabi://bucket
duplicacy add wasabi-temp my-backups-temp wasabi://bucket
duplicacy backup -storage backblaze
duplicacy backup -storage wasabi-temp
duplicacy copy -from backblaze -to wasabi

Sorry to extend this threath a bit, but here there is an example with different naming (0 and 1 as respository_id) with no luck restoring.

initialized storage and preparations

# duplicacy init -e -storage-name openDrive_rclone 0 /mnt/OpenDrive/Duplicacy/Odroid/music
# duplicacy add  -e -copy openDrive_rclone hubic 1 hubic://Duplicacy/oDroid/music
# duplicacy set -key hubic_token -storage hubic -value /tmp/hubic-token.json
# duplicacy set -key hubic_password -storage hubic -value "XXXXX"
# duplicacy set -key password -storage openDrive_rclone  -value "XYXYXY"

Following documentation, started an initial backup on both repositories

# duplicacy backup -stats -threads 10
# duplicacy backup -stats -storage hubic  -threads 10

Copy a new file

# cp ../01-Mr.Brightside.mp3 .
# ls
01-Mr.Brightside.mp3  11 - Everything Will Be Alright.mp3

backup on Primary only

# duplicacy backup -stats -threads 10
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music/
Last backup at revision 1 found
Indexing /data/Downloads/duplicacy_test/Music 
Use 10 uploading threads
Uploaded chunk 3 size 2137571, 209KB/s 00:00:32 23.9%
Uploaded chunk 2 size 3259582, 240KB/s 00:00:15 60.3%
Uploaded chunk 1 size 3539299, 323KB/s 00:00:01 100.0%
Uploaded 01-Mr.Brightside.mp3 (8936452)
Backup for /data/Downloads/duplicacy_test/Music at revision 2 completed
Files: 2 total, 16,807K bytes; 1 new, 8,727K bytes
File chunks: 6 total, 16,807K bytes; 3 new, 8,727K bytes, 8,754K bytes uploaded
Metadata chunks: 3 total, 967 bytes; 3 new, 967 bytes, 1K bytes uploaded
All chunks: 9 total, 16,807K bytes; 6 new, 8,727K bytes, 8,755K bytes uploaded
Total running time: 00:00:57

Copied from Primary to Secondary without running backup on secondary (to test the copy as per the documentation)

# duplicacy  copy -from openDrive_rclone -to hubic -threads 10
Source storage set to /mnt/OpenDrive/Duplicacy/Odroid/music/
Destination storage set to hubic://Duplicacy/oDroid/music/
Chunk 239e584b27a168b16f6bd2c8f45 (2/12) skipped at the destination
Chunk b48131ac50b47ab918b9a447412 (3/12) skipped at the destination
Chunk cf6d5762507976675ff93cdfe3d (4/12) skipped at the destination
Chunk 43fc0eb96e65e1f1de635f022eb (8/12) skipped at the destination
Chunk 4b7d350267d44fcbe22ed9b66eb  (9/12) skipped at the destination
Chunk 2a5fc079907fe6211ef15972552 (10/12) skipped at the destination
Chunk a26620d76a17ae18bad4cc890bd (11/12) copied to the destination
Chunk 8bf6f59782a00564d9f46a2aaa9 (6/12) copied to the destination
Chunk f9f73ef6318aadba0d0aacc6229 (7/12) copied to the destination
Chunk 026dffc51abf55f776745aec3f5 (12/12) copied to the destination
Chunk c2faca6714dbbc0981e703ee36d (5/12) copied to the destination
Chunk dfb8a6fd4ed9c175a8b39372b58 (1/12) copied to the destination
Copy complete, 12 total chunks, 6 chunks copied, 6 skipped
Copied snapshot 0 at revision 1
Copied snapshot 0 at revision 2

List the primary storage :

# duplicacy list -a -storage openDrive_rclone
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music/
Snapshot 0 revision 1 created at 2018-03-25 22:20 -hash
Snapshot 0 revision 2 created at 2018-03-25 22:27

List the secondary storage: Note, there is revision 1 only that belongs to the initial backup and revision 2 that belongs to the copy execution while in the webpage, I can see a folder named 0 inside snapshots in OpenDrive with two files. And two folders inside snapshots named 0 (with 2 files) and 1 (with 1 file) in Hubic.

# duplicacy list -a -storage hubic
Storage set to hubic://Duplicacy/oDroid/music/
Snapshot 0 revision 1 created at 2018-03-25 22:20 -hash
Snapshot 0 revision 2 created at 2018-03-25 22:27
Snapshot 0 revision 1 created at 2018-03-25 22:20 -hash
Snapshot 0 revision 2 created at 2018-03-25 22:27
Snapshot 1 revision 1 created at 2018-03-25 22:23 -hash
Snapshot 1 revision 1 created at 2018-03-25 22:23 -hash

Removed a file

# ls
01-Mr.Brightside.mp3  11 - Everything Will Be Alright.mp3
# rm 01-Mr.Brightside.mp3

try to restore it from secondary at revision 1 (nothing is restore as this file was not on the initial backup, it only exists on revision 2):

# duplicacy restore -r 1  -stats -storage hubic  -- *Brightside*
Storage set to hubic://Duplicacy/oDroid/music/
Restoring /data/Downloads/duplicacy_test/Music to revision 1
Restored /data/Downloads/duplicacy_test/Music to revision 1
Files: 0 total, 0 bytes
Downloaded 0 file, 0 bytes, 0 chunks
Total running time: 00:00:05

Restore from Primary (same, as the file does not exist on revision 1 but 2)

# duplicacy restore -r 1  -stats   -- *Brightside*
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music/
Restoring /data/Downloads/duplicacy_test/Music to revision 1
Restored /data/Downloads/duplicacy_test/Music to revision 1
Files: 0 total, 0 bytes
Downloaded 0 file, 0 bytes, 0 chunks
Total running time: 00:00:06

No file is restored:

# ls
11 - Everything Will Be Alright.mp3

As stated before, this new file only exists on revision 2, hence, try to restore from revision 2.

But… Revision 2 on Secondary does not exist

# duplicacy restore -r 2  -stats -storage hubic  -- *Brightside*
Storage set to hubic://Duplicacy/oDroid/music/
Snapshot 1 at revision 2 does not exist

Revision 2 on Primary - Restored, but again from Primary only…

# duplicacy restore -r 2  -stats   -- *Brightside*
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music/
Restoring /data/Downloads/duplicacy_test/Music to revision 2
Downloaded chunk 1 size 3539299, 1.13MB/s 00:00:05 39.6%
Downloaded chunk 2 size 3259582, 1.08MB/s 00:00:02 76.0%
Downloaded chunk 3 size 2137571, 1.07MB/s 00:00:01 100.0%
Downloaded 01-Mr.Brightside.mp3 (8936452)
Restored /data/Downloads/duplicacy_test/Music to revision 2
Files: 1 total, 8.52M bytes
Downloaded 1 file, 8.52M bytes, 3 chunks
Total running time: 00:00:11

At this point I understood the naming has to be the same in both respositories (or at least, everything points in that direction).

I can perform any test as required.

cVillabrille · 26 March 2018 10:16

Hi,

I have performed another test having only the OpenDrive and a repository folder in the same machine.

I have configure them having different respository_id and unfortunatelly, same result. Only with same repository_id naming can be restored.

Hence it does not depend on hubis slowness (even when that slowned can be true :)).

Another test, different naming, Storage local and opendrive.

# mkdir    /data/Downloads/duplicacy_test
# mkdir -p /mnt/OpenDrive/Duplicacy/Odroid/music3
# mkdir -p /data/Downloads/duplicacy_repository/music3

“duplicacy_test” Folder contains only one file

# ls 
11-Everything_Will_Be_Alright.mp3

Initial configuration:

# duplicacy init -e -storage-name openDrive_rclone 0 /mnt/OpenDrive/Duplicacy/Odroid/music3
# duplicacy add  -e -copy openDrive_rclone local 1 /data/Downloads/duplicacy_repository/music3

First Backup:

# duplicacy backup -storage openDrive_rclone -stats -threads 10
# duplicacy backup -storage local -stats -threads 10

Sync from remote to local

# duplicacy  copy -from openDrive_rclone -to local -threads 10

Add data and second backup with copy

# cp ../01-Mr.Brightside.mp3 .
# duplicacy backup -storage openDrive_rclone -stats -threads 10
# duplicacy  copy -from openDrive_rclone -to local -threads 10

List of snapshots

# duplicacy list -a -storage openDrive_rclone
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music3
Snapshot 0 revision 1 created at 2018-03-26 10:57 -hash
Snapshot 0 revision 2 created at 2018-03-26 11:03
Snapshot 0 revision 3 created at 2018-03-26 11:35
# duplicacy list -a -storage local
Storage set to /data/Downloads/duplicacy_repository/music3
Snapshot 0 revision 1 created at 2018-03-26 10:57 -hash
Snapshot 0 revision 2 created at 2018-03-26 11:03
Snapshot 0 revision 3 created at 2018-03-26 11:35
Snapshot 1 revision 1 created at 2018-03-26 11:03 -hash

Remove one file and test to restore… I can restore only from primary.

# ls
01-Mr.Brightside.mp3  11-Everything_Will_Be_Alright.mp3
# rm 01-Mr.Brightside.mp3

# duplicacy restore -r 3  -stats -storage local  -- *Brightside*
Storage set to /data/Downloads/duplicacy_repository/music3
Snapshot 1 at revision 3 does not exist

# duplicacy restore -r 2  -stats -storage local  -- *Brightside*
Storage set to /data/Downloads/duplicacy_repository/music3
Snapshot 1 at revision 2 does not exist

# duplicacy restore -r 1  -stats -storage local  -- *Brightside*
Storage set to /data/Downloads/duplicacy_repository/music3
Restoring /data/Downloads/duplicacy_test to revision 1
Restored /data/Downloads/duplicacy_test to revision 1
Files: 0 total, 0 bytes
Downloaded 0 file, 0 bytes, 0 chunks
Total running time: 00:00:01

List of Snapshots:

# duplicacy list -a -storage openDrive_rclone
Storage set to /mnt/OpenDrive/Duplicacy/Odroid/music3
Snapshot 0 revision 1 created at 2018-03-26 10:57 -hash
Snapshot 0 revision 2 created at 2018-03-26 11:03
Snapshot 0 revision 3 created at 2018-03-26 11:35
# duplicacy list -a -storage local
Storage set to /data/Downloads/duplicacy_repository/music3
Snapshot 0 revision 1 created at 2018-03-26 10:57 -hash
Snapshot 0 revision 2 created at 2018-03-26 11:03
Snapshot 0 revision 3 created at 2018-03-26 11:35
Snapshot 1 revision 1 created at 2018-03-26 11:03 -hash

gchen · 26 March 2018 15:32

It seems like that when listing the snapshots directory in the hubic storage, the same subdirectory may be returned twice, and that is why you saw two sets of revisions with the same snapshot id.

As for restoring from the second storage, sorry I wasn’t clear about it – you can use the same or different repository id for the default storage and the secondary storage, and the copy operation preserves the repository id. For instance, when all revisions of snapshot 0 are copied from the default storage to the secondary storage, they still belong to snapshot 0. However, if you want to restore a revision of snapshot 0 in the same repository from the secondary storage, the secondary storage must be initialized with the same repository id 0.

That is, when you restore a revision from a storage, you can only restore from the snapshot with the repository id that was provided to the init or add command.

In case you use different repository ids for the default storage and the secondary storage, the only way to restore from a revision that is backed up to the default storage and then copied to the secondary storage, is to create a new repository in a new folder/computer, initialize it with the repository id for the default storage but with the secondary storage as the storage, and then run the restore command.

cVillabrille · 27 March 2018 14:53

Absolutely clear now.

In order to restore from both you need same respository_id.

Thanks a lot for your time, help and comments on this.