Remote revisions with same ids already exist after local drive crash

[steve.francia]

Following what I believe are the best practices shared here I’ve been using Duplicacy to backup my system. I first backup to a local external drive and then copy that to Wasabi. This has worked well for me for many months… until last week. The local drive crashed. I purchased a new one and effectively swapped out the broken one with a fresh clean one… except now there’s a big difference between Wasabi and the local disk. The local disk had to start over beginning at revision #1. The Wasabi copy has ~65 revisions on it.

The problem is that when I go to copy from local disk to Wasabi now it says that the revisions already exist, which isn’t strictly true. Revisions with the same numbers exist, but they are different versions entirely. I would think that Duplicacy would be smart enough to use some sort of hash or even timestamp to produce a unique revision id, but it seems like it just uses an autoincrementing int.

I’m not quite sure how to move forward and I’ve been unsuccessful finding an answer in the docs or forum.

The four options I see all have serious drawbacks.

1. Remove local revisions and copy from Wasabi to local disk as the initial starting point and then resume backups going the opposite direction. This would be expensive and time consuming and during the initial copy any new data (not on Wasabi) is at risk for not having a backup.

2. Remove the Wasabi revisions and start over with that repo. This would also be expensive and time consuming and would have the extra disadvantage of losing all the versions that currently only exist on Wasabi.

3. Manually deleting the revisions 1 - 5 from Wasabi so that the local revisions no longer have conflicting IDs and copy them. Of course this is very manual and I’d need to do it daily to minimize the revision deletion. This would effectively turn it into a FIFO buffer of sorts until I caught up to the 65 or so existing revisions.

4. Just wait until the local copy exceeds the revision count on Wasabi. In approximately 2 - 3 months the revision ID will be unique and copies can commence, but for fairly obvious reasons this isn’t ideal.

Are there any ideas I’m missing? For instance is there a way to just bump the local revision number up a bunch, say start at 1000 instead?

[saspus]

Two options:

1. Restore state of local by copying from wasabi to local. Then resume as usual. Not my first choice for the reasons you have mentioned.

2. Rename the local snapshot. Right now it has the same name but is de facto a new snapshot that has nothing in common with the previous one, including revision history. Two different snapshots having the same name is a problem.

3. Rename wasabi snapshot you were copying to (literally rename the sub folder under snapshots folder).

(“Snapshot” here refers to what duplicacy calls snapshot, which is in reality source dataset ID)

[towerbr]

Assuming that the amount of data is large and/or your connection will not allow Wasabi to be copied to the new disk in the interval between two backups (as you commented on performing new backups while copying), I would make a slightly adapted version of yours first option:

• Make/start a copy from Wasabi to your new disk (no cost, Wasabi does not charge for egress);
• Meanwhile, point your backup directly to Wasabi, new backups (and new revisions) will be made directly to the cloud while your copy is in progress;
• When you finish the first copy, run the copy again to copy only the new backups, this copy will be very fast;
• Point your backup back to disk.

On the last point, I would particularly prefer to have cloud backup as the main one, but each one must evaluate their use case.

[Droolio]

It’s not clear how you initialised your new local storage - did you make it copy-compatible with Wasabi?

If so, your new local copy already has most of the chunks that exist on Wasabi, by virtue of the fact that copy-compatibility implies deterministically identical chunks. There will, of course, be chunks that need to be copied due to differences in metadata and because your source data maybe slightly different since you last backed up to Wasabi, but most of the chunks will be there in both storages.

Thus I’d just go with your first option - copy the last revision(s) from Wasabi to local. This shouldn’t use much bandwidth and whatever chunks that do need to be copied, should be free as towerbr points out. You’ll have to delete or rename your local snapshot directories out of the way first, to avoid conflict.

Or before doing the Wasabi->local copy, set up a fresh backup to local using different, dummy IDs - to ensure the local chunks are fully referenced - then you can delete the local original snapshot IDs, then finally do the Wasabi->local copy.

Once you have the latest revisions copied from Wasabi, doing backups to local will resume from those higher revision numbers.