Help on Migration

[jasonf]

TL;DR If I change the folder path of source repositories, can I use the same ID and storage, and continue to add to the existing revisions? In more detail:

For several years, I’ve been running Duplicacy web on Windows 10 as a service. I have several different IDs, each with its own retention period, etc. The source content is stored on multiple SSDs and HDDs, and destination is B2.

In moving away from Win 10, I’ve split my box into a new Win 11 machine with single SSD and a NAS running Open Media Vault for all of the HDDs.

My main question is how to retain the existing ID revisions and merge the new ones into them. For the source folders that remain the same, but now are on a different path (including some moving from Windows to Linux), can Duplicacy pick up where I left off? Or do these breakaway and start fresh?

For the other IDs that use a list of symlinks, these links will now be split between the systems, and therefore a repository from each. Does Duplicacy manage to deduplicate from the old, or is this a fresh start?

Finally, what is the best approach?

1. Should I run Duplicacy on both Windows and NAS, or just on Windows and use the NAS SMB shares as repository sources?

2. Assuming I run on both, is it better to back up from the PC to the NAS and then copy that to B2 (providing 2 backups of that data)? What’s the correct way to set it up to pick up the existing storage which currently sources from Windows, but now would be the copy?
   old: Win10->B2
   new: Win10->NAS, NAS->B2 via copy

Thanks for your insight.

[Droolio]

@jasonf

For the source folders that remain the same, but now are on a different path (including some moving from Windows to Linux), can Duplicacy pick up where I left off?

Backups should continue from where they left off, though if you’re moving to a different OS there a chance the metadata changes slightly that causes the next backup to take longer (as it may have to re-hash everything), but it shouldn’t upload a whole new lot of chunks - even if you used a different ID.

Think of the ID as simple way for Duplicacy to reference the last revision when doing incremental backups. For all files that appear unchanged, it skips the hashing part to make backups faster.

@jasonf

For the other IDs that use a list of symlinks, these links will now be split between the systems, and therefore a repository from each. Does Duplicacy manage to deduplicate from the old, or is this a fresh start?

Duplicacy deduplicates regardless of ID, so it’s neither fresh or stale.

Strictly speaking, you don’t have to keep the original IDs - you can keep them around and gradually historic revisions will get pruned out.

For me personally - for IDs where you’re copying over the original source data - I’d keep. For IDs where you’re splitting up the source data, I’d probably use different IDs. Duplicacy will take care of deduplication no matter.

@jasonf

1. Should I run Duplicacy on both Windows and NAS, or just on Windows and use the NAS SMB shares as repository sources?

Up to you really. Personally, I’d run them on each. The benefit of the NAS is it can be always on, so can do operations like copy in the background on a schedule. Hashing over SMB will be slower too.

@jasonf

2. Assuming I run on both, is it better to back up from the PC to the NAS and then copy that to B2 (providing 2 backups of that data)? What’s the correct way to set it up to pick up the existing storage which currently sources from Windows, but now would be the copy?
   old: Win10->B2
   new: Win10->NAS, NAS->B2 via copy

New is how I do it. The NAS/server takes care of the copying and you get quicker (local) backups for your Windows machine. Plus you get an extra copy (3-2-1).

[jasonf]

Thanks for the detailed answers, really appreciated. I’ll give it a try (hopefully this weekend) and see how it goes.

When setting up the copy from NAS to B2, how can I pick up where I left off gracefully? Let’s use Documents as an example. Currently it’s backed up from Windows to B2 and is on a revision somewhere in the 30,000s. If I now back up Windows->NAS it will start from rev 1. Then I copy this to B2 - I assume this becomes a new target and has nothing to do with the old ID? I just don’t quite get how to keep going when breaking the chain this way.

[Droolio]

Good spot.

When resuming the original ID, you’ll want to copy that last revision (3x,xxx) back from B2 to the NAS, so the next incremental is sequential.

Obviously that’s not very efficient so there’s a trick - you can ‘pre-seed’ the chunks from your local copy first. Because chunks are deterministic, you can initiate a backup to your NAS using a temporary ID and let it fill in the chunks. Or forgo the temporary ID and delete the resultant revision 1 after completion.

Then you copy B2 to the NAS, carefully specifying the the last revision and the corresponding ID instead of using the -all flag (e.g. -id users-jason -r 30123). It should skip most chunks except a handful for metadata and potential differences in how the original data was chunked over time (fragmented).

You’ll end up with a revision 30123 from which subsequent backups increment from, then you’re good to schedule a copy job from NAS to B2.