Adding a new second computer backup to an existing storage?

C11 · 27 January 2023 23:42

I’m trying to add a new computer to be backed up to an existing storage.
The first time I did this, the new computer’s backup started a new snapshot and the first computer started failing backups.

2023-01-25 12:59:31.078 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot MOTOCAT-Carl-Carl
2023-01-25 12:59:31.686 ERROR SNAPSHOT_PARSE Failed to load files specified in the snapshot MOTOCAT-Carl-Carl at revision 14121: not a list of entries
Failed to load files specified in the snapshot MOTOCAT-Carl-Carl at revision 14121: not a list of entries

I must be missing something obvious… How do I get the new computer to use the de-duplication feature and add it’s stuff to the storage instead of overriding what is there?

The “Storage” settings are the same on both the first and new second computer, except I gave the new computer it’s own Storage “Name”. Is that the mistake?
The “Backup” Settings are the same on both the first and new new computer, except again the Storage “Name” is different, and the directories to be backed up are obviously somewhat different. The backup ID is the same.
Is the problem the different “storage” names on the old and new computers? If not that, I must be missing something obvious…

saspus · 27 January 2023 23:57

(Assuming Backup ID is Snapshot ID)

It does not matter. It’s just how you refer to the same store on different machines. Storage URL is what is important.

This is a problem. Different computers must have different backup IDs.

In other words, this is what a resonable setup will look like

computer1:
Storage Name: MyStorage-> b2:my_bucket
Backup Name: Machine1Backup, uses MyStorage.

computer2:
Storage Name: MyStorage-> b2:my_bucket (no need to use different alias for the storage)
Backup Name: Machine2Backup, uses MyStorage.

As a result, in the “snapshots” “folder” in the my_bucket you will see “subfolders” Machine1Backup and Machine2Backup.

If you are using same IDs your version history will be a mess.

However, this does not explain the " not a list of entries" error.

What is in the “snapshots” folder on the destination?

And separately, duplicacy should reject choices that end up corrupting data store. It’s a duplicacy bug.

C11 · 28 January 2023 01:14

I deleted the snapshot revisions made by the new machine and disabled it, and the first machine started backing up properly again. So it does seem the new machine’s snapshots were causing the problem. I’m guess the first machine pulled down the snapshot from the new machine, and just went WTF… and spit out the error.

So, when building the new “Backup” in the web-GUI, I should select the same “Storage” but instead of “Select an existing backup ID in the storage”, I should just create a new one named I wish?
Thank you! Tomorrow I’ll make the backup ID’s different and try it.

gchen · 28 January 2023 01:24

This error means that 14121 is a revision created with CLI 3.0.0+ but you were trying to run CLI 2.x.x to create a new backup (which needs to open the last revision for file comparison). If you upgrade the CLI to the latest version on your existing computer then it should be able to open these revisions.

C11 · 28 January 2023 01:50

Is there a problem if the two computers are running different versions of the CLI?
The new machine is a mac on 3.1.0 and the first machine is a PC on 2.7.2. The web-gui claims these are the latest stable versions…?

saspus · 28 January 2023 02:00

Yes, there was a breaking change recently.

You have set “Stable” on a PC and “Latest” on the mac (or haven’t rebooted pc or duplicacy service for months (or there is a race condition where duplicacy attempts to download update while network is not yet ready on start)), that’s why you got different versions: They are reported by this api: https://duplicacy.com/latest_cli_version, that right now returns {"latest":"3.1.0","stable":"2.7.2"}

Duplicacy shall keep “minimum required version” in the data store, recorded after every non-backwards-compatible change, and use it to fail adding the updated storage to old client, with a human-readable, descriptive, and actionable message.

C11 · 28 January 2023 04:07

Hmm…
I’m running the web versions on everything.
MAC is set to stable and is on 3.1.0. Really, I double checked.
PC is set to stable and is on 2.7.2 and I just shutdown and restarted the app. It’s not running as a service. It’s not grabbing anything than 2.7.2 on it’s own… How do I prod it into action?

saspus · 28 January 2023 05:44

PC is set to stable and is on 2.7.2

This seems to be correct: current stable is indeed 2.7.2.

This is weird. Not sure how could it end up like so.

What happens in the duplicacy_web.log when you restart duplicacy? (Exit and start it again)? I’d expect it to download 2.7.1.

If it does not even try — delete the duplicacy CLI binaries from ~/.duplicacy_web/bin folder and restart the duplicacy_web. It shall download some CLI now, and it shall be expected to be 2.7.2, per settings.

C11 · 28 January 2023 06:36

Quit and restarted duplicacy on the mac:

2023/01/27 22:30:20 Temporary directory set to /Users/carlliebold/.duplicacy-web/repositories
2023/01/27 22:30:20 Duplicacy Web Edition 1.4.1 (074ED2)
2023/01/27 22:30:20 127.0.0.1:49967 GET /
2023/01/27 22:30:20 The log file backup-20221228-203141.log has been deleted
2023/01/27 22:30:20 The log file check-20221228-203146.log has been deleted
2023/01/27 22:30:20 The log file backup-20221228-204001.log has been deleted
2023/01/27 22:30:20 The log file check-20221228-204005.log has been deleted
2023/01/27 22:30:20 127.0.0.1:49968 GET /assets/fonts/fontawesome-webfont.woff2?v=4.7.0
2023/01/27 22:30:20 127.0.0.1:49967 GET /assets/css/paper-dashboard.css
2023/01/27 22:30:20 127.0.0.1:49968 GET /assets/css/bootstrap.min.css
2023/01/27 22:30:20 127.0.0.1:49969 GET /assets/css/font-awesome.min.css
2023/01/27 22:30:20 127.0.0.1:49967 GET /assets/js/bootbox.min.js
2023/01/27 22:30:20 127.0.0.1:49970 GET /assets/css/duplicacy-web.css
2023/01/27 22:30:20 127.0.0.1:49971 GET /assets/js/jquery-1.10.2.min.js
2023/01/27 22:30:20 127.0.0.1:49972 GET /assets/js/jquery.min.js
2023/01/27 22:30:20 127.0.0.1:49968 GET /assets/js/bootstrap.min.js
2023/01/27 22:30:20 127.0.0.1:49969 GET /assets/js/paper-dashboard.js
2023/01/27 22:30:20 127.0.0.1:49967 GET /assets/js/axios.min.js
2023/01/27 22:30:20 127.0.0.1:49970 GET /assets/js/chartist.min.js
2023/01/27 22:30:20 127.0.0.1:49971 GET /assets/js/polyfill.min.js
2023/01/27 22:30:20 127.0.0.1:49972 GET /assets/img/product-icon.png
2023/01/27 22:30:20 Duplicacy CLI 3.1.0
2023/01/27 22:30:25 127.0.0.1:49967 GET /setting
2023/01/27 22:30:25 127.0.0.1:49967 GET /assets/css/paper-dashboard.css
2023/01/27 22:30:25 127.0.0.1:49968 GET /assets/css/bootstrap.min.css
2023/01/27 22:30:25 127.0.0.1:49970 GET /assets/css/duplicacy-web.css
2023/01/27 22:30:25 127.0.0.1:49971 GET /assets/js/jquery-1.10.2.min.js
2023/01/27 22:30:25 127.0.0.1:49969 GET /assets/css/font-awesome.min.css
2023/01/27 22:30:25 127.0.0.1:49972 GET /assets/js/bootstrap.min.js
2023/01/27 22:30:25 127.0.0.1:49968 GET /assets/js/chartist.min.js
2023/01/27 22:30:25 127.0.0.1:49971 GET /assets/js/bootbox.min.js
2023/01/27 22:30:25 127.0.0.1:49967 GET /assets/js/axios.min.js
2023/01/27 22:30:25 127.0.0.1:49973 GET /assets/js/jquery.min.js
2023/01/27 22:30:25 127.0.0.1:49969 GET /assets/js/paper-dashboard.js
2023/01/27 22:30:25 127.0.0.1:49972 GET /assets/js/duplicacy-web.js
2023/01/27 22:30:25 127.0.0.1:49968 GET /assets/js/polyfill.min.js
2023/01/27 22:30:25 127.0.0.1:49971 GET /assets/img/product-icon.png
2023/01/27 22:30:25 127.0.0.1:49967 GET /assets/fonts/fontawesome-webfont.woff2?v=4.7.0

C11 · 28 January 2023 17:40

So…
I tried this:

If it does not even try — delete the duplicacy CLI binaries from ~/.duplicacy_web/bin folder and restart the duplicacy_web. It shall download some CLI now, and it shall be expected to be 2.7.2, per settings.

And:

ANd then:

So… 3.1.0 seems to be the latest stable release for the M1 Mac. And 2.7.2 on the PC…
What should I make of that?

saspus · 28 January 2023 17:57

Weird. Paging @gchen — AFAIK there is just one version url that duplicacy-web is using to fetch CLI executables, but maybe it changed recently?

C11 · 28 January 2023 18:39

LoL.
So on a whim, I deleted the CLI bin AND the “Duplicacy Web Edition” app from my mac and installed the newest version… (I was on an older version, I think 1.4?).
And the new version DID download 2.7.2! So either it was the old app, or @gchen fixed something in the last hour…

C11 · 28 January 2023 18:44

So back to my question… running both a mac and a PC on 2.7.2.
I can imagine one might end up eventually updating before the other machine. I can’t really make sure those backups are synchronized.
Is it a problem if two machines pointing at the same bucket are on different revs?

saspus · 28 January 2023 19:08

Looks like indeed a new behavior in duplicacy web…

https://duplicacy.com/latest_cli_version still returns the same/

There used to be a way to fix the version of the CLI that web is using, somewhere in the settings.json file, but I can’t find that post.

Within the same major version, it should be fine. Between major versions — definitely not. That’s (one of the) implications of incrementing a major version – that it may break forward compatibility, among other things. In this case this was indeed the case somewhere in 3.0.x

Droolio · 28 January 2023 20:01

Latest Duplicacy Web Edition is v1.6.3, is it up-to-date on both systems?

C11 · 28 January 2023 22:23

Now they are… They were not earlier today.

Hmm… so how do people manage this when the “stable” release bumps up a major rev?
I mean, I might not even notice the change…

saspus · 28 January 2023 23:43

I don’t think it’s handled properly today, if at all.

If duplicacy web checked for the CLI version before each invocation it would not have been a problem as long as all machines involved were on the same channel.

But IIRC it only checks on startup and CLI faceplants if datastore contains newer content.

I think nobody noticed, because there was no breaking change before, and there are very few people who backup to the same datastore from multiple sources, and even fewer that use web_ui, (based on my speculation). But since that 3.0.x there have been a few posts with symptoms similar to yours.

In other words, yes, duolicacy must handle that gracefully but I don’t think it handles it today.

C11 · 29 January 2023 00:10

OK… so it sounds like I really shouldn’t use the same data store. And thus will lose any de-duplication benefits. No biggie for me, but I could imagine others being rather annoyed.

saspus · 29 January 2023 00:26

If you benefit from deduplication across machines — use it. Worst thing that will happen — backups will start failing on the other machines, so you would get an email, and manually update that other machine.

I would not change my backup arrangement to workaround software issues.

Droolio · 29 January 2023 02:39

You just need to update all clients to the latest v3.1.0.

The aforementioned issues only arise when trying to copy v3.x snapshots with v2.x client, or copying v3.x snapshots with v2.x client. Both issues are fixed if updating all clients to v3.1.0 (which can see ‘bad’ snapshots of v3 copied with v2, and can already upgrade v2 to v3).

Your issue, from what I understand, comes from downgrading to an earlier version on the same snapshot ID (which you must admit is extremely unlikely to happen and easy to rectify, since all you get is an error).

Ideally you’ll want to upgrade all clients at the same time, but v2 snapshots of one snapshot ID should be able to coexist with v3 snapshots of another snapshot ID. So backups should be fine, unless you downgrade a client. Trouble arises when doing storage maintenance (check, prune) if not all clients are updated, so update ASAP once the process is begun.

I understand v3.1.0 and subsequent versions now have a version bit for the snapshot format, so future upgrades should be smoother sailing.