iDrive e2 - NoSuchKey

Support
#1

I’ve been backing up to iDrive e2 with Duplicacy Web Edition 1.5.0 since July without incident, and I’ve been doing weekly restores just to make sure everything is working. But I only tried doing a prune last week, and I just took the default - “-keep 0:1800 -keep 7:60 -keep 1:7 -a”. The very first time it ran, it seems to have munged up e2. Now if I try to prune, check, or restore, I get “Failed to list revisions for backup ‘Full’ in the storage IDrive-Home: Failed to download the file snapshots/Full/2: NoSuchKey: The specified key does not exist.” Backup, however, seems to still be working.

What should I do?

Check and Prune both failing with NoSuchKey: The specified key does not exist. (after running fine for months)
#2

Do you see this file snapshots/Full/2 on the iDrive website? It could be that this file was removed by the prune job but for some reason still returned when Duplicacy listed the files under snapshot/Full.

#3

Yes, the file snapshots/Full/2 does appear in the bucket in e2. And I can download it. And when I look at it in a hex editor, it looks like this:
image

which seems to suggest that it isn’t corrupt.

I don’t know if this is important or not, but when I created this bucket, I left Versioning set to “on”, which strikes me as undesirable. So e2 is actually showing 11 different versions of this file.

So it does seem to exist. Any idea why Duplicacy is reporting that it doesn’t exist?

#4

Ehh… duplicacy never modifies files. It creates them, renamed them, and deletes them (hides on some remotes). I don’t see how you could have ended up with 11 revisions. Maybe some glitch with API when duplicacy tried to delete it it creates a ghost version instead?

Can you see and download that file with some third party tool, like Cyberduck or whichever supports e2?

#5

Right, but they do seem to actually be there. From the e2 UI:

image
image779×896 47.4 KB

Seems possible. Note that they were one a day until V11 and then they just stopped. Lots of the files seem like this and newer ones aren’t.

Yes. Cyberduck can see the file. I downloaded it and it’s identical to what I downloaded directly from the e2 UI.

#6

Can you go to the restore to list how many revisions there are? Is there any other revision that has multiple versions?

#7

No. If I try to restore, check, or prune I get the same error - “Failed to list revisions for backup ‘Full’ in the storage IDrive-Home: Failed to download the file snapshots/Full/2: NoSuchKey: The specified key does not exist.” I can never see the revision list.

I’m guessing by “revision” you’re referring to items in the Snapshots folder on e2, correct? If so, yes, it’s a little hard to determine because I have to click into each snapshot file individually and then click into Versions to see how many there are. The e2 UI is awkward to say the least. It looks like all of the first 16 snapshots have multiple versions. I’ve spotchecked a bunch of them after that (there are 168 total) and I haven’t found any after that with multiple versions. There may be some, but I haven’t found one.

#8

Does revision 3 have multiple versions too? If so were they all created after V11 of revision 2?

One scenario where this can happen is if a prune keeps deleting revision 2 while a copy job keeps copying the same revision from another storage. Other than that I agree with @saspus that it must be an API glitch. At least it should return a version when Duplicacy asks for it.

I’ll try to get iDrive people to look at this issue.

#9

It does, yes, 11 of them. And revision 1 has 81 versions.

Close. R3V1 was created the day after R2V1 (7/5 and 7/6). Then all the V2 revisions run daily from 7/23 to 8/1. Then all the V3 revisions run daily from 8/2 to 8/11.

Much appreciated. Let me know if you need screenshots or anything. I have three other buckets that are managed by three other installations of Duplicacy. I did not leave Versioning turned on for any of those, but I haven’t tried running prune on any of them yet either. It doesn’t seem possible to turn off Versioning on a bucket after it has been created.

#10

Something definitely seems to have happened on the e2 side. Revision 2 is now gone from the snapshots folder, along with many others. And now Duplicacy can restore, check, and prune.

This is the second time in the past six months that I’ve found e2 to pretty seriously fumble. It’s marginally acceptable as a light duty backup store, but this would never work in a commercial application.

But I continue to have many versions of some revisions. Like revision 1 is now up to 106 different versions, many of which are now flagged as Deleted=Yes whatever that means. Revision 7 has 19 versions, 14 has 11. Seems to drop off to one version for each revision after that.

Spot checking chunks, I don’t see any that have multiple versions, but you never know. Basically I decided to start pruning because I was getting near 1TB, and I didn’t want to pay overages. But ever since the prune, my usage went up dramatically. Here’s an interesting graph:

image
image986×506 34.5 KB

Basically, you can see my usage slowly increasing. I did the first prune on 12/12/22 and my usage shot up dramatically. It went back down, but then shot up again just after 12/19/22 which is when gchen said he’d get in touch with the e2 folks. Here’s what my object count looks like across the entire period of backups:
image
image975×511 39.2 KB

Why would my object shoot up when I start pruning?

I think I may try to copy the entire bucket to a new bucket that has versioning turned off and go from there.

Anyway, thanks Gilbert for your support.

#11

… at another storage provider. Your second paragraph above is correct, there is no point in kicking the dead horse. There are many storage providers out there that have benefit of actually working. IDrive seems to think that aggressive marketing can make up for lack of engineering. Issues like these at this stage of product maturity are not acceptable. Cut your losses now, before you sink even more of your time fighting the uphill battle. I would specifically recommend Storj, B2, even Wasabi. Just not iDrive or pCloud.

#12

iDrive people said the issue was related to versioning. If you want to get to the bottom of it, you can file a ticket with them and they will enable traces for you. Otherwise, just copy the content to a non-version bucket and the issue should go away.

#13

I hesitated for a day whether to respond… Ah, I can’t just stand here.

This was their response? So not only they don’t have sufficient monitoring in place to triage issues like these, let alone adequate QA, but also they twist this to be a customer’s “wish”, let alone responsibility, “to get to the bottom of it”, requiring a support ticket, no less? Why are they not investigating right now, through holidays, like there is no tomorrow, all while apologizing profusely on the front page, with logging enabled across all buckets for all customers? Storing data is what they exist for – they can’t screw that up and go home for the holidays as if nothing happened.

“Hey, you lost data, we have no idea why and how, but if you really want, we can enable logging, and when you lose data again we will be able to tell you why. And maybe.”. Top-notch integrity and business plan.

“Here is a feature, the feature is broken, we don’t know why, and can’t debug, because we don’t have storage to keep enough logs, but hey, if you don’t use the feature, you won’t see the issue. It was your fault to use the feature as published.”

In my opinion, run, don’t walk from this nonsense, and furthermore, Duplicacy should distance itself from this circus. Whatever kickback arrangement is there with partnership, it’s not worth the risk of having reputation tarnished though association. That’s my personal opinion of a dude watching from the outskirts.

I would at this point also remove support for this storage provider entirely, because backup is only as good as the underlying storage. While bugs and mishaps do happen with any provider, how they are handled is what becomes important. In this case – lack of any insight of what’s going on in their own system is inexcusable. This is not a bug, it’s an entire system design flaw, and this being not the first time when this specific provider drops the ball – how many signs do we need?

1 Like
#14

saspus, I appreciate your words. It’s always nice to find someone more critical of bad support than I am. I’ve run a small software company for almost 30 years.We’ve never advertised or “marketed” because we don’t have to. Our customers sell our product for us. We used to have several competitors, but we have none now because we destroyed all of them with competence.

This is actually my third run-in with iDrive. I tried them years ago when Crashplan ended their non-business service. I tried iDrive’s backup system and it was terrible. And their support then was just as bad as it is now, always trying to push issues back into the customer’s lap.

I know from experience that when a development team has a good product, they leap at opportunities to explore a bug with a user, generally because they don’t have many bugs left, and they haven’t been able to reproduce the few they do have in-house. The team that is content to get a user over a hump and then walk away is one that has more bugs than they know what to do with.

My second run-in with iDrive was just a few months ago. One day, my usage chart dropped to 0, and all my buckets were empty. Here’s what it looked like:
image

And here’s a literal quote from them:

There is no problem with your data; there was a technical glitch on calculating your daily storage usage and that got automatically fixed in next run.

I wrote:

Wow, that’s nonsense. I visually confirmed that all my buckets were empty yesterday and I watched the data getting restored back in to them throughout the day today. What do you say about that?

And they replied:

We are so sorry for the inconvenience; there is no data loss in your account, as we mentioned there was technical glitch on meta information and it got fixed automatically.

It’s pretty shocking. I spend over $3000/year on S3 for years and I’ve never had a single hiccup. I don’t expect a service like e2 to be as good as S3, but I do expect a bit of honesty.

So yes, I will be moving out of e2. Not sure where I’m going. I tried to create a new bucket on e2 with versioning turned off and copy my existing data to it using Cyberduck, but it crapped out. I have very little experience with Cyberduck. I should have used WinSCP which I have years of experience with. So I don’t know whether Cyberduck failed or e2. If it was e2, Cyberduck definitely didn’t handle it gracefully.

But yes, saspus, with respect to Duplicacy, I agree. After Crashplan, I tried many backup systems, and finally landed on Duplicati. I still use it professionally because it meets my needs, but I think Duplicacy is head-and-shoulders better. I’m beginning to use it more for commercial applications. Any kind of partnership with e2 is probably not doing Duplicacy as much good as it’s doing them.

2 Likes