ERROR: Storage has not been initialized for S3

Support
#1

We recently changed our backup destination from Backblaze to S3. Backups were running successfully for about 3 days and then we noticed we weren’t getting any success emails for about a week (this is in a separate topic). Upon investigating the logs, it’s stating:

2019-07-08 16:00:30.843 ERROR STORAGE_NOT_CONFIGURED The storage has not been initialized
[16:00:30] E: Backup operation returned an error:
ERROR The storage has not been initialized

On all of our servers on the same day. The log entry from the day prior to these errors looks completely fine (at least to me, if necessary, I can upload this).

We have our 4 servers going to the same S3 bucket, but in four different subfolders so there will not be conflict. Each folder appears okay (root of the subfolder has a chunks and snapshots folder, if more info is necessary, I can provide more).

Why would I be receiving this?

How to troubleshoot ERROR STORAGE_NOT_CONFIGURED
#2

Do you see a file named config there? This error means that this file doesn’t exist in the storage.

#3

Nope, no config file there. What would have caused that to be deleted? I see chunks in the chunks folder, as well as subfolders within the snapshots folder.

#4

Do you have “versioning” enabled on this bucket?

#5

No, versioning is not enabled on the bucket.

#6

No, versioning is not enabled on the bucket. Should it be?

#7

Not necessarily. I just asked because it could be a way to investigate what happened with the config file.

#8

It’s possible I missed something, but from a quick skim I don’t see any code paths in Duplicacy that would delete a config file.

My first thought would be a misbehaving script or accidental manual deletion. If you have bucket logging enabled it could have more useful info on who deleted the config files and when.

#9

I’ll check with the team regarding if any logging was enabled.

What is very strange though is that this has affected every single server’s folder on S3, not just a single one. If logging is not enabled, I’ll see if they have some sort of script running checking for files to delete.

What is the exact name of this config file (including extension)?

#10

Wanted to report back that it appears there was a script running against the S3 bucket entirely, which deleted files after 5 days (lines up nicely with how long it was running before it stopped working).

Now that we found the culprit, is there a way to restore this config file without having to do a complete re-initialization and back up all of our servers again?

1 Like
#11

Was the storage encrypted? If so, I think that deleting the config is equivalent to deleting the encryption keys – which by design shouldn’t be easily regenerated. So I can’t think of a way to not restart the backups in this scenario.

If the storages weren’t encrypted, I think there might be a chance that you can re-init with the same settings to regenerate the config – but I’m not 100% sure that’ll work. Maybe someone else on here can confirm unless you want to give it a shot.

1 Like
#12

Just a curiosity/doubt: then the chunks and snapshots are also being deleted after 5 days, ie the backup is only for the last 5 days, that’s right?

1 Like
#13

Yeah, I just had a look and one of our largest servers now only had 50mb of data remaining in the bucket. I think unfortunately the repo needs to be reinitialized and uploaded again.

2 Likes
#14

That’s the best right now.