Duplicacy and wasabi stopped working out of the blue

Support
#1

I am having a problem with Duplicacy and Wasabi. I am new the Duplicacy, and after some testing, decided to go with Duplicacy backing up to Wasabi. Everything seemed to be working fine and backing up successfully. However 1st sign of issue was in the middle of the night last night (Sydney/Australia time) at 1:57 I got the error :-

2018-10-29 01:57:27.715 ERROR UPLOAD_CHUNK Failed to upload the chunk 4045b2d894160ed83cf5fc987aa56cd16c4be584e42bb055240f9a0eef6461aa: AccessDenied: Access Denied
[01:57:27] C:\duplicacy-matthew Backup operation returned an error:
ERROR Failed to upload the chunk 4045b2d894160ed83cf5fc987aa56cd16c4be584e42bb055240f9a0eef6461aa: AccessDenied: Access Denied

After trying to restart it, I get the following error :-

C:\duplicacy-backup>duplicacy -log backup -stats
2018-10-29 15:37:01.567 INFO STORAGE_SET Storage set to s3://us-west-1@s3.us-west-1.wasabisys.com/duplicacy-other/jjs
2018-10-29 15:37:02.755 ERROR STORAGE_NOT_CONFIGURED The storage has not been initialized

But I know it is configured and working correctly up to that point and in between I had done nothing to change this, other than run backups. Interesting that I have backups going to 2 different buckets, and they all appear to have stopped working at the same time???

I tried accessing Wasabi with other tools and it all seems to be working fine. eg tried web gui to upload files, and also tried S3 access using CyberDuck and as far as I can tell, Wasabi seems to be working fine, so it looks like some sort of compatibility issue with Duplicacy???

I am not sure if this might be a little clue, but initially suspecting an issue with Wasabi I went to check out Wasabi via the Web console. When I tried to sign on to the console via HTTP I got a swirling Wasabi icon in the web interface that went nowhere. This seemed to affect any PC/Browser, I had previously accessed Wasabi on, but did not affect browsers or PCs I had not accessed Wasabi console on. To fix this, I did a “hard refresh” which is CTRL reload in Chrome and all seemed to work again for web access to the console. I have never had this issue before, but don’t like coincidences so wonder if this issue is related (ie Wasabi have made some changes overnight which caused this problem and broke Duplicacy access to Wasabi.

I have also tried a number of things to get it going. At 1 point I wondered if somehow my Data or bucket had got corrupted, so I created a new bucket and tried a test backup to that Bucket. I did this in the GUI and I got throguht to entering the encryption password, but when I did that I got the error message back “ERROR Failed to configure the storage: AccessDenied: Access Denied”. I even tried doing this to a different Wasabi account in case my account had become corrupted. But nothing worked. But if I access with another tool like CyberDuck, I can’t see any sign of an issue.

I have Duplicacy backing up to a 2nd store (local SFTP) and that seems to be working fine.

So to me it seems like there might have been some change to Wasabi overnight that is causing compatibility problems with Duplicacy and I am stuck.

Note : I am using us-west-1.

Are other people able to still access backups on Wasabi and in particular us-west-1? Any ideas how to fix this problem as I have a lot of Data backed up now and want to avoid having to start again.

Thanks in advance.

Bucket not Found - Wasabi
#2

Further to this issue, I have done some further testing on a new computer with a new repository. What I have is the issue seems to be with us-west-1, while buckets created on us-west-1 no longer seem to work.

I have followed instructions and used configuration settings as outlined here :-

When I create a new bucket and East region, and use the setting above, everything works fine. But when I create a bucket and use the configuration setting as outlined above and it fails with “Error Failed to configure storage: AccessDenied: Access Denied”.

So issue seems to be with Duplicacy and Buckets created in Region West.

Interesting CyberDuck does not seem to use a “region” in the configuration, and seems to work. So I tried no configuring a region in Duplicacy by leaving the region blank. But then I get the error “ERROR Missing Region” Could not find region configuration".

So there seems to be an issue with configuring Duplicacy and Wasabi for region west.

Any advice would be appreciated.

#3

From Supported storage backends it seems that there’s a new format for wasabi:
wasabi://us-west-1@s3.us-west-1.wasabisys.com/bucket/path (us-west-1 region)
but that format is not currently supported on 2.1.1, only in the next dupliacy release.

I’m not sure here but maybe wasabi disabled the older link (s3://region@s3.wasabisys.com/bucket/path), hence your problems?

#4

Thanks for that link. I had not seen that before. But does that link not suggest wasabi:… is supported in 2.1.1 as that is the latest version of the CLI??

I can see that it appears the GUI uses the S3:… version. Out of curiosity I tried changing the S3:… for the wasabi:… version of the command in the CLI (“duplicacy add…” command because it is a 2nd storage), and it seemed to take it, and prompted for the key and secret, as well as the storage secret, but errored out with :-

Failed to configure the storage: AccessDenied: Access Denied
        status code: 403, request id: xxxxxx, host id: xxxxxx-changed-in-case-it-is-confidential

Above is exactly the same error as I get if I try with the S3:… version of the command.

Anyway, I am hoping there is a quick fix as I have large backup that now appears to be useless until this issue is resolved.

#5

I can confirm that this URL works:

minios://us-west-1@s3.us-west-1.wasabisys.com/bucket/path

The difference the minios:// prefix makes is to force the path-style URLs (Working with Amazon S3 Buckets - Amazon Simple Storage Service).

s3c://us-west-1@s3.us-west-1.wasabisys.com/bucket/path also seemed to work, but this s3c backend uses the third-party S3 client library GitHub - goamz/goamz: Amazon AWS Library for Go instead of the official one.

1 Like
#6

That does indeed solve the most immediate problem for me, so thanks heaps for that and the quick response.

For other peoples reference, I just went into the .duplicacy\preferences file and edited them and changed the s3://us-west-1@s3.us-west-1.wasabisys.com/bucket/path to minios://us-west-1@s3.us-west-1.wasabisys.com/bucket/path and it all worked from the CLI. This is all that I needed for my pre existing backups, and then it all ran as it did before. This worked for CLI setup/scheduled backups, as well as GUI scheduled backups.

However, I have a few more questions in getting to the bottom of all aspects of this I am hoping you can help with :-

  1. I assume this means that this can’t be configured just in the GUI any more, or is there some sort of workaround to configure this in the GUI (so I can use the GUI scheduling)? I have come up with a convoluted way which involves 1) starting the job in region east to get the initial configuration on for the repository in the GUI, 2) then I delete the configuration files in the .duplicacy directory and 3) go to the CLI and start the process from the CLI and do the initialise and initial backup (to save the storage password), and 4) then if I go back to the GUI and it will run the modified job if I press start. But this is complex a more than a little misleading in the GUI, because the “Storage” config text in the GUI is wrong and misleading as it still makes it look like it is going to region East. Is there going to be any problems doing it this way other than I have pointed out? Is there a better way of doing this?

  2. when do you expect to have this fixed in a GUI that will work? I believe you have a totally new GUI pending for release soon, so will it be in that GUI, and if so when is the expected ETA?

  3. Supported storage backends mentions a “Billing issue” and talks about the advantages of the :wasabi://… configuration over the S3://… configuration in supporting a rename rather than having to do a copy and delete which has some bill downsides. Can you tell me if the minios://… implementation supports the more efficient rename?

Thanks heaps.

Wasabi us-east-2 region doesn't want to work with Duplicacy (GUI)
#7

Actually, in answering part of my own question above, I have found a simpler way to get a CLI configuration properly in the GUI. Looks like you can configure the correctly on the CLI. Then add a job to the GUI and select the repository and it will pull in the correct config. So that part of the problem is solved.

1 Like
#8

Similar issue on my end with West. Can’t even get duplicacy to see the bucket.

#9

I have discovered another side affect of this issue that I thought I had better point out and help others with if they are having the same issue. I have been struggling to get Duplicacy to run successfully as a service.

My 1st issue getting this going is nothing to with the problem with Wasabi configuration outline above, but just in case anyone else struggled to find it, see Can duplicacy run as a service on Windows? which outlines how to install Duplicacy as a service (you need to run as administrator or run duplicacy --install for administrator).

But the next issue, was related to this issue. Despite the fact that I had duplicacy installed as a service, backups where still not running. They ran fine when I ran them manually, and they even ran fine if I had the GUI open and left that run the backups (seems when you have the GUI open, it pauses the duplicacy service". But if I did not have the GUI open, the backups would not run. If you had a look at the logs, I saw :-

2018-10-31 19:30:30.701 INFO STORAGE_SET Storage set to minios://us-west-1@s3.us-west-1.wasabisys.com/xxxxxx/xxx
2018-10-31 19:30:30.704 INFO PASSWORD_MISSING gui1_s3_id is not found in Keychain/Keyring
2018-10-31 19:30:30.704 INFO PASSWORD_MISSING gui1_s3_secret is not found in Keychain/Keyring
2018-10-31 19:30:30.705 INFO PASSWORD_MISSING gui1_password is not found in Keychain/Keyring
2018-10-31 19:30:30.705 ERROR STORAGE_CONFIG Failed to download the configuration file from the storage: EmptyStaticCreds: static credentials are empty
[19:30:30] C:\duplicacy-backup Backup operation returned an error:
ERROR Failed to download the configuration file from the storage: EmptyStaticCreds: static credentials are empty

So somehow when running as a service, it is not finding the keys, despite it working when I run it manually or let a actual GUI session run it. So then I found this post PASSWORD_MISSING Authorization Failure . It appears the issue is that the CLI does not set the correct flags on the keys to enable the local system account that is used by the service to read the keys, and thus the problem. One of workarounds outlined in that thread is to re-setup the repository using the GUI rather than the CLI. However, as far as I can tell, this is not an option for anyone setting up a storage service in region west with Wasabi due to the problem outline above.

The workaround I have used is to change the service to logon using a local administrators account. This is not ideal, but it does appear to work as a workaround for those looking for a solution to this issue.

But I hope this Wasabi in region west issue will be properly addressed in a future update to avoid this any other issues. Do we have any clue when the new release might be out, and if it in fact fixes the issues with Wasabi / Region West and Duplicacy?

#10

Do remember though, that you not only need to close the GUI, but you have to go to the icon tray and quit it, for the service to resume.

I, too, have done this on the 2 PCs on my network (and suggested a friend do the same), for different reasons. But the service just doesn’t seem to work reliably for me.

On my brother’s PC for example, the Duplicacy service for some reason prevents the OS going into standby mode automatically through inactivity. Even when I manually put the PC to sleep, and later wake it up, the service doesn’t run a backup - no logs, no nothing. So I changed it to run, via Task Scheduler to start up on login, with elevated privileges (since it needs volume shadow copy).

(This old PC has a separate problem where network shares are sometimes inaccessible after waking from sleep, so I get backup errors there in what is actually quite an annoying recurring popup window - but at least it’s running backups more frequently than not at all.)

My friend’s PC was the same, albeit without the standby issues - it just wouldn’t backup on schedule. Changing from service mode to start at login via Task Scheduler, fixed the issue for now.

On my own PC I just decided to do without the service since I leave the PC on 24/7 anyway and I’d quite like the handy icon in the tray (which I can’t have if running as a service) and I don’t want to have to remember to quit it in order for backups to run. It’s really quite awkward.

I didn’t report these issues yet because I know a new GUI is under development. But if the service / tray code is similar, it might behave similarly. Hopefully something can be done to improve reliability and make the whole experience more intuitive and consistent.

#11

I just got back from Wasabi people who said they will fix this in 2 days. Therefore no code changes are needed.

@Droolio the new GUI is taking a different approach towards the service mode. In fact, since the web GUI is already separated from the main program, you can easily run the main program as a service and then control/manage the backup jobs from the browser.

3 Likes