Duplicacy and Backblaze B2 API Costs

ssshayagriva · 6 June 2019 21:02

Hi All,

I’m trying to evaluate Backblaze B2 cloud storage along with duplicacy, and my understanding is that Backblaze charges extra for the API transactions (class B and C) as listed here: Cloud Storage Pricing Comparison: Amazon S3 vs Azure vs B2

My intended use case is mostly storing personal documents and pictures/videos on B2, with mostly file additions. File deletions and renames will be rare except from duplicacy revision pruning. I also intend to test my backups every now and then by downloading a set of random files from B2 and comparing their checksums with my local copy etc.

For such an use case, I was wondering if I should be worried about the B2’s API transaction costs or not. I’m hoping not, if someone could confirm it…In other words, I’m curious to find out what duplicacy use scenarios (if any exist) would end up hitting against the B2 API transaction limits frequently.

Thanks,
G.Sibi

leerspace · 6 June 2019 21:15

How much data are you backing up?

FWIW, when I was using Backblaze B2 to backup a little over 1TB (with the default chunking options), I don’t remember ever getting anywhere near exceeding any of the free API thresholds. Backups and prunes were scheduled to run daily. I don’t remember if I had regular checks set up at the time.

ssshayagriva · 6 June 2019 21:17

Right now, I’m planning to back up about ~700 GB. So your data point is very relevant and useful. Good to know, thanks @leerspace!.

snellgrove · 12 June 2019 16:45

Hello,

This is not my experience.

I’m running the web edition and do a backup to local storage, and a copy to B2.

I’ve not finished the initial full copy to B2 yet but something odd happens after a few days of starting a copy. Initially it’s all upload file, class A transactions which are free.

Then I get hit with loads of class C transactions all of which are logging in / Auth (I forget the exact call, buy I’ll attach a screenshot of the B2 report)

Weirdly, copying carries on working even after hitting the class C cap (which it hits every day)

I’m more than happy to work with developers on sorting this out.

Being the web edition, if I stop the copy I don’t seem to get a log file. Adding a switch for verbose logging is not allowed in the web GUI for some reason, so I can’t offer much explanation currently.

gchen · 13 June 2019 03:56

The number of b2_authorize_account in your screenshot is unusually high. I guess what happened was Backblaze B2 had a glitch and kept returning 401 errors for a while, forcing Duplicacy to re-authorize repeatedly. Looking at the corresponding code, I found two issues there:

github.com

gilbertchen/duplicacy/blob/41668d4bbd3652de45f63ae81479be4ef9ce549c/src/duplicacy_b2client.go#L294-L301


func (client *B2Client) AuthorizeAccount(threadIndex int) (err error) {
	client.Lock.Lock()
	defer client.Lock.Unlock()


	readCloser, _, _, err := client.call(threadIndex, B2AuthorizationURL, http.MethodPost, nil, make(map[string]string))
	if err != nil {
		return err
	}

There is no need for every thread to make the b2_authorize_account call. The first one to hold the lock can proceed, but others can skip the call once the first one finishes the call
There should be an exponential backoff if this call continues to fail.

I’ll be working on a fix tomorrow.

gchen · 14 June 2019 13:59

The fix for the authorization issue has been checked in: Better handling of B2 authorization failures · gilbertchen/duplicacy@045be39 · GitHub

ssshayagriva · 14 June 2019 14:17

@gchen, for fixes such as these, is the expectation that whoever needs the fix would pull it from git, and build their own version of duplicacy? Or once you have enough fixes do you bump up the version number? In other words, I was wondering on your release version policy. Thanks.

ssshayagriva · 16 June 2019 18:04

Just to add to @snellgrove 's data point, I have been uploading to B2 for the first time too using the CLI version on Windows 10, and I saw a jump in the class C transactions in the middle of the upload (I’m still using 2.2.1 and I haven’t picked up @gchen 's fixes yet):

Wesley · 19 June 2019 00:02

I’m in the middle of an initial copy to G2 using the Web GUI and I’m also getting an absurd number of calls to b2_authorize_account. How does one implement the fix?

gchen · 21 June 2019 03:01

I just released a new version 2.2.2 on github. If you restart the web GUI it should automatically download this new version and have the authorization bug fixed.

Wesley · 27 June 2019 00:07

Can confirm after a couple of days that it does seem to be working now. Thanks!