Transferring backup between providers, B2, cloudflare, storj

Thanks for replying good points.

I am currently using Google Drive but are considering moving to Backblaze B2 as it should be cheaper. Still doing my research on B2.

Do you know if egressing to Cloudflare with B2 requires a public bucket and Cloudflare domain/setup or is it an option to just tell B2 to download via Cloudflare instead so it just works with Duplicacy restore and you get no B2 download fees?

Thanks

IIRC bucket shall be public, it was long time ago I played with this. This does not matter though – you data is encrypted anyway.

B2 however is still hot storage. They are competing with Amazon’s hot storage and therefore they can claim they are “cheaper”. For backup hot storage is an overkill, by using even cheap hot storage, you are still overpaying for access performance you don’t need

Consider Amazon cold storage tiers instead. While duplicacy does not support ultra-low cost archival tiers – even instant access ones can provide better cost structure.

For example, S3 Intelligent - Tiering, and S3 Glacier Instant Retrieval costs under $0.004/GB. And you have 100GB/month of free egress

With amazon storage the concern is usually - what if I have restore everything? The restore cost will be astronomical. Yes, kind of. Restore is expensive, if you want all data back asap in its entirety. But this cost must be multiplied by the probably of needing to restore. Which is very low. Ideally, you never will have to restore full backup – but you pay for storing it every month. Hence, taking a hit on restore cost but optimizing the storage costs makes sense.

For small restores, such as revert few files, etc – 100GB free egress per month covers that.

I’m not suggesting to blindly switch to amazon though – you need to be careful about the cost and plan accordingly, but with the right planning this can be very cost and performance effective.

If you go hot storage route – consider Storj, instead of backblaze. The cost per TB stored is lower than B2, egress is lower, you get 150GB monthly traffic and storage free, and, more importantly, this is distributed, geo-redundant storage. I really like it, consider it :slight_smile:

1 Like

Thanks for suggestions.

I had excluded Storj from my considerations but I am now looking at it again.
Not 100% sure why I excluded it so quickly :slight_smile:

  1. Just to be sure are you referring to https://www.storj.io?

  2. It reads as Storj encrypts anything you upload before it is distributed but is that done at their end so there is no overhead on the machine performing the backup with Duplicacy? I am confused it mentions having to select a passphrase for server side encryption several times on the help pages.

  3. On pricing information it mentions additional per-segment fee of $0.0000088 applies - any idea what this means? What is a segment? Is this going to add up to be a large hidden cost?

Huge Thanks

I did too! It was unusual and scary. But then I read white paper, listened to their townhall, read forums, and now I’m convinced :slight_smile:

Yes.

It’s a bit confusing, but yes, everything is encrypted, but the encryption is not tied to the bucket; it’s tied to the objects. Each bucket can contain objects encrypted with different keys, that’s why it asks for passcode when you open the bucket in the web ui – it will only show objects encrypted with that key, and hide others. It causes confusion…

Also, ideally you use their utility “uplink” with access grant to talk to satellites and nodes; but if you want to use their S3 gateway, then the passphrase is managed by the S3 gateway, and you instead use API key and API secret. It’s a bit less secure, but that’s for S3 gateway mode only. I think duplciacy supports access grant directly (not sure about that, but you definitely can use S3 to access it).

This is what the most confusing for new users: you go and create bucket, and you need to specify a passphrase there. Then you go create S3 credentials, and specify another passphrase. When you upload the data to the bucket using S3 credentials, that S3 related passcode will be used. SO why was I forced to created passcode when creating the bucket then? Many people stumble on it, and complain, including myself in that thread: TrueNAS backups and differences between Storj and Storj (iX) - #16 by BrightSilence - Product Discussions - Storj Community Forum (official). They’ve promised to update documentation and UI for bucket, but with the understanding that the passphrase is not tied to bucket but objects – it makes sense.

The intent is to discourage many small files because of large per-file overhead.

Segment is 64MB. large files are split to 64MB segments, smaller than 64MB files are 1 segment. According to Grafana average segment size is about 13MB, so assuming this for 1TB worth of file segment fee will be about $0.67, if my calculator does not fail me. You can configure Duplicacy to produce chunks close to 64MB to optimize usage, but I would not bother.

1 Like

Yep, duplciacy uses access grant, not S3 gateway: Storj storage backend added

Appreciate the replies it is useful to have someone with first hand experience of a new cloud provider.

  1. To avoid unexpected credit card shocks is there configurable spend limits?
  2. When you started to use Storj did you do a bulk upload of your existing Duplicacy backup? Any tips on how to do this?
  3. Still confused with the flow does this sound sensible? Login to Storj. Create a bucket, create an access grant. When asked for passphrase it is safe to use to browser generated one? Make sure you make a secure note of everything. Bulk upload existing backup. Perform a duplicacy init on the storj storage using the access grant and I should then be able to duplicacy list/backup/restore?

Thanks

No, there are talks about it but I don’t think there are at the moment. You can start with free account with 150GB of storage, and play with it. But with duplicacy backup the egress is minimal (way under 150GB of free monthly egress), ingress is free, segment cost negligible, api cost zero, so you are essentially just paying for storage at $0.004/GB/month.

Or you can pay with STORJ tokens (and get 10% discount for doing so). Then you just transfer tokens to your account and you can’t spend more than the amount of tokens you have there. I use storj for storage (not with a backup program, but just to store data) and pay with tokens. Tokens, that I earn as a storage node operator, sharing extra space on my NAS with the network. It’s brilliant :slight_smile:

I don’t use hot storage, including storj, for backup. But if I was doing that now – I would have created copy-compatible duplicacy repository at storj and used duplicacy copy, if my current cloud provider had free egress. Else – I would have started a new backup.

Yes. The passcode you are forced to choose when creating a bucket you can throw away, it’s useless. The passcode you choose when creating the storage grant is the important one. Once duplicacy uploads stuff, to be able to see that stuff in the web ui you woudl need to unlock the bucket with the passcode from the storage grant, not the one that you had to choose when creating a bucket. Otherwise you will just see empty bucket, which may be a bit unnerving :slight_smile:

1 Like

Many thanks for help. I have a few more things I need to sort out regarding Duplicacy setup but those need to be in a different thread. Thanks for splitting the thread BTW I did try to add to the subject yesterday but I could not edit.

Let me know what subject would be more suitable, I’ll change it :slight_smile: I did not put much thought into it… other than the discussion has diverged in the original thread.

No problems with subjects now I was trying to edit the original to mention we were also discussing Storj/B2 but splitting the thread has sorted it.