Best practice for older/static content? (small macbook SSD, lots of photos) & duplicacy settings

Scenario:

I have several hundred gig of older photos/videos that I want to keep backed up, but don’t really want to keep on my macbook because Apple charges outrageous amounts of money to upgrade the SSD ($400 to go from 1tb to 2tb).

Goal:

  • Have remote storage (external USB or NAS) that holds the static media, let’s say 500gb. I also want this backed up securely in the cloud (B2, etc)
  • Have regular incoming media, say 50-100gb a year, which is both on the macbook SSD & backed up both to local storage and cloud.

i.e.

Macbook:

  • Photos from last 2-3 years easily accessible, backed up to external & B2

External:

  • Old static media (500gb)
  • Incoming media

How would you best configure this? Main concerns:

  1. Can I keep all the media in the same folder on the NAS/external device? I would need a duplicacy backup setting that is smart enough not to “sync” - that is - it should never wipe out the old static files on the NAS if the SSD on the laptop doesn’t have it. Worried esp if I use the same \media folder or whatever.

  2. Where in this workflow would you upload to B2? A one-off for old content, and then regularly from laptop/SSD? If so later I would have to move items off of the laptop to be in the NAS only so not sure of the easiest way to handle this.

Hope this makes sense. Trying to avoid paying $10/mo icloud fees or $400 in SSD fees to Apple. For that much money I could just use B2 at $2-3/month and a $100 external drive or put the $400 towards 8tb of NAS storage.

Few thoughts, in no particular oder.

  1. $400 for 1TB of high-performance flash is not even close to outrageous; I’d say not upgrading (and generally getting anything but the highest-end Mac means getting worse value than possible); on the other hand wasting all that goodness on static immutable media would indeed be a waste.

  2. Photos and videos are incompressible, non-deduplicatable, immutable data which do not benefit from duplicacy or any other versioned backup tool. What you want is to write the photo or video to storage and prevent it from changing forever, because the only reason it would change is that it’s corrupted or encrypted by ransomware.

  3. Something tells me that paying for iCloud storage is not in your plans either, so let’s assume you will have some cloud storage (including the one served from your NAS, for this discussion, as you can always install ZeroTier and have your NAS always assessable in the LAN)

  4. We all also assume that you use Photos Library (you can apply the same logic to any other photo database – C1, Lightroom, or what have you)

  5. Also worth mentioning, even though in my mind it’s a given no manual steps should be required; the tiered storage solution should just work.

So, this is what I’m suggesting:

  1. Storage

Configure rclone remote for your cloud storage destination. Be that unlimited Google Drive or Backblaze, or your NAS over SFTP does not matter. That location will keep full collection of your photos. You can replicate it right from there if you want to. They don’t have to be in plain text, on the contrary, I suggest encrypting any data you upload to any location outside of your mac, including your NAS; rclone can do it transparently via “crypt” remote.

Configure your Mac to mount that location to ~/Pictures/Photos\ Library.photoslibrary/originals with local cache enabled, using macFUSE. Unlike how iCloud manages storage where “optimize storage on my Mac” is selected and offloaded files are replaced by stubs and are missing from the filesystem, virtual filesystem mounted by rclone will behave as if all data is present, while in reality no disk space will be used. Only when you try to read files small portions of these will be cached locally. That means, if Photos needs to extract metadata from a specific file, it will be able to do so without downloading the whole file locally via the magic of sparse files. When you add a new file – it will go to the cache immediately, and then slowly get uploaded to the cloud storage in the background.

As a result, you now have a bottomless SSD-level performing local drive, backed by the cloud service in the actual datacenter.

  1. Backup.

Since those are pictures and videos – you don’t need duplciacy. Just rclone copy them via another rclone remote to another cloud storage provider. For example, another B2 bucket, via special keys that will only allow upload, but not rename, delete, or overwrite. Or just configure replication on B2 itself, so you don’t have to upload the same file twice. Or a million other opportunities.

For other data – documents, projects, source code, etc absolutely do use Duplicacy – to a cloud destination from any of the big players regardless of cost – the amount of data will be negligible and there is no difference in paying between $.001/month and $1/month.

Anecdotally this “mountable cloud storage with cache instead of a local drive or NAS” approach works well for me for quite a time now. Not only it works it is infinitely tunable and customizable. I don’t own NAS(es) anymore, and can’t be happier with the result. I do pay for 2TB storage on iCloud though, as full disclosure, because it’s cheap and well-integrated to the OS; and my mac has a 4TB SSD drive, 2TB of which is allocated for rclone cache… regardless, this solution scales very well, and what’s more important – “just works” without requiring you to babysit it ever and does not include any manual step, after initial configuration. I think it’s a win :slight_smile:

1 Like

The duo Rclone + Duplicacy is unbeatable. They are tools that complement each other perfectly. I use a setup similar to what @saspus described, using Rclone for local mount of cloud-stored volumes and cloud sync of media files. And obviously Duplicacy for what he does masterfully: backups (also in the cloud).

Just a note:

Depending on usage, consider the provider’s egress cost (if any). eg B2 has costs for this, Wasabi (S3 compatible) does not.

2 Likes

I would say the opposite is true. With Wasabi you can only download for free the amount of data stored (per month (IIRC)). When you use that as your mountable drive you can easily exceed that depending on your cache configuration and usage pattern. Anecdotally, my traffic to one of the endpoints that host about 1TB of photos last month was 8TB. Wasabi would be way too expensive in this usecase.

B2 on the other hand is member of bandwidth alliance, so you can route egress traffic via cloudflare and download for free. See --b2-download-url rclone flag. Btw duplicacy also supports that usecase.

But I do agree that usage must be analyzed to select appropriate pricing plan

You’re right, I hadn’t thought of this heavy use case. My usage is very light, the monthly traffic is much smaller than the total stored.

I recently had some nasty surprises with large B2 transfers via Cloudflare using Rclone. Rclone only passes the effective file download requests through Cloudflare, the other API calls are direct. For large amounts of files, these calls can be a cost issue.

But I agree that a Rclone mount using B2 through Cloudflare is an interesting solution.

Oh indeed, I keep forgetting that Cloudflare removes the egress cost, but api cost is still there. But I guess that’s fair, I’d rather pay directly for my usage than indirectly for other people’s abuse. Also, B2 has free tiers of up to 2500 transactions per day, and then they are a cent per 5000 or so; if you are seeing significant cost in that – perhaps there is some issue with excessive calls? For example, b2_list_file_names is expensive, so if your rclone kept refreshing the directory structure (does B2 support --fast-list?) it would become very expensive fast. Rclone’s polling defaults are way too aggressive.

It was only once, in a major transfer out of B2. Lesson learned. :upside_down_face: