Little help with include/exclude filter choices on unRAID please :(

Support
#1

I’m finally moving from Borg to Duplicacy on my unRAID server. I’ve got it setup up ok via the Docker, but i’m struggling with the filters (again). I got Duplicacy set up fine on my parents/wife’s Windows PCs but that was not without difficulty. I really don’t remember how it goes, and yes i’ve read the FAQ but i really struggled with it way back then and i struggle with it now.

The first thing that always confuses me is the initial state. Is it everything included or excluded by default?

unRAID has it’s ‘user defined’ shares, and the usual shares you get when you run Dockers and VMs. I’ve got the usual stuff, photos, docs, videos etc. But i also have a couple of subdirectories that i’d like to backup, while excluding everything else in there. Here’s what i think will work, but i’d appreciate some guidance.

Note the Backups & Work directories have some subdirectories i want to include and some directories i want to exclude. And i want to exclude iso files from my Apps directory.

The start point is /mnt/user for those who know unRAID.

-*.Recycle.Bin/
-*.Trash-1000/
-*.Trash-99/
+appdata/*
-appdata/macinabox/
-appdata/shinobi/

+Work/*
-Work/Legacy/

+Backups/unRAID-Cache/
+Backups/unRAID-Cache/appdata/*
-Backups/*

+Apps/*
-Apps/*.iso

+Documents/*
+Photos/*

I’ve just started a -dry-run with different settings which i don’t think is what i’m after. It’s got a few hours to run (overall it’s about 3tb of data). I’ll wait for it to finish and review the log, but i’d figured i’d ask about these settings as i put more thought into it after starting the previous one.

Thanks. Hopefully someone out there uses Duplicacy on unRAID and knows straight away what works best.

#2

it depends on whether you have include or exclude rules or both. E.g. if you only have exclude rules — then everything else is included. But it’s best to explicitly add the default in the end: e.g +* or -* to remove any ambiguity.

-*.Recycle.Bin/
-*.Trash-*/

+appdata/
-appdata/macinabox/
-appdata/shinobi/
+appdata/*

+Work/
-Work/Legacy/
+Work/*

+Backups/
+Backups/unRAID-Cache/
+Backups/unRAID-Cache/appdata/*
-Backups/*

+Apps/
-Apps/*.iso
+Apps/*

+Documents/*
+Photos/*

# finally, exclude everything else
-*

I don’t have unraid, but my suggestion would be to backup everything indiscriminately. Storage is cheap, and storage cost savings don’t justify risk of making a mistake with filters (accidentally excluding important or new stuff) and extra work required configuring and then maintaining the filter list. Best filter list is an empty list.

1 Like
#3

Thank you @saspus :heartbeat:

Are the # comments allowed in the filter within Duplicacy? It would help me in future.

I don’t find storage cheap :frowning:, unless it’s local storage, that’s not too bad, but i would like to also upload to the cloud which is hundreds of dollars a year over 2TB. 2TB seems to be a pretty good sweet spot. My use is personal/family.

Thanks again for the help. I wasn’t too far off and your assistance made it click. Hopefully i’ll understand from now on. I already have 3 paid licenses (Windows machines), looks like i’ll grab a 4th for unRAID :slight_smile: Duplicacy seems to have really gained traction amongst the unRAID community.

1 Like
#4

Yes! But it has to be the first symbol in the line: Include patterns for symbolic links - #16 by gchen

I find the opposite to be the case: local reliable storage (we are talking performant ZFS array that guarantees data integrity with caching and scrubs) is ridiculously expensive: upfront and recurring cost of hardware, maintenance, electricity… but it’s convenient sometimes, so we overpay. Cloud storage provides do that on scale and can offer the service cheaper, and better.

Unfortunately duplicacy does not support cold storage (yet, hopefully)— I’m using Glacier Deep Archive with a competing solution and just as you, family stuff mostly, around 2.5TB total, and I pay under $3/month. This does not even register as an amount of money.

The cheapest service duplicacy can access is STORJ at $4/month/TB — it also happens to be decentralized and super fast, and being hot storage is generally an overkill for backup scenario, but even that 2.5TB gets you to $10/month. Two cups of coffee.

In contrast, running my home server (Time Machine target/plex/homebridge) costs me $33/month in electricity alone!

Good to hear, I’m more of a FreeBSD/TrueNAS guy, but good for duplicacy :).

#5

Yeah, for local i’m just using “another HDD on another device” :smiley: I figure if i get cloud, then i don’t need the ZFS multi-whizbangery. It’s a convenient local backup, with the cloud being the emergency backup. So one big fat HDD and a low-powered, low-cost PC/Pi to run it from.

That’s pretty good. I hadn’t heard of STORJ before. It’s only a little cheaper than Google Workspace, though Workspace also gives other bonuses like email domain. The main problem with Google is the price jump from 2tb to 5tb is pretty significant. STORJ you pay only for what you need.

Ideally i’d set up a system at my parent’s place for my ‘cloud’, but i’m pretty terrified of the security risks. I do not know enough and to simply follow a guide that intends to provide the safe/secure setup, and then likely not properly maintain it ('cause i’m not a network/system administrator) may not the safest thing to do.

1 Like
#6

Google worksplace at $12/month plan comes with unlimited storage, de facto. So if you have huge amount of data — then google drive, being fixed cost, is, essentially, asymptotically approaches free/TB/month.

People used that since it was g-suite and this “wink wink loophole” migrated to Workspace and continues to work. So that’s an option.

I’m personally against that — I prefer to pay for what I use — not for what other customers use on average — but it’s just me. I can find old discussions on this topic on this forum later for you. But if you don’t mind — then this is a viable alternative.

Btw you still pay for domain separately, and hosting/routing email is not a problem today: from cloudflare to apple everyone offers email services for your domain these days.

But I agree — with google you pay for services mostly, storage is incidental to their SAAS, not a main product, so I can see why they don’t enforce any quotas on any plans starting from Standard. Using that storage as a bulk storage however is a questionable idea. It simply wasn’t optimized for that

STORJ is really interesting and way underpriced — comparable multi-region hot storage on Amazon is way over $20/TB. But it’s a startup, so they need to get a foot in the door and to compete with B2. But resilience, durability, and performance (throughput; time to first byte is quite high there) wise — nothing comes close.

#7

I thought they closed that loophole? Is there anything special i need to do to get it? I’m never going to be that guy who uploads 30tb of movies or whatever, but if i can get ~3tb today and know it’ll be 5tb in a few years and i don’t pay more then i would be set.

Please do! :sweat_smile:

#8

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.