Best practice for single NAS + Cloud backup?

[klayhamn]

So, I’m about to set up my first-ever NAS soon (already ordered 3x16TB HDDs ) based on a Synology system (420+)
I’m also pretty sure I want to use Wasabi as the (main) cloud backup since I like the fact that I can easily create buckets with different regions on the same account there (and the no-cost for egress)

What I’m still not sure is what would be the best practice in terms of the backup flow -
At least some of my important data would sit on my own desktop computer – and I would wanna use the NAS as a backup destination for it, but - I would also want the same data to be backed up to the cloud

It is my understanding that if I just backup (using duplicacy) from PC → NAS it already creates chunks at the destination, and then, if i try to backup (to the cloud) the chunks themselves using duplicacy i probably end up with something inefficient -
but - backing up separately to the two destinations also feels a bit weird - since it seems like a lot of duplicate work is being done (and a lot of unnecessary strain on the PC and its HDD’s)

Would the best practice be something like backing up to the primary destination (NAS) first using duplicacy, followed by a simple sync of the data into the cloud, from the NAS?

and then, is that what the copy task is for? and how does the copy differ from something like rclone 's copy - if at all?

[gadget]

@klayhamn

So, I’m about to set up my first-ever NAS soon (already ordered 3x6TB HDDs ) based on a Synology system (420+)

The base 2GB of RAM in the DS420+ is likely to be too low if you’re planning on running any additional services besides SMB and/or NFS along with Duplicacy if the volume of data to be backed up is going to be hundreds of gigabytes, so consider maxing out the empty memory slot (4GB DIMM for a total of 6GB).

@klayhamn

I’m also pretty sure I want to use Wasabi as the (main) cloud backup since I like the fact that I can easily create buckets with different regions on the same account there (and the no-cost for egress)

There has been some debate on this forum regarding Wasabi’s long-term future and actual cost effectiveness compared to alternatives.

I’ve used a lot of cloud storage services, but not Wasabi, so the only advice I can offer is that because you’ll have your own NAS as your first line of defense, even if there were an egress fee the cost would be reasonably low. For example, Backblaze B2 charges $0.01/GB/month for downloads, so restoring 1TB of data (likely equal to at least 2TB of uncompressed files for anything but music and videos) would cost $10 – a small price to pay if recovering from a cloud backup is the last resort. Don’t let an egress fee be a deal breaker for an otherwise great cloud storage provider.

@klayhamn

What I’m still not sure is what would be the best practice in terms of the backup flow -
At least some of my important data would sit on my own desktop computer – and I would wanna use the NAS as a backup destination for it, but - I would also want the same data to be backed up to the cloud

A fairly common scenario…

@klayhamn

It is my understanding that if I just backup (using duplicacy) from PC → NAS it already creates chunks at the destination, and then, if i try to backup (to the cloud) the chunks themselves using duplicacy i probably end up with something inefficient -
but - backing up separately to the two destinations also feels a bit weird - since it seems like a lot of duplicate work is being done (and a lot of unnecessary strain on the PC and its HDD’s)

Would the best practice be something like backing up to the primary destination (NAS) first using duplicacy, followed by a simple sync of the data into the cloud, from the NAS?

Yes, as far as backing up to the NAS first, then uploading from the NAS to the cloud. Efficiency also increases as more devices are added to the same storage destination on the NAS.

Rsync, rclone and even the cloud storage provider’s own sync client could be used to mirror Duplicacy’s storage archive to the cloud. Some people prefer to use just one program for all parts of the process, while others don’t mind combining Duplicacy with rsync, rclone and/or some other transfer tool.

@klayhamn

and then, is that what the copy task is for? and how does the copy differ from something like rclone 's copy - if at all?

Yes, Duplicacy’s copy command is for replicating the chunks for one or more existing snapshots from one supported storage to another. Depending on the needs, the chunks can be bit identical or not (e.g., decrypted and then re-encrypted with a different key).

With rclone, the most common choice is to simply mirror/sync Duplicacy’s backup directory to another storage destination. Rclone has a lot of other features including encryption, but it’s generally not necessary (Duplicacy already offers two encryption options) and would mostly complicate disaster recovery (e.g., more encryption keys to manage).

[klayhamn]

@gadget

The base 2GB of RAM in the DS420+ is likely to be too low if you’re planning on running any additional services besides SMB and/or NFS along with Duplicacy if the volume of data to be backed up is going to be hundreds of gigabytes, so consider maxing out the empty memory slot (4GB DIMM for a total of 6GB).

After looking this up - I saw that many had success with putting in RAM modules of larger size (e.g. 8GB or 16GB) - is that not advised?

@gadget

There has been some debate on this forum regarding Wasabi’s long-term future and actual cost effectiveness compared to alternatives.

What do you mean long-term future? i.e. are they gonna be around?

@gadget

I’ve used a lot of cloud storage services, but not Wasabi, so the only advice I can offer is that because you’ll have your own NAS as your first line of defense, even if there were an egress fee the cost would be reasonably low. For example, Backblaze B2 charges $0.01/GB/month for downloads, so restoring 1TB of data (likely equal to at least 2TB of uncompressed files for anything but music and videos) would cost $10 – a small price to pay if recovering from a cloud backup is the last resort. Don’t let an egress fee be a deal breaker for an otherwise great cloud storage provider.

That’s an interesting POV but I hate the concept of egress costs in principle – it is sort of like having the data held captive for ransom. You could say the monthly storage fee is just another form of “ransom” supposedly… Dunno, maybe it’s all psychological, but it feels to me like it’s just something they could always easily hike up unlike the storage-fee which is more competitive I guess…?

Also, I can easily see myself making a bunch of mistakes or inefficient decisions when restoring data, forcing me to pay for egress more than once - and thus incurring unnecessary costs (i’m always more of a trial-and-error than fully-plan-out kind of person)

In any case, i thought about using both clouds as backup for critical data (only) - considering my critical data is less than 1TB

@gadget

Rsync, rclone and even the cloud storage provider’s own sync client could be used to mirror Duplicacy’s storage archive to the cloud. Some people prefer to use just one program for all parts of the process, while others don’t mind combining Duplicacy with rsync, rclone and/or some other transfer tool.

So it’s just a matter of personal preference? there’s no important distinction there?

@gadget

Yes, Duplicacy’s copy command is for replicating the chunks for one or more existing snapshots from one supported storage to another. Depending on the needs, the chunks can be bit identical or not (e.g., decrypted and then re-encrypted with a different key).

With rclone, the most common choice is to simply mirror/sync Duplicacy’s backup directory to another storage destination. Rclone has a lot of other features including encryption, but it’s generally not necessary (Duplicacy already offers two encryption options) and would mostly complicate disaster recovery (e.g., more encryption keys to manage).

Ya i don’t think i’ll wanna re-encrypt anything — though, this does raise an interesting point that i didn’t fully consider: if i backup to the NAS and then sync to cloud - the files in the NAS are basically not in “usable” form –

so i could do it the other way around: i.e. sync (i.e. copy) to the NAS, and then backup from there to the cloud

What’s considered the better approach?or are there any things to be aware of when choosing one over the other?

[1wanderingexpat]

I don’t use duplicacy to backup any desktop/laptop to my NAS. That way, the data on my NAS is immediately available for restore in the same format it was backed up in (I have mixed OS systems in my LAN). I use Duplicacy-web in a docker container on my NAS to backup my NAS data to cloud.

[klayhamn]

Ya for some reason this kind of setup only occurred to me now

it does make more sense, but I thought maybe the backup process (vs sync) is more resource intensive which might create a bottleneck… do you mind saying which NAS you use (or what kind of resources it has?)

[towerbr]

There are two options:

If the NAS and cloud storages are -bit-identical, you can copy from NAS storage to cloud storage using Duplicacy or a sync tool (Rclone, rsync or cloud provider client).

If the storages in NAS and cloud are not -bit-identical, you can only copy using Duplicacy (because the chunks will be different in the two storages).

I particularly recommend using Duplicacy itself to do the copying (whether the storages are -bit-identical or not), for a simple reason: when it reads the chunks from the source storage, it decrypts and re-encrypts them to upload them to the target storage, so in practice you are doing a check and a copy in the same operation .

[klayhamn]

@towerbr sorry maybe i misunderstood something but —

if I do go with the option of just syncing from the PC to the NAS (rather than a proper de-duplicative “backup”) - there won’t be any chunks, and the only backup flow would be NAS->Cloud
right? would a “duplicacy copy” or an “rclone copy” operation in that scenario be considered “bit-identical” in that case?

Is there anywhere where the topic of “bit identical” is fully explained? cause I’m not sure i understand what it means and I think it might be useful to understand even if i don’t end up needing it

[towerbr]

No, the two options I mentioned are for this scenario you initially proposed:

@klayhamn

At least some of my important data would sit on my own desktop computer – and I would want to use the NAS as a backup destination for it, but - I would also want the same data to be backed up to the cloud

Remember: backup and sync are different things. If a file is corrupted on your computer, the corrupted copy will also be synced to your NAS. If on the other hand you are backing up to your NAS, you can simply restore “yesterday’s version” of the file that was corrupted today. Same reasoning applies for ransomware, etc.

Both options have advantages and disadvantages, depending on how you want to manage your backups and copies. I do a sync to cloud myself and from there I make a backup with Duplicacy to S3 buckets.

So there are two scenarios:

• sync from computer to NAS (with some other tool) and backup from it to cloud with Duplicacy;

• backup from computer to NAS with Duplicacy and copy from it to cloud with Duplicacy.

[klayhamn]

@towerbr

Both options have advantages and disadvantages, depending on how you want to manage your backups and copies. I do a sync to cloud myself and from there I make a backup with Duplicacy to S3 buckets.

Wait did you mean - sync to the NAS? otherwise, which cloud are you syncing to, and how are you backing up from it to S3?

@towerbr

So there are two scenarios:

• sync from computer to NAS (with some other tool) and backup from it to cloud with Duplicacy;
• backup from computer to NAS with Duplicacy and copy from it to cloud with Duplicacy.

right, i also see your point regarding the corruption on computer, i suppose it makes sense to also maintain a direct backup of critical data (personal files, etc.) - but i think for non-critical media (e.g. movies) this would be an overkill probably

[towerbr]

@klayhamn

Wait did you mean - sync to the NAS?

Use a tool like Rclone, rsync, Syncthing, etc to keep your computer files synchronized with NAS.

@klayhamn

which cloud are you syncing to

Common Google Drive.

@klayhamn

and how are you backing up from it to S3?

I use a cloud VM to mount google drive with rclone mount and run Duplicacy backup on VM.

@klayhamn

but i think for non-critical media (e.g. movies) this would be an overkill probably

Here there is an additional detail: media files (movies, music) do not benefit from deduplication. So a more efficient solution is to synchronize them. I use Rclone for this type of file.

[klayhamn]

oh interesting, thanks for the clarification -
do you mind sharing why you chose to replicate your data to both gdrive and s3? is that just a form of redundancy?

@towerbr

Here there is an additional detail: media files (movies, music) do not benefit from deduplication. So a more efficient solution is to synchronize them. I use Rclone for this type of file.

oh… then what type of files do benefit? photos? text?

[towerbr]

@klayhamn

do you mind sharing why you chose to replicate your data to both gdrive and s3?

Sure, but see, you’re mixing the two concepts: I replicate (sync) to google drive, and I backup to S3.

I sync to google drive so the files are available on more than one device (in my case, two computers, my phone and a tablet). So I can work with them in different ways.

And the backup to S3 is a… backup. I have hundreds of versions of each file, and if I need to, for example, restore a version from 1 year ago, I can do it perfectly.

Some people might say, “but why would you need a 1 year old version?” so let me give you a real example that happened to me. A very important spreadsheet got corrupted in February 2021. I don’t know what corrupted it (an error on my disk, an excel error, etc), the fact is that in February of this year, after 1 year without accessing it, when I went to open it to make the “2022 version”, I obviously couldn’t. But since I had the backups, it took a few minutes and I restored the latest uncorrupted version of the spreadsheet, from early February 2021.

@klayhamn

oh… then what type of files do benefit? photos? text?

Files that have many versions or are continually edited. If you have a 1MB text file, edit a few words and save a new version, instead of your backup being 2MB for both versions, deduplication will only upload the few KB changed, and your backup will have, say, 1.1MB. Multiple this across thousands of files and hundreds of versions and the magic of deduplication will be done.

Files that are similar to other files. Imagine that you do a “save as” of the file above with 1MB, generating a new copy. Your backup, instead of having 2MB for the two files, will also be just over 1MB, because most of the chunks of the first file will be used.

The same applies if you have, for example, multiple copies of a photo. The backup of all will be little larger than the size of just one.

My comment about media files is more related to media libraries (movies and music) of which we usually don’t have more than one copy in use.

In other words, in a simplistic way, deduplication will be efficient where “chunks” of files can be used to back up other files.

[sevimo]

@towerbr

Here there is an additional detail: media files (movies, music) do not benefit from deduplication.

Not entirely accurate. High-entropy files like audio/video/etc do not benefit from dedups between different files, but they still benefit from dedups with themselves. The most obvious use case is moving files around (e.g. re-organizing your photo collection). With sync you only get the latest version of your structure (not to mention that your deleted files are gone, though this can be alleviated somewhat). Deduped backup will allow you to store all your previous perturbations of the folder structure without using (much) additional storage for each snapshot.

[kgorlen]

@klayhamn

What I’m still not sure is what would be the best practice in terms of the backup flow -
At least some of my important data would sit on my own desktop computer – and I would wanna use the NAS as a backup destination for it, but - I would also want the same data to be backed up to the cloud

Rather than storing data on desktop PCs, I store it on the NAS and access it via network folders (see Properties -> Location on Windows). That way, it’s not critical to backup the desktop PCs, and all data is accessible from any PC on the LAN. I then backup the NAS to directly attached RAID storage using Duplicacy, and copy that to Backblaze.

I do back up system images of the PCs to the RAID storage a few times a year using the Duplicacy CLI, but I don’t copy these to Backblaze because they are large and non-critical.

[klayhamn]

@kgorlen

Rather than storing data on desktop PCs, I store it on the NAS and access it via network folders (see Properties -> Location on Windows). That way, it’s not critical to backup the desktop PCs, and all data is accessible from any PC on the LAN. I then backup the NAS to directly attached RAID storage using Duplicacy, and copy that to Backblaze.

That’s another great point…
I’m really glad i made this thread - you all bring up stuff I didn’t fully think through

But is the behavior of this “mounted” style of usage good enough to be indistinguishable for regular use? e.g. - if i watch movies directly through the network drive, move files around, do searches, etc. - how much of a performance hit should i expect (assuming the NAS hosts 16TB WD red pros) ?

Also, going through this path leaves me with one less copy of the data, which kind of goes against the 3-2-1 principle, no?

@kgorlen

I do back up system images of the PCs to the RAID storage a few times a year using the Duplicacy CLI, but I don’t copy these to Backblaze because they are large and non-critical.

What do you use to make these images?
what would you consider the main purpose of this process? to be able to maintain uninterrupted operation in the event of a corrupted OS and/or failure of the main SSD etc.?

[kgorlen]

@klayhamn

But is the behavior of this “mounted” style of usage good enough to be indistinguishable for regular use? e.g. - if i watch movies directly through the network drive, move files around, do searches, etc. - how much of a performance hit should i expect (assuming the NAS hosts 16TB WD red pros) ?

I’ve been running everything off the NAS for over 10 years, so I’m not sure if the performance is “indistinguishable”. My most I/O intensive applications are Lightroom and Photoshop, which ran just fine over a 1Gbps connection. I recently upgraded to 10Gbps, and haven’t noticed a dramatic improvement. (Note that the max transfer rate of SATA III hard drives is 6Gbps.) Each PC has a second SSD that I thought I’d use for temp storage when working on large image files, but I rarely bother to use this anymore. Performance might be an issue for something like editing video files.

My recollection is that Windows search doesn’t support network drives. I’m using X1 Search 8.6.2, an old version. Synology has an add-on search tool. (I have a QNAP, and have found their Qsirch app to be virtually useless.) The advantage of a NAS-based search app is that it can maintain a single copy of its index on the NAS without transferring files over the LAN.

@klayhamn

Also, going through this path leaves me with one less copy of the data, which kind of goes against the 3-2-1 principle, no?

There’s a copy on the NAS (two if you count RAID 1 mirroring), a local copy in the RAID storage (another 2 if you count RAID 5 as redundant), and an offsite copy at Backblaze.

@klayhamn

What do you use to make these images?
what would you consider the main purpose of this process? to be able to maintain uninterrupted operation in the event of a corrupted OS and/or failure of the main SSD etc.?

I use the free version of Macrium Reflect, and I’ve also successfully used AOMEI Backupper Free. A system image avoids having to reinstall/reconfigure the OS and apps from scratch in the event of OS corruption or hardware failure, which would save me a couple of man-weeks effort. With all data on the NAS, should a PC fail, one can just use another until it’s repaired/replaced.

There are some files that are useful to have resident on each PC, for example, some application data (Adobe in my case), the desktop, password safes. These I synchronize across all PCs through the NAS using FreeFileSync. Since there’s a copy of these on the NAS, they’re backed up with everything else.

I also sync various folders on our phones and tablets with the NAS, so those are backed up too.

[gadget]

@klayhamn

@gadget

The base 2GB of RAM in the DS420+ is likely to be too low if you’re planning on running any additional services besides SMB and/or NFS along with Duplicacy if the volume of data to be backed up is going to be hundreds of gigabytes, so consider maxing out the empty memory slot (4GB DIMM for a total of 6GB).

After looking this up - I saw that many had success with putting in RAM modules of larger size (e.g. 8GB or 16GB) - is that not advised?

Generally speaking, it’s best to follow a manufacturer’s recommendations but there’s often some leeway in the specifications because the recommendations are chosen based on components readily available at the time and to allow for variances in every component.

Synology’s DS420+ uses Intel’s Celeron J4025 which was released in late 2019. Intel states that the maximum supported RAM is 8GB (ideally as a pair of 4GB modules for dual-channel performance). However, others on this forum, Reddit, YouTube and elsewhere have reported success with higher capacity modules.

Is it advisable? No. Should you do it? Sure, the risk of damage is tiny as long as the memory module is the right type and well made. Pick a module that other DS420+ owners have confirmed to work without any issues.

Having said that, a few things to consider…

Synology says that the DS420+ consumes 28.30 watts when the internal drives are being accessed (based on a stock unit populated with four WD10EFRX, aka “1TB Western Digital Red Plus”).

A WD10EFRX pulls 3.3W during read/write activity, a Celeron J4025 has a 10W TDP at full load, and 2GB of DDR4 requires about 0.75W (rule of thumb is that every 8GB of DDR4 requires 3W of power).

The WD Red Pro has a 7200RPM spindle speed so it requires more power than the Red and Red Plus models. The 6TB Red Pro pulls 7.2W during read/writes (+3.9W compared to a WD10EFRX).

Then there are also the pair of M.2 slots for optional SSDs.

16GB of DDR4 would increase the power draw by just over 21% on the 28.3W baseline reference. It’s extra load on the mainboard, power supply and cooling fans. A mainboard manufacturer would have selected components (e.g., capacitors, resistors, voltage regulators, etc.) for the intended maximum memory capacity. Likewise, Synology would have designed the DS420+ assuming a total of 6GB of DDR4.

I wouldn’t recommend upgrading the RAM immediately. It’s best to stress test the NAS for at least a few days or more just in case there’s a need to call tech support and/or get a RMA. Having an unsupported amount of RAM will complicate tracking down any stability issues (Synology says each year they receive 50+ support tickets as a result of memory upgrades).

After upgrading the RAM, run a full memory test to check for a defective memory module, compatibility, plus stability under extended high load and heat (Synology uses Memtest86 under the hood for its Synology Assistant add-on).

[gadget]

@klayhamn

@gadget

There has been some debate on this forum regarding Wasabi’s long-term future and actual cost effectiveness compared to alternatives.

What do you mean long-term future? i.e. are they gonna be around?

Wasabi Technologies, Inc. was founded in 2015 and started accepting customers in June 2017, so it’s been in business for just over 5 years.

As a private company, reporting to the SEC and the media isn’t required so detailed financial information isn’t readily available.

Most startup companies seek out venture capitalists for seed money in exchange for interest payments, equity and/or profits (e.g., “Shark Tank”). Wasabi decided instead to recruit angel investors and family offices (the former is an individual private investor; the latter is a private company set up to invest a family’s money). There are pros and cons to each one so it doesn’t mean Wasabi made a poor choice.

From Wasabi’s press releases that sometimes mention the results of new funding rounds, it’s been able to raise around $275 million so far (averaging ~$39.3M per year since its founding).

Is $40 million a year in investor funding plus customer revenue enough to cover the office staff, software developers, engineers, electricians, plumbers, security, etc. plus the utilities, facilities and other operating expenses in multiple locations around the world? I’m not sure. For comparison, Walmart spends over $25 million a year maintaining just its website. It could be that Walmart hasn’t been getting the most for its money (unlikely), or that Wasabi has been able to live on a shoestring budget. But based on Wasabi’s repeat funding rounds – most recent one I could find was last year in November 2021 – it’s been dependent on debt and private financing to help cover expenses (there’s very little public info on if/when any corporate bonds are nearing maturity).

Given how many cloud storage providers have come and gone over the last 20 years, the odds aren’t great for Wasabi, especially with Amazon, Google and Microsoft as the competition. The barrier to entry is pretty low with CrashPlan, Jungle Disk and many others being built on top of S3 / GCS / Azure.

But at the same time, the global volume of data keeps expanding by leaps and bounds so the “pie” also keeps getting bigger and bigger.

[1wanderingexpat]

Sorry, was out of touch for a bit; I use a DS920+ It runs a 4-core Intel Celeron at 2ghz. I’ve upgrade the ram to 20 (added a 16gb stick). It has the capacity for 2 nvme cache drives, but the reviews are mixed on them, so I’m not using them at all. It’s a great NAS & the performance is magnificent.

[klayhamn]

@gadget

Given how many cloud storage providers have come and gone over the last 20 years, the odds aren’t great for Wasabi, especially with Amazon, Google and Microsoft as the competition. The barrier to entry is pretty low with CrashPlan, Jungle Disk and many others being built on top of S3 / GCS / Azure.

I see, well - that’s a good point but i also don’t see what’s the big deal for the end-user?
It doesn’t seem likely that the data would just evaporate out-of-the-blue one day,
seems like a way more likely scenario is that they get bought out by a bigger fish

and in any case seems likely that clients should have enough time to find alternatives, no?

@1wanderingexpat

Sorry, was out of touch for a bit; I use a DS920+ It runs a 4-core Intel Celeron at 2ghz. I’ve upgrade the ram to 20 (added a 16gb stick). It has the capacity for 2 nvme cache drives, but the reviews are mixed on them, so I’m not using them at all. It’s a great NAS & the performance is magnificent.

there’s like a $60-70 difference between the 420+ and the 920+ where i live, and even though the different is small I just can’t bring myself to spend it, considering that this is money that could go to a dedicated solution if i ever really needed one, rather than try to make my NAS into a general-purpose server
I read a review (i believe it’s the “NAS compared” channel on youtube or whatever) that said that basically the difference is mostly felt in cases of horizontal scaling (e.g. multiple users doing different things, or various tasks running simultaneously, e.g. surveillance cameras, backups, media streaming, etc.)

---

and regarding the RAM - ya those are good points, i’ll probably see how it operates and then just add the 4GB stick if necessary…