Best Practices for Backup Strategy

[Spudster]

Hi. I’m planning my initial backup strategy. I’m backing up large video and audio files, about 20TB of these.

What are the best practices for this file type?

How should I set up the jobs?

What are the requirements for the local database- how much storage should I allocate for Duplicacy?

Are there any tips for the initial backup? For instance, “don’t reboot until the initial scan is completed”.

Also, any best practices guides and white papers would be appreciated.

Thanks.

[Droolio]

Duplicacy is great for versioning of data. It can chunk, de-duplicate, compress and encrypt your 1TB of data quite well and can keep a snapshot history.

For large media though - personally I’d use a tool like Rclone for copying/synchronising to cloud storage or just a file sync tool for local/external storage.

Local database? Duplicacy doesn’t really use one; you only need space for storage on the backup destination.

Initial backup can be interrupted but it’ll take longer to restart. You won’t lose much bandwidth by aborting a backup, but if you have a lot of data to backup and a slow uplink, you could reduce the initial size by moving files/folders temporarily out of the repository until it’s complete. Then move more data in and run another backup. Rinse repeat.

[Spudster]

For large media though - personally I’d use a tool like Rclone for copying/synchronising to cloud storage or just a file sync tool for local/external storage.

Local database? Duplicacy doesn’t really use one; you only need space for storage on the backup destination.

I need backup not rsync.

It has to have a local database- unless it’s stored on the target which would be ridiculous.

[towerbr]

It has to have a local database- unless it’s stored on the target which would be ridiculous.

Duplicacy in fact does not use a local database, and it is precisely this design that makes it so reliable, not presenting the various problems that other tools that use databases present.

Lock free deduplication algorithm

What is novel about lock-free deduplication is the absence of a centralized indexing database for tracking all existing chunks and for determining which chunks are not needed any more. Instead, to check if a chunk has already been uploaded before, one can just perform a file lookup via the file storage API using the file name derived from the hash of the chunk. This effectively turns a cloud storage offering only a very limited set of basic file operations into a powerful modern backup backend capable of both block-level and file-level deduplication. More importantly, the absence of a centralized indexing database means that there is no need to implement a distributed locking mechanism on top of the file storage.

[Spudster]

Thank you for the link.

Ok. Wow. So every time a backup job is run it has to do a gazillion reads on the local storage and the remote storage?

[towerbr]

Exactly, but since this information about the files is obtained by API calls, they are insanely fast.

Take a look at this example: a 22 GB storage on B2, a backup generated for 11 MB of new files, took 1min40sec.

INFO BACKUP_STATS Files: 18883 total, 22,137M bytes; 47 new, 37,209K bytes
INFO BACKUP_STATS File chunks: 20025 total, 22,471M bytes; 11 new, 10,828K bytes, 9,000K bytes uploaded
INFO BACKUP_STATS Metadata chunks: 10 total, 8,381K bytes; 5 new, 6,736K bytes, 2,427K bytes uploaded
INFO BACKUP_STATS All chunks: 20035 total, 22,480M bytes; 16 new, 17,565K bytes, 11,428K bytes uploaded
INFO BACKUP_STATS Total running time: 00:01:40

In other words, reading information about the 18,883 files already stored is very quick.

reads on the local storage

A little correction here. I think you mean a local repository. Take a look:

About Duplicacy nomenclature

This is an example of nomenclatures used in Duplicacy. This case considers two computers (“John” and “Mary”) backing up to the same storage (a single bucket in B2). Since both computers have “documents” folders, deduplication can save storage space and allow faster backups if the documents used by John and Mary are similar. This example only covers one configuration use case. Separate storages and different configurations may be used. The purpose is just to clarify how the following terms are u…

[Droolio]

Rclone isn’t rsync. (Although I guess it does similar things.) However, you can ‘backup’ your large media files more efficiently by a simple Rclone copy or perhaps a sync with the --backup-dir flag.

Anyway, you asked for best practices and I gave you what I consider a better practice…

Duplicacy is fantastic for most user data (and, ridiculously, doesn’t have a local database) but the process packs files into chunks on the destination. This makes it cumbersome to access directly, and you likely won’t benefit from de-duplication or compression.