Backing up media files: duplicacy or rclone?

Support
#1

Continuing the discussion from Change order of files backing up:

This is an interesting point and I’m starting this new topic to provide a space to discuss this recommendation further, the pros and cons, basically.

So when you say

does this mean that if I have 20 video files (say: 1GB each) on my Dropbox so that they are synced to two computers and both computers are backed up to the same storage, duplicacy will upload 20 GB of media file chunks from one machine and another 20 GB of media file chunks from the other?

This is not how I understood duplicacy, but, I may well be wrong.

So the “versioning not required” argument seems to weigh heavier. And I would add my suspicion that there isn’t much compression to be achieved on most mefja files as they tend to be already heavily compressed.

Another argument for rclone might be that it is very easy to remove movies that you don’t want any more in order to free up storage space. In duplicacy that is basically impossible.

#2

If you are backing up two sources to the same storage the deduplication will save coping your files twice… You’ll have to scan the files, chunk them, zip them and encrypt them twice, but traveling over the network is the biggest bottle neck anyway. Here a backup from my NAS to Backblaze B2 that skips uploading redundant chunks is about 20 times faster than sending the chunks for the first time.

Tho my big media files are readonly, my smaller files (e.g. .flac) have metadata and so versioning is a good thing.

Personally I want all of my files (media or not) in the same place(s) and backed up with the same tools. After the initial backup big files aren’t really a problem anyway. As to splitting, zipping and encrypting, any core on my computer is faster than the network bandwidth so I don’t care about that. I did part of my big media files using a fixed chunk size to a separate Backblaze B2 bucket, but then changed my mind and just use the same one I normally use with the same chunk size settings. A separate backup system for my big files is a whole separate process, it may not provide checksumming/hashing, etc. for quick comparison to my local files…

I’m hoping that at some point we’ll get a mountable file system on top of a snapshot then I’ll have both a good back up with checksums and reasonable access outside my house to my music collection.

1 Like
#3

No, you’re correct to say it will de-duplicate amongst several copies, but it very likely won’t within parts of the same file. (Notable exception maybe if you’ve got remuxed edited clips that haven’t been recompressed.)

It very much depends on how big that media is; 1GB is rather small but it’s getting a bit on the unwieldy size imo.

I use Duplicacy for .mp3 and .flac and photos because there’s a lot of metadata and folder structure that I wouldn’t mind having versioned, but everything else in terms of media doesn’t need to be snapshotted.

As you surmised, it’s a great deal easier to deal with if you want to replace a particular piece of media with a higher quality encode for example, if they were simply accessible as files. Rclone allows you to copy or sync to cloud storage, encrypt the contents on the fly and mount as a drive if necessary.

Both Duplicacy and Rclone work side-by-side in my setup, backing up to the same Google Drive on G Suite. For me, I feel this is the right choice of tool for each job, though it isn’t necessarily so for everyone. :slight_smile:

1 Like
#4

?
Isn’t the purpose of a rolling hash (and variable sized chunks) exactly to resync (especially on longer sequences)? There’s no difference between doing it within a file and file to file.

#5

Well, sure? What I’m saying is you normally won’t find duplicated data within a video file.

#6

Sorry, I got sidetracked by not reading carefully enough.

In my circumstance my media library isn’t useful unless it’s all there, the big files and the small files. I want the small files to be backed up with something like Duplicacy and (personally) I don’t have any downside with backing the big files (which are readonly) with Duplicacy too. But I understand other people have other priorities.