Is Duplicacy compatible with Rsync?

I have a large file(2TB) that has been moved to EBS from my local machine using rsync. Rsync doesnt dedupe. So i am thinking can i set duplicacy to continually run delta uploads made to the file?

or do i have to start a new backup with duplicacy?

Hi Oodenike!

You’ll learn more when you continue reading in this forum, but Duplicacy and rsync do very different things!

  • rsync can syncronize your file (or folders) to a separate folder or remote server. (this is one identical copy of a file or folder) Older versions are lost.

  • Duplicacy makes backups of your file/folders to a separate folder or remote server.
    The backup har a special folder structure with chunks (part of files) and contains all the data from the file(s) but is not a copy of the file, it is a collection of chunks which describes the file(s). Also prevous versions are kept. Due to depuplication the total size might be smaller.

And if only a part of the file changes, the next backup will only create chunks with the changed part(s) of the file.

So if your intention is to make a backup (as opposed to an identical copy), Duplicacy might be better due to deduplication.
(If the file contains many identical parts)
(but realize that these are two very different scenarios)

Hope this makes sense!

(Duplicacy is compatible with rsync in that you could use rsync to copy a full backup set (chunks etc) from one server/locaton to another, but this is has certain conditions!)

1 Like

@akvarius

Thanks for your response. i just started with duplicacy and i have been reading alot of topics here. very informative. We have been using rsync for years to backup files from local to ebs and i know rsync doesnt dedupe. why i asked that question was, i am looking at using duplicacy for that workload but considering do i have to run a new backup with duplicacy being that it has a sequence of how it maps out the files or can i just continue since rsync already backed up the last version so that any change made i can continue with duplicacy.

I think I see what you mean now!
Duplicacy cannot use your already uploaded copy in EBS (since it is not split up in the duplicate format of chunks), so I’m afraid you need to start a new backup.

2 Likes

If your goal is to use Duplicacy as a sync tool, then there is a way to speed up the transfer.

Assuming the big file has originally been synced between host1 and host2. Now on host2 the file has been modified with a few changes and you want to sync the changes back to host1.

Step 1: On host1, back up the old file to a local storage

cd /path/to/file
duplicacy init host1 /path/to/local/storage
duplicacy backup

Step 2: On host2, back up the new file to the storage on host2 (note that it uses a different repository id – host2 rather than host1)

cd /path/to/file
duplicacy init host2 sftp://host2/path/to/host1/storage
duplicacy backup

Because most chunks should been generated in host1’s storage by step1, only a few chunks will be uploaded and this should be significant faster than uploading the entire file.

Step 3: On host1, change the repository id from host1 to host2 by removing the .duplicacy directory and re-initialize the storage (alternatively you can directly edit the .duplicacy/preferences file to change the repository id)

rm -rf .duplicacy
duplicacy init host2 /path/to/local/storage

Step 4: restore the new file:

duplicacy restore -r 1 -overwrite

I think this should be much faster than a direct rsync transfer for large files, because the rsync algorithm is actually very slow compared to the variable-sized chunking algorithm used by Duplicacy.

3 Likes