Duplicacy writes two kinds of chunks:
- File chunks hold the contents of your files. Almost all of the storage space goes to these.
- Metadata chunks hold each snapshot’s file list and chunk sequences. They’re small, but almost every command reads them: backup (to find the previous snapshot), list, check, prune and copy.
Until now, both kinds went into chunks/. That makes it hard to put your file chunks in a cheap archival storage class, like S3 Glacier Deep Archive or a lifecycle rule that moves old objects to cold storage. The rule would catch the metadata chunks too, and then every command would fail.
With this commit, you can now keep metadata chunks in a separate top-level metadata/ directory:
storage/
config
nesting
snapshots/
chunks/ <- file chunks only
metadata/ <- metadata chunks
This means you can:
- Apply a storage-class or lifecycle rule to the
chunks/prefix alone. - Run
list -files,check(without-chunks),pruneand incremental backups even whenchunks/is slow or temporarily unreadable. Onlyrestore,check -chunksandcopyneed to read file chunks.
There’s no new command-line option to enable this feature. Instead, just add separate-metadata to the file named nesting at the root of the storage, or create the file if it doesn’t exist:
{
"separate-metadata": true
}
Things to be aware of:
- It is recommended to turn this option on only for a new storage, before any backups are made.
- All clients backing up to the same storage should have this option enabled.
- Don’t turn the option off once it’s on.
- Nesting levels only apply to
chunks/.metadata/always uses a single level, which is 256 subdirectories.
I would appreciate it if anyone could test this feature. If you need an executable built from this commit, please let me know.