Zstd compression implemented

Announcement
#1

This is the commit: Implement zstd compression · gilbertchen/duplicacy@53b0f3f · GitHub

Zstd compression can be enabled by providing -zstd or -zstd-level <level> to the init or add command. The use of -zstd sets the compression level to default , while -zstd-level allows for the selection of a compression level from fastest , default , better , or best .

The backup command can also accept these 2 options, which means you can switch to zstd without the need to initialize a new storage.

I ran a test on my own project code base. With the default LZ4 algorithm:

Files: 405702 total, 53,027M bytes; 403024 new, 53,027M bytes
File chunks: 10567 total, 53,027M bytes; 9636 new, 49,110M bytes, 32,570M bytes uploaded
Metadata chunks: 69 total, 87,238K bytes; 69 new, 87,238K bytes, 34,928K bytes uploaded
All chunks: 10636 total, 53,112M bytes; 9705 new, 49,195M bytes, 32,604M bytes uploaded
Total running time: 00:07:11

With -zstd:

Files: 405702 total, 53,027M bytes; 403024 new, 53,027M bytes
File chunks: 10567 total, 53,027M bytes; 9636 new, 49,110M bytes, 28,364M bytes uploaded
Metadata chunks: 69 total, 87,238K bytes; 69 new, 87,238K bytes, 23,309K bytes uploaded
All chunks: 10636 total, 53,112M bytes; 9705 new, 49,195M bytes, 28,386M bytes uploaded
Total running time: 00:06:35

So zstd performs slightly faster while using (also slightly) less storage space.

6 Likes
Questions about zstd-compression
Question about zstd? compression
#2

Project code base meaning a whole bunch of text .go files? That’s probably the best case scenario. I can imagine if there are more compressed files (including media) in the mix, the space savings would be significantly less.

How would it work if you use zstd in new backups on the existing storage? Old chunks would be in lz4, while new ones would be in zstd, and :d: would be able to handle both on a case by case basis?

#3

Yes, there shouldn’t be any issue with lz4 and zstd chunks coexisting in the same storage. The compression flag is stored in the header of each chunk so Duplicacy knows which compression algorithm to use.

#4

Hi, this will be implemented in the next version of duplicacy-cli ?

#5

Great to see this implemented! And cool to see it has some benefits compared with lz4, albeit small. Nice to have options though.

It might also be beneficial to add that the author of the Go package implementation of zstd de/compress that this uses, has stated that

Currently the package is heavily optimized for 64 bit processors and will be significantly slower on 32 bit processors.

#6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.