Does Duplicacy utilize compression by default?

mark.whitworth · 2 May 2022 22:58

I’m using Duplicacy web on two of my servers, but now that I’m putting more jobs on them I realize I should use at least some level of compression. It sounds like this should be done at storage initialization, but I don’t remember the option and I definitely didn’t manually set anything.

Do I need to do something to ensure I have compression enabled?

saspus · 3 May 2022 01:41

It’s on by default, with no way to disable it.

IMO having an ability to selectively disable it (e.g. at least by file type or regex) would be an excellent feature. Most home users data is photos and videos and there is zero benefit in burning electricity attempting to compress it.

mark.whitworth · 3 May 2022 03:13

Are you certain? I’d seen some references from way back in 2016 that said “The compression level parameter is passed to the zlib library. Valid values are -1 through 9, with 0 meaning no compression, 9 best compression (slowest), and -1 being the default value (equivalent to level 6)”.

I just wasn’t sure since I didn’t specify the compression option that it would actually treat it per that default setting. The above would also indicate it can be disabled upon storage initialization.

saspus · 3 May 2022 03:34

This is the new setting, instead of former -1…9:

github.com

gilbertchen/duplicacy/blob/e43e848d47176144049b93938f09820c61effc72/src/duplicacy_config.go#L36

    
      
          	"reflect"
          
          
	blake2 "github.com/minio/blake2b-simd"
          )
          
          
// If encryption is turned off, use this key for HMAC-SHA256 or chunk ID generation etc.
          var DEFAULT_KEY = []byte("duplicacy")
          
          
// The new default compression level is 100.  However, in the early versions we use the
          // standard zlib levels of -1 to 9.
          var DEFAULT_COMPRESSION_LEVEL = 100
          
          
// The new banner of the config file (to differentiate from the old format where the salt and iterations are fixed)
          var CONFIG_BANNER = "duplicacy\001"
          
          
// The length of the salt used in the new format
          var CONFIG_SALT_LENGTH = 32
          
          
// The default iterations for key derivation
          var CONFIG_DEFAULT_ITERATIONS = 16384

The meaning of it is just a flаg meaning “use new compression”; which is now (far superior for the intended use) lz4:

github.com

gilbertchen/duplicacy/blob/f2d6de3fff7567740e86cb49801b0698f085cc59/src/duplicacy_chunk.go#L268

    
      
          	// offset is either 0 or the length of banner + nonce
          
          
	if chunk.config.CompressionLevel >= -1 && chunk.config.CompressionLevel <= 9 {
          		deflater, _ := zlib.NewWriterLevel(encryptedBuffer, chunk.config.CompressionLevel)
          		deflater.Write(chunk.buffer.Bytes())
          		deflater.Close()
          	} else if chunk.config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
          		encryptedBuffer.Write([]byte("LZ4 "))
          		// Make sure we have enough space in encryptedBuffer
          		availableLength := encryptedBuffer.Cap() - len(encryptedBuffer.Bytes())
          		maximumLength := lz4.CompressBound(len(chunk.buffer.Bytes()))
          		if availableLength < maximumLength {
          			encryptedBuffer.Grow(maximumLength - availableLength)
          		}
          		written, err := lz4.Encode(encryptedBuffer.Bytes()[offset+4:], chunk.buffer.Bytes())
          		if err != nil {
          			return fmt.Errorf("LZ4 compression error: %v", err)
          		}
          		// written is actually encryptedBuffer[offset + 4:], but we need to move the write pointer
          		// and this seems to be the only way
          		encryptedBuffer.Write(written)

mark.whitworth · 3 May 2022 03:53

LOL. As I was typing that, I did start to think that between the length of time and the improvements in compression over zlib back then that things had likely changed. Thanks for spelling that out. Well, it looks like I’m in okay shape with what I’ve got, storage-wise. I do get your comment about it being wasteful for large volumes of non-compressible media, and hopefully that will get addressed sooner rather than later. I have 10TB+ that fall into that category as well.