Issue with include/exclude pattern

Support
#1

Hello,

i’m trying to figure out how should i set include/exclude patterns.

My repository is the dir “/source/Backup”

I have one folder inside at the moment “omvbackup”

Here the files inside:

steakhutzeee@DK:/srv/dev-disk-by-label-HC2/Backup/omvbackup$ ls
backup-omv-09-Jun-2021_18-22-20.blkid
backup-omv-09-Jun-2021_18-22-20.ddfull.gz
backup-omv-09-Jun-2021_18-22-20.fdisk
backup-omv-09-Jun-2021_18-22-20.grub
backup-omv-09-Jun-2021_18-22-20.grubparts
backup-omv-09-Jun-2021_18-22-20.packages
backup-omv_u-boot-09-Jun-2021_18-22-20.tar.bz

I would like to only backup the files with extension .ddfull.gz and .dd.zst

Tried with regual expression but no luck.

I did this: i:omvbackup/(?i)\.(ddfull.gz|dd.zst)$

Tried some variants but was not able to let it work.

#2

Filters are being applied in order to each file and directory until match is found.

Please clarify what directories are — where is source directory?

Otherwise you can do this

+*/
+*.ddfull.gz
+*.dd.zst
-*

This means - all directories will match. This is needed to get to the actual files.

Then when files are being tested — match to those paths ending with what you wanted

And exclude everything else.

Note, * also matches any number of directory separators.

Also, this is much more readable, and therefore better, than regex

And lastly, if you are backing up disk images and virtual machines you may want to consider fixed chunk size (see Vertical backup)

1 Like
#3

I’m using a B2 bucket to backup my NAS files: docker-compose files, containers configurations files (and some not so big containers’ databases) .
And now i would like to add some microsd clones of the system and some live backup i did with dd.

The source directory is a shared folder on the NAS called “Backup”, inside there is a dir named “omvbackup” containing files, but i only want to backup the .ddfull.gz and .dd.zst.

Never heard of Vertical backup. I will add new files to be backed up monthly more or less.
Should use fixed chunk size in Duplicacy? Could incur on big issues if not?

The .ddfull.gz and .dd.zst files i will backup will be a couple of GBs big, nothing massive.

Thanks!

#4

For large files that have minor modifications (like VMs, disk images, databases, etc) yes.

To configure a storage with fixed chunk size use the -chunk-size, -max-chunk-size and -min-chunk-size options with the same value.

Larger backups, longer upload time, and the like.

#5

Thanks!

Well actually apart from the system images that are 2 GBs i backup basically configuration files of a few KBs

#6

Note that fixed chunk size probably won’t work very well if your disk images are .gz-compressed.

#7

Out of curiosity, can i ask why?

What kind of compression is suitable for a fixed chunk size?

#8

Most types of compression will rearrange and significantly change the pattern of bytes - even if you change one byte - everything that comes after that, will always change subsequent parts of data, since the compression algorithm is essentially dealing with new data that has to be compressed differently.

This’ll be even more pronounced with fixed size chunks.

For most efficiency, raw (and uncompressed) disk images, with fixed size chunks would be more efficient, and result in far less incremental delta between snapshots.

3 Likes
#9

Thanks, it sounds i should stick with default chunk size then.

#10

For me it sounds you should keep your disk image uncompressed, to save space on the destination, provided you update the disk image periodically as opposed this being an one-off snapshot.

Or better yet — test both approaches and see what happens to be more effective in your specific scenario.

4 Likes
#11

I’ll just add to what @saspus says… remember that Duplicacy will de-duplicated and compress chunks of the raw disk image - so you really only have to worry about disk space on the source file (and you can always take care of that with fs-based compression; though it or gz is unlikely to compress your images down massively anyway, unless you have a lot of empty space, which again Duplicacy will happily de-duplicate).

2 Likes
#12

As i can see the content of my .gz archive is a file with .ddfull extension. Dumb question, this is the file to be directly backed up with Duplicacy? Or it is a sort of an archive itself?

#13

judging by the name the .ddfull appears to be a full image made by dd :).

Yes, you can back it up directly.

Side note, on most os-es there is a utility file that can provide you insight about the type of the file.

#14

Little late, but i just tried to clone my microsd and the result was ofc a file of 30GB, so yes also the free space is backed up.

I actually keep just one clone of the microsd so probably it’s not worth the extra space.

The compressed file is just 2 GB.