Reducing memory usage deciding what to back up

Support
#1

We are hitting an issue with a very large backup (in terms of file count, which is in the 100,000s):

duplicacy.exe : runtime: VirtualAlloc of 15532032 bytes failed with errno=1455

We were initially trying to backup with 2GB ram, this failed for the initial backup. Increased memory to 4GB and the initial backup worked, but a subsequent backup failed (where its deciding what needs backing up).

Is there anything we can/ any options we can use to reduce the amount of memory required? e.g. altering the scheme used to determine what to backup up or tell it to use a memory mapped file?

Duplicacy CLI Version 2.2.1

I can get more info is necessary (file count etc).

#2

Have you tried this? Out of Memory on backup, Revisited

#3

It is still failing, its actually during the chunk upload phase

Storage set to F:/AZBACKUP/BACKUPS/RED011
Last backup at revision 1 found
Indexing F:\AZBACKUP\REPOSITORIES\RED011\RED011
Parsing filter file \\?\F:\AZBACKUP\REPOSITORIES\RED011\RED011\.duplicacy\filters
Loaded 0 include/exclude pattern(s)
Discarding file attributes
Skipped chunk 1 size 7717015, 3.68MB/s 01:52:28 0.0%
Skipped chunk 2 size 1992716, 4.63MB/s 01:29:23 0.0%
...
Uploaded chunk 288 size 8533044, 18.33MB/s 00:21:10 6.3%
Uploaded chunk 289 size 16777216, 18.52MB/s 00:20:56 6.4%
duplicacy.exe : runtime: VirtualAlloc of 16908288 bytes failed with errno=1455
fatal error: out of memory

System Memory 4GB. Memory had gone past the 3GB mark for duplicacy.exe the last I checked

Stack Dump

runtime.mallocgc
(0x1010115, 0xc0b5c0, 0xcd6801, 0xc04212a570)
        /usr/local/go/src/runtime/malloc.go:720 +0x8e1 fp=0xc0caa17a78 sp=0xc0caa179d8 pc=0x410c91
runtime.makeslice(0xc0b5c0, 0x1010115, 0x1010115, 0xc05c806ad8, 0x445be6, 0x50)
        /usr/local/go/src/runtime/slice.go:61 +0x7e fp=0xc0caa17aa8 sp=0xc0caa17a78 pc=0x441a1e
github.com/gilbertchen/duplicacy/vendor/github.com/bkaradzic/go-lz4.Encode
(0xc042230004, 0x0, 0xfffffc, 0xc1121bc000, 0x1000000, 0x1000000, 0x0, 0xc0caa17c08, 0x441cac, 0xc0477a7d90, ...)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/vendor/github.com/bkaradzic/go-lz4/
writer.go:112 +0x538 fp=0xc0caa17b68 sp=0xc0caa17aa8 pc=0x7b6ba8
github.com/gilbertchen/duplicacy/src.(*Chunk).Encrypt(0xc0420922d0, 0x1303c88, 0x0, 0x0,
0xc05c43f500, 0x20, 0x0, 0x0)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunk.go:234 
+0x45d fp=0xc0caa17cc0 sp=0xc0caa17b68 pc=0xaf643d
github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Upload(0xc042092190, 0x0, 0xc0420922d0, 0x122, 0x1)
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:131
+0x274 fp=0xc0caa17ef8 sp=0xc0caa17cc0 pc=0xafe084
github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Start.func1(0xc042092190, 0x0)        /Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:60
+0x8a fp=0xc0caa17fd0 sp=0xc0caa17ef8 pc=0xb5c5ca
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc0caa17fd8 sp=0xc0caa17fd0 pc=0x45a8c1
created by github.com/gilbertchen/duplicacy/src.(*ChunkUploader).Start
/Users/chgang/zincbox/go/src/github.com/gilbertchen/duplicacy/src/duplicacy_chunkuploader.go:55
+0x4f
#4

What backup command options are you currently using?

#5

This issue is unfortunately delaying us rolling out our backup solution to our other hosted servers. We have a number of hosted customers spread across a number of hosted servers. We are migrating the backups for these servers to a new system that has duplicacy cli at its core.

We are going to use a work around that will hopefully keep us going in the short term. The issue is this customer has a document store that has over a million entries (and growing). The workaround for now is to split the backup into 3, one part is everything except the content store, and we are going to split the content store into two (odd/even folder names). We currently have 001 and 002 folders (002 is nearly full) next will be 003 by time we get to 004 will probably be in the same boat as the store will likely have around 2 million files.

#6

If you decide to split the backup, this new feature in 2.3.0 will allow you to use a different filters file for each subset: Filters/Include exclude patterns

1 Like
#7

Don’t think it applies in my case. I have one storage, with several snapshots (repositories) each repository links to a mounted read only snapshot of a live disk, and has its own filter.

#8

Eventually got this backup working, by splitting it into 3 (in future more) snapshots:

Main: 118k files, 40GB
000: 960k files, 248GB
001: 947k files, 268GB

These backups have gone through several revisions OK (so far) with 4GB ram.

The backup servers are Azure VMs running windows, and so there is a cost to increasing the available ram, and something we want to avoid.