Limiting disk usage during `duplicacy check -chunks`

lolwut · 10 May 2020 00:36

Preface: I’ve read both Cache usage details and Cache folder is is extremely big! 😱

Both seem to imply that cache usage only grows through subsequent runs as the cache is not cleared until a prune is run. This thread is asking how to limit cache usage within a single run.

Running duplicacy 2.5.1 (51CBF7)

I recently ran a duplicacy check -chunks to verify that my backups had made it to the remote storage without corruption.

duplicacy check -chunks -threads 8
Repository set to /foo
Storage set to gcd://backups/duplicacy/foo
Listing all chunks
2 snapshots and 5 revisions
Total chunk size is 5173G in 1100701 chunks
All chunks referenced by snapshot foo at revision 1 exist
All chunks referenced by snapshot bar at revision 1 exist
All chunks referenced by snapshot bar at revision 2 exist
All chunks referenced by snapshot bar at revision 3 exist
All chunks referenced by snapshot bar at revision 4 exist
Verifying 1097648 chunks

However after a few hours I noticed that disk usage was growing at an alarming rate. Currently sitting around 100GB.

Digging a bit deeper it looks like duplicacy is caching every chunk to disk.

/foo/.duplicacy # du -sh cache/
101.1G  cache/
/foo/.duplicacy/cache/default # find chunks -type f | wc -l
20312

At the current growth it looks like running duplicacy check -chunks requires enough disk to store the whole backup (or at least will attempt to cache the whole backup on disk).

My assumption is that chunk verification shouldn’t need to store more than n threads worth of chunks on disk at a time and might not need to store them on disk at all if they’re sufficiently small (default chunk size is <16M which should fit in memory easily).

Is there any way to stop from caching the chunks (and therefore my whole backup) on disk? Or at least a way to ask it to clean up as soon as it no longer needs the files to minimize the amount of storage needed to run the command?

I think it’s somewhat unreasonable to expect a single system to have enough disk to store the entirety of the backup storage, especially if many systems are backing up to the same location to leverage the deduplication functionality.

gchen · 10 May 2020 01:39

Duplicacy only caches metadata chunks on disk, so subsequent check commands will be much faster.

You can create a post-check script to delete everything in the cache after a check.

lolwut · 10 May 2020 02:30

Duplicacy only caches metadata chunks on disk, so subsequent check commands will be much faster.

Is this completely true? Or does it only apply to check -chunks?

I ran a duplicacy check (without -chunks) right before this and it did not use a noticeable amount of disk space (<1GB).

Is it normal to have >100GB of metadata chunks for a ~5TB backup?

You can create a post-check script to delete everything in the cache after a check.

Sure, but that only works if you have enough storage to cache all the chunks to begin with.

At the current rate is using storage, I don’t think I’ll have enough storage to complete a check -chunks.

Is there any way (or any plans to support) a way of preventing/limiting the caching from happening? ie a -no-cache option to minimize the amount of disk needed for a check -chunks run?

lolwut · 10 May 2020 03:18

Looking at the code, it looks like the chunkDownloader will blindly cache any chunk it’s asked to download.

github.com

gilbertchen/duplicacy/blob/v2.5.1/src/duplicacy_chunkdownloader.go#L460


				LOG_FATAL("DOWNLOAD_CORRUPTED", "The chunk %s has a hash id of %s", chunkID, actualChunkID)
				return false
			}
		}


		break
	}


	if len(cachedPath) > 0 {
		// Save a copy to the local snapshot cache
		err := downloader.snapshotCache.UploadFile(threadIndex, cachedPath, chunk.GetBytes())
		if err != nil {
			LOG_WARN("DOWNLOAD_CACHE", "Failed to add the chunk %s to the snapshot cache: %v", chunkID, err)
		}
	}


	downloadedChunkSize := atomic.AddInt64(&downloader.downloadedChunkSize, int64(chunk.GetLength()))


	if (downloader.showStatistics || IsTracing()) && downloader.totalChunkSize > 0 {


		now := time.Now().Unix()

And the check -chunks routine will add all chunks (even non-metadata chunks it looks like), to the download queue. As it seems to be relying on the chunkDownloader's integrity checking for chunk verification.

github.com

gilbertchen/duplicacy/blob/v2.5.1/src/duplicacy_snapshotmanager.go#L1002


	}


	if showTabular {
		manager.ShowStatisticsTabular(snapshotMap, chunkSizeMap, chunkUniqueMap, chunkSnapshotMap)
	} else if showStatistics {
		manager.ShowStatistics(snapshotMap, chunkSizeMap, chunkUniqueMap, chunkSnapshotMap)
	}


	if checkChunks && !checkFiles {
		LOG_INFO("SNAPSHOT_VERIFY", "Verifying %d chunks", len(*allChunkHashes))
		for chunkHash := range *allChunkHashes {
			manager.chunkDownloader.AddChunk(chunkHash)
		}
		manager.chunkDownloader.WaitForCompletion()
		LOG_INFO("SNAPSHOT_VERIFY", "All %d chunks have been successfully verified", len(*allChunkHashes))
	}
	return true
}


// Print snapshot and revision statistics
func (manager *SnapshotManager) ShowStatistics(snapshotMap map[string][]*Snapshot, chunkSizeMap map[string]int64, chunkUniqueMap map[string]bool,

Which seems to line up with what’s happening here, where all chunks in the backup storage are being cached during check -chunks. A bug perhaps? I don’t see any options that could disable this behavior.

As an aside, this also seems to imply that doing a full restore from backup will require at least 2x the storage of the backup (1x for the chunks, 1x for the actual files being restored).

gchen · 10 May 2020 04:39

Your suspicion is right – every chunk being checked is saved to the cache due to a bug. I just pushed a commit that fixes this bug.

This bug only affects the check command when the -chunks option is specified. A full restore should only store metadata chunks to the cache even without this fix.

Thanks for the finding. I’ll upload a new release tomorrow.

lolwut · 11 May 2020 04:08

Thanks for the quick fix!

gchen · 11 May 2020 04:16

This bug has been fixed by CLI 2.5.2 which has been released at Release Duplicacy Command Line Version 2.5.2 · gilbertchen/duplicacy · GitHub

TheBestPessimist · 1 July 2020 15:00

This topic was automatically closed after 50 days. New replies are no longer allowed.