Since some time, I’m getting daily warnings that I’m reaching 75% of my upload cap.
The bucket is used for backup of multiple hosts; I’ve been basing myself on the stats output of backup runs to try to identify which one. One of these hosts seems the likely culprit, but I’m not exactly getting why…
This is the logfile without UPLOAD_FILE|LIST_SKIP|SKIP_FILE entries:
Running backup command from /root/.duplicacy-web/repositories/localhost/0 to back up /
Options: [-log backup -storage duplicacy-tuxera -threads 8 -stats]
2024-03-06 12:00:02.759 INFO REPOSITORY_SET Repository set to /
2024-03-06 12:00:02.759 INFO STORAGE_SET Storage set to b2://duplicacy-tuxera
2024-03-06 12:00:03.388 INFO BACKBLAZE_URL Download URL is: https://f002.backblazeb2.com
2024-03-06 12:00:09.924 INFO BACKUP_START Last backup at revision 28572 found
2024-03-06 12:00:11.140 INFO BACKUP_INDEXING Indexing /
2024-03-06 12:00:11.140 INFO SNAPSHOT_FILTER Parsing filter file /root/.duplicacy-web/repositories/localhost/0/.duplicacy/filters
2024-03-06 12:00:11.140 INFO SNAPSHOT_FILTER Loaded 28 include/exclude pattern(s)
2024-03-06 12:07:17.548 INFO BACKUP_END Backup for / at revision 28573 completed
2024-03-06 12:07:17.548 INFO BACKUP_STATS Files: 3715438 total, 600,317M bytes; 4582 new, 3,143M bytes
2024-03-06 12:07:17.548 INFO BACKUP_STATS File chunks: 362841 total, 1988G bytes; 83 new, 493,348K bytes, 223,485K bytes uploaded
2024-03-06 12:07:17.548 INFO BACKUP_STATS Metadata chunks: 612 total, 789,836K bytes; 25 new, 42,871K bytes, 20,213K bytes uploaded
2024-03-06 12:07:17.548 INFO BACKUP_STATS All chunks: 363453 total, 1989G bytes; 108 new, 536,219K bytes, 243,698K bytes uploaded
2024-03-06 12:07:17.548 INFO BACKUP_STATS Total running time: 00:07:12
The all chunks size, almost 2TB, is a fair enough match to the issue I’m seeing: this host backs up every 6 hours, so 3 sequential runs get pretty close to that 7.5TB warning I get.
I am guessing that that number is just the total size of everything stored, though, not what’s uploaded, and that the actually uploaded size is the end of that line. That ~240MB seems not unreasonable for daily backup. 2T backup would also suggest an upload speed of 4.5GB/s - that host is in a decent colo, but I’m pretty sure I don’t get a 45 gigabit pipe to myself :-p
However, as this bucket is ONLY used for duplicacy, I am at a total loss as to why I would be hitting my upload caps.
So, I’m looking for confirmation that my understanding of the stats is accurate (why is files and file chunks different?); as wel as potential culprits for hitting the upload cap. Any help is appreciated.
/Johan