Handling Apple Time Machine Sparsebundles

[saspus]

@Christoph

But TimeMachine kept complaining about lack of space. I tried reselecting the volume to which time machine should backup but it complained that it was already in use

You should not be manually mounting and selecting the volume. It should be advertised, via Bonjour, and time machine must mount and own the mount point. What likely happened it was not a clean unmount and/or smbd got stuck holding on to handles. Rebooting the server or killing stale smbd instances should have fixed that.

If would also ignore the existing bundle if hostname of your Mac changed.

When you initialize the time machine on a share it creates a bundle with the specified maximum size. You can control that size at the creation state by placing a special plist file to the share before the bundle is created: .com.apple.TimeMachine.quota.plist. But the best practice is to not set any quotas in the first place. Because see what you did when it wanted more space? You tried to give it more space. Why set quotas in the first place? You do want long backup history.

[Christoph]

@saspus

You should not be manually mounting and selecting the volume. It should be advertised, via Bonjour, and time machine must mount and own the mount point.

I don’t think I did what you mean by “manually mounting”. I was selecting the volume here:

@saspus

Rebooting the server or killing stale smbd instances should have fixed that.

Will remember that for the future.

I tried to emulate this by disabling and re-eenabling the TM share, but I suppose that did not kill the stale smbd instances?

@saspus

When you initialize the time machine on a share it creates a bundle with the specified maximum size.

I thought I read somewhere that the bundle would expand as the backup grows (until it reaches space limits). So this is not the case?

@saspus

But the best practice is to not set any quotas in the first place. Because see what you did when it wanted more space? You tried to give it more space. Why set quotas in the first place? You do want long backup history.

Thing is: the smb share resides on my NAS, which I also use for other stuff. So if I let the TimeMachine backup use as much storage as it wants, it will eventually fill up my entire NAS and all kinds of things will stop working. That’s why I gave it, I think it was 500GB to see how long that would last and then expand it if needed. I didn’t expect those 500 GB to fill up in less than a month.

Now it has 800 GB (once I remove the broken bundle), which probably won’t last very long either, so I’ll have to add a HDD to my NAS. I still have a 4TB drive in my PC that started having faulty sectors a year or so ago so I replaced it but left it in the machine, curious to see when it will die. It still seems to work fine so maybe I can use it in my NAS? From what I understand, zfs will take care of those bad sectors and even an complete disc failure?

[saspus]

@Christoph

I don’t think I did what you mean by “manually mounting”

You did it right.

@Christoph

I thought I read somewhere that the bundle would expand as the backup grows (until it reaches space limits). So this is not the case?

Good question. You can mount the bundle in disk utility and check if it has maximum size. My understanding is it will grow up to that maximum size — being sparse it will only occupy amount of space equal to amount of data written. Ie if you create 1TB sparse bundle and write 1GB there is till take 1GB on disk and will keep growing up until it reaches 1TB

@Christoph

Thing is: the smb share resides on my NAS, which I also use for other stuff. So if I let the TimeMachine backup use as much storage as it wants, it will eventually fill up my entire NAS and all kinds of things will stop working.

It won’t realistically. Time machine backs up everything useful from your Mac - so the first backup will take most space. Subsequent backups are incremental. Which means quota just above that will be reached quickly while adding a bit more will make it take forever to fill. Depending on your data turnover of course. And then you can manually delete old backups if you really want to. Anecdotally my time machine bundle was created in 2008 or 2009 and today it is about 4TB. It survived a lot of migrations, and hardware, and looking at how much data o had on my Mac — just under 2TB — it seems very reasonable growth. Yes, I understand I should probably start over with a new bundle to take advantage of all the improvements made over last decade but hey it works. Not broken don’t fix

Just like with duplicacy, it might be worth watching a couple of backup runs (tmutil compare) to see what data is being picked up to ensure no high turnover random crap is being saved over and over again. For example, Envelope Index is excluded but its satellite sidecar files are not. And they are quite large. Some third party apps not only don’t bother marking their transient files for exclusion but also dump them into inappropriate places. Taking 10 minutes to add those exclusions (see tmutil addexclusion) will make your time machine much more space efficient and extremely slow growing; to the point of your storage growing faster than TM requirements.

500GB sounds very very little.

[Christoph]

@saspus

You can mount the bundle in disk utility and check if it has maximum size.

So this is what I get:

I guess the 816 GB correspond to the 800 GB I allotted to the SMB share. Used is probably what it says: that much data is used by the existent backups. But available?? The only calculation that might yield about 117 GB available would be if the old bundle still used its 460 or so GB that it used before it broke: 460 + 117 + 252 = 829 ≈ 816.

But the thing is, the old bundle supposedly occupies 0 bytes:

So, I don’t know what to make of this. It’s not important, though, just learning my way into MacOS.

@saspus

if you create 1TB sparse bundle and write 1GB there is till take 1GB on disk and will keep growing up until it reaches 1TB

Yes, that’s one type of expansion: Max size is set at time of creation and then expands until that size is reached. But I guess I was expecting another kind of expansion: Max size is set dynamically based on max space available and it expands until it reaches that size. But what happened was obviously: Max size was set to 500 GB because I gave it 500 GB. Then I increased the available space to 800 GB but the mix size stayed at 500 GB. Problem not solved.

However, wonder whether what determines the max size of the bundle: the total space of the share or the available space on the share at the time of creation? If it is the total space of the share, that would prove my strategy of limiting the share right, because without quota max size would have been something like 1.8 TB, which is not a problem in itself but given that other stuff on the NAS keeps growing, the actually available space would be significantly less than the bundle’s max size, thus it would probably fill up the disk to the last byte, which is not good.

@saspus

the first backup will take most space. Subsequent backups are incremental.

This was exactly my thinking: I have a 512 GB SSD which is just about half full. So giving it 500 GB seemed like it would last for a while. (Most of my data is still on my PC and on the NAS and I’m still trying to figure out what the best storage is for Mac, if you don’t want to pay the exorbitant prices for a larger built in SSD…)

@saspus

it might be worth watching a couple of backup runs ( tmutil compare ) to see what data is being picked up to ensure no high turnover random crap is being saved over and over again.

So you are saying I still have to take care of stuff myself, now that I’m an Apple user?

Thanks for letting me know.

The command, however gives me this:

~ % tmutil compare
Unable to locate machine directory for host.

-------------------------------------
Added:         0B
Removed:       0B
Changed:       0B
~ % 

[saspus]

@Christoph

But available?? The only calculation that might yield about 117 GB available would be if the old bundle still used its 460 or so GB that it used before it broke: 460 + 117 + 252 = 829 ≈ 816.

This is indeed strange. Not sure what to make of this.

@Christoph

But the thing is, the old bundle supposedly occupies 0 bytes:

and this is clearly bogus. Perhaps the connection is not connected?

@Christoph

But what happened was obviously: Max size was set to 500 GB because I gave it 500 GB. Then I increased the available space to 800 GB but the mix size stayed at 500 GB. Problem not solved.

perfect! At least this matches my understanding of how time machine is supposed to work :), for better or wors.e

However, wonder whether what determines the max size of the bundle: the total space of the share or the available space on the share at the time of creation?

It’s the latter. I don’t think volumes can be dynamically grown after creation. Back then I created a sparse bundle manually, with 10TB size (even though I had just 1TB of space) with the expectaion that it will grow “infinitely” (until 10TB) as needed.

@Christoph

but given that other stuff on the NAS keeps growing, the actually available space would be significantly less than the bundle’s max size, thus it would probably fill up the disk to the last byte, which is not good.

This problem has a brutal but effective solution – when your storage approaches 80-85% - add more storage. BTRFS filesystem actually needs about 10-15 of free space at all times to function effectively, so if you plan right you should never run out of space. Hence, no need to moderate time machine.

@Christoph

So you are saying I still have to take care of stuff myself, now that I’m an Apple user?

Heh. Right. However the OS won’t be trying to actively sabotage your efforts, so there is that

@Christoph

The command, however gives me this:

is time machine mounted? I get non-bogus output:

alex@obsidian ~ % tmutil compare
- 12.0K                         /Volumes/.timemachine/C18ACDDB-C329-4848-886D-06A31C94B8B2/2021-03-18-223847.backup/2021-03-18-223847.backup/Obsidian - Data/Applications/.DS_Store
- 0B                            /Volumes/.timemachine/C18ACDDB-C329-4848-886D-06A31C94B8B2/2021-03-18-223847.backup/2021-03-18-223847.backup/Obsidian - Data/Applications/.localized
- 196.4M                        /Volumes/.timemachine/C18ACDDB-C329-4848-886D-06A31C94B8B2/2021-03-18-223847.backup/2021-03-18-223847.backup/Obsidian - Data/Applications/1Password 7.app
- 52.4M                         /Volumes/.timemachine/C18ACDDB-C329-4848-886D-06A31C94B8B2/2021-03-18-223847.backup/2021-03-18-223847.backup/Obsidian - Data/Applications/BBEdit.app
- 84.4M                         /Volumes/.timemachine/C18ACDDB-C329-4848-886D-06A31C94B8B2/2021-03-18-223847.backup/2021-03-18-223847.backup/Obsidian - Data/Applications/Backup and Sync.app

Try from the menu bar tell time machine to backup now (or run tmutil startbackup --auto) to ensure the storage is mounted and path as returned by tmutil machinedirectory exists; and then do tmutil compare. It shall work.