Git Sagar
e241364ce9
macOS refuses to mount a volume over /Users (firmlink), and a failing fstab entry also suppresses the automount. Point NFSHomeDirectory at the automounted volume instead. Profile: Dock entries need tile-type/file URL; wait for the wallpaper store before shutting down. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XsESshRCoBoUVWV9qKURUF
169 lines
9.9 KiB
Markdown
169 lines
9.9 KiB
Markdown
# macOS images (Tahoe 26)
|
|
|
|
Pre-installed, Apple-ID-capable macOS VM images built the same way as the
|
|
Windows ones: `makeImage` (unattended install) → templates → `.generalize`
|
|
(user, hostname, fresh SMBIOS identity). Runs on QEMU/KVM with OpenCore.
|
|
|
|
```
|
|
vmix build --image macos.images.tahoe.basic --generalize username=sagar,password=secret,hostname=MAC
|
|
vmix run ./result --macos --vnc :10 --mem 8192
|
|
```
|
|
|
|
Nix: `macos.images.tahoe.{pe,upstream,basic,remote}` and
|
|
`<image>.generalize { username; password; hostname; timezone; locale; seed; … }`.
|
|
|
|
## How it works: the vmix "PE"
|
|
|
|
Apple's Recovery (`BaseSystem.dmg`, a plain journaled HFS+ volume) with **one
|
|
LaunchDaemon added** (`makeRecoveryPE`): at boot it mounts a `VMIX` volume and
|
|
runs `run.sh` from it as root, records the exit status and powers off. That is
|
|
the whole automation surface — the equivalent of Windows PE + Autounattend:
|
|
|
|
* **no GUI is driven**: no OCR, no keystrokes, no screen layouts to learn per
|
|
macOS version; the hook is a launchd plist, stable across releases (same idea
|
|
as AutoNBI/Imagr NetBoot images).
|
|
* **observable**: the guest prints `VMIX-*` markers to `/dev/console`, which the
|
|
build reads from QEMU's serial log (`boot-args serial=3 -v`). Kernel panics and
|
|
reboots show up there too. Screenshots are still taken for debugging.
|
|
* **offline**: no NIC during the install, and the guest blackholes Apple's
|
|
install/verify endpoints so `startosinstall` never waits on the network. The
|
|
only inputs are the pinned `InstallAssistant.pkg` and `BaseSystem.dmg`.
|
|
* **everything else happens offline from the PE too**: templates and generalize
|
|
mount the image's Data volume (rw) and System volume (ro) and edit them
|
|
(`dscl -f` for users, `plutil` for preferences) — the installed macOS is
|
|
never booted for customization, so nothing depends on launchd/BTM approval,
|
|
first-boot agents or auto-login inside the guest. One PE boot ≈ 30 s.
|
|
|
|
### Pipeline
|
|
|
|
1. `makeRecoveryPE` — BaseSystem.dmg → raw HFS+ image + `ch.vmix.pe` daemon.
|
|
2. `makeImage` — QEMU with: OpenCore boot disk (build variant with serial
|
|
console), the PE, the empty target disk, the VMIX volume (`vmix-install.sh`,
|
|
installer app skeleton) and the whole `InstallAssistant.pkg` mapped as a raw
|
|
disk. The guest script erases the target as APFS, unpacks the app and `dd`s
|
|
the pkg into it as `SharedSupport.dmg` (it is a "pkgdmg": xar + koly footer;
|
|
the bare xar member fails with "pkgdmg is missing a footer"), then runs
|
|
`startosinstall`, which reboots itself through the install phases. The
|
|
installed system's first boot ends at the loginwindow: the driver detects the
|
|
bright screen and powers the VM down. OpenCore is then copied into the image's
|
|
own ESP so it boots with plain OVMF.
|
|
3. `customizeImage` — boots the PE with the image attached (OpenCore
|
|
`ScanPolicy` restricted to HFS+ on SATA, so only the PE can boot) and runs the
|
|
template script with `$SYS`/`$DATA` mounted. `pe-lib.sh` has the helpers.
|
|
4. `templates/generalize.nix` — user (dscl, admin, home from the user template),
|
|
auto-login (`kcpassword`), Setup Assistant suppression, hostname, locale,
|
|
timezone, keyboard type, container resize, fresh SMBIOS via a new OpenCore
|
|
ESP (`serial`/`mlb` from macserial, MAC + UUID from `seed`).
|
|
|
|
### Recovery source
|
|
|
|
`recovery.file` in `upstream.json` points at a content-addressed store path for
|
|
the verified Tahoe `BaseSystem.dmg` (Apple's CDN load-balances Sequoia/Tahoe
|
|
during the rollout, so a plain fetch is non-deterministic). Reproduce it on any
|
|
host with `nix store add-path --name macos-tahoe-BaseSystem.dmg BaseSystem.dmg`.
|
|
Drop `recovery.file` to fetch from Apple instead (`fetchRecovery` retries until
|
|
the pinned hash matches).
|
|
|
|
## Reliability
|
|
|
|
Things QEMU does intermittently, and what handles each (all in `vm-driver.py`
|
|
and `vmix-install.sh`; every event is logged with a reason):
|
|
|
|
* `startosinstall` prepare stalls or crawls — the guest kills and retries it on a
|
|
freshly erased target (free-space watchdog + time cap).
|
|
* the installer comes back to the PE instead of the install phase — the PE
|
|
counts boots and simply re-runs the install (max 3).
|
|
* the installed system hangs at the Apple logo on first boot — a `system_reset`
|
|
is issued only when the screen is dark and frozen **and** disk and serial
|
|
console are idle, so a slow-but-working boot is never interrupted.
|
|
* macOS `shutdown -h` halts to a black screen without an ACPI power-off — an
|
|
idle black screen counts as a completed halt.
|
|
* a kernel panic (seen on the serial console) resets the VM.
|
|
* a wedged run fails at the 4 h timeout instead of hanging.
|
|
|
|
`tools/soak.sh <flake> macos.images.tahoe.upstream 3` rebuilds an image N
|
|
times and tabulates outcome, duration, boots, resets, panics and retries.
|
|
Measured 2026-09-09 on the build host (Ryzen 7 7840HS, ZFS), Tahoe 26.6.2,
|
|
VirtualSMC-only, PE install — 3 of 3 builds completed:
|
|
|
|
| run | minutes | kernel boots | prepare tries | panics (self-recovered) | reboot deaths |
|
|
|-----|---------|--------------|---------------|-------------------------|---------------|
|
|
| 1 | 30 | 8 | 1 | 2 | 0 |
|
|
| 2 | 26 | 7 | 1 | 1 | 0 |
|
|
| 3 | 26 | 7 | 1 | 1 | 0 |
|
|
|
|
What still happens: a kernel panic at one of the guest's own reboots during the
|
|
install (GPF in launchd's context right after `MACH Reboot`). XNU reboots
|
|
itself within seconds and the install continues; the driver only intervenes
|
|
if no kernel comes back within 60 s. Known, logged, not yet root-caused.
|
|
|
|
## Debugging
|
|
|
|
`/tmp/vmix-macos/<name>/` on the build host: `driver.log`, `serial.log`
|
|
(kernel + `VMIX-*` markers), periodic PNG screenshots, `qmp.sock`.
|
|
`vmix-run.log` / `system-install.log` from the VMIX volume are printed at the
|
|
end of the build. Add `vncDisplay = ":10"` to watch.
|
|
|
|
## QEMU profile
|
|
|
|
`helpers/qemu.nix`: q35, `Skylake-Client` CPU spoof (works on AMD),
|
|
AppleSMC with the OSK, XHCI keyboard/tablet, AHCI disks, VMware SVGA,
|
|
virtio-net pinned to `PciRoot(0x0)/Pci(0x12,0x0)` so OpenCore marks it built-in
|
|
(en0, required for Apple ID / iMessage). SMBIOS `MacPro7,1` with four DIMMs
|
|
described (avoids the "Memory Modules Misconfigured" warning).
|
|
|
|
OpenCore comes from OSX-KVM's proven ESP, with Lilu / VirtualSMC /
|
|
WhateverGreen replaced by current releases (`upstream.json` → `opencore.kexts`):
|
|
the versions OSX-KVM ships disable themselves on macOS 26, and without
|
|
VirtualSMC the guest's restart path panics on QEMU's SMC stub
|
|
(`SMCWDT smcWriteKey kSMCBadCommand`, nested panic after `MACH Reboot`).
|
|
For the same reason QEMU's `isa-applesmc` is not used any more: its presence
|
|
makes VirtualSMC step aside ("multiple devices present"); VirtualSMC carries
|
|
the OSK itself. Images built before this change still need the stub:
|
|
`vmix run --macos --applesmc`. RestrictEvents (`revpatch=memtab`) silences
|
|
MacPro7,1's "Memory Modules Misconfigured" at login.
|
|
|
|
## Guest agent, shares, persistent home, online templates
|
|
|
|
macOS 13+ ships **Apple's own QEMU guest agent** (`/usr/libexec/AppleQEMUGuestAgent`,
|
|
started by launchd when a virtio console port named `org.qemu.guest_agent.0`
|
|
appears). It is Apple-signed, needs no approval, and offers `guest-exec` as
|
|
root plus `guest-file-*`. vmix uses it everywhere an in-guest agent is needed:
|
|
|
|
* `vmix run --macos` and the NixOS module attach it by default
|
|
(`/tmp/vmix-qga-<pid>.sock`, `/run/vmix/qga-<name>.sock`); talk to it with any
|
|
QGA client, e.g. `printf '{"execute":"guest-exec","arguments":{"path":"/usr/bin/id","capture-output":true}}\n' | socat - UNIX-CONNECT:<sock>`.
|
|
* **online templates** (`bootScript`): `customizeImage` boots the image with the
|
|
agent, runs the script as root (network available, `as_user <cmd>` runs inside
|
|
the logged-in user's session), then shuts down through the agent.
|
|
`templates.software.script { name; script; }`,
|
|
`templates.software.homebrew { formulae; casks; }`,
|
|
`templates.profile.settings { hideWidgets; wallpaper; dockApps; dockAutohide;
|
|
darkMode; showHiddenFiles; }` (wallpaper via the pinned `desktoppr`; Apple
|
|
Events / `osascript` do not work headless — TCC automation consent).
|
|
* **offline software templates** run in the PE: `templates.software.pkg { name;
|
|
src; }` (`installer -target`), `templates.software.app { name; src; }`.
|
|
|
|
`AppleVirtIO.kext` (x86 Tahoe) drives virtio-fs, 9p, block, console, input,
|
|
net, sound, balloon, vsock — QEMU's modern virtio-pci devices work as-is:
|
|
|
|
* **shared folders**: virtio-fs (`virtiofsd` + `vhost-user-fs-pci`, shared
|
|
memory backend). The tag `com.apple.virtio-fs.automount` is mounted by macOS
|
|
itself at `/Volumes/My Shared Files`; further tags are mounted with
|
|
`mount -t virtiofs <tag> <dir>` — the module does that through the guest agent
|
|
for every `shares.<name>` beyond the first. `vmix run --macos --share DIR`.
|
|
(9p does not automount on macOS; the Linux `-virtfs` path is not used.)
|
|
* **ephemeral OS disk + persistent home**: `generalize { persistHome = true; }`
|
|
gives the account its home directory on an APFS volume labelled `vmix-home`
|
|
(`NFSHomeDirectory = /Volumes/vmix-home/<user>`; macOS refuses mounts over
|
|
`/Users`, which is a firmlink). The host provides a virtio-blk disk
|
|
(`macos.homeDisk` in the module, `--home FILE` in the CLI: qcow2/raw file or
|
|
zvol) that `formatVolume` formats as APFS `vmix-home` by booting the PE for
|
|
~35 s on first use; diskarbitrationd mounts it before login and loginwindow
|
|
creates the home directory there on first login. The OS disk can then run
|
|
with `snapshot=on` (`disks.os.persist = false`).
|
|
* **SPICE**: `-vga vmware` (or `std`) is kept as the display device — macOS has
|
|
no QXL/virtio-gpu driver; USB redirection channels work as for other guests
|
|
(`spice.usbRedir`); there is no vdagent for macOS (no clipboard sharing).
|
|
virtio keyboard/tablet (`AppleVirtIOInput`) are available as
|
|
`qemu.virtioInputArgs` but the USB HID pair is the default.
|