Git Sagar
e6e2e9f18d
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XsESshRCoBoUVWV9qKURUF
177 lines
10 KiB
Markdown
177 lines
10 KiB
Markdown
# macOS images (Tahoe 26)
|
||
|
||
Pre-installed, Apple-ID-capable macOS VM images built the same way as the
|
||
Windows ones: `makeImage` (unattended install) → templates → `.generalize`
|
||
(user, hostname, fresh SMBIOS identity). Runs on QEMU/KVM with OpenCore.
|
||
|
||
```
|
||
vmix build --image macos.images.tahoe.basic --generalize username=sagar,password=secret,hostname=MAC
|
||
vmix run ./result --macos --vnc :10 --mem 8192
|
||
```
|
||
|
||
Nix: `macos.images.tahoe.{pe,upstream,basic,remote}` and
|
||
`<image>.generalize { username; password; hostname; timezone; locale; seed; … }`.
|
||
|
||
## How it works: the vmix "PE"
|
||
|
||
Apple's Recovery (`BaseSystem.dmg`, a plain journaled HFS+ volume) with **one
|
||
LaunchDaemon added** (`makeRecoveryPE`): at boot it mounts a `VMIX` volume and
|
||
runs `run.sh` from it as root, records the exit status and powers off. That is
|
||
the whole automation surface — the equivalent of Windows PE + Autounattend:
|
||
|
||
* **no GUI is driven**: no OCR, no keystrokes, no screen layouts to learn per
|
||
macOS version; the hook is a launchd plist, stable across releases (same idea
|
||
as AutoNBI/Imagr NetBoot images).
|
||
* **observable**: the guest prints `VMIX-*` markers to `/dev/console`, which the
|
||
build reads from QEMU's serial log (`boot-args serial=3 -v`). Kernel panics and
|
||
reboots show up there too. Screenshots are still taken for debugging.
|
||
* **offline**: no NIC during the install, and the guest blackholes Apple's
|
||
install/verify endpoints so `startosinstall` never waits on the network. The
|
||
only inputs are the pinned `InstallAssistant.pkg` and `BaseSystem.dmg`.
|
||
* **everything else happens offline from the PE too**: templates and generalize
|
||
mount the image's Data volume (rw) and System volume (ro) and edit them
|
||
(`dscl -f` for users, `plutil` for preferences) — the installed macOS is
|
||
never booted for customization, so nothing depends on launchd/BTM approval,
|
||
first-boot agents or auto-login inside the guest. One PE boot ≈ 30 s.
|
||
|
||
### Pipeline
|
||
|
||
1. `makeRecoveryPE` — BaseSystem.dmg → raw HFS+ image + `ch.vmix.pe` daemon.
|
||
2. `makeImage` — QEMU with: OpenCore boot disk (build variant with serial
|
||
console), the PE, the empty target disk, the VMIX volume (`vmix-install.sh`,
|
||
installer app skeleton) and the whole `InstallAssistant.pkg` mapped as a raw
|
||
disk. The guest script erases the target as APFS, unpacks the app and `dd`s
|
||
the pkg into it as `SharedSupport.dmg` (it is a "pkgdmg": xar + koly footer;
|
||
the bare xar member fails with "pkgdmg is missing a footer"), then runs
|
||
`startosinstall`, which reboots itself through the install phases. The
|
||
installed system's first boot ends at the loginwindow: the driver detects the
|
||
bright screen and powers the VM down. OpenCore is then copied into the image's
|
||
own ESP so it boots with plain OVMF.
|
||
3. `customizeImage` — boots the PE with the image attached (OpenCore
|
||
`ScanPolicy` restricted to HFS+ on SATA, so only the PE can boot) and runs the
|
||
template script with `$SYS`/`$DATA` mounted. `pe-lib.sh` has the helpers.
|
||
4. `templates/generalize.nix` — user (dscl, admin, home from the user template),
|
||
auto-login (`kcpassword`), Setup Assistant suppression, hostname, locale,
|
||
timezone, keyboard type, container resize, fresh SMBIOS via a new OpenCore
|
||
ESP (`serial`/`mlb` from macserial, MAC + UUID from `seed`).
|
||
|
||
### Recovery source
|
||
|
||
`recovery.file` in `upstream.json` points at a content-addressed store path for
|
||
the verified Tahoe `BaseSystem.dmg` (Apple's CDN load-balances Sequoia/Tahoe
|
||
during the rollout, so a plain fetch is non-deterministic). Reproduce it on any
|
||
host with `nix store add-path --name macos-tahoe-BaseSystem.dmg BaseSystem.dmg`.
|
||
Drop `recovery.file` to fetch from Apple instead (`fetchRecovery` retries until
|
||
the pinned hash matches).
|
||
|
||
## Reliability
|
||
|
||
Things QEMU does intermittently, and what handles each (all in `vm-driver.py`
|
||
and `vmix-install.sh`; every event is logged with a reason):
|
||
|
||
* `startosinstall` prepare stalls or crawls — the guest kills and retries it on a
|
||
freshly erased target (free-space watchdog + time cap).
|
||
* the installer comes back to the PE instead of the install phase — the PE
|
||
counts boots and simply re-runs the install (max 3).
|
||
* the installed system hangs at the Apple logo on first boot — a `system_reset`
|
||
is issued only when the screen is dark and frozen **and** disk and serial
|
||
console are idle, so a slow-but-working boot is never interrupted.
|
||
* macOS `shutdown -h` halts to a black screen without an ACPI power-off — an
|
||
idle black screen counts as a completed halt.
|
||
* a kernel panic (seen on the serial console) resets the VM.
|
||
* a wedged run fails at the 4 h timeout instead of hanging.
|
||
|
||
`tools/soak.sh <flake> macos.images.tahoe.upstream 3` rebuilds an image N
|
||
times and tabulates outcome, duration, boots, resets, panics and retries.
|
||
Measured 2026-09-09 on the build host (Ryzen 7 7840HS, ZFS), Tahoe 26.6.2,
|
||
VirtualSMC-only, PE install — 3 of 3 builds completed:
|
||
|
||
| run | minutes | kernel boots | prepare tries | panics (self-recovered) | reboot deaths |
|
||
|-----|---------|--------------|---------------|-------------------------|---------------|
|
||
| 1 | 30 | 8 | 1 | 2 | 0 |
|
||
| 2 | 26 | 7 | 1 | 1 | 0 |
|
||
| 3 | 26 | 7 | 1 | 1 | 0 |
|
||
|
||
What still happens: at roughly one in ten guest-initiated reboots the guest
|
||
either panics (GPF in launchd/kernel_task context shortly after `MACH Reboot`
|
||
or within the first 15 s of the next boot — tmpfs/APFS/zone corruption
|
||
signatures, i.e. memory or register state, not one driver) or never comes back
|
||
(dead after `IOPlatformHaltRestartAction`). XNU reboots itself after a panic;
|
||
the driver resets a dead guest after 60 s, so builds complete. A device bisect
|
||
(`tools`-style 10–30 PE reboots per variant: VMware SVGA vs std VGA, no HDA,
|
||
EHCI input, 1 vCPU) showed the rate is independent of the emulated devices and
|
||
of SMP; Haswell-noTSX does not boot Tahoe. Host: AMD Zen 4, kvm_amd, Intel
|
||
Skylake-Client vCPU model — the FPU-context-switch panic points at XSAVE state
|
||
handling on that combination. Not fixed; a `vmix run` VM that hangs on Restart
|
||
must be reset from the host.
|
||
|
||
## Debugging
|
||
|
||
`/tmp/vmix-macos/<name>/` on the build host: `driver.log`, `serial.log`
|
||
(kernel + `VMIX-*` markers), periodic PNG screenshots, `qmp.sock`.
|
||
`vmix-run.log` / `system-install.log` from the VMIX volume are printed at the
|
||
end of the build. Add `vncDisplay = ":10"` to watch.
|
||
|
||
## QEMU profile
|
||
|
||
`helpers/qemu.nix`: q35, `Skylake-Client` CPU spoof (works on AMD),
|
||
AppleSMC with the OSK, XHCI keyboard/tablet, AHCI disks, VMware SVGA,
|
||
virtio-net pinned to `PciRoot(0x0)/Pci(0x12,0x0)` so OpenCore marks it built-in
|
||
(en0, required for Apple ID / iMessage). SMBIOS `MacPro7,1` with four DIMMs
|
||
described (avoids the "Memory Modules Misconfigured" warning).
|
||
|
||
OpenCore comes from OSX-KVM's proven ESP, with Lilu / VirtualSMC /
|
||
WhateverGreen replaced by current releases (`upstream.json` → `opencore.kexts`):
|
||
the versions OSX-KVM ships disable themselves on macOS 26, and without
|
||
VirtualSMC the guest's restart path panics on QEMU's SMC stub
|
||
(`SMCWDT smcWriteKey kSMCBadCommand`, nested panic after `MACH Reboot`).
|
||
For the same reason QEMU's `isa-applesmc` is not used any more: its presence
|
||
makes VirtualSMC step aside ("multiple devices present"); VirtualSMC carries
|
||
the OSK itself. Images built before this change still need the stub:
|
||
`vmix run --macos --applesmc`. RestrictEvents (`revpatch=memtab`) silences
|
||
MacPro7,1's "Memory Modules Misconfigured" at login.
|
||
|
||
## Guest agent, shares, persistent home, online templates
|
||
|
||
macOS 13+ ships **Apple's own QEMU guest agent** (`/usr/libexec/AppleQEMUGuestAgent`,
|
||
started by launchd when a virtio console port named `org.qemu.guest_agent.0`
|
||
appears). It is Apple-signed, needs no approval, and offers `guest-exec` as
|
||
root plus `guest-file-*`. vmix uses it everywhere an in-guest agent is needed:
|
||
|
||
* `vmix run --macos` and the NixOS module attach it by default
|
||
(`/tmp/vmix-qga-<pid>.sock`, `/run/vmix/qga-<name>.sock`); talk to it with any
|
||
QGA client, e.g. `printf '{"execute":"guest-exec","arguments":{"path":"/usr/bin/id","capture-output":true}}\n' | socat - UNIX-CONNECT:<sock>`.
|
||
* **online templates** (`bootScript`): `customizeImage` boots the image with the
|
||
agent, runs the script as root (network available, `as_user <cmd>` runs inside
|
||
the logged-in user's session), then shuts down through the agent.
|
||
`templates.software.script { name; script; }`,
|
||
`templates.software.homebrew { formulae; casks; }`,
|
||
`templates.profile.settings { hideWidgets; wallpaper; dockApps; dockAutohide;
|
||
darkMode; showHiddenFiles; }` (wallpaper via the pinned `desktoppr`; Apple
|
||
Events / `osascript` do not work headless — TCC automation consent).
|
||
* **offline software templates** run in the PE: `templates.software.pkg { name;
|
||
src; }` (`installer -target`), `templates.software.app { name; src; }`.
|
||
|
||
`AppleVirtIO.kext` (x86 Tahoe) drives virtio-fs, 9p, block, console, input,
|
||
net, sound, balloon, vsock — QEMU's modern virtio-pci devices work as-is:
|
||
|
||
* **shared folders**: virtio-fs (`virtiofsd` + `vhost-user-fs-pci`, shared
|
||
memory backend). The tag `com.apple.virtio-fs.automount` is mounted by macOS
|
||
itself at `/Volumes/My Shared Files`; further tags are mounted with
|
||
`mount -t virtiofs <tag> <dir>` — the module does that through the guest agent
|
||
for every `shares.<name>` beyond the first. `vmix run --macos --share DIR`.
|
||
(9p does not automount on macOS; the Linux `-virtfs` path is not used.)
|
||
* **ephemeral OS disk + persistent home**: `generalize { persistHome = true; }`
|
||
gives the account its home directory on an APFS volume labelled `vmix-home`
|
||
(`NFSHomeDirectory = /Volumes/vmix-home/<user>`; macOS refuses mounts over
|
||
`/Users`, which is a firmlink). The host provides a virtio-blk disk
|
||
(`macos.homeDisk` in the module, `--home FILE` in the CLI: qcow2/raw file or
|
||
zvol) that `formatVolume` formats as APFS `vmix-home` by booting the PE for
|
||
~35 s on first use; diskarbitrationd mounts it before login and loginwindow
|
||
creates the home directory there on first login. The OS disk can then run
|
||
with `snapshot=on` (`disks.os.persist = false`).
|
||
* **SPICE**: `-vga vmware` (or `std`) is kept as the display device — macOS has
|
||
no QXL/virtio-gpu driver; USB redirection channels work as for other guests
|
||
(`spice.usbRedir`); there is no vdagent for macOS (no clipboard sharing).
|
||
virtio keyboard/tablet (`AppleVirtIOInput`) are available as
|
||
`qemu.virtioInputArgs` but the USB HID pair is the default.
|