macOS: drive the install and all customization from a Recovery "PE", no GUI
Replace the screenshot/OCR/keystroke driving of Apple's Recovery with a "PE": BaseSystem.dmg (a journaled HFS+ volume, writable from Linux) with one LaunchDaemon added (makeRecoveryPE) that runs /Volumes/VMIX/run.sh as root at boot, records the status and powers off. launchd loads it alongside its signed cache (verified on Tahoe 26.6.2); same idea as AutoNBI/Imagr NetBoot images. - makeImage: the PE runs vmix-install.sh (erase, installer app, SharedSupport pkgdmg, startosinstall). Progress is read from the serial console (boot-args serial=3 -v, VMIX-* markers) and screenshots (brightness only). Fully offline; prepare now takes ~5 min instead of ~10. - customizeImage: boots the PE with the image attached and runs the template offline against the mounted System/Data volumes; OpenCore ScanPolicy restricted to HFS+/SATA so only the PE can boot. One PE boot ~30 s. The installed macOS is never booted for customization, so nothing depends on launchd/BTM approval or a first-boot agent (removed). - templates rewritten for offline use: generalize creates the user with dscl -f (admin, home, auto-login kcpassword, Setup Assistant suppression, hostname, locale, timezone, keyboard type, container resize); remote-access, no-updates, performance edit the target's plists. - makeBootDisk: build-time OpenCore variant (serial console, ScanPolicy). - vm-driver.py rewritten: passive observation only (serial markers, kernel boots, panics, brightness), disk+serial-aware hang watchdog, reboot-death reset, halt/loginwindow detection. No OCR/tesseract. - OpenCore: four SMBIOS DIMMs for MacPro7,1 (no "Memory Modules Misconfigured" warning). - tools/soak.sh: repeatability harness. Verified on daku: base install 23 min end to end; basic + generalize in three ~30 s PE boots; the result auto-logs into the desktop with the created user. Root cause of the "first-boot hang" (from the serial log): the guest's restart path panics (IOPlatformHaltRestartAction -> AppleSMC, SMCWDT smcWriteKey kSMCBadCommand, nested panic) because the pinned OSX-KVM Lilu disables itself on macOS 26, so VirtualSMC never loads. Handled by the driver (reset within 60 s); kext update to follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XsESshRCoBoUVWV9qKURUF
This commit is contained in:
parent
58a317f5d2
commit
8dc8f4265d
24 changed files with 802 additions and 977 deletions
|
|
@ -1,138 +1,100 @@
|
|||
# vmix macOS images
|
||||
# macOS images (Tahoe 26)
|
||||
|
||||
Unattended macOS (Tahoe / 26) VM images, built the same way as the Windows
|
||||
images: `makeImage` installs the OS once, templates customize it by booting it,
|
||||
`.generalize` creates the user and seals the image.
|
||||
Pre-installed, Apple-ID-capable macOS VM images built the same way as the
|
||||
Windows ones: `makeImage` (unattended install) → templates → `.generalize`
|
||||
(user, hostname, fresh SMBIOS identity). Runs on QEMU/KVM with OpenCore.
|
||||
|
||||
```
|
||||
vmix build --image macos.images.tahoe.basic \
|
||||
--generalize username=sagar,password=secret,hostname=MAC,timezone=Europe/Zurich
|
||||
vmix run ./result --macos --vnc :10 --mem 8192 # VNC on port 5910
|
||||
vmix build --image macos.images.tahoe.basic --generalize username=sagar,password=secret,hostname=MAC
|
||||
vmix run ./result --macos --vnc :10 --mem 8192
|
||||
```
|
||||
|
||||
## How it works
|
||||
Nix: `macos.images.tahoe.{pe,upstream,basic,remote}` and
|
||||
`<image>.generalize { username; password; hostname; timezone; locale; seed; … }`.
|
||||
|
||||
| step | what happens |
|
||||
|---|---|
|
||||
| `fetchRecovery` | BaseSystem.dmg from Apple's recovery servers (fixed-output, pinned by sha256) |
|
||||
| `installerPayload` | takes the App Store `InstallAssistant.pkg` (18 GB, pinned) apart on Linux: the app skeleton (pbzx/cpio) and the byte offset of `SharedSupport.dmg` |
|
||||
| `makeOpenCore` | OSX-KVM's OpenCore ESP with a config.plist rewritten for this image: SMBIOS model, serial + MLB (`macserial`), UUID and ROM = NIC MAC (derived from a seed), NIC marked built-in |
|
||||
| `makeImage` | one QEMU session: Recovery boots via OpenCore → `vm-driver.py` opens Terminal with keystrokes (Ctrl-F2 menu navigation, screen-settle detection + OCR of the menu bar) and types `sh /Volumes/VMIX/run.sh` → `vmix-install.sh` erases the disk, rebuilds `Install macOS Tahoe.app` (skeleton + `SharedSupport.dmg` copied from a raw disk mapped straight out of the pkg), runs `startosinstall --installpackage vmix-agent.pkg` → installer reboots through its phases → first boot runs the **vmix agent** which powers off. OpenCore is then copied into the image's EFI partition, so it boots standalone with OVMF |
|
||||
| `customizeImage` | boots the image with a FAT volume `VMIX`; the agent (LaunchDaemon `ch.vmix.agent`) runs `vmix-run.sh` as root, writes `vmix-run.status`/`.log` back and shuts down |
|
||||
| `templates.generalize` | user (admin) + auto-login (`/etc/kcpassword`), Setup Assistant suppressed, hostname, timezone, no sleep, APFS grown to the disk, then the agent removes itself; a fresh SMBIOS identity is written to the ESP |
|
||||
## How it works: the vmix "PE"
|
||||
|
||||
The vmix agent replaces Windows' Audit Mode RunOnce; `.AppleSetupDone` replaces
|
||||
the OOBE unattend. Everything on the host side runs inside `__noChroot`
|
||||
derivations (KVM + `/tmp`), exactly like the Windows builders.
|
||||
Apple's Recovery (`BaseSystem.dmg`, a plain journaled HFS+ volume) with **one
|
||||
LaunchDaemon added** (`makeRecoveryPE`): at boot it mounts a `VMIX` volume and
|
||||
runs `run.sh` from it as root, records the exit status and powers off. That is
|
||||
the whole automation surface — the equivalent of Windows PE + Autounattend:
|
||||
|
||||
## Generalize options
|
||||
* **no GUI is driven**: no OCR, no keystrokes, no screen layouts to learn per
|
||||
macOS version; the hook is a launchd plist, stable across releases (same idea
|
||||
as AutoNBI/Imagr NetBoot images).
|
||||
* **observable**: the guest prints `VMIX-*` markers to `/dev/console`, which the
|
||||
build reads from QEMU's serial log (`boot-args serial=3 -v`). Kernel panics and
|
||||
reboots show up there too. Screenshots are still taken for debugging.
|
||||
* **offline**: no NIC during the install, and the guest blackholes Apple's
|
||||
install/verify endpoints so `startosinstall` never waits on the network. The
|
||||
only inputs are the pinned `InstallAssistant.pkg` and `BaseSystem.dmg`.
|
||||
* **everything else happens offline from the PE too**: templates and generalize
|
||||
mount the image's Data volume (rw) and System volume (ro) and edit them
|
||||
(`dscl -f` for users, `plutil` for preferences) — the installed macOS is
|
||||
never booted for customization, so nothing depends on launchd/BTM approval,
|
||||
first-boot agents or auto-login inside the guest. One PE boot ≈ 30 s.
|
||||
|
||||
`username password fullName autoLogon hostname locale timezone delayOobeRun`
|
||||
as for Windows (`bgColor` is accepted but ignored), plus the SMBIOS identity:
|
||||
`model serial mlb uuid mac seed`. Anything unset is generated: serial/MLB by
|
||||
macserial (random per build), MAC and UUID deterministically from `seed`
|
||||
(default `hostname-username`). `vmix macserial --model MacPro7,1` prints a
|
||||
ready-to-paste set.
|
||||
### Pipeline
|
||||
|
||||
`delay-oobe-run=true` creates no user and re-arms Setup Assistant for the first
|
||||
real boot.
|
||||
|
||||
## Apple ID / iMessage
|
||||
|
||||
The image satisfies what Dortania lists for iServices: unique serial + MLB for a
|
||||
Tahoe-supported model (`MacPro7,1` by default; `iMac20,1/2`,
|
||||
`MacBookPro16,x` also work), SystemUUID, ROM equal to en0's MAC, and en0 marked
|
||||
built-in (the NIC is pinned to `PciRoot(0x0)/Pci(0x12,0x0)`). The NixOS module
|
||||
and `vmix run --macos` use the MAC recorded in the image (`EFI/vmix/vmix.json`).
|
||||
Give each deployed VM its own generalized image (different `seed`, or explicit
|
||||
`serial=`/`mlb=`) — two VMs with the same identity will be blocked.
|
||||
|
||||
## Runtime
|
||||
|
||||
* `vmix run <qcow2> --macos [--vnc :N] [--mac ..]`
|
||||
* NixOS module: `disks.os.file = vmixLib.macos.images.tahoe.basic.generalize {...}`
|
||||
is auto-detected (`_vmixOsType = "macos"`): Skylake-Client CPU spoof, AppleSMC,
|
||||
USB keyboard/tablet, AHCI system disk, VMware SVGA, pinned NIC with the image's MAC.
|
||||
`macos.cpu`, `macos.mac`, `macos.enable` override the defaults.
|
||||
* `vmix copy` writes the image to a disk but cannot grow APFS from Linux
|
||||
(`diskutil apfs resizeContainer disk0s2 0` in macOS afterwards).
|
||||
|
||||
## Debugging a build
|
||||
|
||||
Screenshots (`NNN-<state>.png`), `driver.log` and the QMP socket of every VM
|
||||
session are in `/tmp/vmix-macos/<image name>/` on the build host. The guest logs
|
||||
(`install.log`, `vmix-run.log`, `vmix-agent.log`) are printed at the end of the
|
||||
build. Pass `vncDisplay = ":10"` to `makeImage`/`customizeImage` (or
|
||||
`--generalize vncDisplay=:10`) to watch live; with a `DISPLAY` an SDL window
|
||||
is used as for Windows.
|
||||
|
||||
## Updating pins (`upstream.json`)
|
||||
|
||||
* installer: URL + SRI hash of a newer `InstallAssistant.pkg`
|
||||
(`nix store prefetch-file --name InstallAssistant.pkg <url>`; Mr. Macintosh's
|
||||
database lists Apple's URLs)
|
||||
* recovery: Apple serves the current build for the board id, so the sha256
|
||||
changes with each point release — copy the "got:" hash from the failed build
|
||||
* opencore: OSX-KVM `OpenCore.qcow2` at a commit; OpenCorePkg release zip (macserial/ocvalidate)
|
||||
|
||||
## Known limits
|
||||
|
||||
* The Recovery bootstrap depends on keyboard navigation of the Recovery UI
|
||||
(Ctrl-F2 → Utilities → Terminal). It self-corrects with screenshots + OCR and
|
||||
falls back to a blind sequence, but a Recovery UI change would need
|
||||
`vm-driver.py` adjusted.
|
||||
* Hosts must run KVM with an AVX2-capable CPU (Intel or AMD; the guest sees a
|
||||
Skylake). `sandbox = relaxed` and the `kvm` system feature, as for Windows.
|
||||
* Software updates inside the VM are disabled by the `noUpdates` template
|
||||
(OTA updates in a VM need the RestrictEvents kext).
|
||||
|
||||
## Current status (2026-09-09): working offline install
|
||||
|
||||
`macos.images.tahoe.upstream` builds a bootable, installed macOS Tahoe 26.6.2
|
||||
qcow2 **fully offline** on the KVM host — no dependency on Apple's servers at build
|
||||
time, just the pinned local `InstallAssistant.pkg` and `BaseSystem.dmg`. The
|
||||
finished image boots standalone (OpenCore from its own ESP) to the macOS
|
||||
loginwindow. Serial/MLB/UUID/ROM are per-image for Apple ID / iMessage.
|
||||
|
||||
How the install is driven (`vm-driver.py`, all by screenshot + OCR over QMP):
|
||||
|
||||
* The whole `InstallAssistant.pkg` is mapped as a raw disk (it is a "pkgdmg":
|
||||
xar + koly footer) and `dd`'d byte-exact into the app as `SharedSupport.dmg` —
|
||||
extracting the bare xar member fails startosinstall with "pkgdmg missing a footer".
|
||||
* No NIC during install + `/etc/hosts` blackhole of Apple's install/verify
|
||||
endpoints, so `startosinstall`'s network calls fail fast instead of hanging —
|
||||
offline prepare, no external dependency. `SecureBootModel=Disabled` lets the
|
||||
sealed volume install without online personalization.
|
||||
* The recovery display is kept awake with a tiny mouse jiggle (a lone keypress
|
||||
does not reset display sleep, and the sleeping display swallows the menu-nav
|
||||
keystrokes); the settle detector uses a coarse fingerprint so the jiggling
|
||||
cursor is not seen as a screen change.
|
||||
* startosinstall prepare is intermittently slow/stalls; a guest watchdog kills and
|
||||
re-erases/retries an attempt that stalls or runs > 9 min.
|
||||
* First boot in QEMU intermittently hangs at the Apple logo; a disk-aware watchdog
|
||||
(`--progress-file`) issues a QMP `system_reset` only when the screen is dark AND
|
||||
the disk is idle, so a slow-but-working boot is never interrupted.
|
||||
* The install reaching the (bright) loginwindow is detected by brightness (the
|
||||
faint gray "password" text does not OCR) and the driver powers the VM down —
|
||||
the image is installed. macOS `shutdown -h now` halts to black without an ACPI
|
||||
power-off, so a black+disk-idle screen is also treated as a completed halt.
|
||||
* OpenCore is then copied into the image's own ESP so it boots standalone with OVMF.
|
||||
1. `makeRecoveryPE` — BaseSystem.dmg → raw HFS+ image + `ch.vmix.pe` daemon.
|
||||
2. `makeImage` — QEMU with: OpenCore boot disk (build variant with serial
|
||||
console), the PE, the empty target disk, the VMIX volume (`vmix-install.sh`,
|
||||
installer app skeleton) and the whole `InstallAssistant.pkg` mapped as a raw
|
||||
disk. The guest script erases the target as APFS, unpacks the app and `dd`s
|
||||
the pkg into it as `SharedSupport.dmg` (it is a "pkgdmg": xar + koly footer;
|
||||
the bare xar member fails with "pkgdmg is missing a footer"), then runs
|
||||
`startosinstall`, which reboots itself through the install phases. The
|
||||
installed system's first boot ends at the loginwindow: the driver detects the
|
||||
bright screen and powers the VM down. OpenCore is then copied into the image's
|
||||
own ESP so it boots with plain OVMF.
|
||||
3. `customizeImage` — boots the PE with the image attached (OpenCore
|
||||
`ScanPolicy` restricted to HFS+ on SATA, so only the PE can boot) and runs the
|
||||
template script with `$SYS`/`$DATA` mounted. `pe-lib.sh` has the helpers.
|
||||
4. `templates/generalize.nix` — user (dscl, admin, home from the user template),
|
||||
auto-login (`kcpassword`), Setup Assistant suppression, hostname, locale,
|
||||
timezone, keyboard type, container resize, fresh SMBIOS via a new OpenCore
|
||||
ESP (`serial`/`mlb` from macserial, MAC + UUID from `seed`).
|
||||
|
||||
### Recovery source
|
||||
|
||||
`recovery.file` in `upstream.json` points at a content-addressed store path for the
|
||||
verified Tahoe `BaseSystem.dmg` (Apple's CDN load-balances Sequoia/Tahoe during the
|
||||
rollout, so a plain fetch is non-deterministic). Reproduce it on any host with
|
||||
`nix store add-path --name macos-tahoe-BaseSystem.dmg BaseSystem.dmg` (same path
|
||||
from the same bytes). Set `recovery.sha256` and remove `recovery.file` to fetch it
|
||||
from Apple instead (subject to the CDN rollout).
|
||||
`recovery.file` in `upstream.json` points at a content-addressed store path for
|
||||
the verified Tahoe `BaseSystem.dmg` (Apple's CDN load-balances Sequoia/Tahoe
|
||||
during the rollout, so a plain fetch is non-deterministic). Reproduce it on any
|
||||
host with `nix store add-path --name macos-tahoe-BaseSystem.dmg BaseSystem.dmg`.
|
||||
Drop `recovery.file` to fetch from Apple instead (`fetchRecovery` retries until
|
||||
the pinned hash matches).
|
||||
|
||||
### Not yet done: generalize / user creation
|
||||
## Reliability
|
||||
|
||||
The base image installs and boots to loginwindow. `.generalize` (user creation,
|
||||
auto-login, hostname) relies on the vmix agent LaunchDaemon running on first boot,
|
||||
but macOS Ventura+ Background Task Management does not auto-run a headless
|
||||
third-party daemon, and neither the pkg `launchctl bootstrap` (installer domain
|
||||
only) nor a cron `@reboot` reliably triggered it. The robust next step is to inject
|
||||
the user record + settings offline from the agent pkg's postinstall (which runs as
|
||||
root on the target during install), instead of a first-boot daemon.
|
||||
Things QEMU does intermittently, and what handles each (all in `vm-driver.py`
|
||||
and `vmix-install.sh`; every event is logged with a reason):
|
||||
|
||||
* `startosinstall` prepare stalls or crawls — the guest kills and retries it on a
|
||||
freshly erased target (free-space watchdog + time cap).
|
||||
* the installer comes back to the PE instead of the install phase — the PE
|
||||
counts boots and simply re-runs the install (max 3).
|
||||
* the installed system hangs at the Apple logo on first boot — a `system_reset`
|
||||
is issued only when the screen is dark and frozen **and** disk and serial
|
||||
console are idle, so a slow-but-working boot is never interrupted.
|
||||
* macOS `shutdown -h` halts to a black screen without an ACPI power-off — an
|
||||
idle black screen counts as a completed halt.
|
||||
* a kernel panic (seen on the serial console) resets the VM.
|
||||
* a wedged run fails at the 4 h timeout instead of hanging.
|
||||
|
||||
`tools/soak.sh <flake> macos.images.tahoe.upstream 3` rebuilds an image N
|
||||
times and tabulates outcome, duration, boots, resets, panics and retries.
|
||||
|
||||
## Debugging
|
||||
|
||||
`/tmp/vmix-macos/<name>/` on the build host: `driver.log`, `serial.log`
|
||||
(kernel + `VMIX-*` markers), periodic PNG screenshots, `qmp.sock`.
|
||||
`vmix-run.log` / `system-install.log` from the VMIX volume are printed at the
|
||||
end of the build. Add `vncDisplay = ":10"` to watch.
|
||||
|
||||
## QEMU profile
|
||||
|
||||
`helpers/qemu.nix`: q35, `Skylake-Client` CPU spoof (works on AMD),
|
||||
AppleSMC with the OSK, XHCI keyboard/tablet, AHCI disks, VMware SVGA,
|
||||
virtio-net pinned to `PciRoot(0x0)/Pci(0x12,0x0)` so OpenCore marks it built-in
|
||||
(en0, required for Apple ID / iMessage). SMBIOS `MacPro7,1` with four DIMMs
|
||||
described (avoids the "Memory Modules Misconfigured" warning).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue