dotfiles/vmix.nix
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

pci: keep multifunction devices together; data disk: diskpart, not Storage cmdlets

Browse source 
Two defects found while getting a GPU through Proxmox to a nested guest.

Passthrough gave every address its own pcie-root-port, which splits a GPU from
its own HDMI audio: 05:00.0 and 05:00.1 arrived in the guest as two devices on
two buses instead of functions 0 and 1 of one device. Navi needs both halves on
one device to reset or power-manage either, so the guest got a card stuck in D3
and a reset that could not be performed. Addresses are now grouped by
everything left of the function digit, and each group goes behind one root port
at one slot with multifunction=on on function 0 -- which is also where the
VBIOS and the VGA route belong.

The data-disk setup used Initialize-Disk/New-Partition/Format-Volume. Only the
first of those works that early in specialize; the rest need services that are
not up yet, and with ErrorActionPreference=Stop the script gave up straight
after writing a GPT header. The result was a 50G disk carrying 24KB of nothing
and a ProfilesDirectory pointing at a volume that never existed. diskpart works
at that stage. It also tries assigning the letter before laying the disk out,
so a disk that already holds a profile is lettered rather than cleaned.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117qMyjpuXsjpVAcpJbFD8g
This commit is contained in:
Git Sagar 2026-09-10 04:58:54 -03:00
parent 97fdbce6d0
commit 0e4619eb39
2 changed files with 43 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

19
nixos/vms/config.nix
View file

@ -84,6 +84,18 @@ let
machineIrqchipArg =
if vmCfg.pci.viommu.enable then ",kernel-irqchip=split"
else optionalString vmCfg.cpu.hideVirtualized ",kernel_irqchip=on";
# Functions of one physical device have to reach the guest as functions
# of one device too. Giving each address its own root port splits a GPU
# from its own HDMI audio, and Navi cannot then reset or power-manage
# either half -- the guest ends up with a card stuck in D3. So group by
# everything left of the function digit and place each group behind a
# single root port, multifunction, at the same slot.
pciDeviceOf = addr: head (splitString "." addr);
pciFunctionOf = addr: last (splitString "." addr);
pciGroups = map
(dev: filter (a: pciDeviceOf a == dev) vmCfg.pci.passthrough)
(unique (map pciDeviceOf vmCfg.pci.passthrough));
# Linux VMs: apply customizeImage with 9p fstab and machine-id setup
linuxOsImage = vmixLib.linux.customizeImage vmCfg.disks.os.file {
@ -226,10 +238,11 @@ let
-device ${vmCfg.nicModel},netdev=macvtap-${macvtap.name},mac=$(ip l show ${macvtap.iface} | awk '/link\/ether/{print $2}') \
-netdev tap,id=macvtap-${macvtap.name},fd=${toString (i+2)} ${toString (i+2)}<>/dev/tap$(ip l show ${macvtap.iface} | awk -F':' '/${macvtap.iface}/{print $1}') \
'') allMacvtaps)} \
${concatStrings (imap1 (i: pciAddr: ''
${concatStrings (imap1 (i: group: ''
-device pcie-root-port,id=pci-passthrough${toString i},chassis=${toString i},slot=${toString i} \
-device vfio-pci,host=${pciAddr},bus=pci-passthrough${toString i}${optionalString (i == 1) "${optionalString vmCfg.pci.vgaPassthrough ",x-vga=on"}${optionalString (vmCfg.pci.romFile != null) ",romfile=${vmCfg.pci.romFile}"}"} \
'') vmCfg.pci.passthrough)} \
'' + concatStrings (imap0 (j: pciAddr: ''
-device vfio-pci,host=${pciAddr},bus=pci-passthrough${toString i},addr=0x0.${pciFunctionOf pciAddr}${optionalString (length group > 1 && j == 0) ",multifunction=on"}${optionalString (i == 1 && j == 0) "${optionalString vmCfg.pci.vgaPassthrough ",x-vga=on"}${optionalString (vmCfg.pci.romFile != null) ",romfile=${vmCfg.pci.romFile}"}"} \
'') group)) pciGroups)} \
${concatMapStrings (usbDev: ''
-device usb-host,vendorid=0x${usbDev.vendorId},productid=0x${usbDev.productId} \
'') vmCfg.usb.hostDevices} \