From fde50b7ca7057f4fe3b6fe0184b1d720d60dd6b1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 02:42:41 -0700 Subject: [PATCH] runbook(park 100): hold kernel metas on NVIDIA-DKMS hosts (fv-ml1 6.12.111 DKMS failure) --- docs/runbooks/fleet-guest-patch-2026-10-05.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/runbooks/fleet-guest-patch-2026-10-05.md b/docs/runbooks/fleet-guest-patch-2026-10-05.md index ef43467..ec4181f 100644 --- a/docs/runbooks/fleet-guest-patch-2026-10-05.md +++ b/docs/runbooks/fleet-guest-patch-2026-10-05.md @@ -46,6 +46,11 @@ same window as its Docker storage move (its holds are listed at the bottom). may upgrade. - **NVIDIA/CUDA user-space (`nvidia-* libnvidia-* cuda-* libcuda*`) is held on the GPU LXCs (esh-ml1, nh3-ml1) and pfi-gx10.** In an LXC it must match the host's kernel driver. +- ⚠ **Kernels on any host that builds NVIDIA with DKMS** (hold `linux-image-amd64 linux-headers-amd64`, or the + `-generic`/`-nvidia` metas on Ubuntu/DGX): `--with-new-pkgs` PULLS A NEW KERNEL. On fv-ml1 tonight, 6.12.111 came in, + NVIDIA 580.65.06 DKMS FAILED to build for it, and dpkg was left half-configured. Had it booted, there would be no + GPU driver and every seat would be down. infra-ops purged it. Check `dkms status` for `nvidia` before patching; if it + is present, hold the kernel metas and park the kernel bump. - **Proxmox hosts:** hold `proxmox-kernel-* pve-kernel-*` and the ZFS stack. If those are the only pending packages, the host counts as current; park the kernel/ZFS bump.