docs(fv-ml1): patched to kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1; ZBM boot and the hand-verified driver path recorded

servers/fv-ml1/README.md

@@ -37,10 +37,24 @@ Primary AI inference host for PFI.
   Move script: `scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh`. ⚠ Lesson from run 1, which aborted on its own guard:
   a live rsync of `/var/lib/docker` without `-x` walks INTO mounted `overlay2/*/merged` views and copies whole container
   root filesystems.
-- ⚠ **Held packages, parked for Prime (2026-10-05):** the ZFS stack (2.3.2 → 2.3.9; root is on ZFS) and
-  `nvidia-container-toolkit` 1.19 → 1.20. **The kernel metas `linux-image-amd64`/`linux-headers-amd64` were REMOVED**:
-  6.12.111 came in with the patch run, NVIDIA 580.65.06 DKMS failed to build for it, and it was purged (still
-  booting 6.12.74). Reinstall the metas only together with an NVIDIA driver that builds on the new kernel.
+- **Patched 2026-10-08 0025–0042 (Prime's go):** kernel **6.12.111** (metas reinstalled; 6.12.41/.73 purged; 6.12.74
+  kept as the fallback, selectable in ZFSBootMenu), ZFS **2.3.9** (kmod + userland), NVIDIA **580.178.04** (from
+  580.65.06; same R580 branch, CUDA 13.0), nvidia-container-toolkit **1.20.1**. One reboot: down 0030:42, ssh back
+  0034:23, all 13 containers healthy at 0041:39. Every fv-ml1 gateway alias answered; the STT round-trip passed.
+  - ⚠ **Boot is ZFSBootMenu** (EFI `\EFI\ZBM\VMLINUZ.EFI`, backup entry too), not GRUB: there is no
+    `/etc/default/grub`. ZBM boots the newest kernel in `/boot` (`org.zfsbootmenu:kernel` is `-`). To fall back,
+    pick the kernel or a snapshot in the ZBM menu over SOL or iKVM. BMC credentials are in the vault as
+    `ana-ml2/bmc-infra-ops` (user `infra-ops`, OPERATOR); SOL is enabled; ipmitool is on nh3-dev.
+  - ⚠ **Where the driver comes from:** the installed apt source is the CUDA *local* repo (`cuda-debian12-13-0-local`,
+    580.65.06 only). NVIDIA's *network* debian12 repo has the R580 updates, but **trixie's apt (sqv) rejects its
+    signing key** (SHA1 binding signature, refused since 2026-02-01), and its debian13 repo carries only 590+. So
+    580.178.04 was installed from 45 `.deb`s fetched directly: InRelease checked with `gpgv` against NVIDIA's
+    keyring → Packages.gz sha256 → each deb's sha256 (`~infra-ops/maint-20261008/fetch-nvidia.py`, amd64/all only,
+    the proprietary `nvidia-kernel-dkms` excluded because this host runs the open modules). The network repo is
+    **not** left configured, so `apt update` stays clean. Do the same for the next driver bump.
+  - Holds re-applied afterwards: kernel metas, the ZFS stack, nvidia-container-toolkit. Bump them deliberately,
+    checking `dkms status` for the new kernel before any reboot.
+  - Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008` (1.3 G); destroy after ~10-15 if nothing regressed.
 
 ## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`