From 423bc946df8f4af54eb1618ceda37227aa8dde43 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 8 Oct 2026 00:42:50 -0700 Subject: [PATCH] docs(fv-ml1): patched to kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1; ZBM boot and the hand-verified driver path recorded --- persistent-memory.md | 4 ++-- servers/fv-ml1/README.md | 22 ++++++++++++++++++---- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 18681d6..6eb9037 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -123,9 +123,8 @@ _As of 2026-10-06 ~2145 PT. The newest subsection is first; older subsections ca - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable. - **Waiting on Prime:** - albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's). - - The fv-ml1 patch window: held ZFS 2.3.9 (Debian stable-security), nvidia-container-toolkit 1.20, and a new kernel only together with a newer NVIDIA driver (6.12.111 broke the 580.65 DKMS build; test-build the driver first, keep 6.12.74 bootable). - The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and `zpool replace` line (the 2084-media-error drive is now `/dev/nvme6`, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the `parakeet` rollback). - - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). Docker engine bumps held on nh3-docker and esh-docker-vm. + - Push Booth commit `ce095c1`. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed). - ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. - **Scheduled / unchecked:** the 10-05 0815 + 0900 infra-hermes jobs (sfsrv-ana `pre-pbs4` snapshot delete; drop `pve/root-pre-pve9` on pfi-pve + esh-pve) left no ops-log lines; ask infra-hermes or read the hosts. esh-nas-pve `nvme/ROOT@pre-pve9` destroy ~10-10 (ours). 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never `zpool upgrade` any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran. - **Worldtree memory gate:** every FAIL since 09-30 (0930T0809, 1004, 1007) is ONE fixture, `u-negative-ephemeral` FILED in one run — the #416 ephemeral residual, ~3 of 42 fixture-runs since bbcc1788 (~1 batch in 5 FAILs on it), NOT a recall regression; recall (filing/user median 0.83–0.92, opener/cross_session/inline_core 100%) has held throughout. Negative is a structural class, so one run's leak fails the aggregate (`_run_structural_failed`); filing alone is judged by median. 10-07: infra-hermes told to classify FAILs by failing row (msg to its skill's daily-duty section); worldtree-dev told #416 is not closed (pull-only, unread 4 days) — reopen is theirs/Prime's. @@ -184,6 +183,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md` - `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md` +- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 → kernel 6.12.111, ZFS 2.3.9, NVIDIA 580.178.04, toolkit 1.20.1 (one reboot, 13/13 seats healthy in ~7 min, every alias answered); Docker 29.4.1 → 29.8.2 on esh-docker-vm and nh3-docker (post office ~25 s blip), all containers back.** ⚠ fv-ml1 boots ZFSBootMenu, not GRUB; trixie's apt rejects NVIDIA's debian12 repo key (SHA1 binding), so driver debs are gpgv/sha256-verified by hand. Holds re-applied on all three. Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008`, destroy ~10-15. SVOS post office is running; its inbox grant is a card for Prime. → `servers/fv-ml1/README.md` - `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11) at svos-dev's ask (Prime's 2240 decision).** Live: app-knowledge (Albok token per A111), plus app-terminal, degraded by design (no seat on the box), on its own origin `svos-terminal.phasefinal.com` (no Basic auth, mesh-only). app-postoffice is declared and stopped (⚠ the bridge handle is Prime's to declare); Miranda's packages are only staged, for U13. Acceptance: 17/17 HTTPS checks and the Chromium phone/desk checks passed. ⚠ The interface fails once on thaw (it starts before the kernel listens); the restart recovers it, but the hook's single status check can race it. Reported to svos-dev. → `servers/nh3-svos/README.md` - `[2026-10-07]` **Gateway: glm-5.3 added; gen-frontier / gen-frontier-reasoning moved 5.2 → 5.3 (Prime); every GLM 5.2/5.3 entry advertises 1,048,576 in / 131,072 out with max_tokens 131,072 as the default.** 5.3 cannot turn reasoning off, so the gen-frontier pair is now effort low / high (measured ~260 vs ~1,490 reasoning tokens). ⚠ z.ai's `thinking: disabled` is light thinking, not zero, on 5.1/5.2/5.3 alike (220–340 reasoning tokens on a hard prompt). Restarted 1725 (61 s), held 2 h for albok-dev's load A/B. The gate script's first two runs could never pass: `grep -c` exits 1 on a zero count. Images: glm-5.x is text only (z.ai rejects image parts); glm-4.6v and glm-5.3-flash read images on our plan, but neither is on the gateway. → `stacks/litellm/conf/config.yaml` - `[2026-10-07]` **Hermes launchers repaired (Miranda relay 8462):** svos-dev's scratch probe had rewritten `~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp}` to exec a /tmp scratch Python. Repointed to `~/.hermes/tools/python-3.14.7…`; the gateway was not restarted. ⚠ Don't run `git` in `~/.hermes/hermes-agent`: a `status` there looped on auto-gc. diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md index 75ba81b..70f1374 100644 --- a/servers/fv-ml1/README.md +++ b/servers/fv-ml1/README.md @@ -37,10 +37,24 @@ Primary AI inference host for PFI. Move script: `scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh`. ⚠ Lesson from run 1, which aborted on its own guard: a live rsync of `/var/lib/docker` without `-x` walks INTO mounted `overlay2/*/merged` views and copies whole container root filesystems. -- ⚠ **Held packages, parked for Prime (2026-10-05):** the ZFS stack (2.3.2 → 2.3.9; root is on ZFS) and - `nvidia-container-toolkit` 1.19 → 1.20. **The kernel metas `linux-image-amd64`/`linux-headers-amd64` were REMOVED**: - 6.12.111 came in with the patch run, NVIDIA 580.65.06 DKMS failed to build for it, and it was purged (still - booting 6.12.74). Reinstall the metas only together with an NVIDIA driver that builds on the new kernel. +- **Patched 2026-10-08 0025–0042 (Prime's go):** kernel **6.12.111** (metas reinstalled; 6.12.41/.73 purged; 6.12.74 + kept as the fallback, selectable in ZFSBootMenu), ZFS **2.3.9** (kmod + userland), NVIDIA **580.178.04** (from + 580.65.06; same R580 branch, CUDA 13.0), nvidia-container-toolkit **1.20.1**. One reboot: down 0030:42, ssh back + 0034:23, all 13 containers healthy at 0041:39. Every fv-ml1 gateway alias answered; the STT round-trip passed. + - ⚠ **Boot is ZFSBootMenu** (EFI `\EFI\ZBM\VMLINUZ.EFI`, backup entry too), not GRUB: there is no + `/etc/default/grub`. ZBM boots the newest kernel in `/boot` (`org.zfsbootmenu:kernel` is `-`). To fall back, + pick the kernel or a snapshot in the ZBM menu over SOL or iKVM. BMC credentials are in the vault as + `ana-ml2/bmc-infra-ops` (user `infra-ops`, OPERATOR); SOL is enabled; ipmitool is on nh3-dev. + - ⚠ **Where the driver comes from:** the installed apt source is the CUDA *local* repo (`cuda-debian12-13-0-local`, + 580.65.06 only). NVIDIA's *network* debian12 repo has the R580 updates, but **trixie's apt (sqv) rejects its + signing key** (SHA1 binding signature, refused since 2026-02-01), and its debian13 repo carries only 590+. So + 580.178.04 was installed from 45 `.deb`s fetched directly: InRelease checked with `gpgv` against NVIDIA's + keyring → Packages.gz sha256 → each deb's sha256 (`~infra-ops/maint-20261008/fetch-nvidia.py`, amd64/all only, + the proprietary `nvidia-kernel-dkms` excluded because this host runs the open modules). The network repo is + **not** left configured, so `apt update` stays clean. Do the same for the next driver bump. + - Holds re-applied afterwards: kernel metas, the ZFS stack, nvidia-container-toolkit. Bump them deliberately, + checking `dkms status` for the new kernel before any reboot. + - Rollback snapshot `zroot/ROOT/debian@pre-patch-20261008` (1.3 G); destroy after ~10-15 if nothing regressed. ## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`