From 23371954f82c0e256a1cf47d21aa49a824473a27 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 5 Oct 2026 02:43:19 -0700 Subject: [PATCH] docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory --- persistent-memory.md | 2 ++ servers/fv-ml1/README.md | 17 +++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 492dcc0..01d9225 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -300,6 +300,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions +- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md` +- `[2026-10-05]` **Fleet guest patching (park 100) handed to infra-hermes 0243** — plan + holds in `docs/runbooks/fleet-guest-patch-2026-10-05.md`; irv-ml1 OFF-LIMITS (Prime); ana-nas parked; report due by 0700. - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md` - `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md` - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml` diff --git a/servers/fv-ml1/README.md b/servers/fv-ml1/README.md index b816817..7cbfbd8 100644 --- a/servers/fv-ml1/README.md +++ b/servers/fv-ml1/README.md @@ -27,7 +27,20 @@ Primary AI inference host for PFI. moving its data-root to tank needs a model-seat outage window (Prime's call). sudo session recordings are archived weekly by `services/sudo-io-archive/`. - **OS:** Debian 13 (trixie), kernel 6.12.x -- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2 +- **Docker:** 29.8.2 (since 2026-10-05), runtimes: runc (default), nvidia, io.containerd.runc.v2 +- ⚠ **`/var/lib/docker` is the zvol `tank/docker`** (sparse 1 TiB, ext4), not zroot, since 2026-10-05 0233 (Prime, + park 99). It is mounted by `/etc/fstab` with `nofail` after `zfs-volume-wait.service`, and + `docker.service.d/10-requires-docker-mount.conf` (`RequiresMountsFor=/var/lib/docker`) stops Docker from ever starting + on an empty root directory. It is a zvol, not a dataset, because Docker's overlay2 refuses a ZFS backing filesystem + (an unpinned daemon would pick the zfs graph driver and lose every image). Freed blocks reach the zvol only through + TRIM: `fstrim.timer` is enabled weekly (the first trim gave back 108 G). Root went from 49% to 8% used. + Move script: `scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh`. ⚠ Lesson from run 1, which aborted on its own guard: + a live rsync of `/var/lib/docker` without `-x` walks INTO mounted `overlay2/*/merged` views and copies whole container + root filesystems. +- ⚠ **Held packages, parked for Prime (2026-10-05):** the ZFS stack (2.3.2 → 2.3.9; root is on ZFS) and + `nvidia-container-toolkit` 1.19 → 1.20. **The kernel metas `linux-image-amd64`/`linux-headers-amd64` were REMOVED**: + 6.12.111 came in with the patch run, NVIDIA 580.65.06 DKMS failed to build for it, and it was purged (still + booting 6.12.74). Reinstall the metas only together with an NVIDIA driver that builds on the new kernel. ## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli` @@ -171,7 +184,7 @@ meter on the circuit. Until then, treat 1650 W as a floor. | `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) | | `/tank/aimodels/mlx/` | MLX packs (Apple-silicon formats, kept for acquisition only; nothing on this host serves them). Created 2026-09-28 for Ternary-Bonsai-2-27B | | `/tank/aimodels/.PROVENANCE.txt` | Provenance sits BESIDE each model dir, not inside it | -| `/var/lib/docker/` | Docker data (on zroot) | +| `/var/lib/docker/` | Docker data (zvol `tank/docker`, ext4, since 2026-10-05) | ## Running stacks