docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory

persistent-memory.md

@@ -300,6 +300,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
 
 ## Recent decisions
 
+- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
+- `[2026-10-05]` **Fleet guest patching (park 100) handed to infra-hermes 0243** — plan + holds in `docs/runbooks/fleet-guest-patch-2026-10-05.md`; irv-ml1 OFF-LIMITS (Prime); ana-nas parked; report due by 0700.
 - `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
 - `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
 - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`

servers/fv-ml1/README.md

@@ -27,7 +27,20 @@ Primary AI inference host for PFI.
   moving its data-root to tank needs a model-seat outage window (Prime's call). sudo session recordings are
   archived weekly by `services/sudo-io-archive/`.
 - **OS:** Debian 13 (trixie), kernel 6.12.x
-- **Docker:** 29.3.1, runtimes: runc (default), nvidia, io.containerd.runc.v2
+- **Docker:** 29.8.2 (since 2026-10-05), runtimes: runc (default), nvidia, io.containerd.runc.v2
+- ⚠ **`/var/lib/docker` is the zvol `tank/docker`** (sparse 1 TiB, ext4), not zroot, since 2026-10-05 0233 (Prime,
+  park 99). It is mounted by `/etc/fstab` with `nofail` after `zfs-volume-wait.service`, and
+  `docker.service.d/10-requires-docker-mount.conf` (`RequiresMountsFor=/var/lib/docker`) stops Docker from ever starting
+  on an empty root directory. It is a zvol, not a dataset, because Docker's overlay2 refuses a ZFS backing filesystem
+  (an unpinned daemon would pick the zfs graph driver and lose every image). Freed blocks reach the zvol only through
+  TRIM: `fstrim.timer` is enabled weekly (the first trim gave back 108 G). Root went from 49% to 8% used.
+  Move script: `scripts/oneoff/fv-ml1-docker-to-tank-20261005.sh`. ⚠ Lesson from run 1, which aborted on its own guard:
+  a live rsync of `/var/lib/docker` without `-x` walks INTO mounted `overlay2/*/merged` views and copies whole container
+  root filesystems.
+- ⚠ **Held packages, parked for Prime (2026-10-05):** the ZFS stack (2.3.2 → 2.3.9; root is on ZFS) and
+  `nvidia-container-toolkit` 1.19 → 1.20. **The kernel metas `linux-image-amd64`/`linux-headers-amd64` were REMOVED**:
+  6.12.111 came in with the patch run, NVIDIA 580.65.06 DKMS failed to build for it, and it was purged (still
+  booting 6.12.74). Reinstall the metas only together with an NVIDIA driver that builds on the new kernel.
 
 ## NVMe drive inventory (`tank`, raidz2-0) — read 2026-09-09 via `nvme-cli`
 
@@ -171,7 +184,7 @@ meter on the circuit. Until then, treat 1650 W as a floor.
 | `/tank/aimodels/llm/` | Legacy GGUF models (790 GB, referenced by llama-swap as `/models/`) |
 | `/tank/aimodels/mlx/` | MLX packs (Apple-silicon formats, kept for acquisition only; nothing on this host serves them). Created 2026-09-28 for Ternary-Bonsai-2-27B |
 | `/tank/aimodels/<model>.PROVENANCE.txt` | Provenance sits BESIDE each model dir, not inside it |
-| `/var/lib/docker/` | Docker data (on zroot) |
+| `/var/lib/docker/` | Docker data (zvol `tank/docker`, ext4, since 2026-10-05) |
 
 ## Running stacks