vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(fv-ml1): docker on tank zvol, held packages, kernel metas removed; memory

Browse Source
This commit is contained in:
vh
2026-10-05 02:43:19 -07:00
parent fde50b7ca7
commit 23371954f8
2 changed files with 17 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+2
View File
@@ -300,6 +300,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
## Recent decisions
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
- `[2026-10-05]` **Fleet guest patching (park 100) handed to infra-hermes 0243** — plan + holds in `docs/runbooks/fleet-guest-patch-2026-10-05.md`; irv-ml1 OFF-LIMITS (Prime); ana-nas parked; report due by 0700.
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`