memory: snapshot — 2026-06-25 (althing v0.17 / nh3-extdev model-B mesh + zellij web pilot + Worldtree #314/#322/#317 config arc; rewrite stale althing Tools-row to v0.17; archive 10 pre-session 2026-06-20 entries)

This commit is contained in:
vh
2026-06-25 00:14:02 -07:00
parent 13bfa4a621
commit 30c883be5d
2 changed files with 123 additions and 137 deletions
+30
View File
@@ -585,6 +585,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab)
_Archived 2026-06-21._
- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96)
_Archived 2026-06-25._
- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c)
_Archived 2026-06-25._
- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c)
_Archived 2026-06-25._
- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85)
_Archived 2026-06-25._
- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context.
_Archived 2026-06-25._
## Tried and abandoned (archived)
- `[2026-04-30]` task-board workflow with
@@ -1137,3 +1152,18 @@ _76 older entries archived to archival-memory.md._
- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first.
_Archived 2026-06-21._
- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur).
_Archived 2026-06-25._
- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free.
_Archived 2026-06-25._
- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching.
_Archived 2026-06-25._
- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end.
_Archived 2026-06-25._
- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_<name>-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder <name> -af`, or `docker volume rm` orphaned ones (builder already removed).
_Archived 2026-06-25._