diff --git a/persistent-memory.md b/persistent-memory.md index dda7f77..767a836 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -101,6 +101,8 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-06-19:_ +- **gateway-chat web chat is now a PERSISTENT container** (ana-docker `:8091`, `stacks/gateway-chat/`, nginx:alpine, homepage tile) — the model-smoking UI; auto-discovers `/v1/models` (new gateway models just appear) + image-upload for vision. **litellm task-aliases live:** `summarizer`/`classifier` → granite-4.1-8b, `summarizer-large` → gen/qwopus (`gen-nt`/`gen-reasoning-nt` were added then removed — redundant with the `strip_empty_tools` hook). **pi on nh3-dev wired to `gen`** (`~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`; vision-capable, thinking off/on). The litellm `*` wildcard still points at decommissioned llama-swap (see Tried-and-abandoned) — a cleanup that'd make stale model names fail legibly. + - **Mistral Small 4 heretic is DISPLACED from GPU 0 (staged-down) — GPU 0 is now the `gen` model (Qwopus).** The heretic abliterated NVFP4 (`stacks/mistral-small-4-heretic/`, built in-house, "heretic stays", build tooling `tools/mistral-small4-nvfp4/`, byte-equiv to official) was downed 2026-06-19 to give GPU 0 to the gen swap. **CONSEQUENCE (operator-acknowledged, per the litellm ⚠️ note): the Worldtree demo+personal `character` backend — bound to `mistral-small-4` — is DARK until repointed.** To restore: down Qwopus + `up` the heretic stack (or serve it elsewhere). Prior doc-drift (official-stack README/comments saying "official NVFP4") now moot for GPU 0. (dd3a5c9, f566f61) - **irv-ml1 VRAM consolidated — ComfyUI owns the full 48 GB A6000.** Pinned comfyui `NVIDIA_VISIBLE_DEVICES=1`; the audio/TTS zoo (chatterbox, parakeet live; vibevoice, yt-voice-clipper config-pinned; kokoro already there) moved to the 3090; downed dia2-2b (17-day stale), ace-step, csm-expressiva. comfy-dev torch-pin applied (`DISABLE_UPGRADES=true` @ torch 2.12.1, SageAttention rebuilt + matched). **WATCH:** the 3090 has ~18.7 GB free for the audio zoo — heavy *concurrent* on-demand audio could pressure it; vibevoice deploys from `/worktank/vibevoice/build` (pre-existing repo-vs-deploy drift). (a8550ad; auto-memory `reference_irv_ml1_comfyui_mmartial`) @@ -216,6 +218,12 @@ _As of 2026-06-19:_ ## Recent decisions +- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed; duplicate-entry aliases per convention). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (a dummy `noop` tool + `tool_choice:none` baked in so a non-empty tools array always reaches vLLM) — operator reversed; redundant with the global `strip_empty_tools` hook. Live-added the new aliases via `/model/new` (no bounce), then finalized them config-sourced on an operator-authorized bounce (which also dropped the lingering gen-nt). (ef45f6d, 2e5ab72, 75bd4c3) + +- `[2026-06-19]` **gateway-chat: model-smoking web chat enhanced + stood up as a PERSISTENT container.** Added `/v1/models` auto-discovery + image-upload (vision smoke) to `tools/gateway-chat.html`; deployed as a static-serve stack (`stacks/gateway-chat/`, nginx:alpine, `http://10.250.50.70:8091` on ana-docker, homepage tile). `conf/index.html` mirrors `tools/gateway-chat.html` (re-sync one-liner in the stack README). (4c40b9f, 740bcae) + +- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — added `gen`/`gen-reasoning` (vision-capable, thinking off/on, 262K ctx) to `~/.pi/agent/models.json` (litellm provider) + a `gen` launcher (`~/.local/bin/gen`) mirroring `glm`. Verified end-to-end (`gen -p` → `PI_GEN_OK`). Local nh3-dev box config, NOT a repo artifact (like the existing `glm` launcher). (untracked by operator choice — local box wiring) + - `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** OpenYourMind/Qwopus3.5-122B-A10B-Kimi-K2.6-destilled-abliterated-NVFP4 on ana-ml2 GPU 0, served-name `qwen3.5-122-a10b` (gen/gen-reasoning/qwen-large route unchanged). Replaced the bjk110 text-only qwen3.5-122b (which had replaced mistral-small-4 earlier same-day). **KEY FINDING: the STABLE vLLM image + fp8 KV reaches the full 262144 — nightly+turboquant-4bit-KV was UNNECESSARY** (the hybrid SSM+attn KV pool is small; 11GB fp8 = 870k tokens = 3.32x concurrency @256K). graphs ON → 92.7 tok/s warm; util 0.95 + `expandable_segments` (0.96 OOMs the FusedMoE workspace); text+image+video + tool-calling qwen3_coder all verified. (20e796c, 5b06514) - `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** (operator asked; recommended against). `nvidia-smi topo -m` = `PIX` (PCIe single-bridge, **NO NVLink** — datacenter-only). TP all-reduces ~twice/layer over PCIe (~64GB/s vs NVLink ~900GB/s) → all-reduce-bound → SLOWER for models that already fit + would evict the 6 GPU-1 services. **One-model-per-card is the optimal layout for non-NVLinked cards** (zero cross-card traffic). If a single >96GB model is ever wanted, the path is PIPELINE parallelism (PP=2, 1 hop/token) + GPU-1 relocation — NOT TP. (untracked by operator choice; "keep it there") @@ -317,6 +325,10 @@ _79 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Deleting a config.yaml model by id returns `Model with id=… not found in db`; only a process restart drops it. So removing a config alias can't go live WITHOUT a bounce — it ages out on the next restart (the cleaned config just won't reload it). The INVERSE works: `/model/new` live-ADDS to the router (DB) with no bounce — but then a config copy + DB copy coexist after a restart (harmless dup; reconcile by `/model/delete`-ing the DB copy). Clean no-bounce-then-bounce add: live-add via API, delete the DB copy before the eventual bounce so config stays single-source. + +- `[2026-06-19]` **the litellm `*` wildcard still routes to DECOMMISSIONED llama-swap (`http://10.250.50.54:9292`) → a stale/typo'd model name returns a misleading `Connection error`, not `model not found`.** Bit a consumer (brokkr) calling the renamed-away `qwen-image-judge` (now `image-judge`): the orphaned name fell through `*` → dead llama-swap → connection error, reading like the judge was down — it was healthy throughout. Until the wildcard is removed or repointed, unknown/typo'd names fail illegibly. (incident 2026-06-19) + - `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace is the floor.** vLLM's FusedMoE kernel allocates a ~3.09 GB transient workspace OUTSIDE its `gpu-memory-utilization` budget, into free VRAM, during graph capture + inference. util 0.96 OOM'd by 0.1 GB on it (`tried to allocate 3.09 GiB, 2.99 free`), worsened by 4.2 GB of PyTorch reserved-but-unallocated fragmentation. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` (reclaims the fragmentation) + leave ~3.2 GB free (util ≤ ~0.95 on a tight-fit MoE). The workspace size is fixed; "fill to 0" is physically impossible for MoE. - `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold single read = 24.8 tok/s; warm steady-state = 92.7 (3 runs identical). A cold read undersells decode ~3–4× — the first gen pays graph-replay/JIT warmup. Measure run 2+ over a ≥256-token output.