vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot refresh — breeze settled, util does not predict residency, handoff defect

Browse Source 
Incremental over 653f7fb, covering the three turns since.

breeze-tts stays on irv-ml1 and the TTS-stack move is parked at id 75. Adds the
full fv-ml1 per-seat residency table and the shuffle arithmetic behind the
recommendation: every arrangement that makes real room on GPU 1 spends the GPU 3
reserve anyway, so if breeze ever moves it should go straight to GPU 3 rather than
evacuating three working seats for the same cost plus a migration.

The finding worth keeping is that --gpu-memory-utilization does not predict resident
VRAM, and is wrong in both directions: cyberprev at util 0.40 holds 8 GB more than
its fraction, gen-small at 0.48 holds 10 GB less. An hour earlier in the session I
reasoned about placement from those fractions and would have been 8 GB out. Read
nvidia-smi --query-compute-apps instead.

Also records a characterized defect in this skill's own handoff generator. Across
two runs in one session it converted operator-deferred items into an imperative
next-steps list and twice invited the next session to commit files marked as
predating it. It fails in the blind spot of the documented fail-loud posture: the
output is structurally valid and exits 0 while inverting the intent. That matters
more than an ordinary bad summary because the handoff is the one artifact a fresh
context reads as instruction. Both runs were corrected in-session; not yet reported
to the skill's owner.
This commit is contained in:
vh
2026-09-15 10:08:57 -07:00
parent 7a33bd9f09
commit 9d36c74572
3 changed files with 75 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+10 -6
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-15 ~09:30 PT (Parakeet STT live on fv-ml1 GPU 0 + LiteLLM `ext-stt`; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency bug fixed. Nothing blocked, nothing mid-flight.)_
_Last updated: 2026-09-15 ~09:45 PT (Parakeet STT live on fv-ml1 GPU 0; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency fixed; breeze stays on irv-ml1, TTS-stack move parked at id 75. Nothing blocked, nothing mid-flight.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,24 +115,28 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight
_As of 2026-09-15 ~09:30 PT._
_As of 2026-09-15 ~09:45 PT._
### Nothing is blocked and nothing is mid-flight. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
### Nothing is in flight and nothing is blocked. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
**Closed this session:**
1. **Parakeet STT live** — fv-ml1 **GPU 0** (not GPU 3), port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is now a deliberate reserve at 2 MiB.**
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8 tools, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator: "i dont want the tools disabled everywhere"); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
1. **Parakeet STT live** — fv-ml1 **GPU 0**, port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is a deliberate reserve at 2 MiB.**
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator ruling); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat.
4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2).
5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy.
6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze).
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
**Settled at the end of the session:** breeze-tts **stays on irv-ml1**; moving it plus `bragi` and `tts-gateway` to fv-ml1 is **parked at id 75**, triggered on evacuating embed/rerank/reward. Full fv-ml1 per-seat residency measured — see the breeze detail file.
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47), TTS-stack move (park id 75). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed.
## Recent decisions
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
- `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`