memory: snapshot refresh — breeze settled, util does not predict residency, handoff defect
Incremental over 653f7fb, covering the three turns since.
breeze-tts stays on irv-ml1 and the TTS-stack move is parked at id 75. Adds the
full fv-ml1 per-seat residency table and the shuffle arithmetic behind the
recommendation: every arrangement that makes real room on GPU 1 spends the GPU 3
reserve anyway, so if breeze ever moves it should go straight to GPU 3 rather than
evacuating three working seats for the same cost plus a migration.
The finding worth keeping is that --gpu-memory-utilization does not predict resident
VRAM, and is wrong in both directions: cyberprev at util 0.40 holds 8 GB more than
its fraction, gen-small at 0.48 holds 10 GB less. An hour earlier in the session I
reasoned about placement from those fractions and would have been 8 GB out. Read
nvidia-smi --query-compute-apps instead.
Also records a characterized defect in this skill's own handoff generator. Across
two runs in one session it converted operator-deferred items into an imperative
next-steps list and twice invited the next session to commit files marked as
predating it. It fails in the blind spot of the documented fail-loud posture: the
output is structurally valid and exits 0 while inverting the intent. That matters
more than an ordinary bad summary because the handoff is the one artifact a fresh
context reads as instruction. Both runs were corrected in-session; not yet reported
to the skill's owner.
This commit is contained in:
@@ -73,6 +73,43 @@ GPU 1 is the constrained card at **0.975 committed / 4,336 MiB free**; GPU 0 has
|
||||
and `tts-gateway` reach each other by name only through `extra_hosts` pins. Those pins
|
||||
travel with them and need re-pointing at the new host.
|
||||
|
||||
## fv-ml1 per-seat residency, measured 2026-09-15 (97,887 MiB per card)
|
||||
|
||||
| GPU | seats (resident MiB) | resident | free |
|
||||
|---|---|---|---|
|
||||
| **0** | cyberprev 47,124 · gen-small 36,942 · parakeet 1,178 | 85,244 | **11,982** |
|
||||
| **1** | meromero 44,056 · erp 27,748 · reward 9,192 · coder 6,292 · embed 3,472 · rerank 2,114 | 92,874 | **4,336** |
|
||||
| **2** | flash-next 95,344 | 95,344 | 1,767 |
|
||||
| **3** | — (deliberate reserve) | 2 | **97,247** |
|
||||
|
||||
## ⚠⚠ `--gpu-memory-utilization` DOES NOT PREDICT RESIDENCY — measure, don't compute
|
||||
|
||||
Capacity planning off the configured util fractions is wrong **in both directions**:
|
||||
|
||||
vllm-cyberprev util 0.40 -> expect ~39,155 MiB ACTUAL 47,124 (+8.0 GB OVER)
|
||||
vllm-gen-small util 0.48 -> expect ~46,986 MiB ACTUAL 36,942 (-10.0 GB UNDER)
|
||||
|
||||
An hour earlier in this session I reasoned about placement from the util fractions and
|
||||
would have been ~8 GB wrong on one card. ⭐ **`util` is a ceiling the engine may not
|
||||
reach and evidently can exceed** (weights, activations and non-KV allocations sit
|
||||
outside the KV budget the fraction governs). Always read
|
||||
`nvidia-smi --query-compute-apps` for the real number.
|
||||
|
||||
## Does a shuffle make room for breeze? — worked through
|
||||
|
||||
- **GPU 0 today: YES, no shuffle.** 11,982 free vs breeze 10,316 → **1.7 GB margin**,
|
||||
against a tenant that grew +1.1 GB in an hour, on the live chat path. Fits; thin.
|
||||
- **GPU 1 today: NO.** 4,336 free.
|
||||
- **Evacuate embed+rerank+reward (14,778 MiB) → GPU 1 has 19,114 free**, breeze fits
|
||||
with ~8.8 GB headroom. **But there is nowhere to put those three except GPU 3** —
|
||||
14,778 does not fit GPU 0's 11,982, and GPU 2 has 1,767.
|
||||
- **Partial shuffle fails:** embed+rerank (5,586) to GPU 0 leaves GPU 1 at 9,922 free —
|
||||
**~400 MB short** of breeze, and breeze is growing.
|
||||
|
||||
⭐ **Conclusion: every arrangement that makes real room on GPU 1 spends the GPU 3
|
||||
reserve anyway.** So if breeze ever goes to FV, put it **straight on GPU 3** rather than
|
||||
evacuating three working seats to achieve the same cost plus a migration.
|
||||
|
||||
## If consolidation onto FV is the goal anyway
|
||||
|
||||
**GPU 3** fits it comfortably (97,247 MiB free) and has none of the margin problem — but
|
||||
|
||||
@@ -111,6 +111,34 @@ possible answer, and it was the right one here.
|
||||
problem; a refusal naming permission rules or auto-mode is a harness one. Different
|
||||
fixes, and neither applies when nothing failed.
|
||||
|
||||
## ⚠ CHARACTERIZED DEFECT: `/snapshot`'s handoff generator turns deferred items into orders
|
||||
|
||||
**n=2, same session, reproducible.** `snapshot_handoff.py` (gen-small) reliably converts
|
||||
"open, operator-deferred, not blocking" into an imperative **Next steps** list, and twice
|
||||
invited the next session to commit files explicitly marked as predating the session.
|
||||
|
||||
run 1: "Execute deferred operator tasks: AI-tab Dormant regrouping, nconnect=8,
|
||||
fused MoE (park id 47)" + "Commit graphify-out/… if they are ready"
|
||||
run 2: six next-steps, FIVE of them deferred/parked items presented as actions,
|
||||
+ the same commit invitation
|
||||
|
||||
⚠ **It fails silently in the skill's blind spot.** The documented failure posture is
|
||||
fail-loud-fall-back — unreachable gateway, timeout, truncation, missing section → write
|
||||
nothing, exit non-zero. **A structurally valid handoff whose content inverts the
|
||||
operator's intent passes every one of those checks** and exits 0.
|
||||
|
||||
⚠ **And this is the one artifact a fresh context inherits as instruction.** It is read
|
||||
immediately after `/clear`, before any other framing, and its Next steps read as a
|
||||
mandate. A wrong one here is not a bad summary; it is a fresh session going and doing
|
||||
belayed work.
|
||||
|
||||
**Mitigation until fixed: read the generated handoff before accepting it**, and invert
|
||||
any deferred item into an explicit *do NOT*. Both runs this session were corrected
|
||||
in-session. Owner is the `/snapshot` skill (galdrabok); not reported yet.
|
||||
|
||||
⭐ Same family as everything above — the instrument produced a plausible artifact and
|
||||
the plausibility is exactly what makes it dangerous.
|
||||
|
||||
## Related
|
||||
|
||||
`2026-09-15-talk-v10-deploy.md` (#2, and the gate built for it),
|
||||
|
||||
+10
-6
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-15 ~09:30 PT (Parakeet STT live on fv-ml1 GPU 0 + LiteLLM `ext-stt`; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency bug fixed. Nothing blocked, nothing mid-flight.)_
|
||||
_Last updated: 2026-09-15 ~09:45 PT (Parakeet STT live on fv-ml1 GPU 0; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency fixed; breeze stays on irv-ml1, TTS-stack move parked at id 75. Nothing blocked, nothing mid-flight.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,24 +115,28 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-15 ~09:30 PT._
|
||||
_As of 2026-09-15 ~09:45 PT._
|
||||
|
||||
### Nothing is blocked and nothing is mid-flight. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
|
||||
### Nothing is in flight and nothing is blocked. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
|
||||
|
||||
**Closed this session:**
|
||||
1. **Parakeet STT live** — fv-ml1 **GPU 0** (not GPU 3), port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is now a deliberate reserve at 2 MiB.**
|
||||
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8 tools, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator: "i dont want the tools disabled everywhere"); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
|
||||
1. **Parakeet STT live** — fv-ml1 **GPU 0**, port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is a deliberate reserve at 2 MiB.**
|
||||
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator ruling); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
|
||||
3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat.
|
||||
4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2).
|
||||
5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy.
|
||||
6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze).
|
||||
|
||||
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
|
||||
**Settled at the end of the session:** breeze-tts **stays on irv-ml1**; moving it plus `bragi` and `tts-gateway` to fv-ml1 is **parked at id 75**, triggered on evacuating embed/rerank/reward. Full fv-ml1 per-seat residency measured — see the breeze detail file.
|
||||
|
||||
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47), TTS-stack move (park id 75). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
|
||||
|
||||
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
- `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
- `[2026-09-15]` **Parakeet STT live on fv-ml1 GPU 0, behind LiteLLM `ext-stt` / `whisper-1`.** ⚠ **Placed on GPU 3 first, which was wrong — operator caught it.** A ~800 MiB seat should ride the card with the most uncommitted headroom (GPU 0, util 0.88, ~13 GB spare), not put the first fingerprint on the one pristine 96 GB card: vLLM sizes KV cache against TOTAL VRAM, so any tenant on an empty card eats a future full-size seat's profiling margin (flash-next needs 93 of 96 GiB). **GPU 3 is now a deliberate reserve at 2 MiB.** Retargeted the existing `stacks/parakeet/` (sherpa-onnx + our own FastAPI wrapper) from irv-ml1; v3 int8, 25 languages. ⚠ **ORT's CUDA EP compiles kernels lazily and the first decode on sm_120 took 45.7 s** — every later call ~0.5 s; a startup warmup in `app.py` now absorbs it, so the first real request is 0.65 s instead of a 45 s hang that no client would wait through. GPU use was **verified by a process on GPU 3 (922 MiB), not by the `provider=cuda` log line**, because ORT falls back to CPU silently and still returns correct text. Silence → `""` (null control), known sentence → near-exact (positive control). → `persistent-memory.d/2026-09-15-parakeet-stt-fv-ml1.md`
|
||||
|
||||
Reference in New Issue
Block a user