feat(intern-decision): decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; eval seat down
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front: 32k refuse-not-trim admission, the seat's bearer token, and /decide + /decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port, token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27. Post-checks on the live seat: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak 16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget. compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step. The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)_
|
||||
_Last updated: 2026-10-12 ~1320 PT (decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
|
||||
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
|
||||
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
|
||||
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
|
||||
| 1 | intern-decision | 32,768/call | cap 14.4 GiB, peak 15,220 | — | — |
|
||||
| 1 | intern-decision = **h2o-lightning-4b** since 2026-10-12 | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
|
||||
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
|
||||
|
||||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
|
||||
@@ -166,10 +166,11 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
|
||||
### intern-decision (replaced SemIf on 2026-09-30)
|
||||
### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12; INC 09-30 → 10-12)
|
||||
|
||||
- **LIVE 0.1.3** at `intern-decision.fv.internal:8033`: semif-compatible `/decide` plus Jev `/v1/systemone`, 32k tokens, a Triton cache volume. Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- **Open:** label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim.
|
||||
- **LIVE: h2o-lightning-4b @ b43f759b** on vLLM 0.30.0 (KV byte-pinned) behind `services/h2o-decision-front` (32k refuse-not-trim admission, bearer, `/decide` + `/decide/shared` translated onto `/v1/systemone`). Same URL, port, token and stack name. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Rollback to INC: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` on fv-ml1. Everything else is in `stacks/intern-decision/README.md`.
|
||||
- ⚠ **Seat headroom under 4-way 32k load is 683 MiB** (peak 16,164 against a 16,847 budget, infra-hermes 10-12). Re-measure before anything else lands on GPU 1.
|
||||
- INC history (semif replacement, Triton warm-up cache): → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`. Its open item (label ~50 real Wyrd/Cicada turns) is moot with INC retired.
|
||||
|
||||
### nh3-pve + nh3-ml1 (2026-09-25/26)
|
||||
|
||||
@@ -178,7 +179,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-12]` **Decision-seat swap (Prime via brokkr): h2o-lightning-4b @ b43f759b replaces Intern-Decision-4B, AFTER albok-dev re-validates Nemi's sensitivity judge on it.** The swap was held at 1234: the seat's only production traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10), and h2o's shim has no /decide. TEMPORARY eval seat `stacks/h2o-eval` up since 1300 on fv-ml1 GPU 3 :8034 (~14.4 GB; Scriberr/MIA still fit beside it), with `services/h2o-decision-front` (32k refuse-not-trim admission from infra-hermes 30f36f2, bearer, the /decide/shared translator, /health model {name, revision}); weights `/tank/aimodels/h2o-lightning-4b-b43f759b` (hashes = HF). **Next, on albok-dev's sign-off (threshold, route):** GPU 1 cutover on :8033 with the same front and byte-pinned KV (budget 16,847 MiB, peak 15,566), rollback = the INC compose; post-checks (health rev, JevBench 205, 422 probe, 16q@32k ~2 s, GPU 1 peak); then `docker compose down` h2o-eval. Thread `01M4RSFC9J2R0J07EEBD4N04W3`.
|
||||
- `[2026-10-12]` **Decision seat swapped to h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 on the GPU 3 eval seat). INC down 130818, h2o healthy on GPU 1 :8033 at 131127; eval seat `stacks/h2o-eval` down 1319.** Post-checks: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q@32k warm 2.0–2.16 s; 4×16q×32k seat peak 16,164 MiB against a 16,847 budget (infra-hermes). The swap was held at 1234 because the seat's only traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10) and h2o's shim has no /decide, hence the front's translator. **Open:** albok-dev's hand re-judge walk report (started 1316, thread `01M4RVMXJQF1KXXQB6KJ34AYYX`); rollback = `rollback-to-inc.sh`. Brokkr thread `01M4RSFC9J2R0J07EEBD4N04W3`.
|
||||
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||||
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime's call, pending:** A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against.
|
||||
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||||
|
||||
Reference in New Issue
Block a user