fix(intern-decision): roll the decision seat back to Intern-Decision-4B; h2o staged with inline-images-only
albok-dev withdrew the judge sign-off on fleet-scale evidence and brokkr concurred: over 10,841 texts judged by both models, h2o flagged 242 adult at 0.5 against INC's 88, and no threshold reproduces INC's set. rollback-to-inc.sh ran 14:03:28, INC healthy 14:03:56, /decide/shared smoke 200 x3. compose.yaml is INC again; the h2o compose is kept as compose.h2o.yaml with switch-to-h2o.sh (GPU 1 free-memory gate, falls back to INC). It now carries Prime's inline-images-only setting: LIGHTNING_IMAGE_URLS=off on the front (verified live: an https image URL is refused before any fetch, inline images still answer) and vLLM --allowed-media-domains media.invalid as a backstop. rollback-to-inc.sh no longer leaves timestamped compose copies behind.
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-12 ~1320 PT (decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||||
_Last updated: 2026-10-12 ~1410 PT (decision seat: h2o 1311–1403, rolled back to INC, h2o staged; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
|
||||
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
|
||||
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
|
||||
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
|
||||
| 1 | intern-decision = **h2o-lightning-4b** since 2026-10-12 | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
|
||||
| 1 | intern-decision = **INC** (h2o 1311–1403 on 10-12, rolled back; staged) | 32,768/call | cap 14.4 GiB, peak 15,220 (rest 8,812) | — | — |
|
||||
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
|
||||
|
||||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
|
||||
@@ -166,11 +166,12 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
|
||||
### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12; INC 09-30 → 10-12)
|
||||
### The decision seat `intern-decision` :8033 (Intern-Decision-4B; h2o staged)
|
||||
|
||||
- **LIVE: Intern-Decision-4B** (`intern-decision-serve:0.1.3`), again since 2026-10-12 1403. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- **h2o-lightning-4b is STAGED** (`stacks/intern-decision/compose.h2o.yaml` + `switch-to-h2o.sh`, with a GPU 1 free-memory gate; weights `/tank/aimodels/h2o-lightning-4b-b43f759b`; front `services/h2o-decision-front`). Inline images only (`LIGHTNING_IMAGE_URLS=off`, verified; vLLM `--allowed-media-domains media.invalid` backstop NOT yet exercised). **It returns only after albok-dev re-fits the judge for it** (a 4.4 contract revision on an eval seat). ⚠ Its seat headroom under 4-way 32k load was 683 MiB.
|
||||
- **Open:** label ~50 real Wyrd/Cicada turns before trusting INC in production; its card makes no contamination claim.
|
||||
|
||||
- **LIVE: h2o-lightning-4b @ b43f759b** on vLLM 0.30.0 (KV byte-pinned) behind `services/h2o-decision-front` (32k refuse-not-trim admission, bearer, `/decide` + `/decide/shared` translated onto `/v1/systemone`). Same URL, port, token and stack name. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Rollback to INC: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` on fv-ml1. Everything else is in `stacks/intern-decision/README.md`.
|
||||
- ⚠ **Seat headroom under 4-way 32k load is 683 MiB** (peak 16,164 against a 16,847 budget, infra-hermes 10-12). Re-measure before anything else lands on GPU 1.
|
||||
- INC history (semif replacement, Triton warm-up cache): → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`. Its open item (label ~50 real Wyrd/Cicada turns) is moot with INC retired.
|
||||
|
||||
### nh3-pve + nh3-ml1 (2026-09-25/26)
|
||||
|
||||
@@ -179,7 +180,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-12]` **Decision seat swapped to h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 on the GPU 3 eval seat). INC down 130818, h2o healthy on GPU 1 :8033 at 131127; eval seat `stacks/h2o-eval` down 1319.** Post-checks: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q@32k warm 2.0–2.16 s; 4×16q×32k seat peak 16,164 MiB against a 16,847 budget (infra-hermes). The swap was held at 1234 because the seat's only traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10) and h2o's shim has no /decide, hence the front's translator. **Open:** albok-dev's hand re-judge walk report (started 1316, thread `01M4RVMXJQF1KXXQB6KJ34AYYX`); rollback = `rollback-to-inc.sh`. Brokkr thread `01M4RSFC9J2R0J07EEBD4N04W3`.
|
||||
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok re-stamps ~260 objects back. Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok).
|
||||
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||||
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime's call, pending:** A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against.
|
||||
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||||
|
||||
Reference in New Issue
Block a user