feat(intern-decision): decision seat back on h2o-lightning-4b under the re-fitted judge

Prime ruled the cutover ("do it", 16:40, via albok-dev) after the judge's
note question was re-fitted (albok-nemi 0.1.18). switch-to-h2o.sh ran 17:02:23,
the GPU 1 gate passed, h2o healthy 17:04:54. Checks on :8033: /health at the
pin, /decide/shared 200, 422 over the cap, inline image answers, image URL
refused by the shim, vLLM media-domain backstop refuses (400). compose.yaml is
h2o again; rollback-to-inc.sh stays the one-step way back.
This commit is contained in:
vh
2026-10-12 17:05:59 -07:00
parent 014a4fd0ec
commit 98254ca077
2 changed files with 72 additions and 66 deletions
+20 -15
View File
@@ -1,21 +1,22 @@
# intern-decision — the decision seat (fv-ml1 GPU 1, :8033)
> **LIVE: Intern-Decision-4B again, since 2026-10-12 1403 PT** (`compose.yaml`, image `intern-decision-serve:0.1.3`).
> Everything from **"The live seat: Intern-Decision-4B"** down describes it.
> **LIVE: h2o-lightning-4b since 2026-10-12 1704 PT** (Prime 1640, via albok-dev). `compose.yaml` = `compose.h2o.yaml`
> (`switch-to-h2o.sh` copies one over the other). Nemi's judge runs the **re-fitted v2 note question**
> (albok-nemi 0.1.18, contract 4.4 rev 1.15) at concurrency 4.
>
> **h2o-lightning-4b is STAGED, not live** (`compose.h2o.yaml`, `switch-to-h2o.sh`). It served 1311–1403 on 2026-10-12
> and was rolled back on albok-dev's word, with brokkr concurring. Prime's plan had made the swap conditional on the
> judge's sign-off, and albok-dev withdrew it on fleet-scale evidence:
> - **adult:** over 10,841 texts judged by both models (same text, prompt and policy), h2o flagged 242 at 0.5 against
> INC's 88. That is all 88 of INC's plus 154 more, most of them at 0.80–0.90 where INC put them at 0.30–0.47. No
> threshold reproduces INC's set (0.70 still flags 166 and loses 6 of INC's).
> - **personal:** INC flagged 5 and h2o 11, with only 2 in common.
> **What happened on 2026-10-12:**
> 1. **1311–1403, h2o's first stint, rolled back.** Under the judge's v1 question, h2o flagged 242 texts adult
> against INC's 88, over 10,841 identical texts.
> 2. **1428–1619, the investigation on GPU 3.** The v1 question asked two things at once: INC answered it
> inconsistently and h2o answered it as written. The shim's floor was ruled out, with identical verdicts on and
> off. Prime ruled "text itself".
> 3. **Under v2 the two seats agree:** work notes INC 0/22, h2o 1/22 at 0.52; nulls 0/18 on both. fleet/memory adult
> went 86 → 0 on INC.
> 4. **Rollback:** `rollback-to-inc.sh`, and set the judge's concurrency back to 1 (see "Switching").
>
> h2o comes back only after albok re-fits the judge for it (prompt and threshold against a labelled set large
> enough to resolve a 2× change, as a 4.4 contract revision, on an eval seat). The h2o sections below describe the
> staged seat, and its numbers are the 1311 cutover's.
> The section headed "The live seat: Intern-Decision-4B" further down describes INC, the rollback target.
### The staged h2o seat (`compose.h2o.yaml`)
### The h2o seat (`compose.h2o.yaml`, live since 1704)
| | |
|---|---|
@@ -97,6 +98,10 @@ Repeat that check on this seat's loopback port (`:18299`) on the next switch. IN
### Switching between the two
The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the
pin, with floor 0.801; `/decide/shared` 200; a 422 over the cap; inline red → red at 0.984; an https image URL refused
by the shim; the vLLM backstop refusing example.com (400); GPU 1 resting at 14,430 MiB.
Both run ON fv-ml1 and share one `.env`.
- **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has
16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies
@@ -106,7 +111,7 @@ Both run ON fv-ml1 and share one `.env`.
over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its
`intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356.
After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is INC today).
After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is h2o since 1704).
⚠ **Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat.** INC scores on one inference
thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's `nemi.yaml`, since
@@ -115,7 +120,7 @@ h2o, vLLM batches, so it goes back to 4.
---
# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and again since 1403)
# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and 1403–1702; now the rollback target)
> **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback.