vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(intern-decision): second rollback to Intern-Decision-4B; h2o staged until the judge's two margins are fixed

Browse Source 
The v2 walk on h2o (17:05-17:45) held notes at 0 adult but moved two margins
past albok-dev's pre-named rollback line: library/nonfiction adult 0 -> 79 (one
textbook chapter at 0.531 on the book profile, INC 0.438, with 77 concepts
inheriting) and fleet/memory personal 6 -> 18. rollback-to-inc.sh ran 17:47:46,
INC healthy 17:48:11. compose.yaml is INC again.
This commit is contained in:
vh
2026-10-12 17:48:43 -07:00
parent 2d85ee5a25
commit d2fb840bfa
3 changed files with 78 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+7 -8
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-10-12 ~1710 PT (decision seat: h2o 1311–1403, rolled back, judge re-fitted, h2o LIVE again 1704; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
_Last updated: 2026-10-12 ~1750 PT (decision seat: h2o tried twice, 1311–1403 and 1704–1747, rolled back both times, INC live, h2o staged; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
| 1 | intern-decision = **h2o-lightning-4b** since 10-12 1704 (INC = rollback target) | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
| 1 | intern-decision = **INC** since 10-12 1748 (h2o staged; live 1311–1403 and 1704–1747) | 32,768/call | cap 14.4 GiB, peak 15,220 (rest 8,812) | — | — |
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
@@ -166,12 +166,11 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12 1704; INC = rollback)
### The decision seat `intern-decision` :8033 (Intern-Decision-4B since 2026-10-12 1748; h2o staged)
- **LIVE: h2o-lightning-4b @ b43f759b** (`compose.h2o.yaml`, `switch-to-h2o.sh`) behind `services/h2o-decision-front`: 32k refuse-not-trim admission, bearer, `/decide/shared` translator. Inline images only (`LIGHTNING_IMAGE_URLS=off` + vLLM `--allowed-media-domains media.invalid`, both verified). Nemi's judge uses the **v2 note question** (albok-nemi 0.1.18) at **concurrency 4**.
- **Rollback:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` (INC `intern-decision-serve:0.1.3`, ~35 s), **and albok-dev sets judge concurrency to 1**. INC scores on one thread (INV-2), and 4 in flight queued into 339 timeouts on 10-12.
- ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1.
- INC history: → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`.
- **LIVE: Intern-Decision-4B** (`intern-decision-serve:0.1.3`). Nemi's judge runs the v2 note question (albok-nemi 0.1.18) at **concurrency 1** (INC scores on one thread, INV-2). Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
- **h2o-lightning-4b is STAGED** (`compose.h2o.yaml` + `switch-to-h2o.sh` with a GPU 1 free-memory gate; weights `/tank/aimodels/h2o-lightning-4b-b43f759b`; front `services/h2o-decision-front`; inline images only, both layers verified). **Rolled back twice on 10-12, each time on albok-dev's word against a pre-named line.** A third attempt waits on albok-dev's two offline fixes going to Prime: (1) the book-profile margin (the Schell chapter at 0.531 vs INC 0.438, via a per-profile threshold or `visibility: public` on published titles); (2) h2o's reading of the personal question (6 → 18 memory entries). ⚠ Seat headroom under 4-way 32k load was 683 MiB.
- On any switch, albok-dev sets judge concurrency: 1 for INC, 4 for h2o.
### nh3-pve + nh3-ml1 (2026-09-25/26)
@@ -181,7 +180,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
## Recent decisions
- `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). albok-dev's v2 fleet walk on h2o followed, with expected counts 0/0/6/2. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word**; h2o is staged until the two fixes reach Prime. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.