vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(pfi): Amendment 1 addendum — h2o confirmation run passes all three clauses

Browse Source 
Admission via thin wrapper (vendor shim unmodified; no shim config knob exists).
Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms;
4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the
unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager
slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause
under the accepted 32,000-token refusing contract; seat swap is the operator's
call.
This commit is contained in:
vh
2026-10-12 12:13:31 -07:00
parent 4098a23dda
commit 14af41b945
1 changed files with 85 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/decision-sweep-followon-bench-2026-10-12.md
+85 -1
View File
@@ -57,7 +57,7 @@ for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit
| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s |
|---|---|---|---|---|---|---|
| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 |
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 |
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 → **154** (measured, see addendum 1a) |
| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 |
| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 |
| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 |
@@ -140,3 +140,87 @@ differ between the morning doc (42) and this one (1012) — both are seeded and
re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that
the choice of seed is worth knowing about, and it survived re-runs under
seed 42 unchanged ([+0.0023, +0.0257] on both).
## Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable)
Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under
h2o is 32,000 tokens (state + questions); the failed clause was the shim's
*silence*, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3,
plus a **thin admission wrapper** (task 2 recorded: no shim config option for
admission exists — `max_state_tokens` only configures the trim). The wrapper
imports the shipped renderers, asks vLLM's own `/tokenize` for the count of
each question's exact prompt, and refuses over-cap with 422 + an explicit
error naming the count (`refused, not trimmed`). No vendor file was edited.
It polices both currencies: per-question prompt max AND the shim's own
shared-head + tails total.
**(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth.**
Every admitted cell reports `usage.input_tokens` exactly equal to the wrapper
count — the trim is now unreachable below the cap, so the report is honest:
| depth | admitted | tokens sent = reported | answer |
|---|---|---|---|
| ~1,000 | yes | 31,700 | supported ✓ |
| ~8,000 | yes | 31,862 | supported ✓ (p: .907/.027/.066) |
| ~16,000 | yes | 29,508 | supported ✓ |
| ~30,000 | yes | 31,885 | supported ✓ |
| over-cap probes (×2) | **refused** | 39,805 / 40,336 tokens | 422, named count, every time |
Evidence answered correctly at every depth — the 10-04 depth-dependent
evidence loss is gone at these sizes (it lived INSIDE the trimmed window).
**Clause (2): clean.**
**(3) Largest-admissible shapes** (state pinned by the wrapper so the 16-q
request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total):
1q cold 213 ms / warm 194 ms; **16q cold 2,123 ms / warm 2,033 ms** (second
pass: 2,141/2,041). Cold = first request after backend ready on a restarted
container (compile cache mounted; calibration probes excluded).
**Clause (3): PASS** — warm 16q is 2.03 s against the 2.93 s bar, and it is
the fastest honest 32k cell of any arm this bench measured.
**(4) Concurrency:** 4 concurrent 16q × 32k on `--max-num-seqs 16`: all four
answered correctly, wall 7.08 s, **peak 15,566 MiB** against the GPU 1 budget
re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 +
34,994). Headroom 1,281 MiB. **Clause (4): PASS** — the clause brokkr flagged
as most likely to bite did not bite; KV pressure at 4× concurrency grows
sub-linearly because all four requests share the prefix.
**(1a) Boot matrix** (reported, not gated): cold launch→ready median **153.6 s**
(min 149.4 / max 185.7, n=3), warm (persistent compile cache) **153.5 s** (n=3),
enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA
graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s.
Two corrections to the main doc's boot row: the earlier ~280 s was measured
without a persistent cache mount and was never reproducible (use ~154 s);
warm cache saves nothing here because boot is graph-capture-bound, and
eager mode — which skips capture entirely — is *slower* (161.9 s), so there
is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost
~2 min of boot per restart.
**(5) Wyrd item by item** (168 scorable rows; 8 disagree, h2o gains 3, loses 5,
net −2 — matching the −0.0167 point estimate; for Wyrd's owner):
| item | h2o | INC | gold | verdict |
|---|---|---|---|---|
| fletcher/f-go-tavern exit2 | stay | tavern | tavern | h2o wrong |
| fletcher/f-leave-town exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-memory exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-wren-comes exit2 | stay | meet_wren | meet_wren | h2o wrong |
| toll/t-granted place | moved | holds | moved | h2o right |
| toll/t-questions exit | departure | stay | stay | h2o wrong |
| vigil/v-to-pub exit2 | stay | outside | stay | h2o right |
| vigil/v-to-pub place | moved | holds | moved | h2o right |
Pattern worth noting: h2o's losses are all fletcher-scene *exit* decisions
(six of them in one scene family); its gains are the toll/vigil *place*
updates. No relabels (zero rows correct-but-renamed).
**Amendment 1 verdict: all three pass clauses hold.** (2) clean, (3) 2.03 s,
(4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now
passes every Supersede clause — with the two honest caveats that its contract
is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and
boot is 4.4x INC's. The seat swap is a decision for the operator and
infra-ops; the bench does not swap anything.
Teardown after the confirmation: wrapper, shim and container stopped; GPU 3
guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB
+ 27 GB materialized serving dirs).