From 14af41b945aa86e5075287697503b966b07685e0 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 12 Oct 2026 12:13:31 -0700 Subject: [PATCH] =?UTF-8?q?docs(pfi):=20Amendment=201=20addendum=20?= =?UTF-8?q?=E2=80=94=20h2o=20confirmation=20run=20passes=20all=20three=20c?= =?UTF-8?q?lauses?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Admission via thin wrapper (vendor shim unmodified; no shim config knob exists). Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms; 4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause under the accepted 32,000-token refusing contract; seat swap is the operator's call. --- ...ecision-sweep-followon-bench-2026-10-12.md | 86 ++++++++++++++++++- 1 file changed, 85 insertions(+), 1 deletion(-) diff --git a/docs/pfi/decision-sweep-followon-bench-2026-10-12.md b/docs/pfi/decision-sweep-followon-bench-2026-10-12.md index ef3371e..404dd44 100644 --- a/docs/pfi/decision-sweep-followon-bench-2026-10-12.md +++ b/docs/pfi/decision-sweep-followon-bench-2026-10-12.md @@ -57,7 +57,7 @@ for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit | arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s | |---|---|---|---|---|---|---| | INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 | -| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 | +| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 → **154** (measured, see addendum 1a) | | CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 | | CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 | | RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 | @@ -140,3 +140,87 @@ differ between the morning doc (42) and this one (1012) — both are seeded and re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that the choice of seed is worth knowing about, and it survived re-runs under seed 42 unchanged ([+0.0023, +0.0257] on both). + +## Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable) + +Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under +h2o is 32,000 tokens (state + questions); the failed clause was the shim's +*silence*, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3, +plus a **thin admission wrapper** (task 2 recorded: no shim config option for +admission exists — `max_state_tokens` only configures the trim). The wrapper +imports the shipped renderers, asks vLLM's own `/tokenize` for the count of +each question's exact prompt, and refuses over-cap with 422 + an explicit +error naming the count (`refused, not trimmed`). No vendor file was edited. +It polices both currencies: per-question prompt max AND the shim's own +shared-head + tails total. + +**(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth.** +Every admitted cell reports `usage.input_tokens` exactly equal to the wrapper +count — the trim is now unreachable below the cap, so the report is honest: + +| depth | admitted | tokens sent = reported | answer | +|---|---|---|---| +| ~1,000 | yes | 31,700 | supported ✓ | +| ~8,000 | yes | 31,862 | supported ✓ (p: .907/.027/.066) | +| ~16,000 | yes | 29,508 | supported ✓ | +| ~30,000 | yes | 31,885 | supported ✓ | +| over-cap probes (×2) | **refused** | 39,805 / 40,336 tokens | 422, named count, every time | + +Evidence answered correctly at every depth — the 10-04 depth-dependent +evidence loss is gone at these sizes (it lived INSIDE the trimmed window). +**Clause (2): clean.** + +**(3) Largest-admissible shapes** (state pinned by the wrapper so the 16-q +request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total): +1q cold 213 ms / warm 194 ms; **16q cold 2,123 ms / warm 2,033 ms** (second +pass: 2,141/2,041). Cold = first request after backend ready on a restarted +container (compile cache mounted; calibration probes excluded). +**Clause (3): PASS** — warm 16q is 2.03 s against the 2.93 s bar, and it is +the fastest honest 32k cell of any arm this bench measured. + +**(4) Concurrency:** 4 concurrent 16q × 32k on `--max-num-seqs 16`: all four +answered correctly, wall 7.08 s, **peak 15,566 MiB** against the GPU 1 budget +re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 + +34,994). Headroom 1,281 MiB. **Clause (4): PASS** — the clause brokkr flagged +as most likely to bite did not bite; KV pressure at 4× concurrency grows +sub-linearly because all four requests share the prefix. + +**(1a) Boot matrix** (reported, not gated): cold launch→ready median **153.6 s** +(min 149.4 / max 185.7, n=3), warm (persistent compile cache) **153.5 s** (n=3), +enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA +graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s. +Two corrections to the main doc's boot row: the earlier ~280 s was measured +without a persistent cache mount and was never reproducible (use ~154 s); +warm cache saves nothing here because boot is graph-capture-bound, and +eager mode — which skips capture entirely — is *slower* (161.9 s), so there +is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost +~2 min of boot per restart. + +**(5) Wyrd item by item** (168 scorable rows; 8 disagree, h2o gains 3, loses 5, +net −2 — matching the −0.0167 point estimate; for Wyrd's owner): + +| item | h2o | INC | gold | verdict | +|---|---|---|---|---| +| fletcher/f-go-tavern exit2 | stay | tavern | tavern | h2o wrong | +| fletcher/f-leave-town exit | meet_wren | stay | stay | h2o wrong | +| fletcher/f-memory exit | meet_wren | stay | stay | h2o wrong | +| fletcher/f-wren-comes exit2 | stay | meet_wren | meet_wren | h2o wrong | +| toll/t-granted place | moved | holds | moved | h2o right | +| toll/t-questions exit | departure | stay | stay | h2o wrong | +| vigil/v-to-pub exit2 | stay | outside | stay | h2o right | +| vigil/v-to-pub place | moved | holds | moved | h2o right | + +Pattern worth noting: h2o's losses are all fletcher-scene *exit* decisions +(six of them in one scene family); its gains are the toll/vigil *place* +updates. No relabels (zero rows correct-but-renamed). + +**Amendment 1 verdict: all three pass clauses hold.** (2) clean, (3) 2.03 s, +(4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now +passes every Supersede clause — with the two honest caveats that its contract +is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and +boot is 4.4x INC's. The seat swap is a decision for the operator and +infra-ops; the bench does not swap anything. + +Teardown after the confirmation: wrapper, shim and container stopped; GPU 3 +guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB ++ 27 GB materialized serving dirs).