Compare commits
17
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6a01da759d | ||
|
|
98254ca077 | ||
|
|
014a4fd0ec | ||
|
|
8d9fcd9ffb | ||
|
|
2b40805b84 | ||
|
|
501bda078d | ||
|
|
f979cca1ea | ||
|
|
c08992f95f | ||
|
|
152137e41d | ||
|
|
2c3ef7a8af | ||
|
|
30f36f2478 | ||
|
|
14af41b945 | ||
|
|
4098a23dda | ||
|
|
0fe081042d | ||
|
|
82987828f8 | ||
|
|
ca9e385316 | ||
|
|
a9bae04ba6 |
@@ -168,7 +168,7 @@ https://svos-terminal.phasefinal.com:443 {
|
||||
|
||||
# The kernel's signed calls come from the box itself (its source on the mesh address); anything else is refused.
|
||||
@not_from_box {
|
||||
path /verbs/* /health
|
||||
path /verbs/* /health /seats/*
|
||||
not remote_ip 100.64.0.9 10.100.50.43 127.0.0.1
|
||||
}
|
||||
handle @not_from_box {
|
||||
@@ -211,6 +211,15 @@ https://svos-terminal.phasefinal.com:443 {
|
||||
handle @health {
|
||||
import svos_terminal_upstream
|
||||
}
|
||||
# U13 fix (2026-10-12): app-terminal's registry base_url is this origin again (browsers frame panes from it),
|
||||
# so the kernel's signed seat-host calls (U5b) come through here too, from the box only (@not_from_box above).
|
||||
@seat_host {
|
||||
method POST
|
||||
path /seats/observe /seats/check /seats/launch /seats/retire-launch
|
||||
}
|
||||
handle @seat_host {
|
||||
import svos_terminal_upstream
|
||||
}
|
||||
handle {
|
||||
respond 404
|
||||
}
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
# D3 vs Intern-Decision-4B — decision-model seat bench (2026-10-12)
|
||||
|
||||
Ordered by Prime; preregistration `brokkr-smithy`
|
||||
`docs/surveys/vllm-sr-decision-landscape-2026-10-11/d3-bench-prereg.md`
|
||||
(c93d7b0, amended b87a7ab), plus Amendment 1 (shape calibration, brokkr) and
|
||||
Amendment 2 (operator: memory budget = what fits on the seat's GPU, e3dbfac).
|
||||
Run by infra-hermes on fv-ml1 GPU 3. **HALTED by operator call at 15:07Z** —
|
||||
"fails on a number of levels… stash the findings and call it." This doc is the
|
||||
stash: everything below was measured; what was *not* run is stated plainly.
|
||||
|
||||
Harness: jevbench v1.2.16 (public 231 = easy/original/hard), the S1 battery
|
||||
(authored144 + perturbations108 + cicada W1/W2 + Wyrd, 1,748 paired scorable
|
||||
rows), and the 10-04 shape-runner lineage with a per-model calibrated 32k
|
||||
builder. All artifacts: `fv-ml1:/tank/bench/arm-*` and `/tank/bench-jev-2026-10-04`
|
||||
(the latter untouched, lkraven-owned).
|
||||
|
||||
## Headline
|
||||
|
||||
The incumbent (intern-decision-serve 0.1.3, pinned Intern-Decision-4B) is not
|
||||
displaced. Every d3 arm that clears the incumbent's accuracy needs either a
|
||||
GPU budget the seat cannot hold at the worst shape, or fails the latency
|
||||
contract, or both. The one arm inside the seat's own memory budget at the
|
||||
worst shape (d3-edge at bs=1) loses 8 battery points and is 28x slower than
|
||||
INC at 32k-class state. The quantization warrant closes: NVFP4 does not buy
|
||||
the 32k shape, because the 32k problem is O(q x state) activation and protocol
|
||||
accounting, not weight bytes.
|
||||
|
||||
## 1. bf16 ladder (Engine primary, `vllm-sr` dev CLI, GPU 3)
|
||||
|
||||
Revisions pinned per arm (edge e51ab425…, lite c91d55d6…, nano ccd7bc1a…,
|
||||
mini 40391eb6…, flash 06d33616…, d3-27 7f257ad7…). jevbench acc = public 231,
|
||||
median of 3 repeats (all repeats bit-exact per arm). Battery = labelled-evidence
|
||||
acc, single condition. Peak VRAM = steady-state serve MiB. p50 = harness per-call
|
||||
median ms.
|
||||
|
||||
| arm | params | jevbench | battery single | peak MiB | p50 ms |
|
||||
|---|---|---|---|---|---|
|
||||
| **INC (baseline)** | 4B | **202** | .910 | 8,822 (32k shape) | ~1.9–2.0s at 32k |
|
||||
| d3-edge | 0.59B | 158 | .749 | 2,398 | 15 |
|
||||
| d3-lite | 1.4B | 168 | .852 | 2,676 | 28 |
|
||||
| d3-nano | 3.8B | 175 | .907 | 5,304 | 29 |
|
||||
| d3-mini | 7.3B | 188 | .945 | 10,540 | 36 |
|
||||
| d3-flash | 8.39B | 190 | .947 | 17,600 | 37 |
|
||||
| d3-27 (vendor path) | 26.09B | 206 | .970 | 50,300 | — |
|
||||
|
||||
INC reproduction: 202/231 x3, bit-exact against the 10-04 record (`out/intern-native`).
|
||||
d3-27 is the only arm that beats INC on the public suite, and it is a path
|
||||
divergence: the Engine's placement estimator (4B/param + 2 GiB) refuses a 26B
|
||||
model on any single card we have; the vendor `d3_server.py` runs it at 50.3 GiB.
|
||||
Gate (jevbench >= INC - 3): PASS at 27B only.
|
||||
|
||||
Argmax cross-check (vendor vs Engine, same snapshot, 231 items): edge 230,
|
||||
lite 229, nano 229, mini 231, flash 231 — mismatches are borderline near-ties
|
||||
(max prob gap 2.7e-2). Engine fidelity: argmax identical on all non-borderline
|
||||
items; probabilities NOT exact (median gap 4.2e-4). The card's "matches exactly"
|
||||
claim is false as written; argmax-equivalence holds.
|
||||
|
||||
## 2. 32k shape cells (Amendment 1: per-model calibrated to ±2% of 32,768 realised tokens)
|
||||
|
||||
Builder length iterated per model in its own tokenizer; every arm converged to
|
||||
32,743 realised tokens for the 1q state. 16q cells carry 16 x 32,743 + framing =
|
||||
**523,900 wire tokens** because the d3 wire protocol counts state once PER
|
||||
QUESTION — O(q x state) accounting, measured, not predicted. INC's equivalent
|
||||
is a single forward over one state. cold = first request after ready; launch to
|
||||
ready wall time is reported separately below and is NOT folded into cold.
|
||||
|
||||
| arm | crit21_short cold ms | 1q32k cold ms | 16q32k cold ms | 16q32k peak MiB |
|
||||
|---|---|---|---|---|
|
||||
| INC | (2-call 16+5, see note) | 1,970 (32,715 tok) | **rejected: >32,768 cap at 33,324; maxfit 16q = 32,708 tok, 1,960 ms** | 8,822 |
|
||||
| d3-edge | 1,242.6 | 895.4 | 20,790 | 28,404 |
|
||||
| d3-lite | 89.1 | 1,139.5 | 18,668 | 28,938 |
|
||||
| d3-nano | 103.3 | 1,442.1 | 23,568 | 33,486 |
|
||||
| d3-mini | 157.1 | 3,565.8 | **deadline-bound** (503 at 30.002s, all retries) | 71,404 |
|
||||
| d3-flash | 242.3 | 4,469.5 | **deadline-bound** | 80,396 |
|
||||
| d3-27 vendor | 642.7 | 8,731.0 | 147,724 (366,172 tok, builder cap) | 96,262 |
|
||||
|
||||
Notes. (a) INC crit21 on 10-04 is a client-side 16+5 split summed across two
|
||||
`/v1/systemone` calls (`bench_shape.py --max-questions 16`); `/v1/systemone`
|
||||
enforces 1..16 questions, so a 21-question crit is structurally out of contract.
|
||||
(b) The Engine's 30 s serving deadline is a configuration of the seat, not a
|
||||
bug; the wall sits between 3.8B (nano completes at 23.6s) and 7.3B (mini).
|
||||
(c) Activation peaks include 32k state at bs=8 (Engine BATCH_SIZE is hard-coded
|
||||
8, `family.py:65`); every bs=8 arm exceeds even Amendment 2's budget at 16q32k.
|
||||
(d) Latency clause (16q32k <= 1.5x INC): INC's honest 32k-class cell is 1.96s,
|
||||
so the clause bar is ~2.9s. NO d3 arm meets it; closest is edge-b1 at 13.0s (6.6x).
|
||||
|
||||
## 3. bs=1 curve (vendor `--batch-size 1`, the only memory knob)
|
||||
|
||||
| arm | 16q32k peak MiB (bs=8 -> bs=1) | 16q32k cold ms (bs=8 -> bs=1) |
|
||||
|---|---|---|
|
||||
| d3-edge | 28,404 -> 4,778 | 20,790 -> 13,019 |
|
||||
| d3-mini | 71,404 -> 18,514 | deadline -> 53,770 |
|
||||
|
||||
The knob buys memory, never time: latency is protocol-intrinsic O(q x state).
|
||||
No setting puts a mini-class model inside the seat at the worst shape (mini-b1
|
||||
still misses Amendment 2's budget by 1.6 GiB *and* fails latency at 18x INC).
|
||||
|
||||
## 4. Stage 4 — control quant trio and the battery sensitivity curve
|
||||
|
||||
d3-lite backbone, RTN (round-to-nearest, no calibration), vendor path,
|
||||
`--verify none` (the manifest hashes the main shard; modified packages can't
|
||||
self-verify). Paired deltas vs the path-matched bf16 vendor baseline, 1,748
|
||||
rows, 95% bootstrap CI, Wyrd split included:
|
||||
|
||||
| arm | weight rel-L2 | jevbench | battery | paired delta [CI] | Wyrd delta [CI] | argmax flips |
|
||||
|---|---|---|---|---|---|---|
|
||||
| lite bf16 vendor (control) | 0 | 168 | .852 | — | — | — |
|
||||
| lite NVFP4 RTN | 9.4% | 164 | .869 | +0.0034 [-0.0086, +0.0154] | -0.019 [-0.055, +0.017] | 117 |
|
||||
| lite INT3 RTN | 19.9% | 158 | .739 | -0.095 [-0.118, -0.072] | -0.131 [-0.176, -0.088] | 476 |
|
||||
|
||||
The battery gate is monotone in weight error and functional: it cannot see 9.4%
|
||||
(NVFP4 RTN lands inside noise), it clearly sees 19.9% (INT3 collapses). This is
|
||||
the calibration warrant's premise confirmed: a quant that survives must move
|
||||
weights *less* than 9.4% AND keep paired deltas inside the band — i.e. the
|
||||
headroom to demonstrate is measured, and it is small.
|
||||
|
||||
## 5. Stage 5 — flash calibrated NVFP4 (the one arm completed before the halt)
|
||||
|
||||
ModelOpt `NVFP4_DEFAULT_CFG`, calibrated on the shared 512-row corpus (sha256
|
||||
e7aa7b18…a31e, seeded `jev-nvfp4-calib-2026-10-12`, no scored item). Served
|
||||
in-process over a `/v1/systemone` shim (`stage5_serve.py`, prepare/run/respond
|
||||
mirroring the vendor contract, 1,405 quantizers): ModelOpt's fake-quant lives
|
||||
in the quantizer forward, so no package rewrite; accuracy is a true fake-quant
|
||||
read, memory is analytic + the byte table below.
|
||||
|
||||
jevbench 194 x3 (bf16 Engine control 190). Survive rules applied by the letter,
|
||||
paired vs `arm-d3-flash`, n=1,748:
|
||||
|
||||
| rule | result | verdict |
|
||||
|---|---|---|
|
||||
| (a) battery pooled delta, CI lower bound >= -0.02 | -0.0011 [-0.0086, +0.0057] | PASS |
|
||||
| (b) public suite within ±3 of bf16 median | +4 (194 vs 190) | **FAIL by the letter** (upward; 10 flips: 6 up, 2 down, 2 both-wrong) |
|
||||
| (c) Wyrd within -0.02 | +0.0071 [-0.0119, +0.0262] | PASS |
|
||||
| (d) argmax flips <= 4 of 231, both directions | 10 | **FAIL by the letter** |
|
||||
|
||||
Byte table (bf16 snapshot index, total 15.63 GiB on disk): quantized linears
|
||||
(attn 3,980 + mlp 9,216 MiB) at 4.5 b/w = 0.28125x -> 3,711 MiB; kept wide
|
||||
(embed+head 1,940, vision 870, other 2) = 2,812 MiB. **Projected export 6,524
|
||||
MiB (~6.37 GiB)** — single-digit, as expected. ModelOpt `export_hf_checkpoint`
|
||||
exists for a measured-export ground truth; not run before the halt.
|
||||
Latency of the fake-quant arm is dequant-matmul timing and is NOT reported as a
|
||||
speedup. **Analytic 32k verdict unchanged: quantization does not touch the
|
||||
O(q x state) activation cost; the 16q32k cell fails identically at any weight width.**
|
||||
|
||||
## 6. Amendment 2 — the seat GPU budget (measured read-only, GPU 1)
|
||||
|
||||
Prime: "our budget is now whatever fits in the current gpu where it's running."
|
||||
|
||||
- GPU 1 total: **97,887 MiB.**
|
||||
- Non-seat processes: `vllm-meromero-r` 43,994 MiB + `vllm-gen-small` 34,994 MiB = 78,988 MiB.
|
||||
- Seat (intern-decision) resting: 8,812 MiB.
|
||||
- Safety margin: 2,048 MiB (brokkr's suggested figure; no seat headroom policy exists).
|
||||
- **Budget: 16,851 MiB (~16.5 GiB).**
|
||||
|
||||
Re-read of every fit clause against 16,851 MiB:
|
||||
- Short shape: fits INC-class (8,822 measured) and every arm through mini
|
||||
(10,540); NVFP4-flash analytic ~6,524 fits; flash bf16 (17,600) misses by
|
||||
749 MiB; 27B (50,300) does not.
|
||||
- 16q32k: only edge-b1 (4,778) and INC-class fit. Every bs=8 arm is 27.6–96.3
|
||||
GiB. The latency clause independently kills mini/flash/INC at that shape.
|
||||
- Net: the wider budget upgrades mini from "memory-out" to "memory-in,
|
||||
latency-out" (53.8s at the only knob setting that fits), and leaves the
|
||||
conclusion unchanged.
|
||||
|
||||
## 7. What was NOT run because of the halt
|
||||
|
||||
- INC bench-copy arms (positive control `inc-bf16-copy` and fake-quant
|
||||
`inc-fakequant`, 249 language tensors RTN-NVFP4, median rel-L2 9.4%, vision
|
||||
kept bf16): the 10-04 `intern_server.py` shim and both packages were built
|
||||
and the arm was mid-boot when the halt landed. The INC fake-quant row is
|
||||
therefore absent — note the production engine cannot load NVFP4 at all
|
||||
(`intern_decision_serve/engine.py` pins the snapshot's `inference.py` by hash
|
||||
and validates dtype against {bfloat16, float16, float32} with an explicit
|
||||
raise; no compressed-tensors path exists in the image).
|
||||
- NVFP4 arms for mini, 27B (fake-quant), nano, lite (calibrated), edge.
|
||||
- ModelOpt measured export (ground-truth on-disk size).
|
||||
- Paired battery deltas + Wyrd splits for the bf16 ladder arms (no quantized
|
||||
sibling to pair against; would need battery re-runs).
|
||||
|
||||
## 8. Teardown & disk
|
||||
|
||||
Engine stack stopped, all bench servers on GPU 3 stopped, `:18080` iptables rule
|
||||
removed, guard read confirms GPU 3 at 2 MiB. Production GPU-0/1 seats untouched
|
||||
(one ~51 s exposure window during cross-checks earlier in the run, zero
|
||||
degradation observed — all RestartCount=0, no Xid).
|
||||
|
||||
Weights KEPT per instruction (Prime rules on deletion):
|
||||
`/tank/bench/models/model-runtime` **80 GB** (all six d3 snapshots + INC copy
|
||||
staged), plus bench working set: `inc-bf16-copy` 8.5 GB, `inc-fakequant` 8.5 GB,
|
||||
`quant-flash-nvfp4` 16 GB, `quant-lite-nvfp4` 1.7 GB, `quant-lite-int3` 1.4 GB.
|
||||
`/tank/bench` total 180 GB; `/tank` at 62% with 3.3 TB free, so this is
|
||||
comfort, not pressure.
|
||||
@@ -0,0 +1,226 @@
|
||||
# Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)
|
||||
|
||||
Ordered by Prime; preregistration `brokkr-smithy`
|
||||
`docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md` @ 2e0539f —
|
||||
the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3
|
||||
(process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench
|
||||
harness as the morning run
|
||||
(`docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md`), unchanged.
|
||||
|
||||
## Headline
|
||||
|
||||
**The incumbent holds.** H2O-lightning-4b is the first non-d3 arm to beat INC
|
||||
on the public suite (205 vs 202) with a positive paired battery delta — but it
|
||||
fails the long-input gate by the letter: its shipped shim silently trims the
|
||||
state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg
|
||||
rules that makes it a **short-route candidate**, not a supersede. Clef-flash is
|
||||
**lateral** in both precisions (battery CI straddles zero, Wyrd significantly
|
||||
negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a
|
||||
**loss** (23 items below INC on public, Wyrd −0.0548).
|
||||
|
||||
## Budget (re-measured at bench time, Amendment 2 method)
|
||||
|
||||
GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small
|
||||
34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB.
|
||||
**Budget = 16,847 MiB.** (16,851 in the morning; 4 MiB drift, immaterial.)
|
||||
|
||||
## Arms and verdicts
|
||||
|
||||
Vendor-intended serving path for every arm, verbatim; no ports, no third-party
|
||||
shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped
|
||||
`h2o_lightning_shim.py`) or RSI (vendor CLI); Clef reuses the 10-04
|
||||
`clef_server.py` stdlib wrapper around the shipped `joint_schema_model.py`
|
||||
(same wrapper that produced the 10-04 numbers; weights byte-identical).
|
||||
|
||||
| arm | path | jevbench (×3) | battery paired Δ vs INC [95% CI] | Wyrd Δ [CI] | neg-control | Brier / ECE |
|
||||
|---|---|---|---|---|---|---|
|
||||
| **INC (reused cells)** | seat 0.1.3 | **202** | — | — | .097 | .178 / .051 |
|
||||
| H2O @ b43f759b | vLLM 0.30.0 + shipped shim | **205** | **+0.0137 [+0.0023, +0.0257]** | −0.0167 [−0.0357, +0.0024] | .028 | .152 / .050 |
|
||||
| CLEF bf16 @ 8b2e5fd1 | vendor loader, max_length=32768 | 191 | +0.0040 [−0.0080, +0.0160] | **−0.0333 [−0.0571, −0.0095]** | .111 | .235 / .053 |
|
||||
| CLEF bnb-int8 | same + LLM.int8() | 190 | +0.0006 [−0.0114, +0.0120] | **−0.0333 [−0.0571, −0.0119]** | .090 | .239 / .043 |
|
||||
| RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev) | `rsi-jev serve --effort auto` | 179 | −0.0149 [−0.0303, +0.0006] | **−0.0548 [−0.0905, −0.0214]** | .208 | .274 / .061 |
|
||||
|
||||
Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same
|
||||
harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the
|
||||
full battery: negative control crushed for H2O (it abstains — the loaded-name
|
||||
trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the
|
||||
run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better
|
||||
Brier than INC, and its ECE is the second-best.
|
||||
|
||||
## Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)
|
||||
|
||||
cold = first request after ready; warm = median of 5; launch→ready its own
|
||||
column. INC crit cells re-measured today on the seat path (2-call 16+5 split
|
||||
for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit
|
||||
16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).
|
||||
|
||||
| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s |
|
||||
|---|---|---|---|---|---|---|
|
||||
| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 |
|
||||
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 → **154** (measured, see addendum 1a) |
|
||||
| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 |
|
||||
| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 |
|
||||
| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 |
|
||||
|
||||
\* reused 32k-run cells; INC's `/v1/systemone` enforces a 32,768-token
|
||||
contract, so its honest 16q32k is a maxfit single forward.
|
||||
|
||||
Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768
|
||||
honours the window but caps there — 16q over 32k state counts state once per
|
||||
request at the wire, and the loader slices to window. Their "warm" at that
|
||||
shape is the fastest honest path, but the state never exceeds the window.
|
||||
RSI realised 33,205 — it has no hard cap; honest 32k+ shape.
|
||||
|
||||
**H2O's trim, the named risk, confirmed:** the shape calibration plateaued at
|
||||
32,082 realised tokens across builder lengths 46,953→51,097 (full history in
|
||||
`arm-h2o/shape32k.stdout`) — the shim head/tail-trims at 32,000 exactly as on
|
||||
10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away).
|
||||
The gate itself scores OK at every depth (the evidence sits mid-state and
|
||||
survives the trim), so the failure mode is silent: **accuracy at 32k is
|
||||
measured on a truncated state and the response says nothing about it.**
|
||||
|
||||
## Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)
|
||||
|
||||
| arm | verdict | evidence |
|
||||
|---|---|---|
|
||||
| INC | clean | reused; no trim observed on seat path |
|
||||
| H2O | **FAIL — silent truncation** | calibration plateau at 32,082; `usage.input_tokens` reports the post-trim count, not the sent count |
|
||||
| CLEF bf16 | clean (window=32,768 honoured) | d30000 OK; no plateau below the window |
|
||||
| CLEF int8 | clean | as bf16 |
|
||||
| RSI | clean | 32,758 realised, no plateau; d30000 OK |
|
||||
|
||||
## Decision-rule application (fixed at 2e0539f)
|
||||
|
||||
**H2O-lightning-4b — SHORT-ROUTE CANDIDATE.** Fits the budget at every shape
|
||||
incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd
|
||||
−0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter,
|
||||
and it is genuinely the fastest at the worst shape because the shim's
|
||||
single-count protocol answers over a *truncated* state). Fails only the gate
|
||||
clause: silent truncation at 32k. Recorded, not acted on. Note for whoever
|
||||
eventually evaluates it for the seat: the trim is inside the vendor's shipped
|
||||
serving code — a supersede needs the vendor to fix the shim, not us to patch
|
||||
around it.
|
||||
|
||||
**CLEF bf16 — LATERAL, memory-out.** Battery CI straddles zero; Wyrd fails;
|
||||
rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the
|
||||
latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget).
|
||||
**CLEF int8 — LATERAL.** Fits short shapes (rest 12,084) but the battery CI
|
||||
straddles zero, so it is not even a short-route candidate; 16q32k fails both
|
||||
memory (23,080) and latency (3,284).
|
||||
**RSI-Jev — LOSS.** Public −23 items (outside ±3), battery Δ −0.0149 with CI
|
||||
[−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its
|
||||
serving numbers are respectable (clean gate, latency pass at 16q32k) — it is
|
||||
the decision quality that loses.
|
||||
|
||||
## Housekeeping
|
||||
|
||||
- Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per
|
||||
prereg; the 7f93b192 sha is the **HF model revision**, not a repo commit —
|
||||
it is not an ancestor of main and was resolved as a snapshot).
|
||||
- Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at
|
||||
source and lists a research-only training source — flagged once, does not
|
||||
block a bench, deployment rights are the operator's call.
|
||||
- Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read
|
||||
2 MiB, no firewall rules were needed (loopback-bound ports only).
|
||||
- Weights KEPT (operator ruling stands): model-runtime now 116 GB total
|
||||
(adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized
|
||||
`pkg-h2o` 9.0 GB and `pkg-clef` 18 GB serving dirs and the RSI-Jev clone
|
||||
72 MB. `/tank` 63% used, 3.2 TB free.
|
||||
- Artifacts: `fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/`
|
||||
(jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally
|
||||
`shape32k-forced.json` recording the trimmed-shape cells and
|
||||
`shape32k.stdout` the plateau history).
|
||||
|
||||
## What was NOT run
|
||||
|
||||
Nothing from the prereg matrix was skipped. One methodology note: INC's
|
||||
battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and
|
||||
bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds
|
||||
differ between the morning doc (42) and this one (1012) — both are seeded and
|
||||
re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that
|
||||
the choice of seed is worth knowing about, and it survived re-runs under
|
||||
seed 42 unchanged ([+0.0023, +0.0257] on both).
|
||||
|
||||
## Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable)
|
||||
|
||||
Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under
|
||||
h2o is 32,000 tokens (state + questions); the failed clause was the shim's
|
||||
*silence*, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3,
|
||||
plus a **thin admission wrapper** (task 2 recorded: no shim config option for
|
||||
admission exists — `max_state_tokens` only configures the trim). The wrapper
|
||||
imports the shipped renderers, asks vLLM's own `/tokenize` for the count of
|
||||
each question's exact prompt, and refuses over-cap with 422 + an explicit
|
||||
error naming the count (`refused, not trimmed`). No vendor file was edited.
|
||||
It polices both currencies: per-question prompt max AND the shim's own
|
||||
shared-head + tails total.
|
||||
|
||||
**(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth.**
|
||||
Every admitted cell reports `usage.input_tokens` exactly equal to the wrapper
|
||||
count — the trim is now unreachable below the cap, so the report is honest:
|
||||
|
||||
| depth | admitted | tokens sent = reported | answer |
|
||||
|---|---|---|---|
|
||||
| ~1,000 | yes | 31,700 | supported ✓ |
|
||||
| ~8,000 | yes | 31,862 | supported ✓ (p: .907/.027/.066) |
|
||||
| ~16,000 | yes | 29,508 | supported ✓ |
|
||||
| ~30,000 | yes | 31,885 | supported ✓ |
|
||||
| over-cap probes (×2) | **refused** | 39,805 / 40,336 tokens | 422, named count, every time |
|
||||
|
||||
Evidence answered correctly at every depth — the 10-04 depth-dependent
|
||||
evidence loss is gone at these sizes (it lived INSIDE the trimmed window).
|
||||
**Clause (2): clean.**
|
||||
|
||||
**(3) Largest-admissible shapes** (state pinned by the wrapper so the 16-q
|
||||
request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total):
|
||||
1q cold 213 ms / warm 194 ms; **16q cold 2,123 ms / warm 2,033 ms** (second
|
||||
pass: 2,141/2,041). Cold = first request after backend ready on a restarted
|
||||
container (compile cache mounted; calibration probes excluded).
|
||||
**Clause (3): PASS** — warm 16q is 2.03 s against the 2.93 s bar, and it is
|
||||
the fastest honest 32k cell of any arm this bench measured.
|
||||
|
||||
**(4) Concurrency:** 4 concurrent 16q × 32k on `--max-num-seqs 16`: all four
|
||||
answered correctly, wall 7.08 s, **peak 15,566 MiB** against the GPU 1 budget
|
||||
re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 +
|
||||
34,994). Headroom 1,281 MiB. **Clause (4): PASS** — the clause brokkr flagged
|
||||
as most likely to bite did not bite; KV pressure at 4× concurrency grows
|
||||
sub-linearly because all four requests share the prefix.
|
||||
|
||||
**(1a) Boot matrix** (reported, not gated): cold launch→ready median **153.6 s**
|
||||
(min 149.4 / max 185.7, n=3), warm (persistent compile cache) **153.5 s** (n=3),
|
||||
enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA
|
||||
graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s.
|
||||
Two corrections to the main doc's boot row: the earlier ~280 s was measured
|
||||
without a persistent cache mount and was never reproducible (use ~154 s);
|
||||
warm cache saves nothing here because boot is graph-capture-bound, and
|
||||
eager mode — which skips capture entirely — is *slower* (161.9 s), so there
|
||||
is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost
|
||||
~2 min of boot per restart.
|
||||
|
||||
**(5) Wyrd item by item** (168 scorable rows; 8 disagree, h2o gains 3, loses 5,
|
||||
net −2 — matching the −0.0167 point estimate; for Wyrd's owner):
|
||||
|
||||
| item | h2o | INC | gold | verdict |
|
||||
|---|---|---|---|---|
|
||||
| fletcher/f-go-tavern exit2 | stay | tavern | tavern | h2o wrong |
|
||||
| fletcher/f-leave-town exit | meet_wren | stay | stay | h2o wrong |
|
||||
| fletcher/f-memory exit | meet_wren | stay | stay | h2o wrong |
|
||||
| fletcher/f-wren-comes exit2 | stay | meet_wren | meet_wren | h2o wrong |
|
||||
| toll/t-granted place | moved | holds | moved | h2o right |
|
||||
| toll/t-questions exit | departure | stay | stay | h2o wrong |
|
||||
| vigil/v-to-pub exit2 | stay | outside | stay | h2o right |
|
||||
| vigil/v-to-pub place | moved | holds | moved | h2o right |
|
||||
|
||||
Pattern worth noting: h2o's losses are all fletcher-scene *exit* decisions
|
||||
(six of them in one scene family); its gains are the toll/vigil *place*
|
||||
updates. No relabels (zero rows correct-but-renamed).
|
||||
|
||||
**Amendment 1 verdict: all three pass clauses hold.** (2) clean, (3) 2.03 s,
|
||||
(4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now
|
||||
passes every Supersede clause — with the two honest caveats that its contract
|
||||
is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and
|
||||
boot is 4.4x INC's. The seat swap is a decision for the operator and
|
||||
infra-ops; the bench does not swap anything.
|
||||
|
||||
Teardown after the confirmation: wrapper, shim and container stopped; GPU 3
|
||||
guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB
|
||||
+ 27 GB materialized serving dirs).
|
||||
+10
-5
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)_
|
||||
_Last updated: 2026-10-12 ~1710 PT (decision seat: h2o 1311–1403, rolled back, judge re-fitted, h2o LIVE again 1704; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
|
||||
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
|
||||
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
|
||||
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
|
||||
| 1 | intern-decision | 32,768/call | cap 14.4 GiB, peak 15,220 | — | — |
|
||||
| 1 | intern-decision = **h2o-lightning-4b** since 10-12 1704 (INC = rollback target) | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
|
||||
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
|
||||
|
||||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
|
||||
@@ -166,10 +166,12 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||||
|
||||
### intern-decision (replaced SemIf on 2026-09-30)
|
||||
### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12 1704; INC = rollback)
|
||||
|
||||
- **LIVE 0.1.3** at `intern-decision.fv.internal:8033`: semif-compatible `/decide` plus Jev `/v1/systemone`, 32k tokens, a Triton cache volume. Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||||
- **Open:** label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim.
|
||||
- **LIVE: h2o-lightning-4b @ b43f759b** (`compose.h2o.yaml`, `switch-to-h2o.sh`) behind `services/h2o-decision-front`: 32k refuse-not-trim admission, bearer, `/decide/shared` translator. Inline images only (`LIGHTNING_IMAGE_URLS=off` + vLLM `--allowed-media-domains media.invalid`, both verified). Nemi's judge uses the **v2 note question** (albok-nemi 0.1.18) at **concurrency 4**.
|
||||
- **Rollback:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` (INC `intern-decision-serve:0.1.3`, ~35 s), **and albok-dev sets judge concurrency to 1**. INC scores on one thread (INV-2), and 4 in flight queued into 339 timeouts on 10-12.
|
||||
- ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1.
|
||||
- INC history: → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`.
|
||||
|
||||
### nh3-pve + nh3-ml1 (2026-09-25/26)
|
||||
|
||||
@@ -178,6 +180,9 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). albok-dev's v2 fleet walk on h2o followed, with expected counts 0/0/6/2. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
|
||||
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||||
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime's call, pending:** A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against.
|
||||
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||||
- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
|
||||
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md`
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# h2o decision front
|
||||
|
||||
The front of the h2o-lightning-4b decision seat on fv-ml1 (seat swap ordered by Prime 2026-10-12, through brokkr).
|
||||
`front.py` sits in front of the vendor's **unmodified** `h2o_lightning_shim.py`, which sits in front of stock vLLM
|
||||
0.30.0. It adds what the seat contract needs and the vendor stack lacks:
|
||||
|
||||
- **Admission** (infra-hermes's `tools/h2o-admission/admission_proxy.py` @ 30f36f2, logic unchanged): count every
|
||||
decision request with h2o's own tokenizer over the shim's own renders, and refuse over 32,000 tokens with 422 + the
|
||||
count. The shim would otherwise trim silently.
|
||||
- **The seat's bearer token** on everything but `GET /health`, as intern-decision-serve has.
|
||||
- **`/decide` and `/decide/shared` continuity** for albok-nemi's sensitivity judge: intern-decision-serve's own
|
||||
mapping (one Jev `choice` question per decision, calls of at most 16, fields `q`/`q1..qN`), re-keyed into the semif
|
||||
result shape (`results[i].native.probabilities[option id]`). `orderings` other than `none` → 422.
|
||||
- **`/health`** with `model: {name, source, revision}`, which the judge stamps into its cache key.
|
||||
|
||||
`run.sh` runs the shim (loopback) and the front in one container (`python:3.12-slim`, host network). Stacks copy both
|
||||
files into `conf/front/`: `stacks/h2o-eval` (TEMPORARY, GPU 3, :8034) today.
|
||||
|
||||
Verified 2026-10-12 1300 on the eval seat: 401 without the token; `/decide/shared` in the judge's shape; a planted
|
||||
personal record → `personal` p_yes .911, a neutral note → .056 (n=1 each, smoke test only); 40,088 tokens → 422 with
|
||||
the count; `/v1/systemone` passthrough with `_admission` counts; reachable from nh3-dev.
|
||||
@@ -0,0 +1,231 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
|
||||
|
||||
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
|
||||
the three things the seat contract needs and the vendor stack does not have:
|
||||
|
||||
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
|
||||
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
|
||||
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
|
||||
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
|
||||
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
|
||||
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
|
||||
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
|
||||
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
|
||||
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
|
||||
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
|
||||
the judge stamps into its cache key.
|
||||
|
||||
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
|
||||
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
|
||||
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
|
||||
"""
|
||||
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
TOKEN = os.environ["FRONT_TOKEN"]
|
||||
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
|
||||
PORT = int(os.environ.get("FRONT_PORT", "8034"))
|
||||
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
|
||||
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
|
||||
CAP = int(os.environ.get("FRONT_CAP", "32000"))
|
||||
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
|
||||
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
|
||||
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
|
||||
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
|
||||
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
|
||||
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
|
||||
MAX_QUESTIONS_PER_CALL = 16
|
||||
MIN_OPTIONS, MAX_OPTIONS = 2, 16
|
||||
|
||||
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
|
||||
shim_mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules["h2o_shim"] = shim_mod
|
||||
spec.loader.exec_module(shim_mod)
|
||||
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
|
||||
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
def __init__(self, code, obj):
|
||||
super().__init__(obj)
|
||||
self.code, self.obj = code, obj
|
||||
|
||||
|
||||
def sent_tokens(body):
|
||||
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
|
||||
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
|
||||
state = body.get("state") or ""
|
||||
if isinstance(state, (dict, list)):
|
||||
state = json.dumps(state)
|
||||
state_txt = shim_mod.render_state(state, CFG.state_compact)
|
||||
all_toks = []
|
||||
for qid, q in body.get("questions", {}).items():
|
||||
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
|
||||
req = urllib.request.Request(VLLM + "/tokenize",
|
||||
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
all_toks.append(json.loads(r.read())["tokens"])
|
||||
mx = max(len(t) for t in all_toks)
|
||||
shared = shim_mod.common_prefix_len(all_toks)
|
||||
return mx, shared + sum(len(t) - shared for t in all_toks)
|
||||
|
||||
|
||||
def admit(body):
|
||||
try:
|
||||
mx, total = sent_tokens(body)
|
||||
except Refused:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001
|
||||
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
|
||||
if mx > CAP or total > CAP:
|
||||
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
|
||||
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
|
||||
"shim_accounted_total": total, "cap": CAP})
|
||||
return mx, total
|
||||
|
||||
|
||||
def post_up(path, obj):
|
||||
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
|
||||
method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as r:
|
||||
return r.status, json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read() or b"{}"
|
||||
try:
|
||||
return e.code, json.loads(raw)
|
||||
except ValueError:
|
||||
return e.code, {"error": raw[:300].decode(errors="replace")}
|
||||
|
||||
|
||||
def systemone(body):
|
||||
mx, total = admit(body)
|
||||
code, obj = post_up("/v1/systemone", body)
|
||||
if isinstance(obj, dict):
|
||||
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
|
||||
return code, obj
|
||||
|
||||
|
||||
def check_decision(d, i):
|
||||
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
|
||||
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
|
||||
opts = d.get("options")
|
||||
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
|
||||
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
|
||||
for o in opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
|
||||
if len({o["id"] for o in opts}) != len(opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
|
||||
if d.get("orderings", "none") != "none":
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
|
||||
"seat (h2o); send orderings 'none' and combine on the client"})
|
||||
|
||||
|
||||
def decide(state, decisions):
|
||||
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
|
||||
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
|
||||
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
|
||||
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
|
||||
for i, d in enumerate(decisions):
|
||||
check_decision(d, i)
|
||||
started, results, sizes, tokens = time.perf_counter(), [], [], []
|
||||
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
|
||||
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
|
||||
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
|
||||
questions = {f: {"type": "choice", "instructions": d["question"],
|
||||
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
|
||||
code, resp = systemone({"state": state, "questions": questions})
|
||||
if code != 200:
|
||||
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
|
||||
sizes.append(len(chunk))
|
||||
tokens.append((resp.get("usage") or {}).get("input_tokens"))
|
||||
for f, d in zip(fields, chunk):
|
||||
answer = (resp.get("answers") or {}).get(f)
|
||||
ids = [o["id"] for o in d["options"]]
|
||||
probs = (answer or {}).get("probabilities") or {}
|
||||
if answer is None or any(i not in probs for i in ids):
|
||||
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
|
||||
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
|
||||
"top": answer.get("choice"), "confidence": answer.get("confidence"),
|
||||
"calibration": resp.get("calibration"), "native": answer,
|
||||
"input_tokens": tokens[-1], "model": MODEL,
|
||||
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
|
||||
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
|
||||
"questions_per_call": sizes, "input_tokens": tokens}}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
|
||||
|
||||
def _send(self, code, obj):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b)))
|
||||
self.end_headers()
|
||||
self.wfile.write(b)
|
||||
|
||||
def _authorized(self):
|
||||
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
|
||||
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/health":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
|
||||
shim = json.loads(r.read())
|
||||
except Exception as e: # noqa: BLE001
|
||||
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
|
||||
ok = bool(shim.get("ok"))
|
||||
return self._send(200 if ok else 503, {
|
||||
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
|
||||
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
|
||||
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
if path == "/v1/models":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
|
||||
return self._send(r.status, json.loads(r.read()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return self._send(e.code, {"error": "upstream"})
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0]
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
n = int(self.headers.get("Content-Length", 0) or 0)
|
||||
if n > 4 * 1024 * 1024:
|
||||
return self._send(413, {"error": "body over 4 MiB"})
|
||||
try:
|
||||
body = json.loads(self.rfile.read(n) or b"null")
|
||||
except ValueError:
|
||||
return self._send(422, {"error": "body is not JSON"})
|
||||
if not isinstance(body, dict):
|
||||
return self._send(422, {"error": "body must be a JSON object"})
|
||||
try:
|
||||
if path == "/v1/systemone":
|
||||
return self._send(*systemone(body))
|
||||
if path == "/api/alpha/decisions":
|
||||
admit(body)
|
||||
return self._send(*post_up(path, body))
|
||||
if path == "/decide/shared":
|
||||
return self._send(200, decide(body.get("state"), body.get("decisions")))
|
||||
if path == "/decide":
|
||||
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
|
||||
out = decide(body.get("state"), [d])
|
||||
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
|
||||
except Refused as r:
|
||||
return self._send(r.code, r.obj)
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
|
||||
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/bash
|
||||
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
|
||||
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
|
||||
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
|
||||
set -eu
|
||||
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
|
||||
--host 127.0.0.1 --port "$SHIM_PORT" &
|
||||
SHIM=$!
|
||||
python3 /front/front.py &
|
||||
FRONT=$!
|
||||
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
kill "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
exit 1
|
||||
@@ -0,0 +1,35 @@
|
||||
# h2o-eval (TEMPORARY) — fv-ml1 GPU 3, :8034
|
||||
|
||||
> **DOWN since 2026-10-12 1619 PT** (round 2 done: the floor was ruled out, with identical verdicts on and off, and albok-nemi 0.1.18
|
||||
> re-fitted the judge's note question). The host keeps `/opt/docker/compose/h2o-eval` and the weights.
|
||||
>
|
||||
> **Round 2 (was UP 2026-10-12 1428–1619 PT)** (Prime 1420, via albok-dev): "I'd like to understand why h2o failed, so
|
||||
> stand it up on gpu3 and refit it and let's get an understanding." albok-dev runs the investigation and the judge
|
||||
> re-fit (thread `01M4S026GWS354WZGXZ5E34F31`). **No load tests or other traffic while their runs are in.** Round 1
|
||||
> (1300–1319) was the pre-cutover validation. Take it down with `docker compose down` when albok-dev is done.
|
||||
>
|
||||
> | port | front | floor | judge stamp (`/health` model) |
|
||||
> |---|---|---|---|
|
||||
> | `:8034` | `h2o-eval-front` | ON, 0.801, as the live cutover ran | `h2o-lightning-4b@b43f759b…`, identical to the cutover's |
|
||||
> | `:8035` | `h2o-eval-front-nofloor` | OFF (`SHIM_NOUL_FLOOR=0`) | `h2o-lightning-4b+noul-floor-0@b43f759b…` |
|
||||
>
|
||||
> Both fronts sit over ONE vLLM (14,420 MiB, KV 135,070 tokens, util-sized because GPU 3 is otherwise empty). Each
|
||||
> `/health` also reports `backend.noul_floor {value, source}`.
|
||||
>
|
||||
> Checked at 1428:
|
||||
> - `/decide/shared` 200 on both, with the cutover's smoke values reproduced (adult 0.052, personal 0.876).
|
||||
> - An https image URL is refused on both: 422 "not accepted here".
|
||||
> - **The vLLM backstop answered for the first time.** An https image URL posted straight to vLLM got 400 "The URL must
|
||||
> be from one of the allowed domains: ['media.invalid']", with no fetch.
|
||||
|
||||
h2o-lightning-4b @ b43f759b behind `services/h2o-decision-front` (copied into `conf/front/`), for albok-dev's
|
||||
re-validation of Nemi's sensitivity judge before the GPU 1 cutover (Prime 2026-10-12 via brokkr). **Remove right after
|
||||
the cutover:** GPU 3 is for transient work only (Scriberr and MIA use it).
|
||||
|
||||
- Weights: `/tank/aimodels/h2o-lightning-4b-b43f759b` (copied from the bench kit; model.safetensors sha256
|
||||
`2d51561f221e1549d91bd1c38b3512a1435dd32950d454a9f040f1d4d11497b6`, tokenizer.json `5f9e4d49…`, shim git oid
|
||||
`fee7807e…`, serve_config `5e6fef1c…`: all equal to Hugging Face at b43f759b).
|
||||
- vLLM exactly as benched: `--max-model-len 40960 --max-num-seqs 16 --gpu-memory-utilization 0.1506`, vision limits as
|
||||
the vendor's serve.sh; healthy ~3 min after `up` (graph capture), ~14.4 GB on GPU 3.
|
||||
- `.env` on the host: `FRONT_TOKEN` = intern-decision's token (= vault `albok/judge-token`).
|
||||
- Remove: `cd /opt/docker/compose/h2o-eval && docker compose down` (the weights stay for the GPU 1 seat).
|
||||
@@ -0,0 +1,90 @@
|
||||
# TRANSIENT h2o-lightning-4b eval seat on fv-ml1 GPU 3 (GPU 3 is for transient work only).
|
||||
# Round 1, 2026-10-12 1300-1319: albok-dev validated Nemi's judge on h2o before the GPU 1 cutover (Prime via brokkr).
|
||||
# Round 2, from 2026-10-12 ~1430 (Prime 1420, via albok-dev): "I'd like to understand why h2o failed, so stand it up
|
||||
# on gpu3 and refit it and let's get an understanding." albok-dev investigates the 154 texts h2o alone called adult
|
||||
# and re-fits the judge. Two fronts over ONE vLLM:
|
||||
# :8034 front the shim's yes/no floor ON (0.801), exactly as the live cutover ran; /health model stamp
|
||||
# h2o-lightning-4b@b43f759b..., identical to the cutover's
|
||||
# :8035 front-nofloor SHIM_NOUL_FLOOR=0 (floor OFF, brokkr's lever); stamp h2o-lightning-4b+noul-floor-0@b43f759b...
|
||||
# Each /health also reports backend.noul_floor {value, source}. NO load tests or other traffic while albok-dev's
|
||||
# runs are in (they name start/end on thread 01M4S026GWS354WZGXZ5E34F31). INC keeps :8033 on GPU 1, untouched.
|
||||
# Serving exactly as benched (docs/pfi/decision-sweep-followon-bench-2026-10-12.md, Amendment 1): stock vLLM 0.30.0
|
||||
# + the vendor's unmodified shim + the admission front (services/h2o-decision-front, copied into conf/front).
|
||||
# Weights: /tank/aimodels/h2o-lightning-4b-b43f759b (hashes match HF at b43f759b; see the stack README).
|
||||
# Inline images only (Prime 2026-10-12): LIGHTNING_IMAGE_URLS=off on both fronts + vLLM --allowed-media-domains.
|
||||
name: h2o-eval
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:v0.30.0
|
||||
container_name: h2o-eval-vllm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:18199:8000"
|
||||
environment:
|
||||
VLLM_USE_FLASHINFER_SAMPLER: "0"
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- h2o-eval-vllm-cache:/root/.cache/vllm
|
||||
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
|
||||
"--max-model-len", "40960", "--max-num-seqs", "16", "--gpu-memory-utilization", "0.1506",
|
||||
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
|
||||
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
|
||||
"--allowed-media-domains", "media.invalid"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["3"]
|
||||
capabilities: [gpu]
|
||||
front:
|
||||
image: python:3.12-slim
|
||||
container_name: h2o-eval-front
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
depends_on: [vllm]
|
||||
env_file: .env
|
||||
environment:
|
||||
SHIM_VLLM: http://127.0.0.1:18199
|
||||
SHIM_PORT: "18191"
|
||||
SHIM_NOUL_FLOOR: "0.801"
|
||||
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
|
||||
FRONT_SHIM: http://127.0.0.1:18191
|
||||
FRONT_VLLM: http://127.0.0.1:18199
|
||||
FRONT_BIND: 0.0.0.0
|
||||
FRONT_PORT: "8034"
|
||||
FRONT_CAP: "32000"
|
||||
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
|
||||
FRONT_SERVE_CONFIG: /model/serve_config.json
|
||||
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- /opt/docker/conf/h2o-eval/front:/front:ro
|
||||
command: ["bash", "/front/run.sh"]
|
||||
front-nofloor:
|
||||
image: python:3.12-slim
|
||||
container_name: h2o-eval-front-nofloor
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
depends_on: [vllm]
|
||||
env_file: .env
|
||||
environment:
|
||||
SHIM_VLLM: http://127.0.0.1:18199
|
||||
SHIM_PORT: "18192"
|
||||
SHIM_NOUL_FLOOR: "0" # the floor OFF (the shim: 0 turns it off)
|
||||
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
|
||||
FRONT_SHIM: http://127.0.0.1:18192
|
||||
FRONT_VLLM: http://127.0.0.1:18199
|
||||
FRONT_BIND: 0.0.0.0
|
||||
FRONT_PORT: "8035"
|
||||
FRONT_CAP: "32000"
|
||||
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
|
||||
FRONT_SERVE_CONFIG: /model/serve_config.json
|
||||
FRONT_MODEL_NAME: h2o-lightning-4b+noul-floor-0 # a distinct judge stamp
|
||||
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- /opt/docker/conf/h2o-eval/front:/front:ro
|
||||
command: ["bash", "/front/run.sh"]
|
||||
volumes:
|
||||
h2o-eval-vllm-cache:
|
||||
@@ -0,0 +1,231 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
|
||||
|
||||
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
|
||||
the three things the seat contract needs and the vendor stack does not have:
|
||||
|
||||
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
|
||||
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
|
||||
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
|
||||
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
|
||||
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
|
||||
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
|
||||
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
|
||||
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
|
||||
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
|
||||
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
|
||||
the judge stamps into its cache key.
|
||||
|
||||
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
|
||||
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
|
||||
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
|
||||
"""
|
||||
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
TOKEN = os.environ["FRONT_TOKEN"]
|
||||
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
|
||||
PORT = int(os.environ.get("FRONT_PORT", "8034"))
|
||||
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
|
||||
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
|
||||
CAP = int(os.environ.get("FRONT_CAP", "32000"))
|
||||
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
|
||||
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
|
||||
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
|
||||
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
|
||||
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
|
||||
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
|
||||
MAX_QUESTIONS_PER_CALL = 16
|
||||
MIN_OPTIONS, MAX_OPTIONS = 2, 16
|
||||
|
||||
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
|
||||
shim_mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules["h2o_shim"] = shim_mod
|
||||
spec.loader.exec_module(shim_mod)
|
||||
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
|
||||
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
def __init__(self, code, obj):
|
||||
super().__init__(obj)
|
||||
self.code, self.obj = code, obj
|
||||
|
||||
|
||||
def sent_tokens(body):
|
||||
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
|
||||
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
|
||||
state = body.get("state") or ""
|
||||
if isinstance(state, (dict, list)):
|
||||
state = json.dumps(state)
|
||||
state_txt = shim_mod.render_state(state, CFG.state_compact)
|
||||
all_toks = []
|
||||
for qid, q in body.get("questions", {}).items():
|
||||
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
|
||||
req = urllib.request.Request(VLLM + "/tokenize",
|
||||
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
all_toks.append(json.loads(r.read())["tokens"])
|
||||
mx = max(len(t) for t in all_toks)
|
||||
shared = shim_mod.common_prefix_len(all_toks)
|
||||
return mx, shared + sum(len(t) - shared for t in all_toks)
|
||||
|
||||
|
||||
def admit(body):
|
||||
try:
|
||||
mx, total = sent_tokens(body)
|
||||
except Refused:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001
|
||||
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
|
||||
if mx > CAP or total > CAP:
|
||||
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
|
||||
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
|
||||
"shim_accounted_total": total, "cap": CAP})
|
||||
return mx, total
|
||||
|
||||
|
||||
def post_up(path, obj):
|
||||
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
|
||||
method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as r:
|
||||
return r.status, json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read() or b"{}"
|
||||
try:
|
||||
return e.code, json.loads(raw)
|
||||
except ValueError:
|
||||
return e.code, {"error": raw[:300].decode(errors="replace")}
|
||||
|
||||
|
||||
def systemone(body):
|
||||
mx, total = admit(body)
|
||||
code, obj = post_up("/v1/systemone", body)
|
||||
if isinstance(obj, dict):
|
||||
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
|
||||
return code, obj
|
||||
|
||||
|
||||
def check_decision(d, i):
|
||||
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
|
||||
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
|
||||
opts = d.get("options")
|
||||
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
|
||||
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
|
||||
for o in opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
|
||||
if len({o["id"] for o in opts}) != len(opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
|
||||
if d.get("orderings", "none") != "none":
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
|
||||
"seat (h2o); send orderings 'none' and combine on the client"})
|
||||
|
||||
|
||||
def decide(state, decisions):
|
||||
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
|
||||
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
|
||||
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
|
||||
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
|
||||
for i, d in enumerate(decisions):
|
||||
check_decision(d, i)
|
||||
started, results, sizes, tokens = time.perf_counter(), [], [], []
|
||||
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
|
||||
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
|
||||
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
|
||||
questions = {f: {"type": "choice", "instructions": d["question"],
|
||||
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
|
||||
code, resp = systemone({"state": state, "questions": questions})
|
||||
if code != 200:
|
||||
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
|
||||
sizes.append(len(chunk))
|
||||
tokens.append((resp.get("usage") or {}).get("input_tokens"))
|
||||
for f, d in zip(fields, chunk):
|
||||
answer = (resp.get("answers") or {}).get(f)
|
||||
ids = [o["id"] for o in d["options"]]
|
||||
probs = (answer or {}).get("probabilities") or {}
|
||||
if answer is None or any(i not in probs for i in ids):
|
||||
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
|
||||
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
|
||||
"top": answer.get("choice"), "confidence": answer.get("confidence"),
|
||||
"calibration": resp.get("calibration"), "native": answer,
|
||||
"input_tokens": tokens[-1], "model": MODEL,
|
||||
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
|
||||
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
|
||||
"questions_per_call": sizes, "input_tokens": tokens}}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
|
||||
|
||||
def _send(self, code, obj):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b)))
|
||||
self.end_headers()
|
||||
self.wfile.write(b)
|
||||
|
||||
def _authorized(self):
|
||||
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
|
||||
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/health":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
|
||||
shim = json.loads(r.read())
|
||||
except Exception as e: # noqa: BLE001
|
||||
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
|
||||
ok = bool(shim.get("ok"))
|
||||
return self._send(200 if ok else 503, {
|
||||
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
|
||||
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
|
||||
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
if path == "/v1/models":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
|
||||
return self._send(r.status, json.loads(r.read()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return self._send(e.code, {"error": "upstream"})
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0]
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
n = int(self.headers.get("Content-Length", 0) or 0)
|
||||
if n > 4 * 1024 * 1024:
|
||||
return self._send(413, {"error": "body over 4 MiB"})
|
||||
try:
|
||||
body = json.loads(self.rfile.read(n) or b"null")
|
||||
except ValueError:
|
||||
return self._send(422, {"error": "body is not JSON"})
|
||||
if not isinstance(body, dict):
|
||||
return self._send(422, {"error": "body must be a JSON object"})
|
||||
try:
|
||||
if path == "/v1/systemone":
|
||||
return self._send(*systemone(body))
|
||||
if path == "/api/alpha/decisions":
|
||||
admit(body)
|
||||
return self._send(*post_up(path, body))
|
||||
if path == "/decide/shared":
|
||||
return self._send(200, decide(body.get("state"), body.get("decisions")))
|
||||
if path == "/decide":
|
||||
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
|
||||
out = decide(body.get("state"), [d])
|
||||
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
|
||||
except Refused as r:
|
||||
return self._send(r.code, r.obj)
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
|
||||
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/bash
|
||||
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
|
||||
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
|
||||
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
|
||||
set -eu
|
||||
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
|
||||
--host 127.0.0.1 --port "$SHIM_PORT" &
|
||||
SHIM=$!
|
||||
python3 /front/front.py &
|
||||
FRONT=$!
|
||||
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
kill "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
exit 1
|
||||
@@ -1,6 +1,128 @@
|
||||
# intern-decision
|
||||
# intern-decision — the decision seat (fv-ml1 GPU 1, :8033)
|
||||
|
||||
> **LIVE since 2026-09-30 0941 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
|
||||
> **LIVE: h2o-lightning-4b since 2026-10-12 1704 PT** (Prime 1640, via albok-dev). `compose.yaml` = `compose.h2o.yaml`
|
||||
> (`switch-to-h2o.sh` copies one over the other). Nemi's judge runs the **re-fitted v2 note question**
|
||||
> (albok-nemi 0.1.18, contract 4.4 rev 1.15) at concurrency 4.
|
||||
>
|
||||
> **What happened on 2026-10-12:**
|
||||
> 1. **1311–1403, h2o's first stint, rolled back.** Under the judge's v1 question, h2o flagged 242 texts adult
|
||||
> against INC's 88, over 10,841 identical texts.
|
||||
> 2. **1428–1619, the investigation on GPU 3.** The v1 question asked two things at once: INC answered it
|
||||
> inconsistently and h2o answered it as written. The shim's floor was ruled out, with identical verdicts on and
|
||||
> off. Prime ruled "text itself".
|
||||
> 3. **Under v2 the two seats agree:** work notes INC 0/22, h2o 1/22 at 0.52; nulls 0/18 on both. fleet/memory adult
|
||||
> went 86 → 0 on INC.
|
||||
> 4. **Rollback:** `rollback-to-inc.sh`, and set the judge's concurrency back to 1 (see "Switching").
|
||||
>
|
||||
> The section headed "The live seat: Intern-Decision-4B" further down describes INC, the rollback target.
|
||||
|
||||
### The h2o seat (`compose.h2o.yaml`, live since 1704)
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) |
|
||||
| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision |
|
||||
| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` |
|
||||
| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check |
|
||||
| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` |
|
||||
| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) |
|
||||
| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below |
|
||||
|
||||
### What the front adds (`services/h2o-decision-front`)
|
||||
|
||||
- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over
|
||||
the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix
|
||||
plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit;
|
||||
refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged.
|
||||
- **The bearer token** on every path except `GET /health`.
|
||||
- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only
|
||||
production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as
|
||||
INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`,
|
||||
`calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's
|
||||
`prompt_sha256`, `readout` and timing fields are gone.
|
||||
- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and
|
||||
returns 503 when the shim is not healthy.
|
||||
- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.
|
||||
|
||||
### Verification at the 1311 cutover (2026-10-12, 1311–1318 PT, on fv-ml1 loopback)
|
||||
|
||||
| check | result | expected |
|
||||
|---|---|---|
|
||||
| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin |
|
||||
| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 |
|
||||
| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) |
|
||||
| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round |
|
||||
| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 |
|
||||
|
||||
The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below.
|
||||
|
||||
**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus
|
||||
the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/`
|
||||
and `/tank/bench/cutover-probe.json`.
|
||||
|
||||
| check | result | reference |
|
||||
|---|---|---|
|
||||
| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench |
|
||||
| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here |
|
||||
| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | |
|
||||
| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property |
|
||||
| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth |
|
||||
| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 |
|
||||
|
||||
⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV
|
||||
size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.
|
||||
|
||||
### Before any (re)create
|
||||
|
||||
GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with
|
||||
`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
|
||||
3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.
|
||||
|
||||
### Images: inline only (Prime 2026-10-12, via brokkr)
|
||||
|
||||
The h2o seat must never fetch an image URL. `compose.h2o.yaml` sets `LIGHTNING_IMAGE_URLS=off` on the front, and the
|
||||
vendor shim then refuses every image URL with 422 "image URLs are not accepted here". Inline `data:` URIs and base64
|
||||
keep working. vLLM also runs with `--allowed-media-domains media.invalid` as a backstop. That is a reserved name that
|
||||
never resolves, and vLLM 0.30.0 checks every http(s) media URL against it before fetching, while `data:` URLs bypass
|
||||
the check.
|
||||
|
||||
The shim switch was verified live at 1402, on the h2o seat with the front recreated alone:
|
||||
- the same `https://media.invalid/red.png` request went from "could not be fetched" (a fetch was attempted) to "not
|
||||
accepted here" (refused before any fetch);
|
||||
- inline red still answered red at 0.984;
|
||||
- before the change, inline red/blue/green answered at 0.98+ and no-image came out near-uniform.
|
||||
|
||||
The vLLM backstop was first exercised on the GPU 3 eval seat at 1428 (same image and flag): an https image URL
|
||||
posted straight to vLLM got 400 "The URL must be from one of the allowed domains: ['media.invalid']", with no fetch.
|
||||
Repeat that check on this seat's loopback port (`:18299`) on the next switch. INC needs neither setting, because its served checkpoint has no vision tower.
|
||||
|
||||
### Switching between the two
|
||||
|
||||
The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the
|
||||
pin, with floor 0.801; `/decide/shared` 200; a 422 over the cap; inline red → red at 0.984; an https image URL refused
|
||||
by the shim; the vLLM backstop refusing example.com (400); GPU 1 resting at 14,430 MiB.
|
||||
|
||||
Both run ON fv-ml1 and share one `.env`.
|
||||
- **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has
|
||||
16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies
|
||||
`compose.h2o.yaml` over `compose.yaml` and waits up to 300 s for `/health`. Only do this once albok has re-fitted
|
||||
the judge.
|
||||
- **To INC:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It stops h2o, copies `compose.inc.yaml`
|
||||
over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its
|
||||
`intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356.
|
||||
|
||||
After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is h2o since 1704).
|
||||
|
||||
⚠ **Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat.** INC scores on one inference
|
||||
thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's `nemi.yaml`, since
|
||||
2026-10-12). With 4 in flight on INC, the 1533–1619 walk queued behind itself and lost 339 calls to 30 s timeouts. On
|
||||
h2o, vLLM batches, so it goes back to 4.
|
||||
|
||||
---
|
||||
|
||||
# The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and 1403–1702; now the rollback target)
|
||||
|
||||
> **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
|
||||
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback.
|
||||
|
||||
**Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
|
||||
@@ -293,7 +415,7 @@ Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`.
|
||||
Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and
|
||||
re-measure the VRAM before changing the cap.
|
||||
|
||||
## Rollback
|
||||
## Rollback (INC → semif)
|
||||
|
||||
`semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word:
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh
|
||||
# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge
|
||||
# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8).
|
||||
# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
|
||||
# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
|
||||
# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
|
||||
# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
|
||||
# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
|
||||
# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
|
||||
# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's
|
||||
# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes.
|
||||
name: intern-decision
|
||||
services:
|
||||
vllm:
|
||||
image: vllm/vllm-openai:v0.30.0
|
||||
container_name: intern-decision-vllm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:18299:8000"
|
||||
environment:
|
||||
VLLM_USE_FLASHINFER_SAMPLER: "0"
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- vllm-cache:/root/.cache/vllm
|
||||
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
|
||||
"--max-model-len", "40960", "--max-num-seqs", "16",
|
||||
"--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
|
||||
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
|
||||
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
|
||||
"--allowed-media-domains", "media.invalid"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["${GPU_ID:-1}"]
|
||||
capabilities: [gpu]
|
||||
front:
|
||||
image: python:3.12-slim
|
||||
container_name: intern-decision
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
depends_on: [vllm]
|
||||
environment:
|
||||
FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
SHIM_VLLM: http://127.0.0.1:18299
|
||||
SHIM_PORT: "18291"
|
||||
SHIM_NOUL_FLOOR: "0.801"
|
||||
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
|
||||
FRONT_SHIM: http://127.0.0.1:18291
|
||||
FRONT_VLLM: http://127.0.0.1:18299
|
||||
FRONT_BIND: 0.0.0.0
|
||||
FRONT_PORT: ${PORT:-8033}
|
||||
FRONT_CAP: "32000"
|
||||
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
|
||||
FRONT_SERVE_CONFIG: /model/serve_config.json
|
||||
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- /opt/docker/conf/intern-decision/front:/front:ro
|
||||
command: ["bash", "/front/run.sh"]
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 300s
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=Decision seat (h2o-lightning-4b)
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
|
||||
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
|
||||
volumes:
|
||||
vllm-cache:
|
||||
@@ -0,0 +1,74 @@
|
||||
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
|
||||
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
|
||||
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
|
||||
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
|
||||
#
|
||||
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
|
||||
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
|
||||
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
|
||||
# fv-ml1 from that dir.
|
||||
#
|
||||
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
|
||||
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
|
||||
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
|
||||
# gets 503 out_of_memory and the service stays up. See the README before changing it.
|
||||
#
|
||||
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
|
||||
# (vault intern-decision/api-token, >= 32 chars).
|
||||
|
||||
name: intern-decision
|
||||
|
||||
services:
|
||||
intern-decision:
|
||||
image: ${IMAGE:?set IMAGE}
|
||||
container_name: intern-decision
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${PORT:-8033}:8000"
|
||||
environment:
|
||||
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
INTERN_DECISION_DEVICE: cuda
|
||||
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
|
||||
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
|
||||
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
|
||||
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
|
||||
# POSTs in progress (queued + scoring) before new ones get 429 busy.
|
||||
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
|
||||
volumes:
|
||||
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
|
||||
- /tank/aimodels/huggingface:/hf:ro
|
||||
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
|
||||
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
|
||||
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
|
||||
- triton-cache:/tmp/triton-cache
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["${GPU_ID:-1}"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=Intern-Decision — typed decisions
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
|
||||
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
# -> intern-decision_triton-cache on the host
|
||||
triton-cache:
|
||||
@@ -1,46 +1,33 @@
|
||||
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
|
||||
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
|
||||
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
|
||||
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
|
||||
#
|
||||
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
|
||||
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
|
||||
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
|
||||
# fv-ml1 from that dir.
|
||||
#
|
||||
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
|
||||
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
|
||||
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
|
||||
# gets 503 out_of_memory and the service stays up. See the README before changing it.
|
||||
#
|
||||
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
|
||||
# (vault intern-decision/api-token, >= 32 chars).
|
||||
|
||||
# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh
|
||||
# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge
|
||||
# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8).
|
||||
# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
|
||||
# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
|
||||
# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
|
||||
# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
|
||||
# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
|
||||
# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
|
||||
# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's
|
||||
# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes.
|
||||
name: intern-decision
|
||||
|
||||
services:
|
||||
intern-decision:
|
||||
image: ${IMAGE:?set IMAGE}
|
||||
container_name: intern-decision
|
||||
vllm:
|
||||
image: vllm/vllm-openai:v0.30.0
|
||||
container_name: intern-decision-vllm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${PORT:-8033}:8000"
|
||||
- "127.0.0.1:18299:8000"
|
||||
environment:
|
||||
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
INTERN_DECISION_DEVICE: cuda
|
||||
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
|
||||
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
|
||||
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
|
||||
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
|
||||
# POSTs in progress (queued + scoring) before new ones get 429 busy.
|
||||
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
|
||||
VLLM_USE_FLASHINFER_SAMPLER: "0"
|
||||
volumes:
|
||||
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
|
||||
- /tank/aimodels/huggingface:/hf:ro
|
||||
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
|
||||
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
|
||||
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
|
||||
- triton-cache:/tmp/triton-cache
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- vllm-cache:/root/.cache/vllm
|
||||
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
|
||||
"--max-model-len", "40960", "--max-num-seqs", "16",
|
||||
"--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
|
||||
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
|
||||
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
|
||||
"--allowed-media-domains", "media.invalid"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
@@ -48,27 +35,41 @@ services:
|
||||
- driver: nvidia
|
||||
device_ids: ["${GPU_ID:-1}"]
|
||||
capabilities: [gpu]
|
||||
front:
|
||||
image: python:3.12-slim
|
||||
container_name: intern-decision
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
depends_on: [vllm]
|
||||
environment:
|
||||
FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
SHIM_VLLM: http://127.0.0.1:18299
|
||||
SHIM_PORT: "18291"
|
||||
SHIM_NOUL_FLOOR: "0.801"
|
||||
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
|
||||
FRONT_SHIM: http://127.0.0.1:18291
|
||||
FRONT_VLLM: http://127.0.0.1:18299
|
||||
FRONT_BIND: 0.0.0.0
|
||||
FRONT_PORT: ${PORT:-8033}
|
||||
FRONT_CAP: "32000"
|
||||
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
|
||||
FRONT_SERVE_CONFIG: /model/serve_config.json
|
||||
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- /opt/docker/conf/intern-decision/front:/front:ro
|
||||
command: ["bash", "/front/run.sh"]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=Intern-Decision — typed decisions
|
||||
- homepage.name=Decision seat (h2o-lightning-4b)
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
|
||||
- homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
|
||||
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
# -> intern-decision_triton-cache on the host
|
||||
triton-cache:
|
||||
vllm-cache:
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
|
||||
|
||||
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
|
||||
the three things the seat contract needs and the vendor stack does not have:
|
||||
|
||||
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
|
||||
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
|
||||
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
|
||||
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
|
||||
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
|
||||
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
|
||||
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
|
||||
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
|
||||
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
|
||||
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
|
||||
the judge stamps into its cache key.
|
||||
|
||||
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
|
||||
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
|
||||
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
|
||||
"""
|
||||
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
TOKEN = os.environ["FRONT_TOKEN"]
|
||||
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
|
||||
PORT = int(os.environ.get("FRONT_PORT", "8034"))
|
||||
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
|
||||
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
|
||||
CAP = int(os.environ.get("FRONT_CAP", "32000"))
|
||||
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
|
||||
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
|
||||
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
|
||||
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
|
||||
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
|
||||
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
|
||||
MAX_QUESTIONS_PER_CALL = 16
|
||||
MIN_OPTIONS, MAX_OPTIONS = 2, 16
|
||||
|
||||
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
|
||||
shim_mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules["h2o_shim"] = shim_mod
|
||||
spec.loader.exec_module(shim_mod)
|
||||
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
|
||||
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
def __init__(self, code, obj):
|
||||
super().__init__(obj)
|
||||
self.code, self.obj = code, obj
|
||||
|
||||
|
||||
def sent_tokens(body):
|
||||
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
|
||||
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
|
||||
state = body.get("state") or ""
|
||||
if isinstance(state, (dict, list)):
|
||||
state = json.dumps(state)
|
||||
state_txt = shim_mod.render_state(state, CFG.state_compact)
|
||||
all_toks = []
|
||||
for qid, q in body.get("questions", {}).items():
|
||||
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
|
||||
req = urllib.request.Request(VLLM + "/tokenize",
|
||||
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
all_toks.append(json.loads(r.read())["tokens"])
|
||||
mx = max(len(t) for t in all_toks)
|
||||
shared = shim_mod.common_prefix_len(all_toks)
|
||||
return mx, shared + sum(len(t) - shared for t in all_toks)
|
||||
|
||||
|
||||
def admit(body):
|
||||
try:
|
||||
mx, total = sent_tokens(body)
|
||||
except Refused:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001
|
||||
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
|
||||
if mx > CAP or total > CAP:
|
||||
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
|
||||
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
|
||||
"shim_accounted_total": total, "cap": CAP})
|
||||
return mx, total
|
||||
|
||||
|
||||
def post_up(path, obj):
|
||||
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
|
||||
method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as r:
|
||||
return r.status, json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read() or b"{}"
|
||||
try:
|
||||
return e.code, json.loads(raw)
|
||||
except ValueError:
|
||||
return e.code, {"error": raw[:300].decode(errors="replace")}
|
||||
|
||||
|
||||
def systemone(body):
|
||||
mx, total = admit(body)
|
||||
code, obj = post_up("/v1/systemone", body)
|
||||
if isinstance(obj, dict):
|
||||
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
|
||||
return code, obj
|
||||
|
||||
|
||||
def check_decision(d, i):
|
||||
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
|
||||
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
|
||||
opts = d.get("options")
|
||||
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
|
||||
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
|
||||
for o in opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
|
||||
if len({o["id"] for o in opts}) != len(opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
|
||||
if d.get("orderings", "none") != "none":
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
|
||||
"seat (h2o); send orderings 'none' and combine on the client"})
|
||||
|
||||
|
||||
def decide(state, decisions):
|
||||
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
|
||||
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
|
||||
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
|
||||
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
|
||||
for i, d in enumerate(decisions):
|
||||
check_decision(d, i)
|
||||
started, results, sizes, tokens = time.perf_counter(), [], [], []
|
||||
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
|
||||
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
|
||||
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
|
||||
questions = {f: {"type": "choice", "instructions": d["question"],
|
||||
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
|
||||
code, resp = systemone({"state": state, "questions": questions})
|
||||
if code != 200:
|
||||
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
|
||||
sizes.append(len(chunk))
|
||||
tokens.append((resp.get("usage") or {}).get("input_tokens"))
|
||||
for f, d in zip(fields, chunk):
|
||||
answer = (resp.get("answers") or {}).get(f)
|
||||
ids = [o["id"] for o in d["options"]]
|
||||
probs = (answer or {}).get("probabilities") or {}
|
||||
if answer is None or any(i not in probs for i in ids):
|
||||
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
|
||||
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
|
||||
"top": answer.get("choice"), "confidence": answer.get("confidence"),
|
||||
"calibration": resp.get("calibration"), "native": answer,
|
||||
"input_tokens": tokens[-1], "model": MODEL,
|
||||
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
|
||||
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
|
||||
"questions_per_call": sizes, "input_tokens": tokens}}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
|
||||
|
||||
def _send(self, code, obj):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b)))
|
||||
self.end_headers()
|
||||
self.wfile.write(b)
|
||||
|
||||
def _authorized(self):
|
||||
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
|
||||
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/health":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
|
||||
shim = json.loads(r.read())
|
||||
except Exception as e: # noqa: BLE001
|
||||
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
|
||||
ok = bool(shim.get("ok"))
|
||||
return self._send(200 if ok else 503, {
|
||||
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
|
||||
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
|
||||
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
if path == "/v1/models":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
|
||||
return self._send(r.status, json.loads(r.read()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return self._send(e.code, {"error": "upstream"})
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0]
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
n = int(self.headers.get("Content-Length", 0) or 0)
|
||||
if n > 4 * 1024 * 1024:
|
||||
return self._send(413, {"error": "body over 4 MiB"})
|
||||
try:
|
||||
body = json.loads(self.rfile.read(n) or b"null")
|
||||
except ValueError:
|
||||
return self._send(422, {"error": "body is not JSON"})
|
||||
if not isinstance(body, dict):
|
||||
return self._send(422, {"error": "body must be a JSON object"})
|
||||
try:
|
||||
if path == "/v1/systemone":
|
||||
return self._send(*systemone(body))
|
||||
if path == "/api/alpha/decisions":
|
||||
admit(body)
|
||||
return self._send(*post_up(path, body))
|
||||
if path == "/decide/shared":
|
||||
return self._send(200, decide(body.get("state"), body.get("decisions")))
|
||||
if path == "/decide":
|
||||
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
|
||||
out = decide(body.get("state"), [d])
|
||||
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
|
||||
except Refused as r:
|
||||
return self._send(r.code, r.obj)
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
|
||||
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/bash
|
||||
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
|
||||
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
|
||||
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
|
||||
set -eu
|
||||
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
|
||||
--host 127.0.0.1 --port "$SHIM_PORT" &
|
||||
SHIM=$!
|
||||
python3 /front/front.py &
|
||||
FRONT=$!
|
||||
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
kill "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
exit 1
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
# One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1:
|
||||
# sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh
|
||||
# It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy).
|
||||
# Used 2026-10-12 1403. The h2o compose is kept as compose.h2o.yaml (canonical in eshpfi); switch-to-h2o.sh goes back.
|
||||
set -euo pipefail
|
||||
cd /opt/docker/compose/intern-decision
|
||||
docker compose down
|
||||
cp compose.inc.yaml compose.yaml
|
||||
docker compose up -d
|
||||
for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done
|
||||
echo "INC not healthy after 180 s"; exit 1
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/bin/bash
|
||||
# Switch the decision seat to h2o-lightning-4b (the staged compose.h2o.yaml), run ON fv-ml1:
|
||||
# sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh
|
||||
# Only once albok-dev has re-fitted Nemi's judge for h2o (2026-10-12 rollback; see the stack README).
|
||||
# It stops INC, checks GPU 1 has room for h2o's measured peak, and either starts h2o or puts INC back.
|
||||
# h2o takes ~3 min to healthy (graph capture). Undo with rollback-to-inc.sh.
|
||||
set -euo pipefail
|
||||
cd /opt/docker/compose/intern-decision
|
||||
NEED_MIB=16900 # h2o seat budget 16,847 MiB (peak 16,164 under 4 x 16q x 32k on 2026-10-12)
|
||||
docker compose down
|
||||
FREE=$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits | tr -d ' ')
|
||||
if [ "$FREE" -lt "$NEED_MIB" ]; then
|
||||
echo "GPU 1 free ${FREE} MiB < ${NEED_MIB}: not starting h2o; restarting INC"; docker compose up -d; exit 2
|
||||
fi
|
||||
cp compose.h2o.yaml compose.yaml
|
||||
docker compose up -d
|
||||
for i in $(seq 1 100); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "h2o healthy"; exit 0; }; sleep 3; done
|
||||
echo "h2o not healthy after 300 s: run rollback-to-inc.sh"; exit 1
|
||||
@@ -0,0 +1,24 @@
|
||||
# h2o admission proxy + gate probe set
|
||||
|
||||
Production-relevant because the seat swap to h2o-lightning-4b (ordered
|
||||
2026-10-12) inherits a hard constraint: the shipped vendor shim SILENTLY
|
||||
trims state at 32,000 tokens. The seat contract is a REFUSING 32,000-token
|
||||
seat, and this proxy is what makes the refusal real. It must sit in front of
|
||||
the shim on any deployment.
|
||||
|
||||
- `admission_proxy.py` — thin wrapper in front of the unmodified shipped
|
||||
`h2o_lightning_shim.py`. Imports the vendor's own renderers, counts each
|
||||
question's exact prompt with vLLM's `/tokenize`, and refuses over-cap with
|
||||
422 + the named count (both currencies: per-question prompt max and the
|
||||
shim's shared-head+tails total). Env: ADMIT_CAP (default 32000), ADMIT_UP
|
||||
(shim URL), ADMIT_VLLM, ADMIT_PORT, ADMIT_SHIM (path to the vendor file).
|
||||
Measured gate overhead: the shape numbers pass with it in-line.
|
||||
- `gate_probe_set.py` — the Amendment 1 confirmation driver (calibrate to the
|
||||
cap, depth-gate cells, cold/warm shapes, 4x concurrency, over-cap refusal).
|
||||
Written for the bench layout on fv-ml1 (`/tank/bench`); paths are hardcoded,
|
||||
adapt before re-running anywhere else.
|
||||
|
||||
Provenance: fv-ml1:/tank/bench/h2o_admission.py @ the confirmation run,
|
||||
results in docs/pfi/decision-sweep-followon-bench-2026-10-12.md (Amendment 1
|
||||
addendum). Measured boot for capacity planning: ~154 s launch-to-ready,
|
||||
graph-capture bound (no warm-cache or eager-mode saving).
|
||||
@@ -0,0 +1,92 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Thin admission wrapper for the h2o confirmation run (Amendment 1, brokkr 63d7a1b).
|
||||
|
||||
Sits IN FRONT of the unmodified shipped h2o_lightning_shim.py and refuses over-cap
|
||||
input loudly, mirroring INC's refusal. Counts with h2o's OWN tokenizer: renders the
|
||||
state exactly as the shim does (imported verbatim from the shipped file) and asks
|
||||
vLLM's /tokenize for the count. Refuses > CAP tokens with an explicit 422; forwards
|
||||
everything else untouched. No trim. The shim is not modified.
|
||||
"""
|
||||
import importlib.util, json, os, sys, time, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
CAP = int(os.environ.get("ADMIT_CAP", "32000"))
|
||||
UP = os.environ.get("ADMIT_UP", "http://127.0.0.1:18091") # the shim
|
||||
VLLM = os.environ.get("ADMIT_VLLM", "http://127.0.0.1:18099")
|
||||
PORT = int(os.environ.get("ADMIT_PORT", "18092"))
|
||||
SHIM_PATH = os.environ.get("ADMIT_SHIM", "/tank/bench/pkg-h2o/h2o_lightning_shim.py")
|
||||
|
||||
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_PATH)
|
||||
shim_mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules["h2o_shim"] = shim_mod
|
||||
spec.loader.exec_module(shim_mod)
|
||||
|
||||
CFG = shim_mod.Contract(json.load(open("/tank/bench/pkg-h2o/serve_config.json")))
|
||||
CFG.max_state_tokens = 10**9 # wrapper must count the UNTRUNCATED render; the shim's own trim is what we police
|
||||
|
||||
def sent_tokens(body):
|
||||
"""Tokens per prompt as vLLM will actually receive them, counted by vLLM's own tokenizer via
|
||||
the shim's own renderers (chat path: tokenize over messages, matching the image-free prompt
|
||||
the shim sends per question). Returns (max_per_prompt, shim_accounted_total).
|
||||
The shim accounts usage as shared_prefix + sum(question tails); we mirror that exactly."""
|
||||
import concurrent.futures
|
||||
state = body.get("state") or ""
|
||||
if isinstance(state, dict):
|
||||
state = json.dumps(state)
|
||||
state_txt = shim_mod.render_state(state, CFG.state_compact) # UNTRUNCATED: what admission guards
|
||||
all_toks = []
|
||||
for qid, q in body.get("questions", {}).items():
|
||||
prompt_text = CFG.question_prompt(state_txt, q, qid)[0] # EXACTLY what the shim sends per question
|
||||
req = urllib.request.Request(VLLM + "/tokenize",
|
||||
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
all_toks.append(json.loads(r.read())["tokens"])
|
||||
mx = max(len(t) for t in all_toks)
|
||||
shared = shim_mod.common_prefix_len(all_toks)
|
||||
total = shared + sum(len(t) - shared for t in all_toks) # the shim's own accounting: head + tails
|
||||
return mx, total
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
def log_message(self, *a): pass
|
||||
def _send(self, code, obj):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code); self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b))); self.end_headers(); self.wfile.write(b)
|
||||
def do_GET(self):
|
||||
try:
|
||||
req = urllib.request.Request(UP + self.path)
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
self._send(r.status, json.loads(r.read()))
|
||||
except urllib.error.HTTPError as e:
|
||||
self._send(e.code, json.loads(e.read() or b"{}"))
|
||||
def do_POST(self):
|
||||
n = int(self.headers.get("Content-Length", 0))
|
||||
raw = self.rfile.read(n)
|
||||
mx = total = None
|
||||
if self.path.split("?")[0] in ("/v1/systemone", "/api/alpha/decisions"):
|
||||
try:
|
||||
body = json.loads(raw)
|
||||
except ValueError:
|
||||
return self._send(422, {"error": "admission: body is not JSON"})
|
||||
try:
|
||||
mx, total = sent_tokens(body)
|
||||
except Exception as e:
|
||||
return self._send(502, {"error": f"admission: tokenizer count failed: {e}"})
|
||||
if mx > CAP or total > CAP:
|
||||
return self._send(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the "
|
||||
f"{CAP}-token seat limit; refused, not trimmed",
|
||||
"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
|
||||
req = urllib.request.Request(UP + self.path, data=raw,
|
||||
headers={"Content-Type": "application/json"}, method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as r:
|
||||
obj = json.loads(r.read())
|
||||
obj.setdefault("_admission", {})["max_prompt_tokens"] = mx if "mx" in dir() else None
|
||||
obj["_admission"]["shim_accounted_total"] = total if "total" in dir() else None
|
||||
self._send(r.status, obj)
|
||||
except urllib.error.HTTPError as e:
|
||||
self._send(e.code, json.loads(e.read() or b"{}"))
|
||||
|
||||
ThreadingHTTPServer(("127.0.0.1", PORT), H).serve_forever()
|
||||
@@ -0,0 +1,150 @@
|
||||
#!/usr/bin/env python3
|
||||
# Amendment 1 confirmation driver for h2o (brokkr 63d7a1b). Stack assumed up:
|
||||
# vLLM :18099, shipped shim :18091, admission wrapper :18092.
|
||||
# Shape sizes are pinned by the WRAPPER's own count (max per-prompt prompt tokens, pre-trim):
|
||||
# the shim's usage.input_tokens reports post-trim and is not trustworthy at the cap.
|
||||
# The gate's filler budget was written for INC's ~3.6 chars/token; h2o's tokenizer runs ~5.1,
|
||||
# so every builder target is iteratively rescaled against the wrapper count.
|
||||
import json, os, sys, time, statistics, subprocess, threading, random, urllib.request, concurrent.futures
|
||||
sys.path.insert(0, "/tank/bench")
|
||||
from gate_longctx_s1 import build_state
|
||||
CAP = 32000
|
||||
MODEL = "h2oai/h2o-lightning-4b"
|
||||
URL = "http://127.0.0.1:18092" # through the admission wrapper
|
||||
OUT = "/tank/bench/arm-h2o"
|
||||
TAG = os.environ.get("CONFIRM_TAG","A")
|
||||
rng = random.Random("h2o-confirm-2026-10-12")
|
||||
|
||||
def post(body, timeout=300):
|
||||
req = urllib.request.Request(URL+"/v1/systemone", data=json.dumps(body).encode(),
|
||||
headers={"Content-Type":"application/json"})
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return json.loads(r.read())
|
||||
def topper(r, qid):
|
||||
a = (r.get("answers") or {}).get(qid) or {}
|
||||
p = a.get("probabilities") or {}
|
||||
return max(p, key=lambda k: float(p[k])) if p else None
|
||||
def questions(nq):
|
||||
return {f"q{i}": {"type":"choice","instructions":f"Does the state satisfy criterion {i}?",
|
||||
"criteria":{"yes":f"topic-{i} mentioned","no":"not mentioned"}} for i in range(nq)}
|
||||
def adm(body):
|
||||
r = post(body)
|
||||
return r, (r.get("_admission") or {})
|
||||
EVID=("DECISIVE: the underwriter's signed memo of 2026-03-14 states the claim was established "
|
||||
"by documents filed before the deadline; the renewal decision must rest on this memo.")
|
||||
GQ={"type":"choice","instructions":"Based only on the state above, does the evidence establish the claim, "
|
||||
"contradict it, or is it insufficient?",
|
||||
"criteria":{"supported":"The evidence establishes the claim","contradicted":"The evidence establishes the opposite",
|
||||
"insufficient":"The evidence does not establish either"}}
|
||||
def count_gate(st):
|
||||
# wrapper counts then forwards; over-cap it REFUSES with the count in the body — either way we learn the count
|
||||
try:
|
||||
r,a = adm({"model":MODEL,"state":st,"questions":{"q1":GQ}})
|
||||
return a.get("max_prompt_tokens")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 422:
|
||||
try: return json.loads(e.read()).get("max_prompt_tokens")
|
||||
except Exception: return None
|
||||
raise
|
||||
|
||||
peak=0; stop=False
|
||||
def poll():
|
||||
global peak, stop
|
||||
while not stop:
|
||||
try: peak=max(peak,int(subprocess.run(["nvidia-smi","-i","3","--query-gpu=memory.used",
|
||||
"--format=csv,noheader,nounits"],capture_output=True,text=True).stdout.split()[0]))
|
||||
except Exception: pass
|
||||
time.sleep(0.05)
|
||||
|
||||
res={"tag":TAG}
|
||||
# ---- largest-admissible state: iterate until wrapper max-prompt is within 160 tok under CAP
|
||||
STATE_FILE = os.environ.get("CONFIRM_STATE","")
|
||||
if os.environ.get("CONFIRM_LOAD") == "1" and STATE_FILE and os.path.exists(STATE_FILE):
|
||||
state32 = open(STATE_FILE).read()
|
||||
hist=[{"note":"state loaded from file; first engine request is the first shape cell"}]
|
||||
target=None
|
||||
else:
|
||||
target=31900; hist=[]
|
||||
for it in range(8):
|
||||
st = build_state(target, None, 0, rng)
|
||||
try:
|
||||
r, a = adm({"model":MODEL,"state":st,"questions":questions(16)}) # pin the TOTAL at 16q, per clause 3
|
||||
mx = a.get("max_prompt_tokens"); tot = a.get("shim_accounted_total")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code != 422: raise
|
||||
b = json.loads(e.read()); mx = b.get("max_prompt_tokens"); tot = b.get("shim_accounted_total")
|
||||
hist.append({"iter":it,"builder_tok_target":int(target),"wrapper_max_prompt":mx,"wrapper_total":tot})
|
||||
if tot is None: raise SystemExit(str(mx))
|
||||
if tot <= CAP and (CAP-tot) <= 160: break
|
||||
target = int(target * max(0.5, min(1.3, (CAP-80)/max(tot,1))))
|
||||
if target is not None: state32 = st
|
||||
res["calibration"]=hist
|
||||
|
||||
def shape(nq, cold_expected=False):
|
||||
body={"model":MODEL,"state":state32,"questions":questions(nq)}
|
||||
t=time.perf_counter(); r=post(body); first=(time.perf_counter()-t)*1000
|
||||
xs=[first]
|
||||
for _ in range(4):
|
||||
t=time.perf_counter(); post(body); xs.append((time.perf_counter()-t)*1000)
|
||||
r2, a2 = adm(body)
|
||||
return {"nq":nq,"first_ms":round(first),"warm_median_ms":round(statistics.median(xs)),
|
||||
"cold_expected":cold_expected,
|
||||
"max_prompt_tokens":a2.get("max_prompt_tokens"),"shim_total":a2.get("shim_accounted_total"),
|
||||
"usage_input_tokens":(r2.get("usage") or {}).get("input_tokens")}
|
||||
|
||||
STATE_FILE=os.environ.get("CONFIRM_STATE","")
|
||||
if STATE_FILE: open(STATE_FILE,"w").write(state32)
|
||||
cold_first=int(os.environ.get("COLD_FIRST","0")) # which nq gets the truly-cold first call (0 = none)
|
||||
th=threading.Thread(target=poll); th.start()
|
||||
res["cells"]={}
|
||||
order=([cold_first] if cold_first in (1,16) else [1,16])
|
||||
order=order+[n for n in (1,16) if n not in order]
|
||||
for nq in order:
|
||||
res["cells"][f"{nq}q32k"]=shape(nq, cold_expected=(nq==cold_first))
|
||||
# ---- concurrency: 4x concurrent 16q at the same state
|
||||
body={"model":MODEL,"state":state32,"questions":questions(16)}
|
||||
t=time.perf_counter()
|
||||
with concurrent.futures.ThreadPoolExecutor(4) as ex:
|
||||
futs=[ex.submit(post,body) for _ in range(4)]
|
||||
oks=[]
|
||||
for f in futs:
|
||||
try: oks.append(topper(f.result(),"q0") is not None)
|
||||
except Exception: oks.append(False)
|
||||
res["concurrency_4x16q"]={"wall_ms":round((time.perf_counter()-t)*1000),"all_ok":all(oks),"per_call_ok":oks}
|
||||
# ---- gate cells: evidence at absolute h2o-token depths ~1k/8k/16k/30k, whole state pinned ~31,900
|
||||
gate=[]
|
||||
for depth_tok in (1000,8000,16000,30000):
|
||||
tgt=31900
|
||||
g=tgt*1.42; hist2=[]
|
||||
for it in range(8):
|
||||
d_scaled = int(depth_tok * (g/tgt)) # relative depth scales with the char budget
|
||||
stg = build_state(int(g/3.6), EVID, d_scaled, rng)
|
||||
mx = count_gate(stg)
|
||||
hist2.append({"chars_target":int(g),"count":mx})
|
||||
if mx is None: break
|
||||
if abs(mx-tgt)<=200: break
|
||||
g = int(g * tgt/mx)
|
||||
try:
|
||||
r,a = adm({"model":MODEL,"state":stg,"questions":{"q1":GQ}})
|
||||
cell={"depth_tok_target":depth_tok,"max_prompt_tokens":a.get("max_prompt_tokens"),
|
||||
"top":topper(r,"q1"),"usage_input_tokens":(r.get("usage") or {}).get("input_tokens"),
|
||||
"refused":False,"cal":hist2[-1]}
|
||||
except urllib.error.HTTPError as e:
|
||||
cell={"depth_tok_target":depth_tok,"refused":True,"detail":json.loads(e.read()),"cal":hist2[-1]}
|
||||
gate.append(cell)
|
||||
time.sleep(1)
|
||||
res["gate"]=gate
|
||||
stop=True; th.join()
|
||||
res["peak_MiB"]=peak
|
||||
# ---- over-cap REFUSAL (the whole point): take state32 and grow it past the cap
|
||||
st_over = state32 + ("Policy renewal review cycle X: coverage schedules, endorsements and rider terms are "
|
||||
"reconciled against the master binder. " * 400)
|
||||
refused=False; detail=None; mx_over=count_gate(st_over)
|
||||
try:
|
||||
r_ok = post({"model":MODEL,"state":st_over,"questions":{"q1":GQ}})
|
||||
detail={"note":"not refused","max_prompt_tokens":(r_ok.get("_admission") or {}).get("max_prompt_tokens")}
|
||||
except urllib.error.HTTPError as e:
|
||||
refused=True; detail={"status":e.code,"body":json.loads(e.read())}
|
||||
res["over_cap_refusal"]={"refused":refused,"counted_tokens":mx_over,"detail":detail}
|
||||
json.dump(res,open(f"{OUT}/confirm-{TAG}.json","w"),indent=1)
|
||||
print("CONFIRM-DONE", json.dumps(res)[:2400])
|
||||
Reference in New Issue
Block a user