vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(intern-decision): decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; eval seat down

Browse Source 
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with
byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front:
32k refuse-not-trim admission, the seat's bearer token, and /decide +
/decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port,
token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27.

Post-checks on the live seat: /health at the pinned revision; JevBench public
205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak
16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget.

compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step.
The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
This commit is contained in:
vh
2026-10-12 13:19:46 -07:00
parent 152137e41d
commit c08992f95f
8 changed files with 468 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stacks/intern-decision/README.md
+82 -3
View File
@@ -1,6 +1,85 @@
# intern-decision
# intern-decision — the decision seat (fv-ml1 GPU 1, :8033)
> **LIVE since 2026-09-30 0941 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via
> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through
> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name
> are unchanged on purpose (brokkr rule 8), so callers change nothing.
> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because
> the rollback goes back to it.
| | |
|---|---|
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) |
| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision |
| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` |
| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check |
| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` |
| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) |
| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below |
### What the front adds (`services/h2o-decision-front`)
- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over
the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix
plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit;
refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged.
- **The bearer token** on every path except `GET /health`.
- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only
production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as
INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`,
`calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's
`prompt_sha256`, `readout` and timing fields are gone.
- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and
returns 503 when the shim is not healthy.
- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.
### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback)
| check | result | expected |
|---|---|---|
| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin |
| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 |
| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) |
| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round |
| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 |
The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below.
**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus
the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/`
and `/tank/bench/cutover-probe.json`.
| check | result | reference |
|---|---|---|
| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench |
| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here |
| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | |
| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property |
| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth |
| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 |
⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV
size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.
### Before any (re)create
GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with
`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.
### Rollback to INC
On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps
the h2o compose as `compose.yaml.h2o-<timestamp>`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC
(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the
`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put
the saved h2o compose back as `compose.yaml` and `up -d`.
---
# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12)
> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback.
**Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
@@ -293,7 +372,7 @@ Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`.
Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and
re-measure the VRAM before changing the cap.
## Rollback
## Rollback (INC → semif)
`semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word: