diff --git a/persistent-memory.md b/persistent-memory.md index 3048b89..aaa63bb 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)_ +_Last updated: 2026-10-12 ~1320 PT (decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -140,7 +140,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca | 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — | | 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) | | 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× | -| 1 | intern-decision | 32,768/call | cap 14.4 GiB, peak 15,220 | — | — | +| 1 | intern-decision = **h2o-lightning-4b** since 2026-10-12 | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) | | 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× | Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`). @@ -166,10 +166,11 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- - **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file. - **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements"). -### intern-decision (replaced SemIf on 2026-09-30) +### The decision seat `intern-decision` :8033 (h2o-lightning-4b since 2026-10-12; INC 09-30 → 10-12) -- **LIVE 0.1.3** at `intern-decision.fv.internal:8033`: semif-compatible `/decide` plus Jev `/v1/systemone`, 32k tokens, a Triton cache volume. Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md` -- **Open:** label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim. +- **LIVE: h2o-lightning-4b @ b43f759b** on vLLM 0.30.0 (KV byte-pinned) behind `services/h2o-decision-front` (32k refuse-not-trim admission, bearer, `/decide` + `/decide/shared` translated onto `/v1/systemone`). Same URL, port, token and stack name. The only production caller is albok-nemi's sensitivity judge (`/decide/shared`, threshold 0.5). Rollback to INC: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh` on fv-ml1. Everything else is in `stacks/intern-decision/README.md`. +- ⚠ **Seat headroom under 4-way 32k load is 683 MiB** (peak 16,164 against a 16,847 budget, infra-hermes 10-12). Re-measure before anything else lands on GPU 1. +- INC history (semif replacement, Triton warm-up cache): → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`. Its open item (label ~50 real Wyrd/Cicada turns) is moot with INC retired. ### nh3-pve + nh3-ml1 (2026-09-25/26) @@ -178,7 +179,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ## Recent decisions -- `[2026-10-12]` **Decision-seat swap (Prime via brokkr): h2o-lightning-4b @ b43f759b replaces Intern-Decision-4B, AFTER albok-dev re-validates Nemi's sensitivity judge on it.** The swap was held at 1234: the seat's only production traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10), and h2o's shim has no /decide. TEMPORARY eval seat `stacks/h2o-eval` up since 1300 on fv-ml1 GPU 3 :8034 (~14.4 GB; Scriberr/MIA still fit beside it), with `services/h2o-decision-front` (32k refuse-not-trim admission from infra-hermes 30f36f2, bearer, the /decide/shared translator, /health model {name, revision}); weights `/tank/aimodels/h2o-lightning-4b-b43f759b` (hashes = HF). **Next, on albok-dev's sign-off (threshold, route):** GPU 1 cutover on :8033 with the same front and byte-pinned KV (budget 16,847 MiB, peak 15,566), rollback = the INC compose; post-checks (health rev, JevBench 205, 422 probe, 16q@32k ~2 s, GPU 1 peak); then `docker compose down` h2o-eval. Thread `01M4RSFC9J2R0J07EEBD4N04W3`. +- `[2026-10-12]` **Decision seat swapped to h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 on the GPU 3 eval seat). INC down 130818, h2o healthy on GPU 1 :8033 at 131127; eval seat `stacks/h2o-eval` down 1319.** Post-checks: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q@32k warm 2.0–2.16 s; 4×16q×32k seat peak 16,164 MiB against a 16,847 budget (infra-hermes). The swap was held at 1234 because the seat's only traffic is albok-nemi's `/decide/shared` judge (57,108 calls 10-05..10-10) and h2o's shim has no /decide, hence the front's translator. **Open:** albok-dev's hand re-judge walk report (started 1316, thread `01M4RVMXJQF1KXXQB6KJ34AYYX`); rollback = `rollback-to-inc.sh`. Brokkr thread `01M4RSFC9J2R0J07EEBD4N04W3`. - `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing. - `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime's call, pending:** A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against. - `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`. diff --git a/stacks/h2o-eval/README.md b/stacks/h2o-eval/README.md index 55a9abb..98396fd 100644 --- a/stacks/h2o-eval/README.md +++ b/stacks/h2o-eval/README.md @@ -1,5 +1,8 @@ # h2o-eval (TEMPORARY) — fv-ml1 GPU 3, :8034 +> **DOWN since 2026-10-12 1319 PT** (`docker compose down`), once the GPU 1 cutover's post-checks passed. The host keeps +> `/opt/docker/compose/h2o-eval` and the weights, so it can come back on GPU 3 for a transient test with `docker compose up -d`. + h2o-lightning-4b @ b43f759b behind `services/h2o-decision-front` (copied into `conf/front/`), for albok-dev's re-validation of Nemi's sensitivity judge before the GPU 1 cutover (Prime 2026-10-12 via brokkr). **Remove right after the cutover:** GPU 3 is for transient work only (Scriberr and MIA use it). diff --git a/stacks/intern-decision/README.md b/stacks/intern-decision/README.md index 1f0ad71..a12e3e1 100644 --- a/stacks/intern-decision/README.md +++ b/stacks/intern-decision/README.md @@ -1,6 +1,85 @@ -# intern-decision +# intern-decision — the decision seat (fv-ml1 GPU 1, :8033) -> **LIVE since 2026-09-30 0941 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. +> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via +> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through +> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name +> are unchanged on purpose (brokkr rule 8), so callers change nothing. +> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because +> the rollback goes back to it. + +| | | +|---|---| +| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) | +| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision | +| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` | +| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check | +| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` | +| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) | +| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below | + +### What the front adds (`services/h2o-decision-front`) + +- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over + the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix + plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit; + refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged. +- **The bearer token** on every path except `GET /health`. +- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only + production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as + INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`, + `calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's + `prompt_sha256`, `readout` and timing fields are gone. +- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and + returns 503 when the shim is not healthy. +- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is. + +### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback) + +| check | result | expected | +|---|---|---| +| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin | +| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 | +| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) | +| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round | +| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 | + +The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below. + +**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus +the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/` +and `/tank/bench/cutover-probe.json`. + +| check | result | reference | +|---|---|---| +| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench | +| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here | +| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | | +| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property | +| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth | +| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 | + +⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV +size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1. + +### Before any (re)create + +GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with +`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about +3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period. + +### Rollback to INC + +On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps +the h2o compose as `compose.yaml.h2o-`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC +(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the +`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put +the saved h2o compose back as `compose.yaml` and `up -d`. + +--- + +# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12) + +> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. > Acceptance on the live service is below. `semif` is stopped and kept as the rollback. **Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`, @@ -293,7 +372,7 @@ Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`. Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and re-measure the VRAM before changing the cap. -## Rollback +## Rollback (INC → semif) `semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word: diff --git a/stacks/intern-decision/compose.inc.yaml b/stacks/intern-decision/compose.inc.yaml new file mode 100644 index 0000000..f4c9fb9 --- /dev/null +++ b/stacks/intern-decision/compose.inc.yaml @@ -0,0 +1,74 @@ +# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on +# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to +# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). +# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). +# +# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the +# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + +# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on +# fv-ml1 from that dir. +# +# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the +# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside +# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more +# gets 503 out_of_memory and the service stays up. See the README before changing it. +# +# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN +# (vault intern-decision/api-token, >= 32 chars). + +name: intern-decision + +services: + intern-decision: + image: ${IMAGE:?set IMAGE} + container_name: intern-decision + restart: unless-stopped + ports: + - "${PORT:-8033}:8000" + environment: + INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + INTERN_DECISION_DEVICE: cuda + INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB} + # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM"). + INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS} + INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64} + # POSTs in progress (queued + scoring) before new ones get 429 busy. + INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32} + volumes: + # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. + - /tank/aimodels/huggingface:/hf:ro + # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket + # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. + # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. + - triton-cache:/tmp/triton-cache + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["${GPU_ID:-1}"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] + interval: 30s + timeout: 10s + retries: 3 + # Startup loads ~9 GB of weights and scores the warm-up three times before it serves. + start_period: 300s + networks: + - tnet + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=Intern-Decision — typed decisions + - homepage.icon=mdi-scale-balance + - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) + - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health + +networks: + tnet: + name: traefik-net + external: true + +volumes: + # -> intern-decision_triton-cache on the host + triton-cache: diff --git a/stacks/intern-decision/compose.yaml b/stacks/intern-decision/compose.yaml index f4c9fb9..9f72ee2 100644 --- a/stacks/intern-decision/compose.yaml +++ b/stacks/intern-decision/compose.yaml @@ -1,46 +1,29 @@ -# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on -# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to -# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). -# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). -# -# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the -# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + -# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on -# fv-ml1 from that dir. -# -# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the -# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside -# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more -# gets 503 out_of_memory and the service stays up. See the README before changing it. -# -# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN -# (vault intern-decision/api-token, >= 32 chars). - +# The decision seat on fv-ml1 GPU 1, address intern-decision.fv.internal:8033 (name kept for continuity, brokkr rule 8). +# Since 2026-10-12: h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 through +# /decide/shared). Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to +# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN), +# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}. +# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README. +# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size); +# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create. name: intern-decision - services: - intern-decision: - image: ${IMAGE:?set IMAGE} - container_name: intern-decision + vllm: + image: vllm/vllm-openai:v0.30.0 + container_name: intern-decision-vllm restart: unless-stopped ports: - - "${PORT:-8033}:8000" + - "127.0.0.1:18299:8000" environment: - INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} - INTERN_DECISION_DEVICE: cuda - INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB} - # Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM"). - INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS} - INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64} - # POSTs in progress (queued + scoring) before new ones get 429 busy. - INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32} + VLLM_USE_FLASHINFER_SAMPLER: "0" volumes: - # Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. - - /tank/aimodels/huggingface:/hf:ro - # Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket - # autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. - # The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. - - triton-cache:/tmp/triton-cache + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - vllm-cache:/root/.cache/vllm + command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b", + "--max-model-len", "40960", "--max-num-seqs", "16", + "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371", + "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}", + "--mm-processor-kwargs", "{\"max_pixels\": 1605632}"] deploy: resources: reservations: @@ -48,27 +31,40 @@ services: - driver: nvidia device_ids: ["${GPU_ID:-1}"] capabilities: [gpu] + front: + image: python:3.12-slim + container_name: intern-decision + restart: unless-stopped + network_mode: host + depends_on: [vllm] + environment: + FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} + SHIM_VLLM: http://127.0.0.1:18299 + SHIM_PORT: "18291" + SHIM_NOUL_FLOOR: "0.801" + FRONT_SHIM: http://127.0.0.1:18291 + FRONT_VLLM: http://127.0.0.1:18299 + FRONT_BIND: 0.0.0.0 + FRONT_PORT: ${PORT:-8033} + FRONT_CAP: "32000" + FRONT_SHIM_FILE: /model/h2o_lightning_shim.py + FRONT_SERVE_CONFIG: /model/serve_config.json + FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a + volumes: + - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro + - /opt/docker/conf/intern-decision/front:/front:ro + command: ["bash", "/front/run.sh"] healthcheck: - test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] + test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"] interval: 30s timeout: 10s retries: 3 - # Startup loads ~9 GB of weights and scores the warm-up three times before it serves. start_period: 300s - networks: - - tnet labels: - homepage.group=AI - Eval & Retrieval - - homepage.name=Intern-Decision — typed decisions + - homepage.name=Decision seat (h2o-lightning-4b) - homepage.icon=mdi-scale-balance - - homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) + - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept) - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health - -networks: - tnet: - name: traefik-net - external: true - volumes: - # -> intern-decision_triton-cache on the host - triton-cache: + vllm-cache: diff --git a/stacks/intern-decision/conf/front/front.py b/stacks/intern-decision/conf/front/front.py new file mode 100644 index 0000000..d3c04ba --- /dev/null +++ b/stacks/intern-decision/conf/front/front.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only. + +It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds +the three things the seat contract needs and the vendor stack does not have: + +1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim + silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own + tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the + largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed. +2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health). +3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes + one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question, + criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed + into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than + "none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which + the judge stamps into its cache key. + +Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000), +FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME, +FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64). +""" +import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +TOKEN = os.environ["FRONT_TOKEN"] +BIND = os.environ.get("FRONT_BIND", "127.0.0.1") +PORT = int(os.environ.get("FRONT_PORT", "8034")) +UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191") +VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199") +CAP = int(os.environ.get("FRONT_CAP", "32000")) +SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py") +SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json") +MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"), + "source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"), + "revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")} +MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64")) +MAX_QUESTIONS_PER_CALL = 16 +MIN_OPTIONS, MAX_OPTIONS = 2, 16 + +spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE) +shim_mod = importlib.util.module_from_spec(spec) +sys.modules["h2o_shim"] = shim_mod +spec.loader.exec_module(shim_mod) +CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG))) +CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices + + +class Refused(Exception): + def __init__(self, code, obj): + super().__init__(obj) + self.code, self.obj = code, obj + + +def sent_tokens(body): + """(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own + tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged).""" + state = body.get("state") or "" + if isinstance(state, (dict, list)): + state = json.dumps(state) + state_txt = shim_mod.render_state(state, CFG.state_compact) + all_toks = [] + for qid, q in body.get("questions", {}).items(): + prompt_text = CFG.question_prompt(state_txt, q, qid)[0] + req = urllib.request.Request(VLLM + "/tokenize", + data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(), + headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=30) as r: + all_toks.append(json.loads(r.read())["tokens"]) + mx = max(len(t) for t in all_toks) + shared = shim_mod.common_prefix_len(all_toks) + return mx, shared + sum(len(t) - shared for t in all_toks) + + +def admit(body): + try: + mx, total = sent_tokens(body) + except Refused: + raise + except Exception as e: # noqa: BLE001 + raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"}) + if mx > CAP or total > CAP: + raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat " + f"limit; refused, not trimmed", "max_prompt_tokens": mx, + "shim_accounted_total": total, "cap": CAP}) + return mx, total + + +def post_up(path, obj): + req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"}, + method="POST") + try: + with urllib.request.urlopen(req, timeout=300) as r: + return r.status, json.loads(r.read()) + except urllib.error.HTTPError as e: + raw = e.read() or b"{}" + try: + return e.code, json.loads(raw) + except ValueError: + return e.code, {"error": raw[:300].decode(errors="replace")} + + +def systemone(body): + mx, total = admit(body) + code, obj = post_up("/v1/systemone", body) + if isinstance(obj, dict): + obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP}) + return code, obj + + +def check_decision(d, i): + if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str): + raise Refused(422, {"error": f"decision {i}: needs string id and question"}) + opts = d.get("options") + if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS: + raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"}) + if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str) + for o in opts): + raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"}) + if len({o["id"] for o in opts}) != len(opts): + raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"}) + if d.get("orderings", "none") != "none": + raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this " + "seat (h2o); send orderings 'none' and combine on the client"}) + + +def decide(state, decisions): + """semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per + decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape.""" + if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS: + raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"}) + for i, d in enumerate(decisions): + check_decision(d, i) + started, results, sizes, tokens = time.perf_counter(), [], [], [] + for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL): + chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL] + fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))] + questions = {f: {"type": "choice", "instructions": d["question"], + "criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)} + code, resp = systemone({"state": state, "questions": questions}) + if code != 200: + raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)}) + sizes.append(len(chunk)) + tokens.append((resp.get("usage") or {}).get("input_tokens")) + for f, d in zip(fields, chunk): + answer = (resp.get("answers") or {}).get(f) + ids = [o["id"] for o in d["options"]] + probs = (answer or {}).get("probabilities") or {} + if answer is None or any(i not in probs for i in ids): + raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"}) + results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids], + "top": answer.get("choice"), "confidence": answer.get("confidence"), + "calibration": resp.get("calibration"), "native": answer, + "input_tokens": tokens[-1], "model": MODEL, + "call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}}) + return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes), + "questions_per_call": sizes, "input_tokens": tokens}} + + +class H(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, fmt, *args): + sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args)) + + def _send(self, code, obj): + b = json.dumps(obj).encode() + self.send_response(code) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(b))) + self.end_headers() + self.wfile.write(b) + + def _authorized(self): + scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ") + return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode()) + + def do_GET(self): + path = self.path.split("?")[0] + if path == "/health": + try: + with urllib.request.urlopen(UP + "/health", timeout=10) as r: + shim = json.loads(r.read()) + except Exception as e: # noqa: BLE001 + return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"}) + ok = bool(shim.get("ok")) + return self._send(200 if ok else 503, { + "status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim", + "endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS, + "max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim}) + if not self._authorized(): + return self._send(401, {"error": "unauthorized"}) + if path == "/v1/models": + try: + with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r: + return self._send(r.status, json.loads(r.read())) + except urllib.error.HTTPError as e: + return self._send(e.code, {"error": "upstream"}) + return self._send(404, {"error": "not found"}) + + def do_POST(self): + path = self.path.split("?")[0] + if not self._authorized(): + return self._send(401, {"error": "unauthorized"}) + n = int(self.headers.get("Content-Length", 0) or 0) + if n > 4 * 1024 * 1024: + return self._send(413, {"error": "body over 4 MiB"}) + try: + body = json.loads(self.rfile.read(n) or b"null") + except ValueError: + return self._send(422, {"error": "body is not JSON"}) + if not isinstance(body, dict): + return self._send(422, {"error": "body must be a JSON object"}) + try: + if path == "/v1/systemone": + return self._send(*systemone(body)) + if path == "/api/alpha/decisions": + admit(body) + return self._send(*post_up(path, body)) + if path == "/decide/shared": + return self._send(200, decide(body.get("state"), body.get("decisions"))) + if path == "/decide": + d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body} + out = decide(body.get("state"), [d]) + return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]}) + except Refused as r: + return self._send(r.code, r.obj) + return self._send(404, {"error": "not found"}) + + +ThreadingHTTPServer((BIND, PORT), H).serve_forever() diff --git a/stacks/intern-decision/conf/front/run.sh b/stacks/intern-decision/conf/front/run.sh new file mode 100755 index 0000000..a902347 --- /dev/null +++ b/stacks/intern-decision/conf/front/run.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and +# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through. +set -eu +python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \ + --host 127.0.0.1 --port "$SHIM_PORT" & +SHIM=$! +python3 /front/front.py & +FRONT=$! +wait -n "$SHIM" "$FRONT" 2>/dev/null || true +kill "$SHIM" "$FRONT" 2>/dev/null || true +exit 1 diff --git a/stacks/intern-decision/rollback-to-inc.sh b/stacks/intern-decision/rollback-to-inc.sh new file mode 100755 index 0000000..093f52f --- /dev/null +++ b/stacks/intern-decision/rollback-to-inc.sh @@ -0,0 +1,12 @@ +#!/bin/bash +# One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1: +# sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh +# It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy). +set -euo pipefail +cd /opt/docker/compose/intern-decision +docker compose down +cp compose.yaml compose.yaml.h2o-$(date +%Y%m%d-%H%M) +cp compose.inc.yaml compose.yaml +docker compose up -d +for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done +echo "INC not healthy after 180 s"; exit 1