feat(intern-decision): decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; eval seat down
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front: 32k refuse-not-trim admission, the seat's bearer token, and /decide + /decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port, token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27. Post-checks on the live seat: /health at the pinned revision; JevBench public 205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak 16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget. compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step. The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
This commit is contained in:
@@ -1,6 +1,85 @@
|
||||
# intern-decision
|
||||
# intern-decision — the decision seat (fv-ml1 GPU 1, :8033)
|
||||
|
||||
> **LIVE since 2026-09-30 0941 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
|
||||
> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via
|
||||
> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through
|
||||
> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name
|
||||
> are unchanged on purpose (brokkr rule 8), so callers change nothing.
|
||||
> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because
|
||||
> the rollback goes back to it.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) |
|
||||
| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision |
|
||||
| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` |
|
||||
| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check |
|
||||
| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` |
|
||||
| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) |
|
||||
| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below |
|
||||
|
||||
### What the front adds (`services/h2o-decision-front`)
|
||||
|
||||
- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over
|
||||
the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix
|
||||
plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit;
|
||||
refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged.
|
||||
- **The bearer token** on every path except `GET /health`.
|
||||
- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only
|
||||
production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as
|
||||
INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`,
|
||||
`calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's
|
||||
`prompt_sha256`, `readout` and timing fields are gone.
|
||||
- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and
|
||||
returns 503 when the shim is not healthy.
|
||||
- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.
|
||||
|
||||
### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback)
|
||||
|
||||
| check | result | expected |
|
||||
|---|---|---|
|
||||
| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin |
|
||||
| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 |
|
||||
| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) |
|
||||
| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round |
|
||||
| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 |
|
||||
|
||||
The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below.
|
||||
|
||||
**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus
|
||||
the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/`
|
||||
and `/tank/bench/cutover-probe.json`.
|
||||
|
||||
| check | result | reference |
|
||||
|---|---|---|
|
||||
| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench |
|
||||
| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here |
|
||||
| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | |
|
||||
| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property |
|
||||
| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth |
|
||||
| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 |
|
||||
|
||||
⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV
|
||||
size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.
|
||||
|
||||
### Before any (re)create
|
||||
|
||||
GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with
|
||||
`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
|
||||
3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.
|
||||
|
||||
### Rollback to INC
|
||||
|
||||
On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps
|
||||
the h2o compose as `compose.yaml.h2o-<timestamp>`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC
|
||||
(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the
|
||||
`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put
|
||||
the saved h2o compose back as `compose.yaml` and `up -d`.
|
||||
|
||||
---
|
||||
|
||||
# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12)
|
||||
|
||||
> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
|
||||
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback.
|
||||
|
||||
**Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
|
||||
@@ -293,7 +372,7 @@ Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`.
|
||||
Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and
|
||||
re-measure the VRAM before changing the cap.
|
||||
|
||||
## Rollback
|
||||
## Rollback (INC → semif)
|
||||
|
||||
`semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word:
|
||||
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
|
||||
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
|
||||
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
|
||||
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
|
||||
#
|
||||
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
|
||||
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
|
||||
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
|
||||
# fv-ml1 from that dir.
|
||||
#
|
||||
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
|
||||
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
|
||||
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
|
||||
# gets 503 out_of_memory and the service stays up. See the README before changing it.
|
||||
#
|
||||
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
|
||||
# (vault intern-decision/api-token, >= 32 chars).
|
||||
|
||||
name: intern-decision
|
||||
|
||||
services:
|
||||
intern-decision:
|
||||
image: ${IMAGE:?set IMAGE}
|
||||
container_name: intern-decision
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${PORT:-8033}:8000"
|
||||
environment:
|
||||
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
INTERN_DECISION_DEVICE: cuda
|
||||
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
|
||||
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
|
||||
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
|
||||
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
|
||||
# POSTs in progress (queued + scoring) before new ones get 429 busy.
|
||||
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
|
||||
volumes:
|
||||
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
|
||||
- /tank/aimodels/huggingface:/hf:ro
|
||||
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
|
||||
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
|
||||
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
|
||||
- triton-cache:/tmp/triton-cache
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids: ["${GPU_ID:-1}"]
|
||||
capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=Intern-Decision — typed decisions
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
|
||||
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
# -> intern-decision_triton-cache on the host
|
||||
triton-cache:
|
||||
@@ -1,46 +1,29 @@
|
||||
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
|
||||
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
|
||||
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
|
||||
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
|
||||
#
|
||||
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
|
||||
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
|
||||
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
|
||||
# fv-ml1 from that dir.
|
||||
#
|
||||
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
|
||||
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
|
||||
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
|
||||
# gets 503 out_of_memory and the service stays up. See the README before changing it.
|
||||
#
|
||||
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
|
||||
# (vault intern-decision/api-token, >= 32 chars).
|
||||
|
||||
# The decision seat on fv-ml1 GPU 1, address intern-decision.fv.internal:8033 (name kept for continuity, brokkr rule 8).
|
||||
# Since 2026-10-12: h2o-lightning-4b @ b43f759b (Prime via brokkr; albok-dev signed off Nemi's judge at 0.5 through
|
||||
# /decide/shared). Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
|
||||
# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
|
||||
# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
|
||||
# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
|
||||
# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
|
||||
# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
|
||||
name: intern-decision
|
||||
|
||||
services:
|
||||
intern-decision:
|
||||
image: ${IMAGE:?set IMAGE}
|
||||
container_name: intern-decision
|
||||
vllm:
|
||||
image: vllm/vllm-openai:v0.30.0
|
||||
container_name: intern-decision-vllm
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "${PORT:-8033}:8000"
|
||||
- "127.0.0.1:18299:8000"
|
||||
environment:
|
||||
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
INTERN_DECISION_DEVICE: cuda
|
||||
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
|
||||
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
|
||||
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
|
||||
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
|
||||
# POSTs in progress (queued + scoring) before new ones get 429 busy.
|
||||
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
|
||||
VLLM_USE_FLASHINFER_SAMPLER: "0"
|
||||
volumes:
|
||||
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
|
||||
- /tank/aimodels/huggingface:/hf:ro
|
||||
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
|
||||
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
|
||||
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
|
||||
- triton-cache:/tmp/triton-cache
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- vllm-cache:/root/.cache/vllm
|
||||
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
|
||||
"--max-model-len", "40960", "--max-num-seqs", "16",
|
||||
"--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
|
||||
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
|
||||
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}"]
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
@@ -48,27 +31,40 @@ services:
|
||||
- driver: nvidia
|
||||
device_ids: ["${GPU_ID:-1}"]
|
||||
capabilities: [gpu]
|
||||
front:
|
||||
image: python:3.12-slim
|
||||
container_name: intern-decision
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
depends_on: [vllm]
|
||||
environment:
|
||||
FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
|
||||
SHIM_VLLM: http://127.0.0.1:18299
|
||||
SHIM_PORT: "18291"
|
||||
SHIM_NOUL_FLOOR: "0.801"
|
||||
FRONT_SHIM: http://127.0.0.1:18291
|
||||
FRONT_VLLM: http://127.0.0.1:18299
|
||||
FRONT_BIND: 0.0.0.0
|
||||
FRONT_PORT: ${PORT:-8033}
|
||||
FRONT_CAP: "32000"
|
||||
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
|
||||
FRONT_SERVE_CONFIG: /model/serve_config.json
|
||||
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
|
||||
volumes:
|
||||
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
|
||||
- /opt/docker/conf/intern-decision/front:/front:ro
|
||||
command: ["bash", "/front/run.sh"]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
|
||||
start_period: 300s
|
||||
networks:
|
||||
- tnet
|
||||
labels:
|
||||
- homepage.group=AI - Eval & Retrieval
|
||||
- homepage.name=Intern-Decision — typed decisions
|
||||
- homepage.name=Decision seat (h2o-lightning-4b)
|
||||
- homepage.icon=mdi-scale-balance
|
||||
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
|
||||
- homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
|
||||
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
|
||||
volumes:
|
||||
# -> intern-decision_triton-cache on the host
|
||||
triton-cache:
|
||||
vllm-cache:
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
|
||||
|
||||
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
|
||||
the three things the seat contract needs and the vendor stack does not have:
|
||||
|
||||
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
|
||||
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
|
||||
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
|
||||
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
|
||||
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
|
||||
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
|
||||
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
|
||||
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
|
||||
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
|
||||
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
|
||||
the judge stamps into its cache key.
|
||||
|
||||
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
|
||||
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
|
||||
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
|
||||
"""
|
||||
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
TOKEN = os.environ["FRONT_TOKEN"]
|
||||
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
|
||||
PORT = int(os.environ.get("FRONT_PORT", "8034"))
|
||||
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
|
||||
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
|
||||
CAP = int(os.environ.get("FRONT_CAP", "32000"))
|
||||
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
|
||||
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
|
||||
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
|
||||
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
|
||||
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
|
||||
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
|
||||
MAX_QUESTIONS_PER_CALL = 16
|
||||
MIN_OPTIONS, MAX_OPTIONS = 2, 16
|
||||
|
||||
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
|
||||
shim_mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules["h2o_shim"] = shim_mod
|
||||
spec.loader.exec_module(shim_mod)
|
||||
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
|
||||
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
|
||||
|
||||
|
||||
class Refused(Exception):
|
||||
def __init__(self, code, obj):
|
||||
super().__init__(obj)
|
||||
self.code, self.obj = code, obj
|
||||
|
||||
|
||||
def sent_tokens(body):
|
||||
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
|
||||
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
|
||||
state = body.get("state") or ""
|
||||
if isinstance(state, (dict, list)):
|
||||
state = json.dumps(state)
|
||||
state_txt = shim_mod.render_state(state, CFG.state_compact)
|
||||
all_toks = []
|
||||
for qid, q in body.get("questions", {}).items():
|
||||
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
|
||||
req = urllib.request.Request(VLLM + "/tokenize",
|
||||
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
all_toks.append(json.loads(r.read())["tokens"])
|
||||
mx = max(len(t) for t in all_toks)
|
||||
shared = shim_mod.common_prefix_len(all_toks)
|
||||
return mx, shared + sum(len(t) - shared for t in all_toks)
|
||||
|
||||
|
||||
def admit(body):
|
||||
try:
|
||||
mx, total = sent_tokens(body)
|
||||
except Refused:
|
||||
raise
|
||||
except Exception as e: # noqa: BLE001
|
||||
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
|
||||
if mx > CAP or total > CAP:
|
||||
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
|
||||
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
|
||||
"shim_accounted_total": total, "cap": CAP})
|
||||
return mx, total
|
||||
|
||||
|
||||
def post_up(path, obj):
|
||||
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
|
||||
method="POST")
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=300) as r:
|
||||
return r.status, json.loads(r.read())
|
||||
except urllib.error.HTTPError as e:
|
||||
raw = e.read() or b"{}"
|
||||
try:
|
||||
return e.code, json.loads(raw)
|
||||
except ValueError:
|
||||
return e.code, {"error": raw[:300].decode(errors="replace")}
|
||||
|
||||
|
||||
def systemone(body):
|
||||
mx, total = admit(body)
|
||||
code, obj = post_up("/v1/systemone", body)
|
||||
if isinstance(obj, dict):
|
||||
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
|
||||
return code, obj
|
||||
|
||||
|
||||
def check_decision(d, i):
|
||||
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
|
||||
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
|
||||
opts = d.get("options")
|
||||
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
|
||||
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
|
||||
for o in opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
|
||||
if len({o["id"] for o in opts}) != len(opts):
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
|
||||
if d.get("orderings", "none") != "none":
|
||||
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
|
||||
"seat (h2o); send orderings 'none' and combine on the client"})
|
||||
|
||||
|
||||
def decide(state, decisions):
|
||||
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
|
||||
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
|
||||
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
|
||||
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
|
||||
for i, d in enumerate(decisions):
|
||||
check_decision(d, i)
|
||||
started, results, sizes, tokens = time.perf_counter(), [], [], []
|
||||
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
|
||||
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
|
||||
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
|
||||
questions = {f: {"type": "choice", "instructions": d["question"],
|
||||
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
|
||||
code, resp = systemone({"state": state, "questions": questions})
|
||||
if code != 200:
|
||||
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
|
||||
sizes.append(len(chunk))
|
||||
tokens.append((resp.get("usage") or {}).get("input_tokens"))
|
||||
for f, d in zip(fields, chunk):
|
||||
answer = (resp.get("answers") or {}).get(f)
|
||||
ids = [o["id"] for o in d["options"]]
|
||||
probs = (answer or {}).get("probabilities") or {}
|
||||
if answer is None or any(i not in probs for i in ids):
|
||||
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
|
||||
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
|
||||
"top": answer.get("choice"), "confidence": answer.get("confidence"),
|
||||
"calibration": resp.get("calibration"), "native": answer,
|
||||
"input_tokens": tokens[-1], "model": MODEL,
|
||||
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
|
||||
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
|
||||
"questions_per_call": sizes, "input_tokens": tokens}}
|
||||
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
protocol_version = "HTTP/1.1"
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
|
||||
|
||||
def _send(self, code, obj):
|
||||
b = json.dumps(obj).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(b)))
|
||||
self.end_headers()
|
||||
self.wfile.write(b)
|
||||
|
||||
def _authorized(self):
|
||||
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
|
||||
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
|
||||
|
||||
def do_GET(self):
|
||||
path = self.path.split("?")[0]
|
||||
if path == "/health":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
|
||||
shim = json.loads(r.read())
|
||||
except Exception as e: # noqa: BLE001
|
||||
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
|
||||
ok = bool(shim.get("ok"))
|
||||
return self._send(200 if ok else 503, {
|
||||
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
|
||||
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
|
||||
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
if path == "/v1/models":
|
||||
try:
|
||||
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
|
||||
return self._send(r.status, json.loads(r.read()))
|
||||
except urllib.error.HTTPError as e:
|
||||
return self._send(e.code, {"error": "upstream"})
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
def do_POST(self):
|
||||
path = self.path.split("?")[0]
|
||||
if not self._authorized():
|
||||
return self._send(401, {"error": "unauthorized"})
|
||||
n = int(self.headers.get("Content-Length", 0) or 0)
|
||||
if n > 4 * 1024 * 1024:
|
||||
return self._send(413, {"error": "body over 4 MiB"})
|
||||
try:
|
||||
body = json.loads(self.rfile.read(n) or b"null")
|
||||
except ValueError:
|
||||
return self._send(422, {"error": "body is not JSON"})
|
||||
if not isinstance(body, dict):
|
||||
return self._send(422, {"error": "body must be a JSON object"})
|
||||
try:
|
||||
if path == "/v1/systemone":
|
||||
return self._send(*systemone(body))
|
||||
if path == "/api/alpha/decisions":
|
||||
admit(body)
|
||||
return self._send(*post_up(path, body))
|
||||
if path == "/decide/shared":
|
||||
return self._send(200, decide(body.get("state"), body.get("decisions")))
|
||||
if path == "/decide":
|
||||
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
|
||||
out = decide(body.get("state"), [d])
|
||||
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
|
||||
except Refused as r:
|
||||
return self._send(r.code, r.obj)
|
||||
return self._send(404, {"error": "not found"})
|
||||
|
||||
|
||||
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
|
||||
# Docker restarts it (restart: unless-stopped). The shim's own SHIM_* env (SHIM_NOUL_FLOOR etc.) passes through.
|
||||
set -eu
|
||||
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
|
||||
--host 127.0.0.1 --port "$SHIM_PORT" &
|
||||
SHIM=$!
|
||||
python3 /front/front.py &
|
||||
FRONT=$!
|
||||
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
kill "$SHIM" "$FRONT" 2>/dev/null || true
|
||||
exit 1
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
# One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1:
|
||||
# sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh
|
||||
# It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy).
|
||||
set -euo pipefail
|
||||
cd /opt/docker/compose/intern-decision
|
||||
docker compose down
|
||||
cp compose.yaml compose.yaml.h2o-$(date +%Y%m%d-%H%M)
|
||||
cp compose.inc.yaml compose.yaml
|
||||
docker compose up -d
|
||||
for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done
|
||||
echo "INC not healthy after 180 s"; exit 1
|
||||
Reference in New Issue
Block a user