esh-pfi-infrastructure/stacks/intern-decision/README.md

# intern-decision — the decision seat (fv-ml1 GPU 1, :8033)

> **Since 2026-10-12 1311 PT the seat serves h2o-lightning-4b**, not Intern-Decision-4B. Prime's order came via
> brokkr-smithy-dev, gated on albok-dev re-validating Nemi's sensitivity judge (signed off at threshold 0.5 through
> `/decide/shared`). INC went down at 130818 and h2o was healthy at 131127. The address, port, token and stack name
> are unchanged on purpose (brokkr rule 8), so callers change nothing.
> Everything from **"History: the Intern-Decision-4B seat"** down describes the previous seat. It is kept because
> the rollback goes back to it.

| | |
|---|---|
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) |
| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision |
| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` |
| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check |
| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` |
| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) |
| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below |

### What the front adds (`services/h2o-decision-front`)

- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over
  the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix
  plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit;
  refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged.
- **The bearer token** on every path except `GET /health`.
- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only
  production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as
  INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`,
  `calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's
  `prompt_sha256`, `readout` and timing fields are gone.
- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and
  returns 503 when the shim is not healthy.
- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.

### Verification on the live seat (2026-10-12, 1311–1313 PT, on fv-ml1 loopback)

| check | result | expected |
|---|---|---|
| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin |
| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 |
| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) |
| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round |
| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 |

The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below.

**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus
the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/`
and `/tank/bench/cutover-probe.json`.

| check | result | reference |
|---|---|---|
| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench |
| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here |
| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | |
| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property |
| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth |
| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 |

⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV
size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.

### Before any (re)create

GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with
`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.

### Rollback to INC

On fv-ml1: `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It runs `docker compose down`, keeps
the h2o compose as `compose.yaml.h2o-<timestamp>`, copies `compose.inc.yaml` over `compose.yaml`, and starts INC
(image `intern-decision-serve:0.1.3`, still on the host; its `intern-decision_triton-cache` volume survives the
`down`). It waits up to 180 s for `/health`. INC's peak is 15,220 MiB, below what h2o frees. To return to h2o: put
the saved h2o compose back as `compose.yaml` and `up -d`.

---

# History: the Intern-Decision-4B seat (2026-09-30 → 2026-10-12)

> **Was LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback.

**Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
the erp/meromero seats and scriberr. It **replaced `semif`** on Prime's ruling of 2026-09-30,
~0510 PT: "replace semif with intern-decision now". The pick came from the previous night's bench,
`docs/pfi/jev-candidates-bench-2026-09-30.md`: the only candidate at least as accurate as SemIf on
our sets, faster at our shape, and inside the memory budget.

[Intern-Decision-4B](https://huggingface.co/internlm/Intern-Decision-4B) (Shanghai AI Lab,
Apache-2.0) is a Qwen3.5-4B finetune. It takes a state plus up to 16 named questions and scores
every question's options in **one forward pass with no decoding**. It reads each answer from the
logits just before a `<decision>` marker. `services/intern-decision-serve/` loads it once and
scores through the checkpoint's **own** `inference.py` (`DecisionEngine.predict`). It keeps
**semif-serve's HTTP surface**, so a semif caller changes only the URL and the token. The contract
is `services/intern-decision-serve/intern-decision-serve.contract.md`.

| | |
|---|---|
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; POSTs need `Authorization: Bearer $(secret get intern-decision/api-token)` |
| **Model** | `internlm/Intern-Decision-4B` @ `0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd`, BF16, from `/tank/aimodels/huggingface`, mounted read-only and offline |
| **Scoring code** | the snapshot's `inference.py`, sha256 `c904e2c6…43b29863`, checked before it is imported. Startup refuses any other file |
| **Stack** | torch `2.10.0+cu128`, transformers `5.17.0`, fla `0.5.2`, causal-conv1d `1.7.0`: the bench's stack |
| **Image** | `intern-decision-serve:<version>`, built on fv-ml1 from `services/intern-decision-serve/` |
| **State** | None. The service never downloads (`HF_HUB_OFFLINE=1`, read-only mount). If the weights are ever lost, re-pull the pinned revision by hand. |
| **Rollback** | `stacks/semif` is stopped, not removed. See **Rollback** below. |

## API (semif-serve's)

```bash
T=$(secret get intern-decision/api-token)
curl -s -H "Authorization: Bearer $T" http://intern-decision.fv.internal:8033/decide -d '{
  "id": "q1", "state": "Health checks passed in all three zones.",
  "question": "Did the deployment succeed?",
  "options": [{"id":"yes","description":"It succeeded."},{"id":"no","description":"It failed."}]}'
```

- `POST /decide` takes one decision, `{id, state, question, options[2..16]}`.
- `POST /decide/shared` takes `{state, decisions: [{id, question, options}]}`.
- `GET /health` reports the pins, the limits and the chunking rule.
- **Order averaging:** `"orderings": "rotations"`, or `"all"` for ≤ 4 options, is still accepted,
  with semif's `combined` block. **It is barely needed:** the bench found Intern-Decision reorders
  8 of 144 labels when the options are reversed, against SemIf's 30, and at one ordering it matched
  SemIf-with-rotations.
- Each result carries semif's fields where they mean something: `option_ids`, `probabilities`,
  `input_tokens`, `prompt_sha256`, `prompt_version`, `model`, `readout`, `probability_status`, and
  on `/decide` also `total_seconds` and `forward_seconds`. New fields:
  - `top`;
  - `confidence`, the model's own;
  - `calibration`, the model's own temperature (T = 1.99241824);
  - `native`, the model's answer object, unchanged;
  - `call` (`index`, `field`, `questions`): which prompt the decision was asked in.
- Past `MAX_QUEUE` (32) requests in progress, new POSTs get `429 busy` before their body is read.

### What a semif caller must know (every deliberate difference is in the contract, "Deltas")

1. **Option ids are shown to the model.** The prompt prints `A = <id>: <description>`, so an
   id is part of the question. Use meaningful or neutral ids, not misleading ones.
2. **Decisions in one `/decide/shared` call are asked together, in one prompt.** An answer can
   depend on the other questions in its call.
   - The bench measured this on Wyrd: 79/84 asked one decision at a time, 77/84 with a turn's
     4 decisions in one prompt.
   - A call holds **at most 16 questions**. More are split greedily in request order: 1–16,
     17–32, … . `/health` says so.
3. **`probabilities` are temperature-scaled** by the model's shipped calibration. SemIf's were
   raw. The argmax is the same either way. This calibration is the vendor's, fitted on its own
   data, not ours.
4. **Gone:**
   - `option_logits` (the model's runtime does not expose logits);
   - the prefix-cache timing fields;
   - `/health.semif_commit`;
   - per-workload calibration. Any `workload` is a 422, exactly as the deployed semif behaved
     with its empty table.
5. **`MAX_TOKENS` is 7,168 per call:** the state plus all its questions. It is lower than the
   model's own 8,192 so that every call the API accepts fits the VRAM cap.
   - The count is taken **before** the forward pass, so a longer call is a clear
     `422 invalid_request` ("Example has N tokens, above 7168; truncation is forbidden"). It is
     never truncated.
   - A request with more decisions is split into more calls, and each call must fit.

## VRAM: 32k-token calls since 2026-09-30 1330 (scriberr moved to GPU 3)

**Current setting: `VRAM_CAP_GIB=14.4`, `MAX_TOKENS=32768`** (Prime: move scriberr to GPU 3 and "extend the jev
endpoint to hit 32k tokens if possible"). With scriberr gone, GPU 1 holds only the static vLLM seats and this
service. This container may therefore use its rest (8,812 MiB) plus GPU 1's nvidia-smi `Free` (6,625 MiB), which is
**15,437 MiB**. The 14.4 GiB cap plus the ~660 MiB outside the allocator puts the card ceiling at ~15,408 MiB.

> ⚠ **2026-10-05:** the 0234 Docker-move restart broke this budget. `vllm-meromero-rp` sized its KV from
> `--gpu-memory-utilization`, re-profiled during a co-boot, and came back 8.7 GiB larger, so `Free` fell to
> 3,981 MiB, about 2.4 GB short of a 32k call. Fixed 2131–2135 the same night: meromero and coder now pin KV in
> bytes (`playbooks/fv-ml1-kv-pin-20261005.yaml`), and `Free` reads **11,801 MiB**, a 20,613 MiB budget, so
> ~5.4 GB is spare at the 15,220 peak. The static seats can no longer drift across restarts.
> **Later that night (2226–2249), the GPU 0/1 rebalance** (`playbooks/fv-ml1-rebalance-20261005.yaml`) left this
> card with gen-small, meromero-rp and this service only. `Free` read 10,864 MiB right after, and should settle
> near ~9.7 GB once gen-small's post-boot growth levels off. That is a ~18.5 GB budget, so ~3.3 GB spare at the 32k peak.

Measured on the live service (per-process nvidia-smi every 0.1 s; single `noul` question unless noted; n=3, deterministic):

| call tokens | card peak MiB | wall (warm) |
|---|---|---|
| 3,187 | 9,306 | 0.15 s |
| 12,187 | 11,206 | 0.68 s |
| 24,187 | 13,552 | 1.49 s |
| 29,987 | 14,692 | 1.92 s |
| **32,768 (limit)** | **15,220** | 2.12 s |
| 32,765, 16 questions | 15,220 | 2.15 s |
| 32,769 | refused 422 before the forward | 0.30 s |

Spare at the limit: 217 MiB. JevBench v1.2.16 through `/v1/systemone` after the change: 202/231, with 0 answer and 0
probability diffs against the bench's r1..r4. `/decide` is unchanged.

⚠ **Cold-shape latency (MEASURED 2026-09-30, was inferred):** the fast kernels (triton + fla) autotune
per shape bucket. **Buckets are 2,048 tokens wide** — proven: after warming one call per bucket, 12
calls at random sizes spread 8k–32k were all warm (worst 2.09 s, the same curve as the warmed sizes;
no +6 s), and every cold call that crossed into its own bucket cost 6.5–9 s while same-bucket repeats
cost <0.3 s. **16 buckets** to MAX_TOKENS=32,768. A full cold warm-up took **109 s** (16 buckets; the
service's own startup warm-up and the script's two calibration probes warm 3 of them along the way;
a second run totals 17 s). Per-bucket cold cost grows with size: 6.5 s at 4k → 8.9 s at 32k.
  - **Durable now:** the autotune cache (TRITON_CACHE_DIR=/tmp/triton-cache, incl. fla's
    *.autotune.json) lives on the named volume `intern-decision_triton-cache`, mounted at a mount
    point baked into the image owned by 10001 (a fresh named volume inherits it; without that, the
    volume is root-owned and Triton cannot write). Proven: 3,200+ files owned by uid 10001 in the
    volume; after `up -d --force-recreate` a previously warmed 32k call answered in 2.11 s.
  - **When to warm:** run `scripts/intern-decision-warmup` (on nh3-dev; token from the vault) after
    an IMAGE CHANGE — a new triton/fla version keys new cache entries. Not needed after ordinary
    recreates. The script reads MAX_TOKENS from /health, aims each bucket just under its edge with a
    two-point live calibration (single-probe "calibration" oscillates ±500 tokens around the edge),
    prints per-bucket wall times, and exits non-zero if any bucket was missed.
  - Cold autotune also costs a few MiB of transient scratch: GPU 1 per-process peak measured
    **15,218 MiB warm** at the 32k limit (budget 15,220), but **15,224** during a COLD 32k autotune.
    The volume means the cold path now happens only after an image change — worth knowing when
    GPU 1 headroom is being read during a deploy.

The table below is the ORIGINAL budget (9.0 GiB / 7,168 tokens, beside scriberr) and is kept for history.

## VRAM (history): fits beside scriberr's peak, whatever the request

**Budget (infra-ops, 2026-09-30):** GPU 1 needs nvidia-smi `Free` ≥ **15,400 MiB** before this
service starts. That is our card peak of 9,876 MiB plus scriberr's peak of 5,496 MiB (with its
120 s slices), rounded up. So both can peak at the same moment. Read nvidia-smi's own `Free`, not
total − used: the driver reserves 640 MiB on every card. The deploy at 0940 read 15,442.

`torch.cuda.set_per_process_memory_fraction` caps only torch's allocator. The two knobs in `.env`
are therefore chosen together and change together:

```
card footprint  <=  VRAM_CAP_GIB 9.0 (9,216 MiB)  +  660 MiB outside the allocator (measured)  =  9,876 MiB
MAX_TOKENS 7168  =  the largest call measured to fit under that cap, so no accepted call reaches the 503
```

| on the live service, GPU 1 (per-process nvidia-smi, 0.1 s sampling, 2,939 samples over 316 s of acceptance load) | MiB |
|---|---|
| **at rest** after startup | **8,812** |
| at rest after the acceptance (512 MiB release slack keeps a little cache) | 8,856 |
| **peak**: largest requests, 64 decisions × 16 options, 4 calls × 7,168 tokens, N = 3 | **9,866**; torch `max_reserved` 8.996 of 9.0 GiB |
| GPU 1 `Free` before the deploy / at rest after / lowest during the acceptance | 15,442 / 6,581 / 5,569, which stays above scriberr's 5,496 |

- ⚠ **At 7,168 tokens the allocator reaches the cap with about 4 MiB to spare.** It fits because
  the allocator frees its cache and retries before it fails.
  - Measured: every call up to the limit answered 200. That is 145 shared calls on the live
    service, including the size search near the boundary, 3 × 4 calls at 7,168 tokens with
    16 questions, and 3 at 7,168 tokens with one question. No 503 was seen.
  - The allocator's cache state depends on request history. If a max-size call ever does return
    503, lower `MAX_TOKENS`. Do not raise the cap.
- Raising either knob needs a fresh measurement (`acceptance/checks.py --checks fit,maxreq`) and
  the budget re-checked against scriberr.
- **Over the cap the answer is 503 `out_of_memory`.** Memory returns to the resting baseline and
  the service keeps answering. This was proven on GPU 3 with a deliberately tight cap.

Measured on GPU 3 before the deploy (alone on the card), which is where the knobs came from:

| measured on fv-ml1 GPU 3 (alone on the card), image 0.1.0 | MiB |
|---|---|
| **at rest** (nvidia-smi, whole card less 2 MiB idle) | **8,820** (torch reserved 8,160 + outside 660) |
| outside the allocator (CUDA context + loaded kernels), at rest / after 2×30 concurrent requests / after the latency shapes / after 4 × 8,191-token calls | 660 / 660 / 662 / 662 |
| card peak at cap 9.0 GiB (fit search and a 64 × 8,191-token request, which got 503) | **9,876** |
| peak with 4 × 8,191-token calls at cap 9.25 GiB (answered 200) / uncapped | 10,134 / 10,422 (allocator 9,760) |
| largest 16-question call that answers 200: at cap 9.0 / at 9.25 GiB | 7,168 / 8,191 tokens |
| startup peak (weights + vision tower until the swap + warm-up) | allocator 8.86 GiB. **A cap below ~8.9 GiB cannot start** (it fails closed; 8.5 was refused) |

- **One inference thread is load-bearing** (contract INV-2). torch keeps CUDA state per host
  thread (cuBLAS handles and workspaces). The first build scored on anyio's 40-thread pool: a
  concurrent burst added 252 MiB outside the cap and 326 MiB inside it, and the peak reached
  10,392 MiB. Load, warm-up and every call now run on one thread.
- **Released after bursts:** when a call leaves reserved memory more than 512 MiB over the resting
  baseline, the engine calls `empty_cache()`. Measured: back to 7.97 GiB after every large
  request.
- The rest footprint is 0.9 GB below the bench's 9,736 MiB. The service takes no images, so the
  vision tower (0.62 GiB) is swapped for a stub after the first warm-up, and the warm-up cache is
  released. Startup proves the swap changes nothing: the warm-up answer must be bit-identical
  (INV-7).

## Latency

**Live, from nh3-dev** (0942 PT). The URL is `intern-decision.fv.internal:8033`, and the round trip
is 22.9 ms on average (19.5–27.7 ms over 10 pings). GPU 1 is shared with the vLLM seats and
scriberr. The harness is `bench_shape.py` from the Jev bench, 3 runs × 20 requests; each cell is
the median with the run-median range. "server" is the service's own time.

| shape | end to end from nh3-dev | server | GPU 3 loopback before deploy | SemIf 0.1.4 (README, from nh3-dev) |
|---|---|---|---|---|
| 1 decision, short (~180 tok) | 53.9 ms (53.3–54.2) | 35.2 | 36.4 | 69 |
| **21 binary criteria**, one `/decide/shared` (2 calls: 16 + 5) | **114.3 ms** (114.1–114.3) | 80.3 | 82.3 | 159 |
| 1 decision over the ~3,900-token state | 211.2 ms (210.8–211.6) | 186.8 | 191.4 | |
| **16 criteria over the ~3,900-token state** (1 call, 4,579 tok) | **238.3 ms** (237.8–238.7) | 205.3 | 216.8 | (bench loopback: 505) |
| largest request (4 calls × 7,168 tok), N = 3 | 1,519–1,543 ms | | | |

Server-side time on GPU 1 matches GPU 3 within a few ms, so the vLLM neighbours were not
measurably slowing it at 0942. The rest of the end-to-end time is the network (~23 ms round trip
plus the HTTP exchange).

## Acceptance (2026-09-30)

Raw results: `services/intern-decision-serve/acceptance/gpu3-2026-09-30/` (`compare.json`, one
directory per process lifetime). **The harness is the bench's own.** `bench_sets.py
--backend semif` speaks semif-serve's API, so it drove this service unchanged. `compare.py`
scores it as the bench's `analyze.py` did.

### On GPU 3 before the deploy (the harness and the floor)

| check (GPU 3, cap 9.25 GiB, N = 3 fresh processes unless noted) | result | bench / floor |
|---|---|---|
| **positive control**, pooled 259, single ordering | **240, 240, 240** | bench 240 (4 repeats); the pooled set resolves ±4 pts |
| positive control, Wyrd /84, single | **79, 79, 79** | bench 79 |
| row by row against the bench's native rows (560 rows, single / rotations / negative) | **0 top changes, max Δp 0.000** in every repeat: bit-identical | bench floor: 0 labels moved across 4 restarts |
| rotations, pooled / Wyrd | 236 / 77 (×3) | bench 236 / 77 |
| **negative control** (descriptions rotated, 144): same top / follows the description / right vs the original gold | **10 / 122 / 14** (×3) | bench 10 / 122 / 14 |
| A-vs-A in the process (authored144 twice) | 0/144 flips, Δp 0.0 (×3) | |
| A-vs-A across restarts (r1~r2, r1~r3, r2~r3; 560 rows, single and rotations) | 0 flips, Δp 0.0 | |
| Wyrd, one `/decide/shared` per turn (4 decisions in one prompt) | 78/84 (×3); 1/84 row differs from the bench's 77 | expected: field names are positional here, decision names in the bench (contract) |
| more than 16 questions: 20 decisions in one request, against the same decisions asked as 16 + 4 | 2 calls `[16, 4]`, all 20 tops equal, Δp 0, same prompt hashes | |
| 401: no token / wrong token, both POSTs; `/health` open | 401 / 401; 200 | |
| 429: 48 concurrent requests, `MAX_QUEUE` 32 | 32 answered, 16 × 429 `busy` | |
| largest request at `MAX_TOKENS` 8,192 (64 × 16 options, 4 × 8,191 tok), cap 9.25 GiB | **200** ×3; card peak 10,134 MiB | superseded by the 9.0 / 7,168 knobs (VRAM) |
| **over the cap** (the same request at a deliberately tight cap of 8.9 GiB) | **503 `out_of_memory`**; reserved 7.969 GiB before, 7.969 after, next request 200; then all 560 single rows answered, bit-identical to the bench | |
| fit under the cap: the largest 16-question call that answers 200 | 8,191 tok (the whole API) at 9.25 GiB; 7,168 tok at 9.0 GiB | |
| cap below startup | 8.5 GiB: startup refused (OOM loading), fails closed | |

### On the live service (GPU 1, 0942–0946 PT, through `http://intern-decision.fv.internal:8033` from nh3-dev)

Raw results: `services/intern-decision-serve/acceptance/gpu1-live/`.

| check | result | reference |
|---|---|---|
| **positive control**, pooled 259, single ordering | **240** | bench 240; GPU 3 240 ×3 |
| positive control, Wyrd /84 | **79** | bench 79 |
| row by row against the bench's native rows (560 single + 144 negative) | **0 top changes, max Δp 0.000** | |
| negative control: same top / follows the description / right vs gold | **10 / 122 / 14** | bench 10 / 122 / 14 |
| 401 without or with a wrong token (both POSTs); `/health` open | 401 / 401; 200 | |
| largest accepted request: 64 decisions × 16 options, 4 calls × 7,168 tok | **200** ×3 | |
| one question at 7,168 tok | 200 ×3; one token more (7,171) → **422** "above 7168" | |
| any 503 across the whole live acceptance | **none** (server log: 128 + 145 × 200, 18 × 422, 4 × 401) | |

The live pass is N = 1 for the positive control. It is read against the GPU 3 floor: the same
image moved 0 of 560 rows across 3 restarts, and 0 against the bench's 4 restarts.

## Building and deploying

```bash
# from nh3-dev. /opt/docker/src is root-owned, so create the version dir with sudo first.
ssh infra-ops@10.251.50.54 'sudo -n install -d -o infra-ops -g infra-ops /opt/docker/src/intern-decision-serve-X.Y.Z | cat'
tar -C services/intern-decision-serve -cf - --exclude=.venv --exclude=.pytest_cache --exclude=__pycache__ \
    --exclude=acceptance --exclude='*.egg-info' . \
  | ssh infra-ops@10.251.50.54 'tar -x -C /opt/docker/src/intern-decision-serve-X.Y.Z'
# on fv-ml1
cd /opt/docker/src/intern-decision-serve-X.Y.Z && docker build -t intern-decision-serve:X.Y.Z .
# from nh3-dev: compose.yaml, .env.example and this README (never .env)
scripts/deploy-stack.sh fv-ml1 intern-decision
# on fv-ml1: .env is infra-ops's, mode 0600. Rewrite it IN PLACE to keep the owner and mode.
cd /opt/docker/compose/intern-decision && new=$(sed 's/^IMAGE=.*/IMAGE=intern-decision-serve:X.Y.Z/' .env) \
  && printf '%s\n' "$new" > .env && docker compose config -q && docker compose up -d
# IMAGE CHANGE ONLY (not ordinary recreates — the triton-cache volume carries those):
# warm every length bucket once the container is healthy (~2 min, prints per-bucket times):
scripts/intern-decision-warmup        # token from vault; MAX_TOKENS read from /health
```

**Before any deploy onto GPU 1**, check two things. If either fails, stop; do not squeeze
scriberr.
- nvidia-smi's own `Free` on GPU 1 must be at least **15,400 MiB**:
  `nvidia-smi -i 1 --query-gpu=memory.free --format=csv`. That is our card peak of 9,876 MiB plus
  scriberr's 5,496, rounded up. Do not use total − used, which misses the driver's 640 MiB
  reserve.
- Scriberr must not be running a job. This command must print 0:
  `docker logs --since 2m scriberr | grep -c "Processing single-track job"`.

Startup fails closed. A container that never reaches healthy did not pass its own checks: the
`inference.py` hash, the pinned snapshot, the warm-up, the text-only swap and the prompt hash.
Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`.

**To move the model forward**, change `REVISION` and `INFERENCE_PY_SHA256` in `config.py`.
Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and
re-measure the VRAM before changing the cap.

## Rollback (INC → semif)

`semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word:

```bash
cd /opt/docker/compose/intern-decision && docker compose stop     # first: the two do not both fit GPU 1
cd /opt/docker/compose/semif && docker compose start
```

Then check GPU 1's free memory against scriberr's peak. Callers go back to `:8032` and
`secret get semif/api-token`.