esh-pfi-infrastructure/stacks/intern-decision/README.md

intern-decision — the decision seat (fv-ml1 GPU 1, :8033)

LIVE: Intern-Decision-4B again, since 2026-10-12 1403 PT (compose.yaml, image intern-decision-serve:0.1.3). Everything from "The live seat: Intern-Decision-4B" down describes it.

h2o-lightning-4b is STAGED, not live (compose.h2o.yaml, switch-to-h2o.sh). It served 1311–1403 on 2026-10-12 and was rolled back on albok-dev's word, with brokkr concurring. Prime's plan had made the swap conditional on the judge's sign-off, and albok-dev withdrew it on fleet-scale evidence:

• adult: over 10,841 texts judged by both models (same text, prompt and policy), h2o flagged 242 at 0.5 against INC's 88. That is all 88 of INC's plus 154 more, most of them at 0.80–0.90 where INC put them at 0.30–0.47. No threshold reproduces INC's set (0.70 still flags 166 and loses 6 of INC's).
• personal: INC flagged 5 and h2o 11, with only 2 in common.

h2o comes back only after albok re-fits the judge for it (prompt and threshold against a labelled set large enough to resolve a 2× change, as a 4.4 contract revision, on an eval seat). The h2o sections below describe the staged seat, and its numbers are the 1311 cutover's.

The staged h2o seat (compose.h2o.yaml)

URL	http://10.251.50.54:8033 = http://intern-decision.fv.internal:8033. /health is open; everything else needs Authorization: Bearer $(secret get intern-decision/api-token) (the same value as albok/judge-token)
Model	h2oai/h2o-lightning-4b @ b43f759b6385d21182f87c3829931ec3d4dfa98a, from /tank/aimodels/h2o-lightning-4b-b43f759b (read-only). model.safetensors sha256 2d51561f…d4d11497b6; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision
Serving	vllm/vllm-openai:v0.30.0 (container intern-decision-vllm, loopback :18299), args exactly as benched: --max-model-len 40960 --max-num-seqs 16, the vendor's vision limits, VLLM_USE_FLASHINFER_SAMPLER=0
KV	pinned in bytes: --kv-cache-memory 4810363371 (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. --gpu-memory-utilization 0.1506 now only gates vLLM's boot check
Front	container intern-decision (python:3.12-slim, host network, :8033), running conf/front/run.sh: the vendor's unmodified h2o_lightning_shim.py on loopback :18291, behind services/h2o-decision-front/front.py
State	none beyond the intern-decision_vllm-cache volume (compile cache)
Rollback	compose.inc.yaml is the INC compose, same .env. See Rollback to INC below

What the front adds (services/h2o-decision-front)

• Admission, refuse-not-trim, at 32,000 tokens. It counts with h2o's own tokenizer through vLLM /tokenize, over the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix plus tails). Over the cap gets 422 "admission: N max-prompt / M total tokens against the 32000-token seat limit; refused, not trimmed". The logic is infra-hermes's tools/h2o-admission/admission_proxy.py (30f36f2), unchanged.
• The bearer token on every path except GET /health.
• /decide and /decide/shared, translated onto /v1/systemone. This keeps albok-nemi's judge, the seat's only production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as INC did. orderings other than none → 422. Each result carries probabilities, top, confidence, calibration, native (the shim's answer object, unchanged), input_tokens, model and call. INC's prompt_sha256, readout and timing fields are gone.
• /health reports model {name, source, revision}, the cap, the endpoints and vLLM's max_model_len, and returns 503 when the shim is not healthy.
• /v1/systemone and /api/alpha/decisions pass through after admission, and /v1/models passes through as is.

Verification at the 1311 cutover (2026-10-12, 1311–1318 PT, on fv-ml1 loopback)

check	result	expected
/health	ok, model h2o-lightning-4b @ b43f759b…, cap 32,000, max_model_len 40,960	the pin
over the cap: one question over a ~40k-token state	422, "40078 max-prompt / 40078 total … refused, not trimmed"	422
16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3	200; 2.105 / 2.061 / 1.999 s	~2 s (bench)
/decide/shared smoke: two yes/no decisions over a line holding a phone number and a street address	200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876	the right way round
GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls	14,420 / 15,590 MiB	budget 16,847, bench peak 15,566

The 16q@32k and /decide/shared rows show the path works; the speed and memory numbers are infra-hermes's, below.

infra-hermes, 1316–1318, same seat: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus the Amendment 1 gate probe driver). Bearer passed by --key-env. Artifacts are on fv-ml1 in /tank/bench/cutover/jevbench/ and /tank/bench/cutover-probe.json.

check	result	reference
JevBench public, one pass	205/231	205 ×3 in the Amendment 1 bench
depth cells: evidence at depth 1k / 8k / 16k / 30k	supported at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens	the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here
over the cap	422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count
usage.input_tokens against the admitted count	equal in every admitted cell	the admission honesty property
largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens)	first 251 / 2,395 ms; warm median 226 / 2,159 ms	bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth
4 concurrent 16q × 32k calls	all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: 95,178 MiB = meromero 43,998 + gen-small 34,994 + seat 16,164	seat budget 16,847, so 683 MiB headroom; the bench peak was 15,566

⚠ 683 MiB is the seat's margin under 4-way 32k load. That is tighter than the bench's peak, because of the byte-pinned KV size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.

Before any (re)create

GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm, and stop if it fails. The vLLM container takes about 3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.

Images: inline only (Prime 2026-10-12, via brokkr)

The h2o seat must never fetch an image URL. compose.h2o.yaml sets LIGHTNING_IMAGE_URLS=off on the front, and the vendor shim then refuses every image URL with 422 "image URLs are not accepted here". Inline data: URIs and base64 keep working. vLLM also runs with --allowed-media-domains media.invalid as a backstop. That is a reserved name that never resolves, and vLLM 0.30.0 checks every http(s) media URL against it before fetching, while data: URLs bypass the check.

The shim switch was verified live at 1402, on the h2o seat with the front recreated alone:

• the same https://media.invalid/red.png request went from "could not be fetched" (a fetch was attempted) to "not accepted here" (refused before any fetch);
• inline red still answered red at 0.984;
• before the change, inline red/blue/green answered at 0.98+ and no-image came out near-uniform.

The vLLM backstop was first exercised on the GPU 3 eval seat at 1428 (same image and flag): an https image URL posted straight to vLLM got 400 "The URL must be from one of the allowed domains: ['media.invalid']", with no fetch. Repeat that check on this seat's loopback port (:18299) on the next switch. INC needs neither setting, because its served checkpoint has no vision tower.

Switching between the two

Both run ON fv-ml1 and share one .env.

• To h2o: sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh. It stops INC and checks that GPU 1 has 16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies compose.h2o.yaml over compose.yaml and waits up to 300 s for /health. Only do this once albok has re-fitted the judge.
• To INC: sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh. It stops h2o, copies compose.inc.yaml over compose.yaml and waits up to 180 s. Image intern-decision-serve:0.1.3 and its intern-decision_triton-cache volume survive the down. Used 2026-10-12, 140328 → healthy 140356.

After either switch, make stacks/intern-decision/compose.yaml match the host again (it is INC today).

⚠ Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat. INC scores on one inference thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's nemi.yaml, since 2026-10-12). With 4 in flight on INC, the 1533–1619 walk queued behind itself and lost 339 calls to 30 s timeouts. On h2o, vLLM batches, so it goes back to 4.

---

The live seat: Intern-Decision-4B (2026-09-30 → 2026-10-12 1308, and again since 1403)

LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT on fv-ml1 GPU 1, with cap 9.0 GiB and MAX_TOKENS 7,168. Acceptance on the live service is below. semif is stopped and kept as the rollback.

Intern-Decision-4B typed decisions on fv-ml1 GPU 1, the utility card beside vllm-coder, the erp/meromero seats and scriberr. It replaced semif on Prime's ruling of 2026-09-30, ~0510 PT: "replace semif with intern-decision now". The pick came from the previous night's bench, docs/pfi/jev-candidates-bench-2026-09-30.md: the only candidate at least as accurate as SemIf on our sets, faster at our shape, and inside the memory budget.

Intern-Decision-4B (Shanghai AI Lab, Apache-2.0) is a Qwen3.5-4B finetune. It takes a state plus up to 16 named questions and scores every question's options in one forward pass with no decoding. It reads each answer from the logits just before a <decision> marker. services/intern-decision-serve/ loads it once and scores through the checkpoint's own inference.py (DecisionEngine.predict). It keeps semif-serve's HTTP surface, so a semif caller changes only the URL and the token. The contract is services/intern-decision-serve/intern-decision-serve.contract.md.

URL	http://10.251.50.54:8033 = http://intern-decision.fv.internal:8033. /health is open; POSTs need Authorization: Bearer $(secret get intern-decision/api-token)
Model	internlm/Intern-Decision-4B @ 0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd, BF16, from /tank/aimodels/huggingface, mounted read-only and offline
Scoring code	the snapshot's inference.py, sha256 c904e2c6…43b29863, checked before it is imported. Startup refuses any other file
Stack	torch 2.10.0+cu128, transformers 5.17.0, fla 0.5.2, causal-conv1d 1.7.0: the bench's stack
Image	intern-decision-serve:<version>, built on fv-ml1 from services/intern-decision-serve/
State	None. The service never downloads (HF_HUB_OFFLINE=1, read-only mount). If the weights are ever lost, re-pull the pinned revision by hand.
Rollback	stacks/semif is stopped, not removed. See Rollback below.

API (semif-serve's)

T=$(secret get intern-decision/api-token)
curl -s -H "Authorization: Bearer $T" http://intern-decision.fv.internal:8033/decide -d '{
  "id": "q1", "state": "Health checks passed in all three zones.",
  "question": "Did the deployment succeed?",
  "options": [{"id":"yes","description":"It succeeded."},{"id":"no","description":"It failed."}]}'

• POST /decide takes one decision, {id, state, question, options[2..16]}.
• POST /decide/shared takes {state, decisions: [{id, question, options}]}.
• GET /health reports the pins, the limits and the chunking rule.
• Order averaging: "orderings": "rotations", or "all" for ≤ 4 options, is still accepted, with semif's combined block. It is barely needed: the bench found Intern-Decision reorders 8 of 144 labels when the options are reversed, against SemIf's 30, and at one ordering it matched SemIf-with-rotations.
• Each result carries semif's fields where they mean something: option_ids, probabilities, input_tokens, prompt_sha256, prompt_version, model, readout, probability_status, and on /decide also total_seconds and forward_seconds. New fields:
  • top;
  • confidence, the model's own;
  • calibration, the model's own temperature (T = 1.99241824);
  • native, the model's answer object, unchanged;
  • call (index, field, questions): which prompt the decision was asked in.
• Past MAX_QUEUE (32) requests in progress, new POSTs get 429 busy before their body is read.

What a semif caller must know (every deliberate difference is in the contract, "Deltas")

1. Option ids are shown to the model. The prompt prints A = <id>: <description>, so an id is part of the question. Use meaningful or neutral ids, not misleading ones.
2. Decisions in one /decide/shared call are asked together, in one prompt. An answer can depend on the other questions in its call.
   • The bench measured this on Wyrd: 79/84 asked one decision at a time, 77/84 with a turn's 4 decisions in one prompt.
   • A call holds at most 16 questions. More are split greedily in request order: 1–16, 17–32, … . /health says so.
3. probabilities are temperature-scaled by the model's shipped calibration. SemIf's were raw. The argmax is the same either way. This calibration is the vendor's, fitted on its own data, not ours.
4. Gone:
   • option_logits (the model's runtime does not expose logits);
   • the prefix-cache timing fields;
   • /health.semif_commit;
   • per-workload calibration. Any workload is a 422, exactly as the deployed semif behaved with its empty table.
5. MAX_TOKENS is 7,168 per call: the state plus all its questions. It is lower than the model's own 8,192 so that every call the API accepts fits the VRAM cap.
   • The count is taken before the forward pass, so a longer call is a clear 422 invalid_request ("Example has N tokens, above 7168; truncation is forbidden"). It is never truncated.
   • A request with more decisions is split into more calls, and each call must fit.

VRAM: 32k-token calls since 2026-09-30 1330 (scriberr moved to GPU 3)

Current setting: VRAM_CAP_GIB=14.4, MAX_TOKENS=32768 (Prime: move scriberr to GPU 3 and "extend the jev endpoint to hit 32k tokens if possible"). With scriberr gone, GPU 1 holds only the static vLLM seats and this service. This container may therefore use its rest (8,812 MiB) plus GPU 1's nvidia-smi Free (6,625 MiB), which is 15,437 MiB. The 14.4 GiB cap plus the ~660 MiB outside the allocator puts the card ceiling at ~15,408 MiB.

⚠ 2026-10-05: the 0234 Docker-move restart broke this budget. vllm-meromero-rp sized its KV from --gpu-memory-utilization, re-profiled during a co-boot, and came back 8.7 GiB larger, so Free fell to 3,981 MiB, about 2.4 GB short of a 32k call. Fixed 2131–2135 the same night: meromero and coder now pin KV in bytes (playbooks/fv-ml1-kv-pin-20261005.yaml), and Free reads 11,801 MiB, a 20,613 MiB budget, so ~5.4 GB is spare at the 15,220 peak. The static seats can no longer drift across restarts. Later that night (2226–2249), the GPU 0/1 rebalance (playbooks/fv-ml1-rebalance-20261005.yaml) left this card with gen-small, meromero-rp and this service only. Free read 10,864 MiB right after, and should settle near ~9.7 GB once gen-small's post-boot growth levels off. That is a ~18.5 GB budget, so ~3.3 GB spare at the 32k peak.

Measured on the live service (per-process nvidia-smi every 0.1 s; single noul question unless noted; n=3, deterministic):

call tokens	card peak MiB	wall (warm)
3,187	9,306	0.15 s
12,187	11,206	0.68 s
24,187	13,552	1.49 s
29,987	14,692	1.92 s
32,768 (limit)	15,220	2.12 s
32,765, 16 questions	15,220	2.15 s
32,769	refused 422 before the forward	0.30 s

Spare at the limit: 217 MiB. JevBench v1.2.16 through /v1/systemone after the change: 202/231, with 0 answer and 0 probability diffs against the bench's r1..r4. /decide is unchanged.

⚠ Cold-shape latency (MEASURED 2026-09-30, was inferred): the fast kernels (triton + fla) autotune per shape bucket. Buckets are 2,048 tokens wide — proven: after warming one call per bucket, 12 calls at random sizes spread 8k–32k were all warm (worst 2.09 s, the same curve as the warmed sizes; no +6 s), and every cold call that crossed into its own bucket cost 6.5–9 s while same-bucket repeats cost <0.3 s. 16 buckets to MAX_TOKENS=32,768. A full cold warm-up took 109 s (16 buckets; the service's own startup warm-up and the script's two calibration probes warm 3 of them along the way; a second run totals 17 s). Per-bucket cold cost grows with size: 6.5 s at 4k → 8.9 s at 32k.

• Durable now: the autotune cache (TRITON_CACHE_DIR=/tmp/triton-cache, incl. fla's *.autotune.json) lives on the named volume intern-decision_triton-cache, mounted at a mount point baked into the image owned by 10001 (a fresh named volume inherits it; without that, the volume is root-owned and Triton cannot write). Proven: 3,200+ files owned by uid 10001 in the volume; after up -d --force-recreate a previously warmed 32k call answered in 2.11 s.
• When to warm: run scripts/intern-decision-warmup (on nh3-dev; token from the vault) after an IMAGE CHANGE — a new triton/fla version keys new cache entries. Not needed after ordinary recreates. The script reads MAX_TOKENS from /health, aims each bucket just under its edge with a two-point live calibration (single-probe "calibration" oscillates ±500 tokens around the edge), prints per-bucket wall times, and exits non-zero if any bucket was missed.
• Cold autotune also costs a few MiB of transient scratch: GPU 1 per-process peak measured 15,218 MiB warm at the 32k limit (budget 15,220), but 15,224 during a COLD 32k autotune. The volume means the cold path now happens only after an image change — worth knowing when GPU 1 headroom is being read during a deploy.

The table below is the ORIGINAL budget (9.0 GiB / 7,168 tokens, beside scriberr) and is kept for history.

VRAM (history): fits beside scriberr's peak, whatever the request

Budget (infra-ops, 2026-09-30): GPU 1 needs nvidia-smi Free ≥ 15,400 MiB before this service starts. That is our card peak of 9,876 MiB plus scriberr's peak of 5,496 MiB (with its 120 s slices), rounded up. So both can peak at the same moment. Read nvidia-smi's own Free, not total − used: the driver reserves 640 MiB on every card. The deploy at 0940 read 15,442.

torch.cuda.set_per_process_memory_fraction caps only torch's allocator. The two knobs in .env are therefore chosen together and change together:

card footprint  <=  VRAM_CAP_GIB 9.0 (9,216 MiB)  +  660 MiB outside the allocator (measured)  =  9,876 MiB
MAX_TOKENS 7168  =  the largest call measured to fit under that cap, so no accepted call reaches the 503

on the live service, GPU 1 (per-process nvidia-smi, 0.1 s sampling, 2,939 samples over 316 s of acceptance load)	MiB
at rest after startup	8,812
at rest after the acceptance (512 MiB release slack keeps a little cache)	8,856
peak: largest requests, 64 decisions × 16 options, 4 calls × 7,168 tokens, N = 3	9,866; torch max_reserved 8.996 of 9.0 GiB
GPU 1 Free before the deploy / at rest after / lowest during the acceptance	15,442 / 6,581 / 5,569, which stays above scriberr's 5,496

• ⚠ At 7,168 tokens the allocator reaches the cap with about 4 MiB to spare. It fits because the allocator frees its cache and retries before it fails.
  • Measured: every call up to the limit answered 200. That is 145 shared calls on the live service, including the size search near the boundary, 3 × 4 calls at 7,168 tokens with 16 questions, and 3 at 7,168 tokens with one question. No 503 was seen.
  • The allocator's cache state depends on request history. If a max-size call ever does return 503, lower MAX_TOKENS. Do not raise the cap.
• Raising either knob needs a fresh measurement (acceptance/checks.py --checks fit,maxreq) and the budget re-checked against scriberr.
• Over the cap the answer is 503 out_of_memory. Memory returns to the resting baseline and the service keeps answering. This was proven on GPU 3 with a deliberately tight cap.

Measured on GPU 3 before the deploy (alone on the card), which is where the knobs came from:

measured on fv-ml1 GPU 3 (alone on the card), image 0.1.0	MiB
at rest (nvidia-smi, whole card less 2 MiB idle)	8,820 (torch reserved 8,160 + outside 660)
outside the allocator (CUDA context + loaded kernels), at rest / after 2×30 concurrent requests / after the latency shapes / after 4 × 8,191-token calls	660 / 660 / 662 / 662
card peak at cap 9.0 GiB (fit search and a 64 × 8,191-token request, which got 503)	9,876
peak with 4 × 8,191-token calls at cap 9.25 GiB (answered 200) / uncapped	10,134 / 10,422 (allocator 9,760)
largest 16-question call that answers 200: at cap 9.0 / at 9.25 GiB	7,168 / 8,191 tokens
startup peak (weights + vision tower until the swap + warm-up)	allocator 8.86 GiB. A cap below ~8.9 GiB cannot start (it fails closed; 8.5 was refused)

• One inference thread is load-bearing (contract INV-2). torch keeps CUDA state per host thread (cuBLAS handles and workspaces). The first build scored on anyio's 40-thread pool: a concurrent burst added 252 MiB outside the cap and 326 MiB inside it, and the peak reached 10,392 MiB. Load, warm-up and every call now run on one thread.
• Released after bursts: when a call leaves reserved memory more than 512 MiB over the resting baseline, the engine calls empty_cache(). Measured: back to 7.97 GiB after every large request.
• The rest footprint is 0.9 GB below the bench's 9,736 MiB. The service takes no images, so the vision tower (0.62 GiB) is swapped for a stub after the first warm-up, and the warm-up cache is released. Startup proves the swap changes nothing: the warm-up answer must be bit-identical (INV-7).

Latency

Live, from nh3-dev (0942 PT). The URL is intern-decision.fv.internal:8033, and the round trip is 22.9 ms on average (19.5–27.7 ms over 10 pings). GPU 1 is shared with the vLLM seats and scriberr. The harness is bench_shape.py from the Jev bench, 3 runs × 20 requests; each cell is the median with the run-median range. "server" is the service's own time.

shape	end to end from nh3-dev	server	GPU 3 loopback before deploy	SemIf 0.1.4 (README, from nh3-dev)
1 decision, short (~180 tok)	53.9 ms (53.3–54.2)	35.2	36.4	69
21 binary criteria, one /decide/shared (2 calls: 16 + 5)	114.3 ms (114.1–114.3)	80.3	82.3	159
1 decision over the ~3,900-token state	211.2 ms (210.8–211.6)	186.8	191.4
16 criteria over the ~3,900-token state (1 call, 4,579 tok)	238.3 ms (237.8–238.7)	205.3	216.8	(bench loopback: 505)
largest request (4 calls × 7,168 tok), N = 3	1,519–1,543 ms

Server-side time on GPU 1 matches GPU 3 within a few ms, so the vLLM neighbours were not measurably slowing it at 0942. The rest of the end-to-end time is the network (~23 ms round trip plus the HTTP exchange).

Acceptance (2026-09-30)

Raw results: services/intern-decision-serve/acceptance/gpu3-2026-09-30/ (compare.json, one directory per process lifetime). The harness is the bench's own. bench_sets.py --backend semif speaks semif-serve's API, so it drove this service unchanged. compare.py scores it as the bench's analyze.py did.

On GPU 3 before the deploy (the harness and the floor)

check (GPU 3, cap 9.25 GiB, N = 3 fresh processes unless noted)	result	bench / floor
positive control, pooled 259, single ordering	240, 240, 240	bench 240 (4 repeats); the pooled set resolves ±4 pts
positive control, Wyrd /84, single	79, 79, 79	bench 79
row by row against the bench's native rows (560 rows, single / rotations / negative)	0 top changes, max Δp 0.000 in every repeat: bit-identical	bench floor: 0 labels moved across 4 restarts
rotations, pooled / Wyrd	236 / 77 (×3)	bench 236 / 77
negative control (descriptions rotated, 144): same top / follows the description / right vs the original gold	10 / 122 / 14 (×3)	bench 10 / 122 / 14
A-vs-A in the process (authored144 twice)	0/144 flips, Δp 0.0 (×3)
A-vs-A across restarts (r1r2, r1r3, r2~r3; 560 rows, single and rotations)	0 flips, Δp 0.0
Wyrd, one /decide/shared per turn (4 decisions in one prompt)	78/84 (×3); 1/84 row differs from the bench's 77	expected: field names are positional here, decision names in the bench (contract)
more than 16 questions: 20 decisions in one request, against the same decisions asked as 16 + 4	2 calls [16, 4], all 20 tops equal, Δp 0, same prompt hashes
401: no token / wrong token, both POSTs; /health open	401 / 401; 200
429: 48 concurrent requests, MAX_QUEUE 32	32 answered, 16 × 429 busy
largest request at MAX_TOKENS 8,192 (64 × 16 options, 4 × 8,191 tok), cap 9.25 GiB	200 ×3; card peak 10,134 MiB	superseded by the 9.0 / 7,168 knobs (VRAM)
over the cap (the same request at a deliberately tight cap of 8.9 GiB)	503 out_of_memory; reserved 7.969 GiB before, 7.969 after, next request 200; then all 560 single rows answered, bit-identical to the bench
fit under the cap: the largest 16-question call that answers 200	8,191 tok (the whole API) at 9.25 GiB; 7,168 tok at 9.0 GiB
cap below startup	8.5 GiB: startup refused (OOM loading), fails closed

On the live service (GPU 1, 0942–0946 PT, through http://intern-decision.fv.internal:8033 from nh3-dev)

Raw results: services/intern-decision-serve/acceptance/gpu1-live/.

check	result	reference
positive control, pooled 259, single ordering	240	bench 240; GPU 3 240 ×3
positive control, Wyrd /84	79	bench 79
row by row against the bench's native rows (560 single + 144 negative)	0 top changes, max Δp 0.000
negative control: same top / follows the description / right vs gold	10 / 122 / 14	bench 10 / 122 / 14
401 without or with a wrong token (both POSTs); /health open	401 / 401; 200
largest accepted request: 64 decisions × 16 options, 4 calls × 7,168 tok	200 ×3
one question at 7,168 tok	200 ×3; one token more (7,171) → 422 "above 7168"
any 503 across the whole live acceptance	none (server log: 128 + 145 × 200, 18 × 422, 4 × 401)

The live pass is N = 1 for the positive control. It is read against the GPU 3 floor: the same image moved 0 of 560 rows across 3 restarts, and 0 against the bench's 4 restarts.

Building and deploying

# from nh3-dev. /opt/docker/src is root-owned, so create the version dir with sudo first.
ssh infra-ops@10.251.50.54 'sudo -n install -d -o infra-ops -g infra-ops /opt/docker/src/intern-decision-serve-X.Y.Z | cat'
tar -C services/intern-decision-serve -cf - --exclude=.venv --exclude=.pytest_cache --exclude=__pycache__ \
    --exclude=acceptance --exclude='*.egg-info' . \
  | ssh infra-ops@10.251.50.54 'tar -x -C /opt/docker/src/intern-decision-serve-X.Y.Z'
# on fv-ml1
cd /opt/docker/src/intern-decision-serve-X.Y.Z && docker build -t intern-decision-serve:X.Y.Z .
# from nh3-dev: compose.yaml, .env.example and this README (never .env)
scripts/deploy-stack.sh fv-ml1 intern-decision
# on fv-ml1: .env is infra-ops's, mode 0600. Rewrite it IN PLACE to keep the owner and mode.
cd /opt/docker/compose/intern-decision && new=$(sed 's/^IMAGE=.*/IMAGE=intern-decision-serve:X.Y.Z/' .env) \
  && printf '%s\n' "$new" > .env && docker compose config -q && docker compose up -d
# IMAGE CHANGE ONLY (not ordinary recreates — the triton-cache volume carries those):
# warm every length bucket once the container is healthy (~2 min, prints per-bucket times):
scripts/intern-decision-warmup        # token from vault; MAX_TOKENS read from /health

Before any deploy onto GPU 1, check two things. If either fails, stop; do not squeeze scriberr.

• nvidia-smi's own Free on GPU 1 must be at least 15,400 MiB: nvidia-smi -i 1 --query-gpu=memory.free --format=csv. That is our card peak of 9,876 MiB plus scriberr's 5,496, rounded up. Do not use total − used, which misses the driver's 640 MiB reserve.
• Scriberr must not be running a job. This command must print 0: docker logs --since 2m scriberr | grep -c "Processing single-track job".

Startup fails closed. A container that never reaches healthy did not pass its own checks: the inference.py hash, the pinned snapshot, the warm-up, the text-only swap and the prompt hash. Read docker logs intern-decision. Record the deploy with scripts/ops-log.

To move the model forward, change REVISION and INFERENCE_PY_SHA256 in config.py. Re-run the bench's sets through the service (services/intern-decision-serve/acceptance/), and re-measure the VRAM before changing the cap.

Rollback (INC → semif)

semif is stopped, not removed, and is kept as the rollback. Only on Prime's word:

cd /opt/docker/compose/intern-decision && docker compose stop     # first: the two do not both fit GPU 1
cd /opt/docker/compose/semif && docker compose start

Then check GPU 1's free memory against scriberr's peak. Callers go back to :8032 and secret get semif/api-token.