vh
47cad33dd1
SemIf's shared scorer trims one token at the state boundary. When an object state's last value ends in ')', ';' or '}', the JSON that follows re-merges two tokens back, so score_shared refused the request with 422. The engine now wraps semif_phase1.shared._state_prefix to keep only the tokens the full prompts share. Each row scores the same token sequence; only the prefill/suffix split moves. Startup proves the fix is in effect, not just installed (heid bug hunt SKAL, folded). It checks that the hook is callable and is what score_shared resolves, that an ordinary state keeps upstream's whole prefix, and that a merge-prone state scores through the shared path. Real tokenizer: 154 states, 23 refused before and 0 after, with no ordinary or authored144 prefix changed. Acceptance: 144/144 parity. Shared vs direct 71/72; the miss is a bf16 tie that flipped across a plain restart (see README).
11 KiB
semif
SemIf option-logit decisions on fv-ml1 GPU 1, the utility card beside
vllm-coder, the erp/meromero seats and scriberr. Deployed 2026-09-27 at Prime's
request. Hosting assessment: infra-hermes, 2026-09-25.
SemIf (TheoLeeCJ/SemIf-OpenJev, MIT)
asks a small model a typed question and reads the answer straight from the logits of
the option letters, after one forward pass with no decoding. Upstream ships only a
batch CLI, so services/semif-serve/ wraps its two torch scorers in a small FastAPI
service. The contract is services/semif-serve/semif-serve.contract.md.
| URL | http://10.251.50.54:8032 (/health is open; POSTs need Authorization: Bearer $(secret get semif/api-token)) |
| Model | Qwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, BF16, from /tank/aimodels/huggingface (read-only, offline) |
| SemIf | commit 23cf1f39fc9534fe81437200959b6dfc7106e45a; torch 2.10.0+cu128, transformers 5.17.0, the same stack SemIf's committed predictions were made on |
| Image | semif-serve:<version>, built on fv-ml1 from services/semif-serve/ |
| State | none. If the weights are ever lost, re-pull them by hand at the pinned revision (the service itself never downloads: HF_HUB_OFFLINE=1, read-only mount). No backup needed beyond the host's /opt/docker restic. |
API
T=$(secret get semif/api-token)
curl -s -H "Authorization: Bearer $T" http://10.251.50.54:8032/decide -d '{
"id": "q1", "state": "Health checks passed in all three zones.",
"question": "Did the deployment succeed?",
"options": [{"id":"yes","description":"It succeeded."},{"id":"no","description":"It failed."}]}'
POST /decide: one decision, returning SemIf's result dict unchanged (option_ids,probabilities,option_logits,prompt_sha256,model, …).POST /decide/shared:{state, decisions: [{id, question, options}]}. It prefills the state once and scores every criterion in one batch. Use it when many questions share one long state.GET /health: the pins, limits and calibrated workloads.- Order averaging (0.1.3): add
"orderings": "rotations"to a decision, or"all"for ≤ 4 options. The option list is asked in every rotation inside ONE shared batch. The reply keeps each ordering's native result underorderingsand addscombined: {method, orderings, probabilities, top, agreement, spread}. Use it for anything real: a small model leans toward the first-listed option on ambiguous inputs, and averaging cancels that. Through the service on SemIf's labelled sets, accuracy goes from 78.6% to 88.1% (group-bootstrap 95% CI +5.1..+14.3 pts, 252 rows).agreementis the cheap confidence signal: unanimous rows are 94.5% accurate, split rows 76.4%. The orderings count toward the decision cap.workloadcalibration is not available together withorderingsyet (422). - Past
MAX_QUEUE(32) requests in progress, new POSTs get429 busybefore their body is read. - ⚠ Rotations cost options², not options. The options live in each row's suffix, and
the shared prefix is only the state. So
rotationsover n options sends n rows each carrying all n options. Measured 2026-09-27 on a short state, over 16 options of about 40 tokens each: 850 ms with rotations against 109 ms for one ordering (10,304 against 644 suffix tokens, 5 calls after warm-up). One cold call at that size returned 503. The VRAM table below covers binary decisions only. For many options, use one ordering or shorter option text. - Object states ending in
),;or}work as of 0.1.4. Until then, if the state was an object whose last value ended in one of those characters, the service returned 422 "The fixed state prefix does not match every full prompt". SemIf trims one token at the state boundary, and the JSON that follows re-merged two tokens back. The fix (contract INV-7) moves only where the shared prefix ends; every row still scores the same tokens. Measured with the real tokenizer: of 154 states (22 endings × 7 shapes) SemIf alone refused 23, and with the fix none. None of the 131 ordinary states, nor any of the authored144 states, changed its prefix. Startup proves the fix is live by scoring{"person_said": "ok :)"}through the shared path.
⚠ Probabilities are uncalibrated
SemIf labels its output "conditional option score; uncalibrated as decision confidence", and means it: on WANLI the model is right ~64% of the time while
reporting far higher confidence. Before a caller thresholds on probabilities, it
brings labelled rows (≥ ~150) for its workload. We fit one temperature T with
SemIf's benchmarks/calibrate.py, add {"<workload>": T} to
/opt/docker/conf/semif/calibration.json (stacks/semif/conf/), and restart. The
caller then passes "workload": "<name>" and gets a calibrated block beside the
native scores. The argmax never changes.
VRAM: a hard cap, released after every burst
VRAM_CAP_GIB=12 becomes torch.cuda.set_per_process_memory_fraction before the
weights load. At rest the process holds ~8.7 GB (nvidia-smi); the weights are 7.84
GiB. After any call that grows torch's reserved memory past the post-warm-up baseline
- 512 MiB, the engine calls
empty_cache(), so a burst returns to the card and does not squeeze scriberr, which shares GPU 1. A request that would exceed the cap gets503 out_of_memory, memory returns to baseline, and the service stays up. Both behaviours were verified on the card (0.1.0 held 11.9 GiB after an OOM, and 12.6 GB after a large request; 0.1.2 returns to 7.85 GiB in both cases).
What fits under 12 GiB (measured on 0.1.3, /decide/shared, binary decisions;
0.1.2 figures in brackets, before the fast kernels):
| state size (prefix tokens) | max rows in one request |
|---|---|
| ~140 | 63 (52) |
| ~520 | 51 (43) |
| ~1,960 | 26 (19) |
| ~3,900 | 16 (13) |
Rows = decisions × orderings, so rotations over 3 options uses 3 rows per decision.
/decide fits at the full 4,096-token limit. Past the table you get a 503, so split
the decisions across requests.
Fast kernels (0.1.3)
The image ships Qwen3.5's fast kernels, flash-linear-attention 0.5.2 and
causal-conv1d 1.7.0 (a prebuilt cu12/torch2.10 wheel). Without them transformers
logs that it falls back to "much slower" reference PyTorch paths. They are adopted
because an A/B on the empty GPU 3 showed:
- parity improved: 144/144 vs upstream (142/144 without), so upstream evidently ran with them;
- long inputs got much faster: a ~2,000-token
/decidewent from 169 to 92 ms server-side.
Short 3-rotation batches cost ~3–6 ms more; everything else is equal or faster.
⚠ triton builds a C shim at runtime, so the image carries gcc. Without it the
warm-up fails, and startup fails closed. Build without the kernels:
--build-arg EXTRAS="--extra model".
Latency (0.1.3, from nh3-dev, 3 runs × 20, network floor ~33 ms)
| request | end to end | server |
|---|---|---|
/decide, short (~130 tok) |
69 ms | 35 ms |
/decide, ~2,000-token state |
131 ms | 92 ms |
| 3 rotations, short | 115 ms | 78 ms |
| 6 orderings, short | 118 ms | 81 ms |
| 3 rotations, ~2,000-token state | 200 ms | 158 ms |
Acceptance (2026-09-27, v0.1.4)
Raw: services/semif-serve/acceptance/result-2026-09-27-v0.1.4.json. Parity with upstream
144/144 (identical prompt hashes, max prob gap 0.060). Deterministic within the process
(A-vs-A gap 0.0). The negative control fails as it should (14/144). Shared vs direct 71/72.
The one miss is an exact bf16 tie in the shared result (0.444/0.444). INV-7 did not move that
row's prefix. After a plain docker restart of the same image, the row read 0.369/0.537 and
agreed.
⚠ So "deterministic" holds within one process, not across restarts. Logits come in bf16 steps (0.125 here), and a near-tie can land differently after a restart, most likely because the fast kernels autotune at startup. That is n=1 row across one restart, and the cross-restart floor is otherwise unmeasured. When comparing two versions, compare them against that floor and not against zero.
Acceptance (2026-09-27, v0.1.3)
Raw: services/semif-serve/acceptance/result-2026-09-27-v0.1.3.json,
averaging-2026-09-27-v0.1.3.json. v0.1.3 matches upstream on 144/144 (identical
prompt hashes, max prob gap 0.049), is deterministic, fails the negative control as it
should (14/144), and shared matches direct on 72/72. The v0.1.2 table below is the
reference-kernel baseline.
Acceptance (2026-09-27, v0.1.2)
Raw: services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json.
| check | result |
|---|---|
| parity with SemIf's committed torch predictions (authored144) | 142/144 same top choice; 144/144 identical prompt SHA-256; max prob gap 0.093 |
| noise floor (same 144 twice) | 144/144, gap 0.0: deterministic |
| negative control (option descriptions rotated) | 14/144: the check catches a wrong answer |
| shared vs direct (36 shared states, 72 rows) | 72/72, max gap 0.045 |
| 21 binary criteria over one state, from nh3-dev | shared 159 ms (3 runs, 159–160) vs 21 sequential calls 981 ms |
The two parity misses are exact bf16 ties in our output (top-2 margin 0.000), where upstream's prefix-cache path gave margins of 0.054 and 0.185; one of the two now matches the label. So the misses come from the numeric path, not the wrapper. The speed figure includes one network round trip (~27 ms). The burst release costs ~16 ms on it (0.1.1 measured 143 ms without the release).
Building and deploying
# from nh3-dev. /opt/docker/src is root-owned, so create the version dir with sudo first.
ssh infra-ops@10.251.50.54 'sudo -n install -d -o infra-ops -g infra-ops /opt/docker/src/semif-serve-X.Y.Z'
tar -C services/semif-serve -cf - --exclude=.venv --exclude=.pytest_cache --exclude=__pycache__ \
--exclude=acceptance --exclude=spike --exclude='*.egg-info' . \
| ssh infra-ops@10.251.50.54 'tar -x -C /opt/docker/src/semif-serve-X.Y.Z'
# on fv-ml1
cd /opt/docker/src/semif-serve-X.Y.Z && docker build -t semif-serve:X.Y.Z .
# ⚠ /opt/docker/compose/semif is root-owned, so `sed -i` cannot write its temp file there. .env
# itself is infra-ops's: rewrite it IN PLACE, which keeps its owner and mode (0600).
cd /opt/docker/compose/semif && new=$(sed 's/^IMAGE=.*/IMAGE=semif-serve:X.Y.Z/' .env) \
&& printf '%s\n' "$new" > .env && docker compose up -d
Startup fails closed (INV-3, INV-7), so a container that does not reach healthy did not pass
its own checks: read docker logs semif. Record the deploy with scripts/ops-log.
To move SemIf forward: bump the commit in services/semif-serve/pyproject.toml
(and SEMIF_COMMIT in config.py), run uv lock, rebuild, and re-run the
acceptance. Upstream is research code that changes weekly, which is why it is
pinned.