vh
47cad33dd1
SemIf's shared scorer trims one token at the state boundary. When an object state's last value ends in ')', ';' or '}', the JSON that follows re-merges two tokens back, so score_shared refused the request with 422. The engine now wraps semif_phase1.shared._state_prefix to keep only the tokens the full prompts share. Each row scores the same token sequence; only the prefill/suffix split moves. Startup proves the fix is in effect, not just installed (heid bug hunt SKAL, folded). It checks that the hook is callable and is what score_shared resolves, that an ordinary state keeps upstream's whole prefix, and that a merge-prone state scores through the shared path. Real tokenizer: 154 states, 23 refused before and 0 after, with no ordinary or authored144 prefix changed. Acceptance: 144/144 parity. Shared vs direct 71/72; the miss is a bf16 tie that flipped across a plain restart (see README).
193 lines
11 KiB
Markdown
193 lines
11 KiB
Markdown
# semif
|
||
|
||
**SemIf option-logit decisions** on **fv-ml1 GPU 1**, the utility card beside
|
||
`vllm-coder`, the erp/meromero seats and scriberr. Deployed 2026-09-27 at Prime's
|
||
request. Hosting assessment: infra-hermes, 2026-09-25.
|
||
|
||
SemIf ([TheoLeeCJ/SemIf-OpenJev](https://github.com/TheoLeeCJ/SemIf-OpenJev), MIT)
|
||
asks a small model a typed question and reads the answer straight from the logits of
|
||
the option letters, after **one forward pass with no decoding**. Upstream ships only a
|
||
batch CLI, so `services/semif-serve/` wraps its two torch scorers in a small FastAPI
|
||
service. The contract is `services/semif-serve/semif-serve.contract.md`.
|
||
|
||
| | |
|
||
|---|---|
|
||
| **URL** | `http://10.251.50.54:8032` (`/health` is open; POSTs need `Authorization: Bearer $(secret get semif/api-token)`) |
|
||
| **Model** | `Qwen/Qwen3.5-4B` @ `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`, BF16, from `/tank/aimodels/huggingface` (read-only, offline) |
|
||
| **SemIf** | commit `23cf1f39fc9534fe81437200959b6dfc7106e45a`; torch `2.10.0+cu128`, transformers `5.17.0`, the same stack SemIf's committed predictions were made on |
|
||
| **Image** | `semif-serve:<version>`, built on fv-ml1 from `services/semif-serve/` |
|
||
| **State** | none. If the weights are ever lost, re-pull them by hand at the pinned revision (the service itself never downloads: `HF_HUB_OFFLINE=1`, read-only mount). No backup needed beyond the host's `/opt/docker` restic. |
|
||
|
||
## API
|
||
|
||
```bash
|
||
T=$(secret get semif/api-token)
|
||
curl -s -H "Authorization: Bearer $T" http://10.251.50.54:8032/decide -d '{
|
||
"id": "q1", "state": "Health checks passed in all three zones.",
|
||
"question": "Did the deployment succeed?",
|
||
"options": [{"id":"yes","description":"It succeeded."},{"id":"no","description":"It failed."}]}'
|
||
```
|
||
|
||
- `POST /decide`: one decision, returning SemIf's result dict unchanged (`option_ids`,
|
||
`probabilities`, `option_logits`, `prompt_sha256`, `model`, …).
|
||
- `POST /decide/shared`: `{state, decisions: [{id, question, options}]}`. It prefills
|
||
the state once and scores every criterion in one batch. Use it when many questions
|
||
share one long state.
|
||
- `GET /health`: the pins, limits and calibrated workloads.
|
||
- **Order averaging (0.1.3):** add `"orderings": "rotations"` to a decision, or
|
||
`"all"` for ≤ 4 options. The option list is asked in every rotation inside ONE
|
||
shared batch. The reply keeps each ordering's native result under `orderings` and adds
|
||
`combined: {method, orderings, probabilities, top, agreement, spread}`. **Use it for
|
||
anything real:** a small model leans toward the first-listed option on ambiguous
|
||
inputs, and averaging cancels that. Through the service on SemIf's labelled sets,
|
||
accuracy goes from 78.6% to **88.1%** (group-bootstrap 95% CI +5.1..+14.3 pts, 252
|
||
rows). **`agreement` is the cheap confidence signal**: unanimous rows are 94.5%
|
||
accurate, split rows 76.4%. The orderings count toward the decision cap. `workload`
|
||
calibration is not available together with `orderings` yet (422).
|
||
- Past `MAX_QUEUE` (32) requests in progress, new POSTs get `429 busy` before their
|
||
body is read.
|
||
- ⚠ **Rotations cost options², not options.** The options live in each row's suffix, and
|
||
the shared prefix is only the state. So `rotations` over n options sends n rows each
|
||
carrying all n options. Measured 2026-09-27 on a short state, over 16 options of about
|
||
40 tokens each: **850 ms with rotations against 109 ms for one ordering** (10,304 against
|
||
644 suffix tokens, 5 calls after warm-up). One cold call at that size returned 503. The
|
||
VRAM table below covers binary decisions only. For many options, use one ordering or
|
||
shorter option text.
|
||
- **Object states ending in `)`, `;` or `}` work as of 0.1.4.** Until then, if the state was
|
||
an object whose last value ended in one of those characters, the service returned 422 "The
|
||
fixed state prefix does not match every full prompt". SemIf trims one token at the state
|
||
boundary, and the JSON that follows re-merged two tokens back. The fix (contract INV-7)
|
||
moves only where the shared prefix ends; every row still scores the same tokens. Measured
|
||
with the real tokenizer: of 154 states (22 endings × 7 shapes) SemIf alone refused 23, and
|
||
with the fix none. None of the 131 ordinary states, nor any of the authored144 states,
|
||
changed its prefix. Startup proves the fix is live by scoring `{"person_said": "ok :)"}`
|
||
through the shared path.
|
||
|
||
## ⚠ Probabilities are uncalibrated
|
||
|
||
SemIf labels its output `"conditional option score; uncalibrated as decision
|
||
confidence"`, and means it: on WANLI the model is right ~64% of the time while
|
||
reporting far higher confidence. **Before a caller thresholds on `probabilities`, it
|
||
brings labelled rows (≥ ~150) for its workload.** We fit one temperature `T` with
|
||
SemIf's `benchmarks/calibrate.py`, add `{"<workload>": T}` to
|
||
`/opt/docker/conf/semif/calibration.json` (`stacks/semif/conf/`), and restart. The
|
||
caller then passes `"workload": "<name>"` and gets a `calibrated` block beside the
|
||
native scores. The argmax never changes.
|
||
|
||
## VRAM: a hard cap, released after every burst
|
||
|
||
`VRAM_CAP_GIB=12` becomes `torch.cuda.set_per_process_memory_fraction` before the
|
||
weights load. At rest the process holds **~8.7 GB** (nvidia-smi); the weights are 7.84
|
||
GiB. After any call that grows torch's reserved memory past the post-warm-up baseline
|
||
+ 512 MiB, the engine calls `empty_cache()`, so a burst returns to the card and does
|
||
not squeeze scriberr, which shares GPU 1. A request that would exceed the cap gets
|
||
`503 out_of_memory`, memory returns to baseline, and the service stays up. Both
|
||
behaviours were verified on the card (0.1.0 held 11.9 GiB after an OOM, and 12.6 GB
|
||
after a large request; 0.1.2 returns to 7.85 GiB in both cases).
|
||
|
||
**What fits under 12 GiB** (measured on 0.1.3, `/decide/shared`, binary decisions;
|
||
0.1.2 figures in brackets, before the fast kernels):
|
||
|
||
| state size (prefix tokens) | max rows in one request |
|
||
|---|---|
|
||
| ~140 | 63 (52) |
|
||
| ~520 | 51 (43) |
|
||
| ~1,960 | 26 (19) |
|
||
| ~3,900 | 16 (13) |
|
||
|
||
Rows = decisions × orderings, so `rotations` over 3 options uses 3 rows per decision.
|
||
|
||
`/decide` fits at the full 4,096-token limit. Past the table you get a 503, so split
|
||
the decisions across requests.
|
||
|
||
## Fast kernels (0.1.3)
|
||
|
||
The image ships Qwen3.5's fast kernels, `flash-linear-attention` 0.5.2 and
|
||
`causal-conv1d` 1.7.0 (a prebuilt cu12/torch2.10 wheel). Without them transformers
|
||
logs that it falls back to "much slower" reference PyTorch paths. They are adopted
|
||
because an A/B on the empty GPU 3 showed:
|
||
- **parity improved**: 144/144 vs upstream (142/144 without), so upstream evidently
|
||
ran with them;
|
||
- **long inputs got much faster**: a ~2,000-token `/decide` went from 169 to 92 ms
|
||
server-side.
|
||
|
||
Short 3-rotation batches cost ~3–6 ms more; everything else is equal or faster.
|
||
⚠ triton builds a C shim at runtime, so the image carries `gcc`. Without it the
|
||
warm-up fails, and startup fails closed. Build without the kernels:
|
||
`--build-arg EXTRAS="--extra model"`.
|
||
|
||
## Latency (0.1.3, from nh3-dev, 3 runs × 20, network floor ~33 ms)
|
||
|
||
| request | end to end | server |
|
||
|---|---|---|
|
||
| `/decide`, short (~130 tok) | 69 ms | 35 ms |
|
||
| `/decide`, ~2,000-token state | 131 ms | 92 ms |
|
||
| 3 rotations, short | 115 ms | 78 ms |
|
||
| 6 orderings, short | 118 ms | 81 ms |
|
||
| 3 rotations, ~2,000-token state | 200 ms | 158 ms |
|
||
|
||
## Acceptance (2026-09-27, v0.1.4)
|
||
|
||
Raw: `services/semif-serve/acceptance/result-2026-09-27-v0.1.4.json`. Parity with upstream
|
||
144/144 (identical prompt hashes, max prob gap 0.060). Deterministic within the process
|
||
(A-vs-A gap 0.0). The negative control fails as it should (14/144). Shared vs direct 71/72.
|
||
The one miss is an exact bf16 tie in the shared result (0.444/0.444). INV-7 did not move that
|
||
row's prefix. After a plain `docker restart` of the same image, the row read 0.369/0.537 and
|
||
agreed.
|
||
|
||
⚠ **So "deterministic" holds within one process, not across restarts.** Logits come in bf16
|
||
steps (0.125 here), and a near-tie can land differently after a restart, most likely because
|
||
the fast kernels autotune at startup. That is n=1 row across one restart, and the cross-restart
|
||
floor is otherwise unmeasured. When comparing two versions, compare them against that floor
|
||
and not against zero.
|
||
|
||
## Acceptance (2026-09-27, v0.1.3)
|
||
|
||
Raw: `services/semif-serve/acceptance/result-2026-09-27-v0.1.3.json`,
|
||
`averaging-2026-09-27-v0.1.3.json`. v0.1.3 matches upstream on **144/144** (identical
|
||
prompt hashes, max prob gap 0.049), is deterministic, fails the negative control as it
|
||
should (14/144), and shared matches direct on 72/72. The v0.1.2 table below is the
|
||
reference-kernel baseline.
|
||
|
||
## Acceptance (2026-09-27, v0.1.2)
|
||
|
||
Raw: `services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`.
|
||
|
||
| check | result |
|
||
|---|---|
|
||
| parity with SemIf's committed torch predictions (authored144) | **142/144** same top choice; **144/144** identical prompt SHA-256; max prob gap 0.093 |
|
||
| noise floor (same 144 twice) | 144/144, gap 0.0: deterministic |
|
||
| negative control (option descriptions rotated) | 14/144: the check catches a wrong answer |
|
||
| shared vs direct (36 shared states, 72 rows) | 72/72, max gap 0.045 |
|
||
| 21 binary criteria over one state, from nh3-dev | shared **159 ms** (3 runs, 159–160) vs 21 sequential calls 981 ms |
|
||
|
||
The two parity misses are **exact bf16 ties in our output** (top-2 margin 0.000),
|
||
where upstream's prefix-cache path gave margins of 0.054 and 0.185; one of the two
|
||
now matches the label. So the misses come from the numeric path, not the wrapper. The
|
||
speed figure includes one network round trip (~27 ms). The burst release costs ~16 ms
|
||
on it (0.1.1 measured 143 ms without the release).
|
||
|
||
## Building and deploying
|
||
|
||
```bash
|
||
# from nh3-dev. /opt/docker/src is root-owned, so create the version dir with sudo first.
|
||
ssh infra-ops@10.251.50.54 'sudo -n install -d -o infra-ops -g infra-ops /opt/docker/src/semif-serve-X.Y.Z'
|
||
tar -C services/semif-serve -cf - --exclude=.venv --exclude=.pytest_cache --exclude=__pycache__ \
|
||
--exclude=acceptance --exclude=spike --exclude='*.egg-info' . \
|
||
| ssh infra-ops@10.251.50.54 'tar -x -C /opt/docker/src/semif-serve-X.Y.Z'
|
||
# on fv-ml1
|
||
cd /opt/docker/src/semif-serve-X.Y.Z && docker build -t semif-serve:X.Y.Z .
|
||
# ⚠ /opt/docker/compose/semif is root-owned, so `sed -i` cannot write its temp file there. .env
|
||
# itself is infra-ops's: rewrite it IN PLACE, which keeps its owner and mode (0600).
|
||
cd /opt/docker/compose/semif && new=$(sed 's/^IMAGE=.*/IMAGE=semif-serve:X.Y.Z/' .env) \
|
||
&& printf '%s\n' "$new" > .env && docker compose up -d
|
||
```
|
||
|
||
Startup fails closed (INV-3, INV-7), so a container that does not reach healthy did not pass
|
||
its own checks: read `docker logs semif`. Record the deploy with `scripts/ops-log`.
|
||
|
||
To move SemIf forward: bump the commit in `services/semif-serve/pyproject.toml`
|
||
(and `SEMIF_COMMIT` in `config.py`), run `uv lock`, rebuild, and **re-run the
|
||
acceptance**. Upstream is research code that changes weekly, which is why it is
|
||
pinned.
|