esh-pfi-infrastructure/persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md

[2026-09-04/05] vLLM RUNS on sm_121 — and the blocker was ninja not being on PATH, not the silicon

Long-standing open question closed: vLLM 0.28.0 serves on the GB10 (aarch64, sm_121) from a stock wheel, no source build. Model loads, torch.compile completes (~29 s), CUDA graphs capture.

⚠ The one trap, and it looks exactly like an sm_121 kernel problem: FlashInfer JIT-builds its sampling kernel at first use and needs ninja on PATH. It ships as a vLLM dependency but lives in the venv bin/, so the failure surfaces as FileNotFoundError: 'ninja' from deep inside a profile_run traceback. Same shape as the python3-dev trap that bit the training harness on this box, and the third present-but-not-on-PATH false-absence on this hardware (after nvcc).

Launch with both ~/vllm-env/bin and /usr/local/cuda/bin on PATH. Harmless and expected: Using default MoE config ... device_name=NVIDIA_GB10 — nobody has tuned MoE kernels for this chip.

⚠ Open WebUI sends tool_choice: "auto" on every request, which vLLM 400s unless launched with --enable-auto-tool-choice --tool-call-parser gemma4. Taken from the working sibling seat (gemma4-charrp), which is where the canonical flags live. Deliberately did NOT copy --reasoning-parser gemma4 — that seat pairs it with --default-chat-template-kwargs '{"enable_thinking": false}', and adding it alone moves output into reasoning_content with a null content, which OWUI renders as an empty reply. A 400 traded for a blank box.

Run 4 — the corpus arm

Run 4 adds an airoboros-3.2 instruct root (7,229 rows) and displaces kvasir 38% → 18% context share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped training-eligibility override operator-2026-09-04-rnd-run4 (the SECOND grant; run-1's was one-run-scoped, a run 5 needs a third).

Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather than accepting a "confirm this":

1. sample_kind: "instruct-dialogue" — core.py:521-544 dispatches on the literal string and raises on anything outside rp-dialogue/prose-chunk/actual-play-passage. Would have hard-stopped the encode on the first airoboros row. brokkr had labelled it non-blocking.
2. Recipe targets collapsed three dialogue roots into one descriptive label with root_sha256: null on three of four. Preflight resolves roots_dir/<dataset_id>/clean-v1/CLEANROOT.json literally and requires the sha.

⚠ I did NOT fill the missing shas in myself, though the values were known and it would have taken two minutes. root_sha256 is the author's assertion and preflight exists to check the deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate. brokkr then went further and recomputed his shas from shard bytes rather than reading them back out of the deployed CLEANROOT, which had the same defect one step removed.

The kvasir prefix cut has three independent confirmations: my build measured 1,613 samples / 3,347,622 ctx = 47.4% of run-3 kvasir, which reproduces the recipe's 18/38 share ratio (47.37%); the harness's own [mix] block then reported ctx 0.1805 against the 18.0% target.

⚠ str.splitlines() splits on U+2028, which json.dumps leaves unescaped under ensure_ascii=False. The harness splitlines() in four places and four roots DO carry U+2028 (fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file iteration (safe), and the splitlines() paths touch only files written with the default ensure_ascii=True. Latent, one flag away from live, not acting. Found by brokkr the hard way.