54 lines
3.7 KiB
Markdown
54 lines
3.7 KiB
Markdown
# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
|
|
|
|
Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
|
|
stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
|
|
|
|
⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
|
|
sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
|
|
lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
|
|
a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
|
|
this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
|
|
|
|
Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
|
|
`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
|
|
|
|
⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
|
|
`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
|
|
(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
|
|
`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
|
|
'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
|
|
null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
|
|
|
|
## Run 4 — the corpus arm
|
|
|
|
Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
|
|
share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
|
|
training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
|
|
one-run-scoped, a run 5 needs a third).
|
|
|
|
**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
|
|
than accepting a "confirm this":**
|
|
|
|
1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
|
|
raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
|
|
hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
|
|
2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
|
|
null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
|
|
literally and requires the sha.
|
|
|
|
⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
|
|
taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
|
|
deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
|
|
brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
|
|
back out of the deployed CLEANROOT, which had the same defect one step removed.
|
|
|
|
**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
|
|
3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
|
|
(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
|
|
|
|
⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
|
|
`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
|
|
(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
|
|
iteration (safe), and the splitlines() paths touch only files written with the default
|
|
`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
|