From 70f7c0e4a2811bcb333690d65e309a97b4e9b67b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 9 Sep 2026 22:44:13 -0700 Subject: [PATCH] =?UTF-8?q?docs(training-playbook):=20=C2=A73.11=20gains?= =?UTF-8?q?=20a=20fourth=20check=20=E2=80=94=20hybrid=20linear-attention?= =?UTF-8?q?=20carriers=20without=20a=20fused=20kernel?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The base-viability pre-flight had three checks (fits / MoE expert mapping / LoRA support) and would have passed Qwen3.5-0.8B-Base clean while it trained 2.6x slower than a dense model 2.3x its size. Check 4 closes that: read `layer_types` for a linear_attention majority AND probe for mamba_ssm / causal_conv1d / fla / kernels. It is the intersection that is slow -- a hybrid shape with the kernel present is fine, a dense shape does not care. Carries the measured table (gx10 GB10, n=10/arm, spreads 0.6-2.6%), plus the two things a hybrid Base checkpoint brings that a dense one does not: a vision tower and MTP head that target_modules="all-linear" would train on text, and the module rename that AutoModelForCausalLM introduces relative to the vLLM serving class; and unsafe cross-document packing, since SSM state ignores the attention mask. Section heading corrected from "three greps" to "four checks". The example was made runnable and verified on the box rather than shipped untested. --- docs/pfi/training-throughput-playbook.md | 66 +++++++++++++++++++++++- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 6079df3..1cc720b 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -325,10 +325,12 @@ The point is that it should be a *decision* made before the window, not a discovery made after — because the alternative it forecloses may be an architecture choice, and by then you have already trained. -## 3.11 Base-viability pre-flight — three greps, before you pick +## 3.11 Base-viability pre-flight — four checks, before you pick Run this on any candidate base BEFORE committing a training window. Each check -is minutes; skipping them cost a night in 2026-08. +is minutes; skipping them cost a night in 2026-08. Check 4 was added 2026-09-09 +after a measurement showed the *newest* carrier in a sweep training 2.6x slower +than a dense one 2.3x its size. **1. Does it fit for TRAINING?** BF16 weights on one card, with room for the real peak — not the weight figure. @@ -369,6 +371,66 @@ from `LlamaForCausalLM`. `mistral_large_3.py` greps as 0 for both and inherits only MRO resolution is right. (Same failure as asserting a substring instead of an effective value.) +**4. Is it a HYBRID linear-attention model — and is the fused kernel installed?** +⚠ **Newest is not fastest, and the penalty is an order of magnitude, not a +percentage.** Modern "small" checkpoints increasingly interleave Mamba-style +`linear_attention` blocks with a minority of real attention layers. Without a +fused kernel (`mamba_ssm`, `causal_conv1d`, or `fla`) `transformers` runs a +reference implementation, and the whole premise of picking a small carrier dies. + +```python +import importlib +from transformers import AutoConfig + +path = "/home/infra-ops/carriers/Qwen3.5-0.8B-Base" # or a hub repo id +cfg = AutoConfig.from_pretrained(path) +tc = getattr(cfg, "text_config", None) or cfg # multimodal configs nest it +lt = list(getattr(tc, "layer_types", []) or []) +print("full_attention:", lt.count("full_attention"), + "linear_attention:", lt.count("linear_attention")) + +for m in ("mamba_ssm", "causal_conv1d", "fla", "kernels"): + try: + importlib.import_module(m); print(f"{m:14s} OK") + except Exception: + print(f"{m:14s} MISSING") +``` + +Both halves are needed: a hybrid shape with the kernel present is fine, and a +dense shape does not care either way. It is the **intersection** that is slow. +The reusable instrument is `scripts/training-probes/bench_lora_step.py`, whose +raw output for this comparison is committed beside it. + +**Measured, pfi-gx10 (GB10), 2026-09-09** — identical harness, n=10 per arm, +seq 4096, LoRA r=32 on `q,k,v,o`+MLP, bf16 `sdpa`, grad-ckpt on, spreads +0.6–2.6%, no fused kernel present: + +| carrier | shape | params | tok/s | +|---|---|---|---| +| `Qwen3-1.7B-Base` | dense | 1.755 B | **1,415** | +| `Qwen3-0.6B-Base` | dense | 0.616 B | **2,399** | +| `Qwen3.5-0.8B-Base` | 18 linear / 6 full | 0.765 B | **540** | + +The dense 1.755 B carrier trains **2.6x faster than the hybrid 0.765 B one** on +2.3x the parameters — ~6x per parameter — while adapting *more* LoRA modules +(196 vs 96, since dense has attention in every layer). Grad checkpointing is not +the cause (19%, and it saves 2.6x memory: keep it on). Batching is not the lever: +both families are at the box's roofline at batch 1 (dense 1,415→1,439 and hybrid +540→546 tok/s at batch 4), so the gap is the kernel path, not a batching artefact. + +⚠ **Two more things a hybrid checkpoint brings that a dense one does not.** +(a) Its `*-Base` release may be **multimodal**: `Qwen3.5-*-Base` ships a vision +tower (153 `model.visual.*` Linear tensors at 0.8B, 297 at 2B) plus an MTP head, +all of which `target_modules="all-linear"` attaches LoRA to and then trains on +pure text — the same defect the Gemma-4 harness audit caught. Loading through +`AutoModelForCausalLM` drops both for free, **but it renames modules** +(`model.layers.N.*` vs the serving class's `model.language_model.layers.N.*`), so +adapter binding needs §3.10's serving-path proof and a sampled-target-changed +check. (b) **Cross-document packing is unsafe**: SSM recurrent state runs along +the sequence and an attention mask does not reset it, so packed documents bleed +in every linear-attention layer. One document per sequence, or prove the +boundary signal is honoured. + **Worked results, 2026-08-25:** | base | fits (1) | MoE mapping (2) | LoRA (3) | verdict |