diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 6079df3..1cc720b 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -325,10 +325,12 @@ The point is that it should be a *decision* made before the window, not a discovery made after — because the alternative it forecloses may be an architecture choice, and by then you have already trained. -## 3.11 Base-viability pre-flight — three greps, before you pick +## 3.11 Base-viability pre-flight — four checks, before you pick Run this on any candidate base BEFORE committing a training window. Each check -is minutes; skipping them cost a night in 2026-08. +is minutes; skipping them cost a night in 2026-08. Check 4 was added 2026-09-09 +after a measurement showed the *newest* carrier in a sweep training 2.6x slower +than a dense one 2.3x its size. **1. Does it fit for TRAINING?** BF16 weights on one card, with room for the real peak — not the weight figure. @@ -369,6 +371,66 @@ from `LlamaForCausalLM`. `mistral_large_3.py` greps as 0 for both and inherits only MRO resolution is right. (Same failure as asserting a substring instead of an effective value.) +**4. Is it a HYBRID linear-attention model — and is the fused kernel installed?** +⚠ **Newest is not fastest, and the penalty is an order of magnitude, not a +percentage.** Modern "small" checkpoints increasingly interleave Mamba-style +`linear_attention` blocks with a minority of real attention layers. Without a +fused kernel (`mamba_ssm`, `causal_conv1d`, or `fla`) `transformers` runs a +reference implementation, and the whole premise of picking a small carrier dies. + +```python +import importlib +from transformers import AutoConfig + +path = "/home/infra-ops/carriers/Qwen3.5-0.8B-Base" # or a hub repo id +cfg = AutoConfig.from_pretrained(path) +tc = getattr(cfg, "text_config", None) or cfg # multimodal configs nest it +lt = list(getattr(tc, "layer_types", []) or []) +print("full_attention:", lt.count("full_attention"), + "linear_attention:", lt.count("linear_attention")) + +for m in ("mamba_ssm", "causal_conv1d", "fla", "kernels"): + try: + importlib.import_module(m); print(f"{m:14s} OK") + except Exception: + print(f"{m:14s} MISSING") +``` + +Both halves are needed: a hybrid shape with the kernel present is fine, and a +dense shape does not care either way. It is the **intersection** that is slow. +The reusable instrument is `scripts/training-probes/bench_lora_step.py`, whose +raw output for this comparison is committed beside it. + +**Measured, pfi-gx10 (GB10), 2026-09-09** — identical harness, n=10 per arm, +seq 4096, LoRA r=32 on `q,k,v,o`+MLP, bf16 `sdpa`, grad-ckpt on, spreads +0.6–2.6%, no fused kernel present: + +| carrier | shape | params | tok/s | +|---|---|---|---| +| `Qwen3-1.7B-Base` | dense | 1.755 B | **1,415** | +| `Qwen3-0.6B-Base` | dense | 0.616 B | **2,399** | +| `Qwen3.5-0.8B-Base` | 18 linear / 6 full | 0.765 B | **540** | + +The dense 1.755 B carrier trains **2.6x faster than the hybrid 0.765 B one** on +2.3x the parameters — ~6x per parameter — while adapting *more* LoRA modules +(196 vs 96, since dense has attention in every layer). Grad checkpointing is not +the cause (19%, and it saves 2.6x memory: keep it on). Batching is not the lever: +both families are at the box's roofline at batch 1 (dense 1,415→1,439 and hybrid +540→546 tok/s at batch 4), so the gap is the kernel path, not a batching artefact. + +⚠ **Two more things a hybrid checkpoint brings that a dense one does not.** +(a) Its `*-Base` release may be **multimodal**: `Qwen3.5-*-Base` ships a vision +tower (153 `model.visual.*` Linear tensors at 0.8B, 297 at 2B) plus an MTP head, +all of which `target_modules="all-linear"` attaches LoRA to and then trains on +pure text — the same defect the Gemma-4 harness audit caught. Loading through +`AutoModelForCausalLM` drops both for free, **but it renames modules** +(`model.layers.N.*` vs the serving class's `model.language_model.layers.N.*`), so +adapter binding needs §3.10's serving-path proof and a sampled-target-changed +check. (b) **Cross-document packing is unsafe**: SSM recurrent state runs along +the sequence and an attention mask does not reset it, so packed documents bleed +in every linear-attention layer. One document per sequence, or prove the +boundary signal is honoured. + **Worked results, 2026-08-25:** | base | fits (1) | MoE mapping (2) | LoRA (3) | verdict |