diff --git a/docs/pfi/gemma4-erp-tune-sizing.md b/docs/pfi/gemma4-erp-tune-sizing.md index c5e8959..a75d6d6 100644 --- a/docs/pfi/gemma4-erp-tune-sizing.md +++ b/docs/pfi/gemma4-erp-tune-sizing.md @@ -316,11 +316,34 @@ Also fold in: - **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA context, graphs, or non-torch overhead — the same misreading that OOM'd the char-rp seat. -- **Serving the result is not settled.** LoRA-on-NVFP4 hot-swap was a silent - no-op on vLLM 0.24.0 (#47639, proven quant-agnostic). Retest on the tagged - `vllm/vllm-openai:v0.27.1` already on disk. **If it still no-ops, the harness - must emit merged weights** — and Eitri needs that requirement while he is - early, not after the run. +- **SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason + we expected.** The open question was whether LoRA-on-NVFP4 hot-swap still + silently no-ops (it did on vLLM 0.24.0, #47639). Retested on + `vllm/vllm-openai:latest` with the NVFP4A16 base plus the run's own + checkpoint adapter. It does not no-op — **it refuses to start:** + + AttributeError: To support LoRA for MoE model, + 'get_expert_mapping' must be implemented + + ⚠ **This is architectural, not quantization-related.** The check lives in + `vllm/lora/utils.py::process_packed_modules_mapping` and branches on whether + the model is MoE; quantization is not in the condition. `gemma4.py`, + `gemma4_mm.py`, `gemma4_mtp.py` and `gemma4_unified.py` contain **zero** + occurrences of `get_expert_mapping` (deepseek_v2, glm4_moe, ernie45_moe and + others do implement it). **vLLM cannot serve a LoRA on Gemma-4 at all — + BF16 or quantized.** Merging is the only path for this architecture. + + Note this holds even though our adapter never touches experts: + `validate_adapter_parameters` forbids per-expert params, so all 205 targets + are attention + dense MLP. The refusal is about the *model* being MoE, not + about what the adapter targets. + + Silver lining worth recording: a loud refusal is strictly better than the + 0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name + and passes every check that does not compare against base. + + The merge → quantize → serve pipeline is implemented and validated end to + end at [`scripts/erp-tune-serve/`](../../scripts/erp-tune-serve/). --- @@ -380,6 +403,78 @@ Independently, the profiler kernel table (device rows only — see playbook §3. **Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two independent methods, 2.6 points apart.** +### 6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20% + +`brokkr-smithy-dev`'s panel (arm: Bil) closed the fold by reading torch 2.13.0 +and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather +than explaining it: + + nominal work billed 27.1 TFLOPS × 34.85 s = 9.4e14 FLOP + dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 FLOP + padded full layers lose the causal skip = +3.5e14 FLOP + ------------------------------------------------------------------ + work actually performed ≈ 1.8e15 + in 34.85 s ≈ 51–61 TFLOPS + ≈ 17–20% OF PEAK + +**We divided the *intended* (windowed) FLOPs by the wall time the *dense* +reality took.** 17–20% is squarely inside the honest stock band. The hardware +is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic +the architecture specifies, and the excess is the sliding window being computed +and then thrown away. + +Source-verified mechanism, no longer hypothesis: + +| file | finding | +|---|---| +| `masking_utils.py:292-301` `_ignore_causal_mask_sdpa` | requires `kv_length < local_attention_size` to skip the mask. 16384 ≥ 1024, so **the sliding mask ALWAYS materialises at this seq len** — not sometimes, always | +| `sdp_utils_cpp.h:259-267`, `sdp_utils.cpp:933` | flash rejects **any** explicit mask | +| `sdp_utils.cpp:647`, `Context.h:480-485` | **cuDNN is unreachable on sm_120 twice over** — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12 | +| `attention.cu:1196/1759`, `kernel_forward.h:282-290` | mem-efficient has no mask gate and no head_dim cap, computes **full n×n** with the mask as additive bias; it trims only for `is_causal` | + +Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding +layers land on mem-efficient computing dense O(n²), and **no backend on this +stack can rescue it.** cuDNN sliding-window does not exist at all — there is no +window argument in the public SDPA signature. + +⚠ **Masked SDPA also blocks `enable_gqa`**, so KV gets `repeat_kv`-expanded on +every layer — extra memory traffic riding on top of the extra FLOPs. + +### 6.1b Backend eligibility, measured — every source claim confirmed + +Shapes-only, random weights, `sdpa_kernel()` pinning one backend at a time. +A forced failure is information: it identifies eligibility rather than +preference. + +**Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:** + +| mask case | FLASH | EFFICIENT | CUDNN | MATH | +|---|---|---|---|---| +| `None` + `is_causal` | **8.8 ms** | 21.6 ms | refused | 217.5 ms | +| explicit 1024-band 4D | refused | **55.5 ms** | refused | — | +| explicit causal+pad 4D | refused | 57.1 ms | refused | — | + +**Global layers (5 of 30) — H_q16/H_kv2, D=512:** + +| mask case | FLASH | EFFICIENT | CUDNN | MATH | +|---|---|---|---|---| +| `None` + `is_causal` | refused | **45.9 ms** | refused | 334.1 ms | +| explicit 1024-band 4D | refused | 104.4 ms | refused | — | +| explicit causal+pad 4D | refused | 107.9 ms | refused | — | + +Refusal reasons, straight from the runtime warnings: + +- `Flash Attention does not support non-null attn_mask` — kills flash for all + 25 sliding layers, always, at this seq len +- `Flash attention requires q,k,v … less than or equal to 256. Got 512` — + kills flash for the 5 global layers **even fully causal**. Dvalin was right + that they are a first-class hole +- `head_dim should be no more than 128` — kills cuDNN on **both** shapes + +**The sliding layers run at 55.5 ms where a maskless flash path would cost +8.8 ms — a 6.3× penalty, and it is unreachable through any config on this +stack.** That is the whole problem in one row. + ### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card `fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in