vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gemma4-erp-tune): merged weights are mandatory — vLLM cannot LoRA any Gemma-4

Browse Source 
The §5 open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops
as it did on vLLM 0.24.0 (#47639), with merged weights as the fallback if it
did. Retested on vllm/vllm-openai:latest against the NVFP4A16 base plus the
live run's checkpoint adapter.

It does not no-op. It refuses to start:

    AttributeError: To support LoRA for MoE model,
                    'get_expert_mapping' must be implemented

And the reason is bigger than the quant. The check is in
vllm/lora/utils.py::process_packed_modules_mapping and branches on whether the
model is MoE — quantization is not in the condition. gemma4.py, gemma4_mm.py,
gemma4_mtp.py and gemma4_unified.py contain zero occurrences of
get_expert_mapping, while deepseek_v2, glm4_moe and ernie45_moe do implement
it. So vLLM cannot serve a LoRA on Gemma-4 at all, BF16 or quantized. Merging
is not a workaround for a quantization limitation; it is the only path for this
architecture.

This holds even though the adapter never touches experts —
validate_adapter_parameters forbids per-expert params, so all 205 targets are
attention and dense MLP. The refusal is about the model being MoE, not about
what the adapter targets.

Worth recording that the current behaviour is an improvement: a loud refusal
beats the 0.24.0 silent no-op, which would ship a base model wearing the tune's
name and pass every check that does not compare against base.
This commit is contained in:
Vuong Hoang
2026-08-25 01:32:51 -07:00
parent ab980e9345
commit 8de5f7a73c
1 changed files with 100 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/gemma4-erp-tune-sizing.md
+100 -5
View File
@@ -316,11 +316,34 @@ Also fold in:
- **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
context, graphs, or non-torch overhead — the same misreading that OOM'd the
char-rp seat.
- **Serving the result is not settled.** LoRA-on-NVFP4 hot-swap was a silent
no-op on vLLM 0.24.0 (#47639, proven quant-agnostic). Retest on the tagged
`vllm/vllm-openai:v0.27.1` already on disk. **If it still no-ops, the harness
must emit merged weights** — and Eitri needs that requirement while he is
early, not after the run.
- **SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason
we expected.** The open question was whether LoRA-on-NVFP4 hot-swap still
silently no-ops (it did on vLLM 0.24.0, #47639). Retested on
`vllm/vllm-openai:latest` with the NVFP4A16 base plus the run's own
checkpoint adapter. It does not no-op — **it refuses to start:**
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
⚠ **This is architectural, not quantization-related.** The check lives in
`vllm/lora/utils.py::process_packed_modules_mapping` and branches on whether
the model is MoE; quantization is not in the condition. `gemma4.py`,
`gemma4_mm.py`, `gemma4_mtp.py` and `gemma4_unified.py` contain **zero**
occurrences of `get_expert_mapping` (deepseek_v2, glm4_moe, ernie45_moe and
others do implement it). **vLLM cannot serve a LoRA on Gemma-4 at all —
BF16 or quantized.** Merging is the only path for this architecture.
Note this holds even though our adapter never touches experts:
`validate_adapter_parameters` forbids per-expert params, so all 205 targets
are attention + dense MLP. The refusal is about the *model* being MoE, not
about what the adapter targets.
Silver lining worth recording: a loud refusal is strictly better than the
0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name
and passes every check that does not compare against base.
The merge → quantize → serve pipeline is implemented and validated end to
end at [`scripts/erp-tune-serve/`](../../scripts/erp-tune-serve/).
---
@@ -380,6 +403,78 @@ Independently, the profiler kernel table (device rows only — see playbook §3.
**Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two
independent methods, 2.6 points apart.**
### 6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20%
`brokkr-smithy-dev`'s panel (arm: Bil) closed the fold by reading torch 2.13.0
and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather
than explaining it:
nominal work billed 27.1 TFLOPS × 34.85 s = 9.4e14 FLOP
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 FLOP
padded full layers lose the causal skip = +3.5e14 FLOP
------------------------------------------------------------------
work actually performed ≈ 1.8e15
in 34.85 s ≈ 51–61 TFLOPS
≈ 17–20% OF PEAK
**We divided the *intended* (windowed) FLOPs by the wall time the *dense*
reality took.** 17–20% is squarely inside the honest stock band. The hardware
is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic
the architecture specifies, and the excess is the sliding window being computed
and then thrown away.
Source-verified mechanism, no longer hypothesis:
| file | finding |
|---|---|
| `masking_utils.py:292-301` `_ignore_causal_mask_sdpa` | requires `kv_length < local_attention_size` to skip the mask. 16384 ≥ 1024, so **the sliding mask ALWAYS materialises at this seq len** — not sometimes, always |
| `sdp_utils_cpp.h:259-267`, `sdp_utils.cpp:933` | flash rejects **any** explicit mask |
| `sdp_utils.cpp:647`, `Context.h:480-485` | **cuDNN is unreachable on sm_120 twice over** — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12 |
| `attention.cu:1196/1759`, `kernel_forward.h:282-290` | mem-efficient has no mask gate and no head_dim cap, computes **full n×n** with the mask as additive bias; it trims only for `is_causal` |
Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding
layers land on mem-efficient computing dense O(n²), and **no backend on this
stack can rescue it.** cuDNN sliding-window does not exist at all — there is no
window argument in the public SDPA signature.
⚠ **Masked SDPA also blocks `enable_gqa`**, so KV gets `repeat_kv`-expanded on
every layer — extra memory traffic riding on top of the extra FLOPs.
### 6.1b Backend eligibility, measured — every source claim confirmed
Shapes-only, random weights, `sdpa_kernel()` pinning one backend at a time.
A forced failure is information: it identifies eligibility rather than
preference.
**Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:**
| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | **8.8 ms** | 21.6 ms | refused | 217.5 ms |
| explicit 1024-band 4D | refused | **55.5 ms** | refused | — |
| explicit causal+pad 4D | refused | 57.1 ms | refused | — |
**Global layers (5 of 30) — H_q16/H_kv2, D=512:**
| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | refused | **45.9 ms** | refused | 334.1 ms |
| explicit 1024-band 4D | refused | 104.4 ms | refused | — |
| explicit causal+pad 4D | refused | 107.9 ms | refused | — |
Refusal reasons, straight from the runtime warnings:
- `Flash Attention does not support non-null attn_mask` — kills flash for all
25 sliding layers, always, at this seq len
- `Flash attention requires q,k,v … less than or equal to 256. Got 512` —
kills flash for the 5 global layers **even fully causal**. Dvalin was right
that they are a first-class hole
- `head_dim should be no more than 128` — kills cuDNN on **both** shapes
**The sliding layers run at 55.5 ms where a maskless flash path would cost
8.8 ms — a 6.3× penalty, and it is unreachable through any config on this
stack.** That is the whole problem in one row.
### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card
`fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in