esh-pfi-infrastructure/docs/pfi/gemma4-erp-tune-sizing.md

# Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication

_Measured 2026-08-24 on `ana-ml2` against
`/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated trainee)._

Division of labour for this run: **Eitri writes the harness, brokkr-smithy-dev
audits, infra-ops owns the GPU window and executes.** This document is the
sizing infra-ops owes; it is arithmetic against the real checkpoint and the
real card, not an estimate.

---

## 1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE

**The proposed shape was QLoRA r64. It cannot be run as specified**, and the
reason is structural rather than a tuning preference.

The checkpoint stores each layer's 128 experts as **two fused 3-D
`nn.Parameter` tensors**, not as 128 `nn.Linear` modules:

    model.language_model.layers.N.experts.gate_up_proj    BF16 [128, 1408, 2816]
    model.language_model.layers.N.experts.down_proj       BF16 [128, 2816,  704]

Note the absence of a `.weight` suffix — compare `mlp.down_proj.weight`
(an `nn.Linear`) against `experts.down_proj` (a bare parameter). That is the
tell, and it is decisive: **`bitsandbytes` 4-bit replacement walks `nn.Linear`
modules.** A fused 3-D parameter is not one, so it is skipped and stays BF16.

What `load_in_4bit=True` would actually buy on this model:

| block | params | BF16 | after bnb NF4 | saved |
|---|---:|---:|---:|---:|
| **MoE experts** (fused 3-D — **NOT quantized**) | 22.84 B | 42.54 GiB | **42.54 GiB** | **0** |
| lm attention (`nn.Linear`) | 1.11 B | 2.07 GiB | 0.52 GiB | 1.55 |
| dense shared MLP (`nn.Linear`) | 0.54 B | 1.00 GiB | 0.25 GiB | 0.75 |
| vision tower (`nn.Linear`) | 0.57 B | 1.06 GiB | 0.27 GiB | 0.79 |
| embed (tied, normally kept BF16) | 0.74 B | 1.38 GiB | 1.38 GiB | 0 |
| router + norms | 0.01 B | 0.02 GiB | 0.02 GiB | 0 |
| **total** | **25.81 B** | **48.07 GiB** | **~44.98 GiB** | **~3.1 GiB** |

**88.5% of the model is in tensors bitsandbytes cannot touch.** "QLoRA" here
means paying the NF4 dequant tax on 6% of the weights to save 6% of the
footprint. The premise does not survive contact with the checkpoint.

> **Eitri: do not hard-code a `BitsAndBytesConfig` / `load_in_4bit` path.**
> It will not error loudly — it will load, report a 4-bit model, and quietly
> leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.

**The one thing that could overturn this** is a third-party fork shipping
custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth
is the candidate). **Not chased, deliberately** — see §4, where the run fits in
BF16 without displacing anything the fleet depends on, which collapses QLoRA's
value to zero. If it is ever revisited, it must be *before* the harness
hard-codes a quantization path, not after.

**Verdict: plain LoRA on BF16 weights.**

---

## 2. What the run actually costs

Adapter targeting `q_proj,k_proj,v_proj,o_proj` at r64, computed from the real
tensor shapes:

| | layers | per layer | total |
|---|---:|---:|---:|
| sliding-attention (q 4096, kv 2048, o 4096) | 25 | 1,507,328 | 37,683,200 |
| full-attention (q 8192, kv 1024, o 8192) | 5 | 1,654,784 | 8,273,920 |
| **trainable** | | | **45,957,120** (0.178% of base) |

⚠ **`v_proj` DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29.** Those are the
`full_attention` layers, and `attention_k_eq_v: true` means one projection
serves both K and V. Consequences the harness must respect:

- PEFT matches by name suffix, so a `v_proj` target **silently produces no
  adapter** on those five layers. Do not assert a fixed adapter count.
- Adapting `k_proj` on a global layer **adapts K and V simultaneously** — a
  different intervention than on the sliding layers. If that asymmetry matters
  to the recipe, say so explicitly rather than discovering it in the loss curve.

### Memory budget, batch 1, `max_seq_len` 8192

| item | GiB | note |
|---|---:|---|
| base weights BF16 | 48.07 | measured: 25,805,936,206 params × 2 B |
| adapters + grads + AdamW fp32 m/v | 0.75 | 45.96 M trainable — rounding error |
| checkpointed layer inputs | 1.29 | 30 × 8192 × 2816 × 2 B |
| recompute peak, one layer | ~2.5 | 8192 tok × top-8 of 128, `moe_intermediate 704` |
| loss head, **fused/chunked CE** | ~2.0 | see the warning below |
| CUDA context + cuBLAS + fragmentation | ~3.0 | the item `--gpu-memory-utilization` never covered |
| **total** | **~57.6** | |

Marginal cost per extra sequence in the micro-batch: **~2.5 GiB.**

| micro-batch | GiB |
|---:|---:|
| 1 | 54.3 |
| 2 | 56.8 |
| **4** | **61.8** |
| 6 | 66.8 |
| 8 | 71.8 |

### ⚠ The loss head is the whole ballgame, and it is not in the brief

`vocab_size` is **262,144** and `final_logit_softcapping` is **30.0**. One
8192-token sequence produces **2.147 billion logits**. Through a naive HF
`ForCausalLM` loss that is:

    BF16 logits          4.0 GiB
    fp32 upcast          8.0 GiB
    softcap tanh saved   8.0 GiB   (autograd keeps the pre-cap tensor)
    softmax + grad       8.0 GiB
    ------------------------------
    ~28-30 GiB transient, at BATCH 1

Naive CE at batch 1 lands the run at **~85.6 GiB on a 95.6 GiB card** — it will
appear to work and then OOM on the first long sample. At micro-batch 4 it is
~120 GiB and never starts. **Fused/chunked linear cross-entropy is mandatory,
not an optimization.**

⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with
softcapping may not have one. Three ways out, in order of preference —
(a) generic `LigerFusedLinearCrossEntropyLoss` wired against the lm_head with
softcapping applied inside the chunk; (b) `cut-cross-entropy`; (c) hand-rolled
sequence-chunked CE. **This must be proven on a 10-step smoke run before the
window is booked**, because everything else in this document assumes it works.

### Step count

58.2 M tokens / 20,576 samples = **2,829 tokens/sample average** — well under
8192, so packing matters.

- Packed to 8192: **7,104 sequences.** At micro-batch 4 × grad-accum 4
  (effective 16) → **444 optimizer steps for the whole epoch.**
- ⚠ That is a *small* step count. A "checkpoint every 100 steps" default gives
  four checkpoints across a multi-hour run. This is exactly why the amendment
  asked for **wall-clock-interval checkpointing, not step-count** — the case is
  now concrete, not hypothetical.
- ⚠ **Packing must use `position_ids` + varlen/block-diagonal attention.** Naive
  concatenation bleeds samples into each other. `sliding_window` is 1024 on 25
  of 30 layers so the damage is bounded there — but the 5 `full_attention`
  layers see the entire packed sequence.

**Open question for brokkr/Eitri:** what fraction of the 20,576 samples exceed
8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting
the ends off RP scenes, which is where the signal lives.

### Runtime

Active parameters per token ≈ **3.67 B** (2.93 B routed + attention, plus the
0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute
≈ 6 × active × tokens = **1.28e18 FLOPs** for the epoch.

At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts
are not efficient — **4 to 10 hours, most likely ~6.** Treat as a band, not a
number; it will be measured on the smoke run.

---

## 3. Where it fits (measured 2026-08-24, 18:20 PDT)

Card total: 97,887 MiB = **95.60 GiB** each.

| | GPU0 | GPU1 |
|---|---|---|
| resident before the window | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |

Three placements were on the table:

- **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
  short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
  38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
  current number is budgeting against a moving one.
- **GPU1 with `mog-sec` stopped: 76,070 MiB free.** Fits, but shares a card with
  four small seats and Scriberr.
- **GPU0 with `gen` MOVED OFF: the whole card.** ← what was chosen.

---

## 4. The window, as executed

**Operator call, 2026-08-24: move `gen` to GPU1 and stand `sec` down, so GPU0 is
emptied completely rather than shared.** This is strictly better than training
beside `gen`: the tune gets 95.60 GiB with no co-tenant, and the fleet's general
seat never goes dark beyond its own ~5-minute restart.

    before:  GPU0 [ gen 42.5 ]                     GPU1 [ sec 55.3 | small seats 20.7 ]
    after:   GPU0 [ ---- empty, 95.60 GiB ---- ]   GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]

`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin:

| | `gen` | `sec` |
|---|---|---|
| aliases | 7 (`gen`, `gen-reasoning`, `chat-judge`, `image-judge`, summarizer/classifier family) | 2 (`sec`, `sec-reasoning`) |
| standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
| measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |

Traffic to `sec` arrives from `10.250.50.70` (the LiteLLM gateway), so the
aliases will fail at the gateway for the duration. Per the standing rule, let
them fail — **do not route `sec` to another model as a stand-in.**

Both directions are playbooks, and **the order in each is load-bearing**:

    scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
    scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml

⚠ `gen` runs at `--gpu-memory-utilization 0.43`, which vLLM reads as a fraction
of **total** card memory: 42,091 MiB must be *free at startup* or the engine
refuses to boot. GPU1 has 19,446 MiB free while `mog-sec` is up. **Recreating
`gen` onto GPU1 before stopping `mog-sec` takes the fleet's main seat down and
leaves it down.** The open playbook stops `mog-sec` first and hard-gates on the
freed memory; the close playbook mirrors it, because `mog-sec` needs 50,901 MiB
of its own and cannot start until `gen` has vacated GPU1.

⚠ Invoke elway as `infra-ops@10.250.50.54`, not the `ana-ml2` ssh-target — that
resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
probe.

### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic

Everything above was arithmetic. This was run on the real checkpoint on GPU0
with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer:

| config | peak | verdict |
|---|---:|---|
| naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare |
| naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB |
| chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ |
| **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** |
| chunked CE, bsz4 seq 16384 | **OOM** | — |

**The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB
estimated.** The estimate modelled gradient checkpointing as storing only layer
inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens ×
top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size
an MoE run from dense-model intuition — measure it.**

Two predictions did land exactly, which is why the rest of the model of the thing
is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and
**74,342,400 trainable params** at r64.

The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq
8192.** Chunking is not an optimisation, it is what makes brokkr's 16384
recommendation reachable at all.

Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure.

### Revised run parameters, now that it is a whole card

**FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2,
`gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer
steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear.

`max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap
drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated
*entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon —
which is 60% of the mix and the axis the seat exists for. Prose and fireball
truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.

⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the
full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751
the typical step sits far below it.

⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()`
with it. Dropping checkpointing looks like ~17% off wall-clock and instead
forces micro-batch 1. Worse, the second call is the silent one: **without
`enable_input_require_grads()` the frozen base produces no gradient through the
checkpointed blocks, every adapter stays at its initialisation, and the run
completes successfully with an inert adapter.** `prepare_model_for_kbit_training`
used to do it as a side effect of the 4-bit path — so removing 4-bit removes it
too, and nothing warns you.

### Harness changes this required (eitri-smithy `62b556b`)

`9d64257` as audited would not have run here. Four fixes:

1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config
   key, defaulting off, per §1.
2. Sequence-chunked CE replacing the model's own loss (the measured table above).
3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own
   stale 365-line template and there was **no override parameter anywhere**, so
   the upstream-template requirement was not expressible in the code.
4. Gradient checkpointing + `enable_input_require_grads()`.

Plus `training_eligibility_override` / `overridden_blockers` /
`substitute_controls` in the provenance manifest, and `device_map` pinned to
device 0 so the run cannot stray onto the card holding the inference seats.

Also fold in:

- **Scriberr STAYS on GPU1.** (An earlier draft of this document suggested moving
  it to GPU0; that was written when training was going to live on GPU1, and it is
  now exactly backwards. GPU0 is the training card and wants no co-tenant.)
- **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
  (36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
  would come close to filling it. `/tank` has 4.0 TB.
- **The run is still resumable-by-design** (INV-T7 + wall-clock checkpointing).
  Nothing about a dedicated card removes that requirement — a 4–10 hour window
  is long enough that an unresumable run is a bad bet regardless of who owns the
  GPU.

---

## 5. Standing warnings that apply to this run

- **Never render training examples through the base's own
  `chat_template.jinja`.** Every third-party Gemma-4 derivative ships a stale
  one; the trainee's is 365 lines against upstream's 390. Use
  `/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. Training through
  the wrong template is train/serve skew with no error — it presents as a
  tuning failure.
- **Base path and chat-template path are config keys, not constants.** The
  trainee base already moved once (stock BF16 → `-heretic-bf16`).
- **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
  context, graphs, or non-torch overhead — the same misreading that OOM'd the
  char-rp seat.
- **SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason
  we expected.** The open question was whether LoRA-on-NVFP4 hot-swap still
  silently no-ops (it did on vLLM 0.24.0, #47639). Retested on
  `vllm/vllm-openai:latest` with the NVFP4A16 base plus the run's own
  checkpoint adapter. It does not no-op — **it refuses to start:**

      AttributeError: To support LoRA for MoE model,
                      'get_expert_mapping' must be implemented

  ⚠ **This is architectural, not quantization-related.** The check lives in
  `vllm/lora/utils.py::process_packed_modules_mapping` and branches on whether
  the model is MoE; quantization is not in the condition. `gemma4.py`,
  `gemma4_mm.py`, `gemma4_mtp.py` and `gemma4_unified.py` contain **zero**
  occurrences of `get_expert_mapping` (deepseek_v2, glm4_moe, ernie45_moe and
  others do implement it). **vLLM cannot serve a LoRA on Gemma-4 at all —
  BF16 or quantized.** Merging is the only path for this architecture.

  Note this holds even though our adapter never touches experts:
  `validate_adapter_parameters` forbids per-expert params, so all 205 targets
  are attention + dense MLP. The refusal is about the *model* being MoE, not
  about what the adapter targets.

  Silver lining worth recording: a loud refusal is strictly better than the
  0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name
  and passes every check that does not compare against base.

  The merge → quantize → serve pipeline is implemented and validated end to
  end at [`scripts/erp-tune-serve/`](../../scripts/erp-tune-serve/).

---

## 6. Round-1 aborted; throughput root-caused (measured 2026-08-24 22:00 PDT)

Run-01 launched, reached step 19 of 1,312 at ~35–46 s/it, and was **killed by
operator instruction** — not a crash, not an OOM. ETA was ~13.9 h at 8.6% MFU
and the operator elected to root-cause before spending the window.

Nothing was destroyed: the 609 MB encode cache, `order-manifest.jsonl`,
`truncation-report.json` and `resume-run-01.sh` are all preserved at
`/tank/erp-tune/run-01/`. **There are no checkpoints** — the first was due at
step 100, so brokkr's `lora_B` inert-adapter gate never ran. That question is
open and moves to the restart.

Model-agnostic lessons from this investigation are in
[`training-throughput-playbook.md`](training-throughput-playbook.md); the
probes are at [`scripts/training-probes/`](../../scripts/training-probes/).
What follows is Gemma-4-specific.

> ## ⚠⚠ CORRECTION 2026-08-26 — MUCH OF THIS SECTION MEASURES THE WRONG PROCESS
>
> **The benchmarks below were run against the SERVING seat with
> `attn_implementation="sdpa"` set explicitly. Training was running
> `flex_attention` the whole time.** `ATTN_IMPLEMENTATION = "flex_attention"`
> was a module constant passed unconditionally into `from_pretrained`, and
> run 1's step-time distribution (n=1,445; min 11.84 / p50 19.75 / p99 30.52 /
> max 45.79 s/it, the max being step 1's compile) confirms it stayed compiled —
> a dynamo fallback sits in the hundreds of seconds per step.
>
> **FALLS** — describes sdpa, not the training run:
> the three-point scaling fit and its 68% quadratic share; the kernel table
> (`fmha_cutlassF/B` sm80, `EFFICIENT_ATTENTION`, attention 65.2%); the **8.6%
> MFU** figure quoted above and throughout; the projection that elementwise
> becomes the largest line item post-fix; and "adopt `flex_attention`" as the
> round-two headline lever — **which round one already had.**
>
> **SURVIVES** — measured on the live training run:
> the padding/bucketing win (44.3 → 20.1 s/it); the zero-pad fast-path
> second-order effect; the eval-battery noise-floor work.
>
> ⚠ **Do not assume the direction of the correction.** Training's real MFU is
> *unmeasured*, not obviously better. Flex with a BlockMask ought to beat
> dense-masked sdpa, but that is a prediction and this investigation has been
> unkind to those.
>
> The root cause was procedural, not technical, and it is written up as
> playbook **§4.3**: run 1 recorded no attention backend in its provenance, so
> the benchmark/trainer delta was invisible and nobody enumerated it. Run 2
> onward records `attn_implementation_requested` **and** `_resolved`, plus the
> torch/transformers versions and dynamo's compile counters.

### 6.1 Where the step time goes

Real checkpoint, GPU0, `attn_implementation="sdpa"`, PEFT + gradient
checkpointing + the chunked CE, fwd+bwd, best-of-2 after warmup:

| shape | time | peak |
|---|---:|---:|
| 2 × 2,048 | 1.776 s | 53.2 GiB |
| 2 × 8,192 | 11.570 s | 62.3 GiB |
| 2 × 16,384 | **35.017 s** | 76.6 GiB |

Fitting `t(w) = A·w + B·w²` over all three (per-sequence `w`, batch 2):

    A = 6.8715e-04 s/token        B = 8.8509e-08 s/token²

| w | predicted | measured | linear | quadratic | quad share |
|---:|---:|---:|---:|---:|---:|
| 2,048 | 1.779 | 1.776 | 1.407 | 0.371 | 20.9% |
| 8,192 | 11.569 | 11.570 | 5.629 | 5.940 | 51.3% |
| 16,384 | 35.017 | 35.017 | 11.258 | 23.759 | **67.8%** |

**Two terms, three points, residuals under 3 ms across an 8× range.** No fixed
per-batch term was needed, which refutes the launch-bound hypothesis outright —
~3,840 expert-GEMM launches per forward are not the cost.

Independently, the profiler kernel table (device rows only — see playbook §3.4):

| device kernel | ms | of step |
|---|---:|---:|
| `fmha_cutlassB_bf16_aligned_128x64_k65536_sm80` (attn BWD) | 16,144.6 | 46.1% |
| `fmha_cutlassF_bf16_aligned_32x128_gmem_sm80` (attn FWD) | 6,691.2 | 19.1% |
| `cutlass_80_tensorop_bf16_s16816gemm` ×3 (dense GEMM) | 2,774.0 | 7.9% |
| elementwise / vectorized / unrolled ×9 | 4,787.4 | 13.7% |
| gather / Memcpy DtoD / dropout | 951.6 | 2.7% |
| **attention total** | **22,835.8** | **65.2%** |

**Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two
independent methods, 2.6 points apart.**

### 6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20%

`brokkr-smithy-dev`'s panel (arm: Bil) closed the fold by reading torch 2.13.0
and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather
than explaining it:

    nominal work billed     27.1 TFLOPS × 34.85 s        =  9.4e14 FLOP
    dense-sliding extra     25 layers, 2 seqs, 4 passes  = +8.2e14 FLOP
    padded full layers      lose the causal skip         = +3.5e14 FLOP
    ------------------------------------------------------------------
    work actually performed                              ≈  1.8e15
    in 34.85 s                                           ≈  51–61 TFLOPS
                                                         ≈  17–20% OF PEAK

**We divided the *intended* (windowed) FLOPs by the wall time the *dense*
reality took.** 17–20% is squarely inside the honest stock band. The hardware
is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic
the architecture specifies, and the excess is the sliding window being computed
and then thrown away.

Source-verified mechanism, no longer hypothesis:

| file | finding |
|---|---|
| `masking_utils.py:292-301` `_ignore_causal_mask_sdpa` | requires `kv_length < local_attention_size` to skip the mask. 16384 ≥ 1024, so **the sliding mask ALWAYS materialises at this seq len** — not sometimes, always |
| `sdp_utils_cpp.h:259-267`, `sdp_utils.cpp:933` | flash rejects **any** explicit mask |
| `sdp_utils.cpp:647`, `Context.h:480-485` | **cuDNN is unreachable on sm_120 twice over** — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12 |
| `attention.cu:1196/1759`, `kernel_forward.h:282-290` | mem-efficient has no mask gate and no head_dim cap, computes **full n×n** with the mask as additive bias; it trims only for `is_causal` |

Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding
layers land on mem-efficient computing dense O(n²), and **no backend on this
stack can rescue it.** cuDNN sliding-window does not exist at all — there is no
window argument in the public SDPA signature.

⚠ **Masked SDPA also blocks `enable_gqa`**, so KV gets `repeat_kv`-expanded on
every layer — extra memory traffic riding on top of the extra FLOPs.

### 6.1b Backend eligibility, measured — every source claim confirmed

Shapes-only, random weights, `sdpa_kernel()` pinning one backend at a time.
A forced failure is information: it identifies eligibility rather than
preference.

**Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:**

| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | **8.8 ms** | 21.6 ms | refused | 217.5 ms |
| explicit 1024-band 4D | refused | **55.5 ms** | refused | — |
| explicit causal+pad 4D | refused | 57.1 ms | refused | — |

**Global layers (5 of 30) — H_q16/H_kv2, D=512:**

| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | refused | **45.9 ms** | refused | 334.1 ms |
| explicit 1024-band 4D | refused | 104.4 ms | refused | — |
| explicit causal+pad 4D | refused | 107.9 ms | refused | — |

Refusal reasons, straight from the runtime warnings:

- `Flash Attention does not support non-null attn_mask` — kills flash for all
  25 sliding layers, always, at this seq len
- `Flash attention requires q,k,v … less than or equal to 256. Got 512` —
  kills flash for the 5 global layers **even fully causal**. Dvalin was right
  that they are a first-class hole
- `head_dim should be no more than 128` — kills cuDNN on **both** shapes

**The sliding layers run at 55.5 ms where a maskless flash path would cost
8.8 ms — a 6.3× penalty, and it is unreachable through any config on this
stack.** That is the whole problem in one row.

### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card

`fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in
this path at all, and the forward is additionally on `gmem` — the
global-memory fallback tier of the memory-efficient backend, selected when the
working set will not fit in shared memory.

This is the mechanism behind the 100%-SM / 27-TFLOPS / 304-TFLOPS-capable
reading: the chip is saturated running a kernel generation behind on the
dominant cost centre.

The candidate fix is a purpose-built kernel for this architecture's mixed
256/512 head-dim split — `zzhhjjj/gemma-triton-flash-attn`
(`register_triton_attention()`, then `_attn_implementation = "triton_gqa"`),
reported 9.23× over SDPA at N=16K D=256 SWA and 2.94× fwd+bwd at D=512.
`flex_attention` + `BlockMask` is the no-new-dependency alternative.

⚠ **Prefer a UNIFORM backend over a per-layer split.** vLLM special-cased this
exact mixed-head-dim architecture and measured mixed backends **8% slower** than
uniform. And `attn_implementation` is all-or-nothing at `from_pretrained` /
`set_attn_implementation` — per-layer routing requires a custom function
registered on `ALL_ATTENTION_FUNCTIONS` branching on `module.head_dim` /
`sliding_window`.

⚠ **FA2 is not available for this model**: it caps head_dim at 256 and the 5
global layers are at 512. FA3 is Hopper-only. Do not bet on FA4 on sm_120.

### 6.3 Masking is CORRECT — and padding is what costs

Band structure asserted directly against the real config at n=16,384:

    sliding_attention   max 1,024 allowed/row, saturates at row 1,023   PASS

Constraints were **not** silently dropped; the 25 sliding layers were genuinely
windowed. Run-01 was training the model we intended.

The same probe found the mechanism nobody had measured:

| 2D mask supplied | `full_attention` mask returned |
|---|---|
| `None` | **`None`** → `is_causal` fast path AVAILABLE |
| all-ones (no padding) | **`None`** → `is_causal` fast path AVAILABLE |
| right-padded (what `collate_mixed` emits) | 4D `16384²` → **fast path LOST** |

**Padding is what pins the 5 global layers to an explicit mask.** The 25
sliding layers get a 4D tensor either way — `sdpa_attention_forward` sets
`is_causal=True` only when `attention_mask is None`, and a 1024 window cannot
be expressed as `is_causal`.

Isolated, same width, only the mask differing:

    2 × 16,384, no padding          35.244 s   26,048 loss targets
    2 × 16,384, 50% pad on row 1    38.567 s   19,640 loss targets

**9.4% slower for 24% less work.**

### 6.4 The corpus is 29.9% padding — and bucketing is the biggest win available

Measured off the preserved encode cache in true `SequentialSampler` order:

    records                  20,982   (3,583 rp-dialogue / 12,003 prose-chunk / 5,396 actual-play)
    seq len min/mean/max     142 / 2,752 / 16,384
    micro-batches (mb=2)     10,491
    real tokens              57,733,156
    padded tokens            82,337,318
    PADDING WASTE            29.9%
    mb width p50/p90/p99     2,092 / 10,634 / 16,341
    micro-batches at 16,384  3 of 10,491  (0.0%)

⚠ Note the last line against §6.1: **the 2 × 16,384 benchmark shape occurs in
three micro-batches out of 10,491.** Weighted over the real distribution the
quadratic share is ~51%, not 67.8%.

**Bucket-to-pair, shuffle-to-mix** (brokkr's design, validated on measured
lengths — form micro-batches within length buckets, then shuffle the resulting
*micro-batches* globally):

| bucket | waste | predicted step | zero-pad mb | roots/accum window |
|---:|---:|---:|---:|---:|
| current | 29.9% | 44.3 s → 16.13 h | 0.1% | 3.68 |
| **2** | **0.0%** | **28.6 s → 10.40 h** | **78.3%** | 3.56 |
| 8 | 0.0% | 28.6 s → 10.41 h | 65.3% | 3.54 |
| 32 | 0.1% | 28.6 s → 10.43 h | 41.9% | 3.55 |
| 128 | 0.6% | 28.8 s → 10.51 h | 14.7% | 3.55 |
| 512 | 2.4% | 29.7 s → 10.82 h | 4.1% | 3.61 |

**≥35.5% wall clock, no kernel work, no new dependency, peak memory unchanged.**

Two findings that changed the design:

- **Bucket size is not a diversity knob.** Roots per accumulation window are
  flat at 3.54–3.61 across a 256× range. The global micro-batch shuffle does
  all the mixing. Use the tightest bucket.
- **35.5% is a floor.** Zero-pad micro-batches go 0.1% → 78.3%, which puts the
  5 global layers back on `is_causal` for most of the run (§6.3). The cost
  model does not capture that. Direction certain, magnitude not yet measured at
  representative shapes.

⚠ **Source-homogeneity is a real hazard here** — length correlates hard with
root (kvasir short, chunked RP windows long), so length-homogeneous batches are
root-homogeneous batches. The global micro-batch shuffle is what prevents an
accumulation window drawing its whole gradient from one source. It is
load-bearing, not decoration.

### 6.5 The chunked CE is fine — do not swap it

    2 × 16,384   CE forward  374 ms  of 35.329 s  = 1.1%
    2 ×  4,096   CE forward   93 ms  of  4.387 s  = 2.1%

⚠ **Forward only** — the `torch.utils.checkpoint` recompute runs inside
`.backward()`, outside the timing window. Even at 3× it is ~3%.

`liger-kernel` fused linear CE is a ~1–3% lever on this shape. §2's finding
stands unchanged: chunking is what makes seq 16384 *reachable*, and it is not
what makes it slow.

### 6.6 MoE is ~8% — stop optimising it

Dense GEMM is 7.9% of the step, confirming the earlier decomposition bound of
~10% from the kernel side.

On `grouped_mm`: **the trace does not adjudicate it.** Run-01 was relaunched on
`eager`, so the profile shows the *default* path — 25,463 `aten::mm` dispatches
in one fwd+bwd, far more than the ~90 a grouped path would produce, so the
default is per-expert sequential. Whether the flag changes that when set is a
different measurement and was not run. At 7.9% it is not worth running.

### 6.7 Restart parameters for round 2

**Do not relaunch without the sampler change.** It is the only lever that wins
under every branch of the diagnosis.

1. **Implement bucket-to-pair + shuffle-to-mix** in the harness, tightest
   bucket, global micro-batch shuffle. Expected ~16.1 h → ~10.4 h or better.
2. **Re-assert the mask band structure** after the sampler change —
   `scripts/training-probes/step0_mask.py`, 30 s, no GPU. The sampler touches
   batch composition, which is what drives mask construction.
3. **Resume with `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch
   command** — it begins `rm -rf /tank/erp-tune/run-01` and would destroy the
   609 MB encode cache (2.5 min to reuse, ~4.3 h to rebuild). ⚠ A sampler change
   alters record *order*, not encoding, so the cache stays valid — but bump
   `encode_version` if anything upstream of `input_ids` changes.
4. **Run the `lora_B` inert-adapter gate at step 100.** It never ran in round 1.
   Norm every `lora_B` tensor in the checkpoint: all-non-zero = real, all-zero =
   INERT (kill the run), partial = module-selection problem. This is the one
   failure that stays invisible until brokkr's acceptance gate reports
   base-identical numbers.
5. **The corpus override is ONE RUN ONLY** (`operator-2026-08-25-rnd-run`). A
   second run needs a second operator grant.
6. **Attention backend is round 2's second lever**, gated on an A/B on the
   replica — not on argument. It can run while the tuned job trains.

⚠ GPU0 is currently **reserved and idle** by operator instruction; `sec` /
mog-sec remains down. The window is still open, so
`playbooks/ana-ml2-training-window-close.yaml` has NOT been run.