esh-pfi-infrastructure/docs/pfi/gemma4-erp-tune-sizing.md

Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication

Measured 2026-08-24 on ana-ml2 against /tank/aimodels/gemma4-26b-a4b-it-heretic-bf16 (llmfan46 abliterated trainee).

Division of labour for this run: Eitri writes the harness, brokkr-smithy-dev audits, infra-ops owns the GPU window and executes. This document is the sizing infra-ops owes; it is arithmetic against the real checkpoint and the real card, not an estimate.

---

1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE

The proposed shape was QLoRA r64. It cannot be run as specified, and the reason is structural rather than a tuning preference.

The checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter tensors, not as 128 nn.Linear modules:

model.language_model.layers.N.experts.gate_up_proj    BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj       BF16 [128, 2816,  704]

Note the absence of a .weight suffix — compare mlp.down_proj.weight (an nn.Linear) against experts.down_proj (a bare parameter). That is the tell, and it is decisive: bitsandbytes 4-bit replacement walks nn.Linear modules. A fused 3-D parameter is not one, so it is skipped and stays BF16.

What load_in_4bit=True would actually buy on this model:

block	params	BF16	after bnb NF4	saved
MoE experts (fused 3-D — NOT quantized)	22.84 B	42.54 GiB	42.54 GiB	0
lm attention (nn.Linear)	1.11 B	2.07 GiB	0.52 GiB	1.55
dense shared MLP (nn.Linear)	0.54 B	1.00 GiB	0.25 GiB	0.75
vision tower (nn.Linear)	0.57 B	1.06 GiB	0.27 GiB	0.79
embed (tied, normally kept BF16)	0.74 B	1.38 GiB	1.38 GiB	0
router + norms	0.01 B	0.02 GiB	0.02 GiB	0
total	25.81 B	48.07 GiB	~44.98 GiB	~3.1 GiB

88.5% of the model is in tensors bitsandbytes cannot touch. "QLoRA" here means paying the NF4 dequant tax on 6% of the weights to save 6% of the footprint. The premise does not survive contact with the checkpoint.

Eitri: do not hard-code a BitsAndBytesConfig / load_in_4bit path. It will not error loudly — it will load, report a 4-bit model, and quietly leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.

The one thing that could overturn this is a third-party fork shipping custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth is the candidate). Not chased, deliberately — see §4, where the run fits in BF16 without displacing anything the fleet depends on, which collapses QLoRA's value to zero. If it is ever revisited, it must be before the harness hard-codes a quantization path, not after.

Verdict: plain LoRA on BF16 weights.

---

2. What the run actually costs

Adapter targeting q_proj,k_proj,v_proj,o_proj at r64, computed from the real tensor shapes:

	layers	per layer	total
sliding-attention (q 4096, kv 2048, o 4096)	25	1,507,328	37,683,200
full-attention (q 8192, kv 1024, o 8192)	5	1,654,784	8,273,920
trainable			45,957,120 (0.178% of base)

⚠ v_proj DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29. Those are the full_attention layers, and attention_k_eq_v: true means one projection serves both K and V. Consequences the harness must respect:

• PEFT matches by name suffix, so a v_proj target silently produces no adapter on those five layers. Do not assert a fixed adapter count.
• Adapting k_proj on a global layer adapts K and V simultaneously — a different intervention than on the sliding layers. If that asymmetry matters to the recipe, say so explicitly rather than discovering it in the loss curve.

Memory budget, batch 1, max_seq_len 8192

item	GiB	note
base weights BF16	48.07	measured: 25,805,936,206 params × 2 B
adapters + grads + AdamW fp32 m/v	0.75	45.96 M trainable — rounding error
checkpointed layer inputs	1.29	30 × 8192 × 2816 × 2 B
recompute peak, one layer	~2.5	8192 tok × top-8 of 128, moe_intermediate 704
loss head, fused/chunked CE	~2.0	see the warning below
CUDA context + cuBLAS + fragmentation	~3.0	the item --gpu-memory-utilization never covered
total	~57.6

Marginal cost per extra sequence in the micro-batch: ~2.5 GiB.

micro-batch	GiB
1	54.3
2	56.8
4	61.8
6	66.8
8	71.8

⚠ The loss head is the whole ballgame, and it is not in the brief

vocab_size is 262,144 and final_logit_softcapping is 30.0. One 8192-token sequence produces 2.147 billion logits. Through a naive HF ForCausalLM loss that is:

BF16 logits          4.0 GiB
fp32 upcast          8.0 GiB
softcap tanh saved   8.0 GiB   (autograd keeps the pre-cap tensor)
softmax + grad       8.0 GiB
------------------------------
~28-30 GiB transient, at BATCH 1

Naive CE at batch 1 lands the run at ~85.6 GiB on a 95.6 GiB card — it will appear to work and then OOM on the first long sample. At micro-batch 4 it is ~120 GiB and never starts. Fused/chunked linear cross-entropy is mandatory, not an optimization.

⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with softcapping may not have one. Three ways out, in order of preference — (a) generic LigerFusedLinearCrossEntropyLoss wired against the lm_head with softcapping applied inside the chunk; (b) cut-cross-entropy; (c) hand-rolled sequence-chunked CE. This must be proven on a 10-step smoke run before the window is booked, because everything else in this document assumes it works.

Step count

58.2 M tokens / 20,576 samples = 2,829 tokens/sample average — well under 8192, so packing matters.

• Packed to 8192: 7,104 sequences. At micro-batch 4 × grad-accum 4 (effective 16) → 444 optimizer steps for the whole epoch.
• ⚠ That is a small step count. A "checkpoint every 100 steps" default gives four checkpoints across a multi-hour run. This is exactly why the amendment asked for wall-clock-interval checkpointing, not step-count — the case is now concrete, not hypothetical.
• ⚠ Packing must use position_ids + varlen/block-diagonal attention. Naive concatenation bleeds samples into each other. sliding_window is 1024 on 25 of 30 layers so the damage is bounded there — but the 5 full_attention layers see the entire packed sequence.

Open question for brokkr/Eitri: what fraction of the 20,576 samples exceed 8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting the ends off RP scenes, which is where the signal lives.

Runtime

Active parameters per token ≈ 3.67 B (2.93 B routed + attention, plus the 0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute ≈ 6 × active × tokens = 1.28e18 FLOPs for the epoch.

At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts are not efficient — 4 to 10 hours, most likely ~6. Treat as a band, not a number; it will be measured on the smoke run.

---

3. Where it fits (measured 2026-08-24, 18:20 PDT)

Card total: 97,887 MiB = 95.60 GiB each.

	GPU0	GPU1
resident before the window	vllm-gen 42,508 MiB (up 3 h)	vllm-mog-sec 56,624 MiB + embed 3,304 + reward 9,512 + coder 6,158 + rerank-a3 2,170; Scriberr pinned here, loads on demand
free	54,741 MiB = 53.46 GiB	19,446 MiB = 18.99 GiB

Three placements were on the table:

• GPU0 beside gen: does not fit. 53.46 GiB free against ~57.6 GiB needed — short by ~4 GiB. And gen is only three hours old: measured footprint runs 38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the current number is budgeting against a moving one.
• GPU1 with mog-sec stopped: 76,070 MiB free. Fits, but shares a card with four small seats and Scriberr.
• GPU0 with gen MOVED OFF: the whole card. ← what was chosen.

---

4. The window, as executed

Operator call, 2026-08-24: move gen to GPU1 and stand sec down, so GPU0 is emptied completely rather than shared. This is strictly better than training beside gen: the tune gets 95.60 GiB with no co-tenant, and the fleet's general seat never goes dark beyond its own ~5-minute restart.

before:  GPU0 [ gen 42.5 ]                     GPU1 [ sec 55.3 | small seats 20.7 ]
after:   GPU0 [ ---- empty, 95.60 GiB ---- ]   GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]

sec is genuinely in use and this is not free — but it is the smaller blast radius by a wide margin:

	gen	sec
aliases	7 (gen, gen-reasoning, chat-judge, image-judge, summarizer/classifier family)	2 (sec, sec-reasoning)
standing role	the fleet's general seat; a documented always-available dependency in global CLAUDE.md	M.O.G.-SEC, niche
measured traffic	765 busy-engine log lines in 24 h — continuously in use	bursty; peak 8 concurrent, last request ~5 h ago

Traffic to sec arrives from 10.250.50.70 (the LiteLLM gateway), so the aliases will fail at the gateway for the duration. Per the standing rule, let them fail — do not route sec to another model as a stand-in.

Both directions are playbooks, and the order in each is load-bearing:

scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml

⚠ gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction of total card memory: 42,091 MiB must be free at startup or the engine refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up. Recreating gen onto GPU1 before stopping mog-sec takes the fleet's main seat down and leaves it down. The open playbook stops mog-sec first and hard-gates on the freed memory; the close playbook mirrors it, because mog-sec needs 50,901 MiB of its own and cannot start until gen has vacated GPU1.

⚠ Invoke elway as infra-ops@10.250.50.54, not the ana-ml2 ssh-target — that resolves to lkraven, which has no NOPASSWD sudo, and elway aborts at its sudo probe.

⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic

Everything above was arithmetic. This was run on the real checkpoint on GPU0 with synthetic tokens (/tank/erp-tune/smoke_ce.py), and it moves the answer:

config	peak	verdict
naive CE, bsz1 seq 8192	81.93 GiB	fits, ~14 GiB spare
naive CE, bsz1 seq 16384	OOM	tried to allocate 16.00 GiB
chunked CE, bsz1 seq 16384	65.66 GiB	✅
chunked CE, bsz2 seq 16384	79.71 GiB	✅ the run config
chunked CE, bsz4 seq 16384	OOM	—

The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB estimated. The estimate modelled gradient checkpointing as storing only layer inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens × top-8 of 128 experts, plus scatter/gather buffers) is far heavier. Do not size an MoE run from dense-model intuition — measure it.

Two predictions did land exactly, which is why the rest of the model of the thing is trustworthy: 205 target modules (q30/k30/v25/o30/gate30/up30/down30) and 74,342,400 trainable params at r64.

The headline: chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq 8192. Chunking is not an optimisation, it is what makes brokkr's 16384 recommendation reachable at all.

Base load peak: 49,221 MiB, confirming the 48.07 GiB weight figure.

Revised run parameters, now that it is a whole card

FINAL, measured: max_seq_len 16384, per_device_batch_size 2, gradient_accumulation_steps 8 → effective batch 16, ~1,280 optimizer steps, 79.71 GiB of 95.60 with ~15.9 GiB clear.

max_seq_len went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap drops 6.2% of samples but 22.4% of TOKENS (61.2M → 47.5M), concentrated entirely in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon — which is 60% of the mix and the axis the seat exists for. Prose and fireball truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.

⚠ The 79.71 GiB figure is worst case — every sample in the micro-batch at the full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751 the typical step sits far below it.

⚠ Keep gradient checkpointing ON, and keep enable_input_require_grads() with it. Dropping checkpointing looks like ~17% off wall-clock and instead forces micro-batch 1. Worse, the second call is the silent one: without enable_input_require_grads() the frozen base produces no gradient through the checkpointed blocks, every adapter stays at its initialisation, and the run completes successfully with an inert adapter. prepare_model_for_kbit_training used to do it as a side effect of the 4-bit path — so removing 4-bit removes it too, and nothing warns you.

Harness changes this required (eitri-smithy 62b556b)

9d64257 as audited would not have run here. Four fixes:

1. runtime.py hardcoded BitsAndBytesConfig(load_in_4bit=True) — now a config key, defaulting off, per §1.
2. Sequence-chunked CE replacing the model's own loss (the measured table above).
3. chat_template_path — apply_chat_template resolved the checkpoint's own stale 365-line template and there was no override parameter anywhere, so the upstream-template requirement was not expressible in the code.
4. Gradient checkpointing + enable_input_require_grads().

Plus training_eligibility_override / overridden_blockers / substitute_controls in the provenance manifest, and device_map pinned to device 0 so the run cannot stray onto the card holding the inference seats.

Also fold in:

• Scriberr STAYS on GPU1. (An earlier draft of this document suggested moving it to GPU0; that was written when training was going to live on GPU1, and it is now exactly backwards. GPU0 is the training card and wants no co-tenant.)
• Package as a uv venv on /tank, not a Docker image. Root is at 91% (36 GB free) and /var/lib/docker lives on it; a PyTorch training image would come close to filling it. /tank has 4.0 TB.
• The run is still resumable-by-design (INV-T7 + wall-clock checkpointing). Nothing about a dedicated card removes that requirement — a 4–10 hour window is long enough that an unresumable run is a bad bet regardless of who owns the GPU.

---

5. Standing warnings that apply to this run

• Never render training examples through the base's own chat_template.jinja. Every third-party Gemma-4 derivative ships a stale one; the trainee's is 365 lines against upstream's 390. Use /tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja. Training through the wrong template is train/serve skew with no error — it presents as a tuning failure.

• Base path and chat-template path are config keys, not constants. The trainee base already moved once (stock BF16 → -heretic-bf16).

• --gpu-memory-utilization sizes the KV cache only. It does not cover CUDA context, graphs, or non-torch overhead — the same misreading that OOM'd the char-rp seat.

• SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason we expected. The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops (it did on vLLM 0.24.0, #47639). Retested on vllm/vllm-openai:latest with the NVFP4A16 base plus the run's own checkpoint adapter. It does not no-op — it refuses to start:

  AttributeError: To support LoRA for MoE model,
                  'get_expert_mapping' must be implemented
  

  ⚠ This is architectural, not quantization-related. The check lives in vllm/lora/utils.py::process_packed_modules_mapping and branches on whether the model is MoE; quantization is not in the condition. gemma4.py, gemma4_mm.py, gemma4_mtp.py and gemma4_unified.py contain zero occurrences of get_expert_mapping (deepseek_v2, glm4_moe, ernie45_moe and others do implement it). vLLM cannot serve a LoRA on Gemma-4 at all — BF16 or quantized. Merging is the only path for this architecture.

  Note this holds even though our adapter never touches experts: validate_adapter_parameters forbids per-expert params, so all 205 targets are attention + dense MLP. The refusal is about the model being MoE, not about what the adapter targets.

  Silver lining worth recording: a loud refusal is strictly better than the 0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name and passes every check that does not compare against base.

  The merge → quantize → serve pipeline is implemented and validated end to end at scripts/erp-tune-serve/.

---

6. Round-1 aborted; throughput root-caused (measured 2026-08-24 22:00 PDT)

Run-01 launched, reached step 19 of 1,312 at ~35–46 s/it, and was killed by operator instruction — not a crash, not an OOM. ETA was ~13.9 h at 8.6% MFU and the operator elected to root-cause before spending the window.

Nothing was destroyed: the 609 MB encode cache, order-manifest.jsonl, truncation-report.json and resume-run-01.sh are all preserved at /tank/erp-tune/run-01/. There are no checkpoints — the first was due at step 100, so brokkr's lora_B inert-adapter gate never ran. That question is open and moves to the restart.

Model-agnostic lessons from this investigation are in training-throughput-playbook.md; the probes are at scripts/training-probes/. What follows is Gemma-4-specific.

⚠⚠ CORRECTION 2026-08-26 — MUCH OF THIS SECTION MEASURES THE WRONG PROCESS

The benchmarks below were run against the SERVING seat with attn_implementation="sdpa" set explicitly. Training was running flex_attention the whole time. ATTN_IMPLEMENTATION = "flex_attention" was a module constant passed unconditionally into from_pretrained, and run 1's step-time distribution (n=1,445; min 11.84 / p50 19.75 / p99 30.52 / max 45.79 s/it, the max being step 1's compile) confirms it stayed compiled — a dynamo fallback sits in the hundreds of seconds per step.

FALLS — describes sdpa, not the training run: the three-point scaling fit and its 68% quadratic share; the kernel table (fmha_cutlassF/B sm80, EFFICIENT_ATTENTION, attention 65.2%); the 8.6% MFU figure quoted above and throughout; the projection that elementwise becomes the largest line item post-fix; and "adopt flex_attention" as the round-two headline lever — which round one already had.

SURVIVES — measured on the live training run: the padding/bucketing win (44.3 → 20.1 s/it); the zero-pad fast-path second-order effect; the eval-battery noise-floor work.

⚠ Do not assume the direction of the correction. Training's real MFU is unmeasured, not obviously better. Flex with a BlockMask ought to beat dense-masked sdpa, but that is a prediction and this investigation has been unkind to those.

The root cause was procedural, not technical, and it is written up as playbook §4.3: run 1 recorded no attention backend in its provenance, so the benchmark/trainer delta was invisible and nobody enumerated it. Run 2 onward records attn_implementation_requested and _resolved, plus the torch/transformers versions and dynamo's compile counters.

6.1 Where the step time goes

Real checkpoint, GPU0, attn_implementation="sdpa", PEFT + gradient checkpointing + the chunked CE, fwd+bwd, best-of-2 after warmup:

shape	time	peak
2 × 2,048	1.776 s	53.2 GiB
2 × 8,192	11.570 s	62.3 GiB
2 × 16,384	35.017 s	76.6 GiB

Fitting t(w) = A·w + B·w² over all three (per-sequence w, batch 2):

A = 6.8715e-04 s/token        B = 8.8509e-08 s/token²

w	predicted	measured	linear	quadratic	quad share
2,048	1.779	1.776	1.407	0.371	20.9%
8,192	11.569	11.570	5.629	5.940	51.3%
16,384	35.017	35.017	11.258	23.759	67.8%

Two terms, three points, residuals under 3 ms across an 8× range. No fixed per-batch term was needed, which refutes the launch-bound hypothesis outright — ~3,840 expert-GEMM launches per forward are not the cost.

Independently, the profiler kernel table (device rows only — see playbook §3.4):

device kernel	ms	of step
fmha_cutlassB_bf16_aligned_128x64_k65536_sm80 (attn BWD)	16,144.6	46.1%
fmha_cutlassF_bf16_aligned_32x128_gmem_sm80 (attn FWD)	6,691.2	19.1%
cutlass_80_tensorop_bf16_s16816gemm ×3 (dense GEMM)	2,774.0	7.9%
elementwise / vectorized / unrolled ×9	4,787.4	13.7%
gather / Memcpy DtoD / dropout	951.6	2.7%
attention total	22,835.8	65.2%

Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two independent methods, 2.6 points apart.

6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20%

brokkr-smithy-dev's panel (arm: Bil) closed the fold by reading torch 2.13.0 and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather than explaining it:

nominal work billed     27.1 TFLOPS × 34.85 s        =  9.4e14 FLOP
dense-sliding extra     25 layers, 2 seqs, 4 passes  = +8.2e14 FLOP
padded full layers      lose the causal skip         = +3.5e14 FLOP
------------------------------------------------------------------
work actually performed                              ≈  1.8e15
in 34.85 s                                           ≈  51–61 TFLOPS
                                                     ≈  17–20% OF PEAK

We divided the intended (windowed) FLOPs by the wall time the dense reality took. 17–20% is squarely inside the honest stock band. The hardware is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic the architecture specifies, and the excess is the sliding window being computed and then thrown away.

Source-verified mechanism, no longer hypothesis:

file	finding
masking_utils.py:292-301 _ignore_causal_mask_sdpa	requires kv_length < local_attention_size to skip the mask. 16384 ≥ 1024, so the sliding mask ALWAYS materialises at this seq len — not sometimes, always
sdp_utils_cpp.h:259-267, sdp_utils.cpp:933	flash rejects any explicit mask
sdp_utils.cpp:647, Context.h:480-485	cuDNN is unreachable on sm_120 twice over — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12
attention.cu:1196/1759, kernel_forward.h:282-290	mem-efficient has no mask gate and no head_dim cap, computes full n×n with the mask as additive bias; it trims only for is_causal

Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding layers land on mem-efficient computing dense O(n²), and no backend on this stack can rescue it. cuDNN sliding-window does not exist at all — there is no window argument in the public SDPA signature.

⚠ Masked SDPA also blocks enable_gqa, so KV gets repeat_kv-expanded on every layer — extra memory traffic riding on top of the extra FLOPs.

6.1b Backend eligibility, measured — every source claim confirmed

Shapes-only, random weights, sdpa_kernel() pinning one backend at a time. A forced failure is information: it identifies eligibility rather than preference.

Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:

mask case	FLASH	EFFICIENT	CUDNN	MATH
None + is_causal	8.8 ms	21.6 ms	refused	217.5 ms
explicit 1024-band 4D	refused	55.5 ms	refused	—
explicit causal+pad 4D	refused	57.1 ms	refused	—

Global layers (5 of 30) — H_q16/H_kv2, D=512:

mask case	FLASH	EFFICIENT	CUDNN	MATH
None + is_causal	refused	45.9 ms	refused	334.1 ms
explicit 1024-band 4D	refused	104.4 ms	refused	—
explicit causal+pad 4D	refused	107.9 ms	refused	—

Refusal reasons, straight from the runtime warnings:

• Flash Attention does not support non-null attn_mask — kills flash for all 25 sliding layers, always, at this seq len
• Flash attention requires q,k,v … less than or equal to 256. Got 512 — kills flash for the 5 global layers even fully causal. Dvalin was right that they are a first-class hole
• head_dim should be no more than 128 — kills cuDNN on both shapes

The sliding layers run at 55.5 ms where a maskless flash path would cost 8.8 ms — a 6.3× penalty, and it is unreachable through any config on this stack. That is the whole problem in one row.

6.2 ⚠ The attention kernels are Ampere, on a Blackwell card

fmha_cutlass*_sm80 on sm_120. There is no Blackwell-tuned attention kernel in this path at all, and the forward is additionally on gmem — the global-memory fallback tier of the memory-efficient backend, selected when the working set will not fit in shared memory.

This is the mechanism behind the 100%-SM / 27-TFLOPS / 304-TFLOPS-capable reading: the chip is saturated running a kernel generation behind on the dominant cost centre.

The candidate fix is a purpose-built kernel for this architecture's mixed 256/512 head-dim split — zzhhjjj/gemma-triton-flash-attn (register_triton_attention(), then _attn_implementation = "triton_gqa"), reported 9.23× over SDPA at N=16K D=256 SWA and 2.94× fwd+bwd at D=512. flex_attention + BlockMask is the no-new-dependency alternative.

⚠ Prefer a UNIFORM backend over a per-layer split. vLLM special-cased this exact mixed-head-dim architecture and measured mixed backends 8% slower than uniform. And attn_implementation is all-or-nothing at from_pretrained / set_attn_implementation — per-layer routing requires a custom function registered on ALL_ATTENTION_FUNCTIONS branching on module.head_dim / sliding_window.

⚠ FA2 is not available for this model: it caps head_dim at 256 and the 5 global layers are at 512. FA3 is Hopper-only. Do not bet on FA4 on sm_120.

6.3 Masking is CORRECT — and padding is what costs

Band structure asserted directly against the real config at n=16,384:

sliding_attention   max 1,024 allowed/row, saturates at row 1,023   PASS

Constraints were not silently dropped; the 25 sliding layers were genuinely windowed. Run-01 was training the model we intended.

The same probe found the mechanism nobody had measured:

2D mask supplied	full_attention mask returned
None	None → is_causal fast path AVAILABLE
all-ones (no padding)	None → is_causal fast path AVAILABLE
right-padded (what collate_mixed emits)	4D 16384² → fast path LOST

Padding is what pins the 5 global layers to an explicit mask. The 25 sliding layers get a 4D tensor either way — sdpa_attention_forward sets is_causal=True only when attention_mask is None, and a 1024 window cannot be expressed as is_causal.

Isolated, same width, only the mask differing:

2 × 16,384, no padding          35.244 s   26,048 loss targets
2 × 16,384, 50% pad on row 1    38.567 s   19,640 loss targets

9.4% slower for 24% less work.

6.4 The corpus is 29.9% padding — and bucketing is the biggest win available

Measured off the preserved encode cache in true SequentialSampler order:

records                  20,982   (3,583 rp-dialogue / 12,003 prose-chunk / 5,396 actual-play)
seq len min/mean/max     142 / 2,752 / 16,384
micro-batches (mb=2)     10,491
real tokens              57,733,156
padded tokens            82,337,318
PADDING WASTE            29.9%
mb width p50/p90/p99     2,092 / 10,634 / 16,341
micro-batches at 16,384  3 of 10,491  (0.0%)

⚠ Note the last line against §6.1: the 2 × 16,384 benchmark shape occurs in three micro-batches out of 10,491. Weighted over the real distribution the quadratic share is ~51%, not 67.8%.

Bucket-to-pair, shuffle-to-mix (brokkr's design, validated on measured lengths — form micro-batches within length buckets, then shuffle the resulting micro-batches globally):

bucket	waste	predicted step	zero-pad mb	roots/accum window
current	29.9%	44.3 s → 16.13 h	0.1%	3.68
2	0.0%	28.6 s → 10.40 h	78.3%	3.56
8	0.0%	28.6 s → 10.41 h	65.3%	3.54
32	0.1%	28.6 s → 10.43 h	41.9%	3.55
128	0.6%	28.8 s → 10.51 h	14.7%	3.55
512	2.4%	29.7 s → 10.82 h	4.1%	3.61

≥35.5% wall clock, no kernel work, no new dependency, peak memory unchanged.

Two findings that changed the design:

• Bucket size is not a diversity knob. Roots per accumulation window are flat at 3.54–3.61 across a 256× range. The global micro-batch shuffle does all the mixing. Use the tightest bucket.
• 35.5% is a floor. Zero-pad micro-batches go 0.1% → 78.3%, which puts the 5 global layers back on is_causal for most of the run (§6.3). The cost model does not capture that. Direction certain, magnitude not yet measured at representative shapes.

⚠ Source-homogeneity is a real hazard here — length correlates hard with root (kvasir short, chunked RP windows long), so length-homogeneous batches are root-homogeneous batches. The global micro-batch shuffle is what prevents an accumulation window drawing its whole gradient from one source. It is load-bearing, not decoration.

6.5 The chunked CE is fine — do not swap it

2 × 16,384   CE forward  374 ms  of 35.329 s  = 1.1%
2 ×  4,096   CE forward   93 ms  of  4.387 s  = 2.1%

⚠ Forward only — the torch.utils.checkpoint recompute runs inside .backward(), outside the timing window. Even at 3× it is ~3%.

liger-kernel fused linear CE is a ~1–3% lever on this shape. §2's finding stands unchanged: chunking is what makes seq 16384 reachable, and it is not what makes it slow.

6.6 MoE is ~8% — stop optimising it

Dense GEMM is 7.9% of the step, confirming the earlier decomposition bound of ~10% from the kernel side.

On grouped_mm: the trace does not adjudicate it. Run-01 was relaunched on eager, so the profile shows the default path — 25,463 aten::mm dispatches in one fwd+bwd, far more than the ~90 a grouped path would produce, so the default is per-expert sequential. Whether the flag changes that when set is a different measurement and was not run. At 7.9% it is not worth running.

6.7 Restart parameters for round 2

Do not relaunch without the sampler change. It is the only lever that wins under every branch of the diagnosis.

1. Implement bucket-to-pair + shuffle-to-mix in the harness, tightest bucket, global micro-batch shuffle. Expected ~16.1 h → ~10.4 h or better.
2. Re-assert the mask band structure after the sampler change — scripts/training-probes/step0_mask.py, 30 s, no GPU. The sampler touches batch composition, which is what drives mask construction.
3. Resume with /tank/erp-tune/resume-run-01.sh, NEVER the original launch command — it begins rm -rf /tank/erp-tune/run-01 and would destroy the 609 MB encode cache (2.5 min to reuse, ~4.3 h to rebuild). ⚠ A sampler change alters record order, not encoding, so the cache stays valid — but bump encode_version if anything upstream of input_ids changes.
4. Run the lora_B inert-adapter gate at step 100. It never ran in round 1. Norm every lora_B tensor in the checkpoint: all-non-zero = real, all-zero = INERT (kill the run), partial = module-selection problem. This is the one failure that stays invisible until brokkr's acceptance gate reports base-identical numbers.
5. The corpus override is ONE RUN ONLY (operator-2026-08-25-rnd-run). A second run needs a second operator grant.
6. Attention backend is round 2's second lever, gated on an A/B on the replica — not on argument. It can run while the tuned job trains.

⚠ GPU0 is currently reserved and idle by operator instruction; sec / mog-sec remains down. The window is still open, so playbooks/ana-ml2-training-window-close.yaml has NOT been run.