vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gemma4-erp-tune): size the run against the checkpoint — QLoRA is structurally unavailable

Browse Source 
The proposed shape was QLoRA r64. It cannot be run as specified. The
checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter
tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj
[128,2816,704] — no .weight suffix, so they are parameters, not modules).
bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params /
42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit
saves ~3.1 GiB of 48.07 and does not error while doing it.

Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per
additional 8192-token sequence.

Two sizing items were absent from the brief and both are load-bearing:

- vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping
  30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at
  ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a
  95.6 GiB card — it starts, then OOMs on the first long sample. Fused or
  chunked linear CE is mandatory and must be smoke-proven before a window
  is booked, since Liger may not carry a Gemma-4 MoE patch.
- v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the
  full-attention layers). A v_proj target silently produces no adapter
  there, and k_proj adapts K and V simultaneously. 45.96M trainable at
  r64 across q/k/v/o.

Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and
gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on
GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr.
Recommend standing down sec (2 aliases, last request ~5h ago) rather than
gen (7 aliases, 765 busy-engine log lines in 24h).

Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours
at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at
effective batch 16, which makes the wall-clock-checkpointing amendment
concrete rather than hypothetical.

Package as a uv venv on /tank: root is 91% full (36 GB) with
/var/lib/docker on it.
This commit is contained in:
vh
2026-08-24 18:28:30 -07:00
parent 9d0e628643
commit c507db9ac0
2 changed files with 236 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/gemma4-erp-tune-sizing.md
+235
View File
@@ -0,0 +1,235 @@
# Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication
_Measured 2026-08-24 on `ana-ml2` against
`/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated trainee)._
Division of labour for this run: **Eitri writes the harness, brokkr-smithy-dev
audits, infra-ops owns the GPU window and executes.** This document is the
sizing infra-ops owes; it is arithmetic against the real checkpoint and the
real card, not an estimate.
---
## 1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE
**The proposed shape was QLoRA r64. It cannot be run as specified**, and the
reason is structural rather than a tuning preference.
The checkpoint stores each layer's 128 experts as **two fused 3-D
`nn.Parameter` tensors**, not as 128 `nn.Linear` modules:
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
Note the absence of a `.weight` suffix — compare `mlp.down_proj.weight`
(an `nn.Linear`) against `experts.down_proj` (a bare parameter). That is the
tell, and it is decisive: **`bitsandbytes` 4-bit replacement walks `nn.Linear`
modules.** A fused 3-D parameter is not one, so it is skipped and stays BF16.
What `load_in_4bit=True` would actually buy on this model:
| block | params | BF16 | after bnb NF4 | saved |
|---|---:|---:|---:|---:|
| **MoE experts** (fused 3-D — **NOT quantized**) | 22.84 B | 42.54 GiB | **42.54 GiB** | **0** |
| lm attention (`nn.Linear`) | 1.11 B | 2.07 GiB | 0.52 GiB | 1.55 |
| dense shared MLP (`nn.Linear`) | 0.54 B | 1.00 GiB | 0.25 GiB | 0.75 |
| vision tower (`nn.Linear`) | 0.57 B | 1.06 GiB | 0.27 GiB | 0.79 |
| embed (tied, normally kept BF16) | 0.74 B | 1.38 GiB | 1.38 GiB | 0 |
| router + norms | 0.01 B | 0.02 GiB | 0.02 GiB | 0 |
| **total** | **25.81 B** | **48.07 GiB** | **~44.98 GiB** | **~3.1 GiB** |
**88.5% of the model is in tensors bitsandbytes cannot touch.** "QLoRA" here
means paying the NF4 dequant tax on 6% of the weights to save 6% of the
footprint. The premise does not survive contact with the checkpoint.
> **Eitri: do not hard-code a `BitsAndBytesConfig` / `load_in_4bit` path.**
> It will not error loudly — it will load, report a 4-bit model, and quietly
> leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.
**The one thing that could overturn this** is a third-party fork shipping
custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth
is the candidate). **Not chased, deliberately** — see §4, where the run fits in
BF16 without displacing anything the fleet depends on, which collapses QLoRA's
value to zero. If it is ever revisited, it must be *before* the harness
hard-codes a quantization path, not after.
**Verdict: plain LoRA on BF16 weights.**
---
## 2. What the run actually costs
Adapter targeting `q_proj,k_proj,v_proj,o_proj` at r64, computed from the real
tensor shapes:
| | layers | per layer | total |
|---|---:|---:|---:|
| sliding-attention (q 4096, kv 2048, o 4096) | 25 | 1,507,328 | 37,683,200 |
| full-attention (q 8192, kv 1024, o 8192) | 5 | 1,654,784 | 8,273,920 |
| **trainable** | | | **45,957,120** (0.178% of base) |
⚠ **`v_proj` DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29.** Those are the
`full_attention` layers, and `attention_k_eq_v: true` means one projection
serves both K and V. Consequences the harness must respect:
- PEFT matches by name suffix, so a `v_proj` target **silently produces no
adapter** on those five layers. Do not assert a fixed adapter count.
- Adapting `k_proj` on a global layer **adapts K and V simultaneously** — a
different intervention than on the sliding layers. If that asymmetry matters
to the recipe, say so explicitly rather than discovering it in the loss curve.
### Memory budget, batch 1, `max_seq_len` 8192
| item | GiB | note |
|---|---:|---|
| base weights BF16 | 48.07 | measured: 25,805,936,206 params × 2 B |
| adapters + grads + AdamW fp32 m/v | 0.75 | 45.96 M trainable — rounding error |
| checkpointed layer inputs | 1.29 | 30 × 8192 × 2816 × 2 B |
| recompute peak, one layer | ~2.5 | 8192 tok × top-8 of 128, `moe_intermediate 704` |
| loss head, **fused/chunked CE** | ~2.0 | see the warning below |
| CUDA context + cuBLAS + fragmentation | ~3.0 | the item `--gpu-memory-utilization` never covered |
| **total** | **~57.6** | |
Marginal cost per extra sequence in the micro-batch: **~2.5 GiB.**
| micro-batch | GiB |
|---:|---:|
| 1 | 54.3 |
| 2 | 56.8 |
| **4** | **61.8** |
| 6 | 66.8 |
| 8 | 71.8 |
### ⚠ The loss head is the whole ballgame, and it is not in the brief
`vocab_size` is **262,144** and `final_logit_softcapping` is **30.0**. One
8192-token sequence produces **2.147 billion logits**. Through a naive HF
`ForCausalLM` loss that is:
BF16 logits 4.0 GiB
fp32 upcast 8.0 GiB
softcap tanh saved 8.0 GiB (autograd keeps the pre-cap tensor)
softmax + grad 8.0 GiB
------------------------------
~28-30 GiB transient, at BATCH 1
Naive CE at batch 1 lands the run at **~85.6 GiB on a 95.6 GiB card** — it will
appear to work and then OOM on the first long sample. At micro-batch 4 it is
~120 GiB and never starts. **Fused/chunked linear cross-entropy is mandatory,
not an optimization.**
⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with
softcapping may not have one. Three ways out, in order of preference —
(a) generic `LigerFusedLinearCrossEntropyLoss` wired against the lm_head with
softcapping applied inside the chunk; (b) `cut-cross-entropy`; (c) hand-rolled
sequence-chunked CE. **This must be proven on a 10-step smoke run before the
window is booked**, because everything else in this document assumes it works.
### Step count
58.2 M tokens / 20,576 samples = **2,829 tokens/sample average** — well under
8192, so packing matters.
- Packed to 8192: **7,104 sequences.** At micro-batch 4 × grad-accum 4
(effective 16) → **444 optimizer steps for the whole epoch.**
- ⚠ That is a *small* step count. A "checkpoint every 100 steps" default gives
four checkpoints across a multi-hour run. This is exactly why the amendment
asked for **wall-clock-interval checkpointing, not step-count** — the case is
now concrete, not hypothetical.
- ⚠ **Packing must use `position_ids` + varlen/block-diagonal attention.** Naive
concatenation bleeds samples into each other. `sliding_window` is 1024 on 25
of 30 layers so the damage is bounded there — but the 5 `full_attention`
layers see the entire packed sequence.
**Open question for brokkr/Eitri:** what fraction of the 20,576 samples exceed
8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting
the ends off RP scenes, which is where the signal lives.
### Runtime
Active parameters per token ≈ **3.67 B** (2.93 B routed + attention, plus the
0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute
≈ 6 × active × tokens = **1.28e18 FLOPs** for the epoch.
At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts
are not efficient — **4 to 10 hours, most likely ~6.** Treat as a band, not a
number; it will be measured on the smoke run.
---
## 3. Where it fits (measured 2026-08-24, 18:20 PDT)
Card total: 97,887 MiB = **95.60 GiB** each.
| | GPU0 | GPU1 |
|---|---|---|
| resident | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |
- **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
current number is budgeting against a moving one.
- **GPU1 as-is: nowhere near.** 18.99 GiB.
- **GPU1 with `vllm-mog-sec` stopped: 76,070 MiB = 74.29 GiB free.** Fits
micro-batch 4 (61.8 GiB) with ~12 GiB clear even after reserving ~6 GiB for
Scriberr's on-demand whisper load.
---
## 4. Recommendation
**Run on GPU1. Stand down `sec`, not `gen`.**
| | `gen` | `sec` |
|---|---|---|
| aliases | 7 (`gen`, `gen-reasoning`, `chat-judge`, `image-judge`, summarizer/classifier family) | 2 (`sec`, `sec-reasoning`) |
| standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
| measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |
`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin, and it is the difference between a 4–10 hour window
that nobody outside the security work notices and one that takes the fleet's
default model offline for a working day.
Also fold in, cheaply and reversibly:
- **Move Scriberr to GPU0 for the window** (`device_ids: ["0"]`, one compose
edit + `up -d`). GPU0 will be sitting on ~53 GiB free with `gen` up, and it
removes contention from the training card entirely.
- **Micro-batch 4, grad-accum 4** (effective 16, 444 steps). Micro-batch 6 is
the measured ceiling; the gap is deliberate margin. A seat crash-looped
earlier the same day on 0.6 GiB of assumed headroom — this document does not
repeat that.
- **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
(36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
would come close to filling it. `/tank` has 4.0 TB.
### If `sec` may not be stood down
Fallback is GPU0 with `gen` stopped — the run fits with ~38 GiB to spare, which
buys micro-batch 8 and a shorter wall-clock. Make it **resumable and split** in
that case: INV-T7 plus wall-clock checkpointing already allows the window to be
broken into two or three shorter stretches with `gen` restored between them,
rather than one long outage.
---
## 5. Standing warnings that apply to this run
- **Never render training examples through the base's own
`chat_template.jinja`.** Every third-party Gemma-4 derivative ships a stale
one; the trainee's is 365 lines against upstream's 390. Use
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. Training through
the wrong template is train/serve skew with no error — it presents as a
tuning failure.
- **Base path and chat-template path are config keys, not constants.** The
trainee base already moved once (stock BF16 → `-heretic-bf16`).
- **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
context, graphs, or non-torch overhead — the same misreading that OOM'd the
char-rp seat.
- **Serving the result is not settled.** LoRA-on-NVFP4 hot-swap was a silent
no-op on vLLM 0.24.0 (#47639, proven quant-agnostic). Retest on the tagged
`vllm/vllm-openai:v0.27.1` already on disk. **If it still no-ops, the harness
must emit merged weights** — and Eitri needs that requirement while he is
early, not after the run.