vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(gemma4-erp-tune): replace the estimates with measurements — they were 3x optimistic

Browse Source 
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.

    naive CE   bsz1 seq 8192    81.93 GiB
    naive CE   bsz1 seq16384    OOM
    chunked CE bsz1 seq16384    65.66 GiB
    chunked CE bsz2 seq16384    79.71 GiB   <- the run config
    chunked CE bsz4 seq16384    OOM

The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.

Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.

The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.

Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
This commit is contained in:
vh
2026-08-24 19:01:49 -07:00
parent 47ec3d1a97
commit 33433e0d1e
1 changed files with 65 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/pfi/gemma4-erp-tune-sizing.md
+65 -16
View File
@@ -218,26 +218,75 @@ of its own and cannot start until `gen` has vacated GPU1.
resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
probe.
### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic
Everything above was arithmetic. This was run on the real checkpoint on GPU0
with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer:
| config | peak | verdict |
|---|---:|---|
| naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare |
| naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB |
| chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ |
| **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** |
| chunked CE, bsz4 seq 16384 | **OOM** | — |
**The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB
estimated.** The estimate modelled gradient checkpointing as storing only layer
inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens ×
top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size
an MoE run from dense-model intuition — measure it.**
Two predictions did land exactly, which is why the rest of the model of the thing
is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and
**74,342,400 trainable params** at r64.
The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq
8192.** Chunking is not an optimisation, it is what makes brokkr's 16384
recommendation reachable at all.
Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure.
### Revised run parameters, now that it is a whole card
| micro-batch | GiB of 95.60 | |
|---:|---:|---|
| 4 | 61.8 | (the shared-card recommendation, now superseded) |
| **8** | **71.8** | **recommended — ~24 GiB clear** |
| 12 | 81.8 | ~14 GiB clear |
| 16 | 91.8 | too tight |
**FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2,
`gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer
steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear.
**Micro-batch 8, grad-accum 1 → 888 optimizer steps** (up from 444 on the shared
card). Two independent wins: the step count doubles, which matters at only one
epoch, and 8 × 8192 tokens gives ~4,096 rows per expert per step against ~512 at
micro-batch 1 — a far healthier GEMM on 704-wide experts, so MFU should land at
the top of the 10–25% band rather than the bottom.
`max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap
drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated
*entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon —
which is 60% of the mix and the axis the seat exists for. Prose and fireball
truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.
⚠ **Keep gradient checkpointing ON.** With a whole card it is tempting to drop
it (~17% off wall-clock by removing the recompute forward), but activations then
run ~24 GiB per sequence, which forces micro-batch 1 — and micro-batch 1 craters
MoE GEMM efficiency by 8×. Going wide beats going shallow here. Do not
"optimize" this later without re-reading this paragraph.
⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the
full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751
the typical step sits far below it.
⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()`
with it. Dropping checkpointing looks like ~17% off wall-clock and instead
forces micro-batch 1. Worse, the second call is the silent one: **without
`enable_input_require_grads()` the frozen base produces no gradient through the
checkpointed blocks, every adapter stays at its initialisation, and the run
completes successfully with an inert adapter.** `prepare_model_for_kbit_training`
used to do it as a side effect of the 4-bit path — so removing 4-bit removes it
too, and nothing warns you.
### Harness changes this required (eitri-smithy `62b556b`)
`9d64257` as audited would not have run here. Four fixes:
1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config
key, defaulting off, per §1.
2. Sequence-chunked CE replacing the model's own loss (the measured table above).
3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own
stale 365-line template and there was **no override parameter anywhere**, so
the upstream-template requirement was not expressible in the code.
4. Gradient checkpointing + `enable_input_require_grads()`.
Plus `training_eligibility_override` / `overridden_blockers` /
`substitute_controls` in the provenance manifest, and `device_map` pinned to
device 0 so the run cannot stray onto the card holding the inference seats.
Also fold in: