diff --git a/docs/pfi/gemma4-erp-tune-sizing.md b/docs/pfi/gemma4-erp-tune-sizing.md index 4d7c7d9..39c3ace 100644 --- a/docs/pfi/gemma4-erp-tune-sizing.md +++ b/docs/pfi/gemma4-erp-tune-sizing.md @@ -218,26 +218,75 @@ of its own and cannot start until `gen` has vacated GPU1. resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo probe. +### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic + +Everything above was arithmetic. This was run on the real checkpoint on GPU0 +with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer: + +| config | peak | verdict | +|---|---:|---| +| naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare | +| naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB | +| chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ | +| **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** | +| chunked CE, bsz4 seq 16384 | **OOM** | — | + +**The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB +estimated.** The estimate modelled gradient checkpointing as storing only layer +inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens × +top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size +an MoE run from dense-model intuition — measure it.** + +Two predictions did land exactly, which is why the rest of the model of the thing +is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and +**74,342,400 trainable params** at r64. + +The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq +8192.** Chunking is not an optimisation, it is what makes brokkr's 16384 +recommendation reachable at all. + +Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure. + ### Revised run parameters, now that it is a whole card -| micro-batch | GiB of 95.60 | | -|---:|---:|---| -| 4 | 61.8 | (the shared-card recommendation, now superseded) | -| **8** | **71.8** | **recommended — ~24 GiB clear** | -| 12 | 81.8 | ~14 GiB clear | -| 16 | 91.8 | too tight | +**FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2, +`gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer +steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear. -**Micro-batch 8, grad-accum 1 → 888 optimizer steps** (up from 444 on the shared -card). Two independent wins: the step count doubles, which matters at only one -epoch, and 8 × 8192 tokens gives ~4,096 rows per expert per step against ~512 at -micro-batch 1 — a far healthier GEMM on 704-wide experts, so MFU should land at -the top of the 10–25% band rather than the bottom. +`max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap +drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated +*entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon — +which is 60% of the mix and the axis the seat exists for. Prose and fireball +truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only. -⚠ **Keep gradient checkpointing ON.** With a whole card it is tempting to drop -it (~17% off wall-clock by removing the recompute forward), but activations then -run ~24 GiB per sequence, which forces micro-batch 1 — and micro-batch 1 craters -MoE GEMM efficiency by 8×. Going wide beats going shallow here. Do not -"optimize" this later without re-reading this paragraph. +⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the +full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751 +the typical step sits far below it. + +⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()` +with it. Dropping checkpointing looks like ~17% off wall-clock and instead +forces micro-batch 1. Worse, the second call is the silent one: **without +`enable_input_require_grads()` the frozen base produces no gradient through the +checkpointed blocks, every adapter stays at its initialisation, and the run +completes successfully with an inert adapter.** `prepare_model_for_kbit_training` +used to do it as a side effect of the 4-bit path — so removing 4-bit removes it +too, and nothing warns you. + +### Harness changes this required (eitri-smithy `62b556b`) + +`9d64257` as audited would not have run here. Four fixes: + +1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config + key, defaulting off, per §1. +2. Sequence-chunked CE replacing the model's own loss (the measured table above). +3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own + stale 365-line template and there was **no override parameter anywhere**, so + the upstream-template requirement was not expressible in the code. +4. Gradient checkpointing + `enable_input_require_grads()`. + +Plus `training_eligibility_override` / `overridden_blockers` / +`substitute_controls` in the provenance manifest, and `device_map` pinned to +device 0 so the run cannot stray onto the card holding the inference seats. Also fold in: