diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 486c81a..6079df3 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -861,36 +861,93 @@ that, because the instrument was dark. If breaking it changes nothing, you have documentation, not a test. -#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect +#### ⚠ 4.6.3 ⭐⭐ A short-answer gate cannot see LENGTH BEHAVIOUR — and the cost is measured in runs -Run 2's reasoning battery reported **zero truncations and zero degenerates on -both arms, across all four passes.** The same tune, measured on long-form -generation in the same session: +> **⚠ THIS ENTRY WAS FIRST WRITTEN WRONG, TWICE, AND THE CORRECTIONS ARE THE +> LESSON.** It originally reported an *output-stability regression* — "truncated +> 0→38/384, degenerate 0→19/384" — as a novel run-2 finding. Both halves of that +> framing were false. Kept visible rather than edited over, because the +> retraction path is more instructive than the conclusion. - truncated base 0/384 -> tuned 38/384 (9.9%) - degenerate base 0/384 -> tuned 19/384 (4.9%) +**What was actually true.** Every one of the 46 flags across every run was +`too_short` (rp turns of 3-14 words). The two collapse guards — +`repeated_trigrams >50%` and `non_latin >5%` — **fired zero times, on any run, +on any seed block.** The model never emitted repetitive garbage once. It did not +destabilise. -**A real output-stability regression, invisible to the reasoning gate**, because -that gate's answers are short and the defect only appears past a length its -tasks never reach. Had the gate been composed only of short-answer tasks, the -tune would have passed it cleanly. +**Correction 1: it was not new.** Run 1's own gate record already carried +"tuned lost 18/192 to truncation+degeneracy against base's 1-2," with the +lopsided-exclusion caveat attached and unresolved. Two runs, two *different* +base models, same effect — which makes it a property of the **recipe** (corpus, +mix, objective), not of the base swap. Nobody read the prior run's record before +calling it novel. -This is not a bug in the battery. It is a **coverage** property: an instrument -measures the regime it samples, and output length is a regime. The same shape -applies to context length, conversation depth, batch composition, and any other -axis where your gate's operating point is narrower than production's. +**Correction 2: it was not degeneracy, and it was not even a separate finding.** +It is the **left tail of a length distribution that had been measured and +reported in the same message**: -**When you assemble a gate set, enumerate the regimes your tasks actually span** -and name the ones they do not — then decide deliberately whether the gap is -acceptable, rather than discovering it from a downstream complaint. Ours was -found only because a separate long-form instrument ran in the same window. + PIPPA = 70.3% of bot-turn demonstrations, median 67 words + -> model learns short rp turns + -> rp length distribution shifts down and goes bimodal + -> its lower tail crosses a 15-word floor + -> flagged -> pooled into a "degeneracy" budget -> breaches 10% -> VOID -⚠ The corollary is about *sequencing*, and it is cheap: put at least one -long-form generation in the gate, and put it **early**. A regression that only -manifests at length is exactly the one you want to find before the short tasks -give you four passes of clean numbers to feel good about. +Truncation is the same mechanism mirrored: story output grew 669 → 727 words and +the 1,500-token cap clipped *that* tail. Both halves are **thresholds calibrated +on the base's output shape, applied to a model with a different output shape** — +§4.6.1, which both parties had already written down and neither applied. -(Run-2 gate, brokkr-smithy-dev, 2026-08-26.) +**The lesson that survives, in its sharper form.** A gate composed only of +short-answer tasks **cannot see length behaviour at all** — neither genuine +collapse nor a learned length prior. And because it could not, **the same effect +went two full runs before anyone named it.** The cost of a blind spot in a gate +set is measured in *runs*, not in findings. + +⚠ **The fix is not to move the threshold.** Moving a floor to make a number look +better is the failure this whole section exists to name. The principled fix is +that **two different properties were pooled into one budget**: + + too_short -> LENGTH CONFORMANCE. Report as a DISTRIBUTION + (median, p25 against stated targets). A pass/fail + floor can be satisfied by moving the number. + repeated_trigrams -> COLLAPSE. This is what a VOID budget should govern. + non_latin -> COLLAPSE. + +Separating them stops the VOID firing on a model that never collapsed **without +relaxing anything**. Measured: pooled, VOID fired on 6 of 8 seed blocks; +separated, **zero** would have fired. + +##### ⚠ 4.6.3.1 A trip point inside the serving stack's own jitter will flip + +Same seed block, same weights, same config, three observations: + + 9/94 = 9.6% 12/95 = 12.6% 9/94 = 9.6% sd 1.77 pp + +**Identical everything, and the rate moves three points** — vLLM nondeterminism +under batching, because load changes batch composition. A guard whose trip point +sits inside that band flips run to run, and the next person sees a VOID appear or +vanish and reasonably concludes one run was wrong. **Neither is.** + +This is a distinct defect from an inert gate: not one that *cannot* fail, but one +that fails *non-deterministically* — worse in one specific way, because it +produces disagreement between honest observers rather than silence. + +Say it precisely. Not "the gate is non-deterministic" but **"the trip point sits +inside the stack's own jitter"** — the cause is specific and the fix is to move +the trip point off the jitter, not to make the guard deterministic. + +⚠ **When you measure a rate to settle this, split the design.** Distinct seed +blocks measure the *model's* rate; repeated same-seed runs measure the *serving +stack's* contribution. Pooled, you cannot tell which variance you are planning +around. Measured here: block-to-block sd 2.78 pp against a binomial expectation +of 3.29 pp at n=96 — **no excess between-block variance at all.** The rate was a +stable property of the model; eight samples of ninety-six merely looked erratic. + +⚠ **Bind a measured rate to the corpus it came from, in its name.** +`under_floor_rate_run02 = 11.78% [9.49, 14.07] @ floor 15, corpus = run-02 mix`. +It is a property of that mix, not of the tune, and the day the mix changes it is +obsolete. A bare number in a doc outlives its validity silently — the same +stale-derived-artifact shape as §4.1 and §4.7. #### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null