docs(training-playbook): 4.6.3 was wrong twice — correct it, and keep the retraction visible
The entry reported an 'output-stability regression' as a novel run-2 finding. Both halves were false and the corrections are more instructive than the original conclusion, so they stay in-line rather than being edited over. Not new: run 1's own gate record already carried the same effect with a caveat attached and unresolved. Two runs across two different base models makes it a property of the RECIPE, not of the base swap -- which also means a third run that changes the base again will not fix it. Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns of 3-14 words, and the two collapse guards fired ZERO times on any run. It is the left tail of a length distribution that had been measured and reported in the same message. Truncation is the same mechanism mirrored on the story side. Both are thresholds calibrated on the base's output shape applied to a model with a different one -- 4.6.1, which both parties had written down and neither applied. The surviving lesson is sharper: a short-answer gate cannot see length behaviour AT ALL, and because it could not, the effect went two full runs before anyone named it. The cost of a gate-set blind spot is measured in runs. Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is non-deterministic' but 'the trip point sits inside the jitter', because the fix follows from the precise statement. Includes the split-design rule for measuring such a rate, and the rule that a measured rate must carry its corpus in its name.
This commit is contained in:
@@ -861,36 +861,93 @@ that, because the instrument was dark.
|
||||
|
||||
If breaking it changes nothing, you have documentation, not a test.
|
||||
|
||||
#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect
|
||||
#### ⚠ 4.6.3 ⭐⭐ A short-answer gate cannot see LENGTH BEHAVIOUR — and the cost is measured in runs
|
||||
|
||||
Run 2's reasoning battery reported **zero truncations and zero degenerates on
|
||||
both arms, across all four passes.** The same tune, measured on long-form
|
||||
generation in the same session:
|
||||
> **⚠ THIS ENTRY WAS FIRST WRITTEN WRONG, TWICE, AND THE CORRECTIONS ARE THE
|
||||
> LESSON.** It originally reported an *output-stability regression* — "truncated
|
||||
> 0→38/384, degenerate 0→19/384" — as a novel run-2 finding. Both halves of that
|
||||
> framing were false. Kept visible rather than edited over, because the
|
||||
> retraction path is more instructive than the conclusion.
|
||||
|
||||
truncated base 0/384 -> tuned 38/384 (9.9%)
|
||||
degenerate base 0/384 -> tuned 19/384 (4.9%)
|
||||
**What was actually true.** Every one of the 46 flags across every run was
|
||||
`too_short` (rp turns of 3-14 words). The two collapse guards —
|
||||
`repeated_trigrams >50%` and `non_latin >5%` — **fired zero times, on any run,
|
||||
on any seed block.** The model never emitted repetitive garbage once. It did not
|
||||
destabilise.
|
||||
|
||||
**A real output-stability regression, invisible to the reasoning gate**, because
|
||||
that gate's answers are short and the defect only appears past a length its
|
||||
tasks never reach. Had the gate been composed only of short-answer tasks, the
|
||||
tune would have passed it cleanly.
|
||||
**Correction 1: it was not new.** Run 1's own gate record already carried
|
||||
"tuned lost 18/192 to truncation+degeneracy against base's 1-2," with the
|
||||
lopsided-exclusion caveat attached and unresolved. Two runs, two *different*
|
||||
base models, same effect — which makes it a property of the **recipe** (corpus,
|
||||
mix, objective), not of the base swap. Nobody read the prior run's record before
|
||||
calling it novel.
|
||||
|
||||
This is not a bug in the battery. It is a **coverage** property: an instrument
|
||||
measures the regime it samples, and output length is a regime. The same shape
|
||||
applies to context length, conversation depth, batch composition, and any other
|
||||
axis where your gate's operating point is narrower than production's.
|
||||
**Correction 2: it was not degeneracy, and it was not even a separate finding.**
|
||||
It is the **left tail of a length distribution that had been measured and
|
||||
reported in the same message**:
|
||||
|
||||
**When you assemble a gate set, enumerate the regimes your tasks actually span**
|
||||
and name the ones they do not — then decide deliberately whether the gap is
|
||||
acceptable, rather than discovering it from a downstream complaint. Ours was
|
||||
found only because a separate long-form instrument ran in the same window.
|
||||
PIPPA = 70.3% of bot-turn demonstrations, median 67 words
|
||||
-> model learns short rp turns
|
||||
-> rp length distribution shifts down and goes bimodal
|
||||
-> its lower tail crosses a 15-word floor
|
||||
-> flagged -> pooled into a "degeneracy" budget -> breaches 10% -> VOID
|
||||
|
||||
⚠ The corollary is about *sequencing*, and it is cheap: put at least one
|
||||
long-form generation in the gate, and put it **early**. A regression that only
|
||||
manifests at length is exactly the one you want to find before the short tasks
|
||||
give you four passes of clean numbers to feel good about.
|
||||
Truncation is the same mechanism mirrored: story output grew 669 → 727 words and
|
||||
the 1,500-token cap clipped *that* tail. Both halves are **thresholds calibrated
|
||||
on the base's output shape, applied to a model with a different output shape** —
|
||||
§4.6.1, which both parties had already written down and neither applied.
|
||||
|
||||
(Run-2 gate, brokkr-smithy-dev, 2026-08-26.)
|
||||
**The lesson that survives, in its sharper form.** A gate composed only of
|
||||
short-answer tasks **cannot see length behaviour at all** — neither genuine
|
||||
collapse nor a learned length prior. And because it could not, **the same effect
|
||||
went two full runs before anyone named it.** The cost of a blind spot in a gate
|
||||
set is measured in *runs*, not in findings.
|
||||
|
||||
⚠ **The fix is not to move the threshold.** Moving a floor to make a number look
|
||||
better is the failure this whole section exists to name. The principled fix is
|
||||
that **two different properties were pooled into one budget**:
|
||||
|
||||
too_short -> LENGTH CONFORMANCE. Report as a DISTRIBUTION
|
||||
(median, p25 against stated targets). A pass/fail
|
||||
floor can be satisfied by moving the number.
|
||||
repeated_trigrams -> COLLAPSE. This is what a VOID budget should govern.
|
||||
non_latin -> COLLAPSE.
|
||||
|
||||
Separating them stops the VOID firing on a model that never collapsed **without
|
||||
relaxing anything**. Measured: pooled, VOID fired on 6 of 8 seed blocks;
|
||||
separated, **zero** would have fired.
|
||||
|
||||
##### ⚠ 4.6.3.1 A trip point inside the serving stack's own jitter will flip
|
||||
|
||||
Same seed block, same weights, same config, three observations:
|
||||
|
||||
9/94 = 9.6% 12/95 = 12.6% 9/94 = 9.6% sd 1.77 pp
|
||||
|
||||
**Identical everything, and the rate moves three points** — vLLM nondeterminism
|
||||
under batching, because load changes batch composition. A guard whose trip point
|
||||
sits inside that band flips run to run, and the next person sees a VOID appear or
|
||||
vanish and reasonably concludes one run was wrong. **Neither is.**
|
||||
|
||||
This is a distinct defect from an inert gate: not one that *cannot* fail, but one
|
||||
that fails *non-deterministically* — worse in one specific way, because it
|
||||
produces disagreement between honest observers rather than silence.
|
||||
|
||||
Say it precisely. Not "the gate is non-deterministic" but **"the trip point sits
|
||||
inside the stack's own jitter"** — the cause is specific and the fix is to move
|
||||
the trip point off the jitter, not to make the guard deterministic.
|
||||
|
||||
⚠ **When you measure a rate to settle this, split the design.** Distinct seed
|
||||
blocks measure the *model's* rate; repeated same-seed runs measure the *serving
|
||||
stack's* contribution. Pooled, you cannot tell which variance you are planning
|
||||
around. Measured here: block-to-block sd 2.78 pp against a binomial expectation
|
||||
of 3.29 pp at n=96 — **no excess between-block variance at all.** The rate was a
|
||||
stable property of the model; eight samples of ninety-six merely looked erratic.
|
||||
|
||||
⚠ **Bind a measured rate to the corpus it came from, in its name.**
|
||||
`under_floor_rate_run02 = 11.78% [9.49, 14.07] @ floor 15, corpus = run-02 mix`.
|
||||
It is a property of that mix, not of the tune, and the day the mix changes it is
|
||||
obsolete. A bare number in a doc outlives its validity silently — the same
|
||||
stale-derived-artifact shape as §4.1 and §4.7.
|
||||
|
||||
#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null
|
||||
|
||||
|
||||
Reference in New Issue
Block a user