docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect
Run 2's reasoning battery reported zero truncations and zero degenerates on both arms across four passes. The same tune, measured on long-form generation in the same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real output-stability regression, structurally invisible to that gate because its answers are short. Not a bug in the battery -- a coverage property. An instrument measures the regime it samples, and output length is a regime. Generalises to context length, conversation depth, and any axis where the gate's operating point is narrower than production's. The actionable form: enumerate the regimes your gate set spans, name the ones it does not, and decide deliberately rather than discovering the gap downstream. Corollary on sequencing -- put a long-form generation in the gate and put it early, because a length-dependent regression is exactly the one you want found before four clean short-task passes make everyone comfortable.
This commit is contained in:
@@ -861,6 +861,37 @@ that, because the instrument was dark.
|
|||||||
|
|
||||||
If breaking it changes nothing, you have documentation, not a test.
|
If breaking it changes nothing, you have documentation, not a test.
|
||||||
|
|
||||||
|
#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect
|
||||||
|
|
||||||
|
Run 2's reasoning battery reported **zero truncations and zero degenerates on
|
||||||
|
both arms, across all four passes.** The same tune, measured on long-form
|
||||||
|
generation in the same session:
|
||||||
|
|
||||||
|
truncated base 0/384 -> tuned 38/384 (9.9%)
|
||||||
|
degenerate base 0/384 -> tuned 19/384 (4.9%)
|
||||||
|
|
||||||
|
**A real output-stability regression, invisible to the reasoning gate**, because
|
||||||
|
that gate's answers are short and the defect only appears past a length its
|
||||||
|
tasks never reach. Had the gate been composed only of short-answer tasks, the
|
||||||
|
tune would have passed it cleanly.
|
||||||
|
|
||||||
|
This is not a bug in the battery. It is a **coverage** property: an instrument
|
||||||
|
measures the regime it samples, and output length is a regime. The same shape
|
||||||
|
applies to context length, conversation depth, batch composition, and any other
|
||||||
|
axis where your gate's operating point is narrower than production's.
|
||||||
|
|
||||||
|
**When you assemble a gate set, enumerate the regimes your tasks actually span**
|
||||||
|
and name the ones they do not — then decide deliberately whether the gap is
|
||||||
|
acceptable, rather than discovering it from a downstream complaint. Ours was
|
||||||
|
found only because a separate long-form instrument ran in the same window.
|
||||||
|
|
||||||
|
⚠ The corollary is about *sequencing*, and it is cheap: put at least one
|
||||||
|
long-form generation in the gate, and put it **early**. A regression that only
|
||||||
|
manifests at length is exactly the one you want to find before the short tasks
|
||||||
|
give you four passes of clean numbers to feel good about.
|
||||||
|
|
||||||
|
(Run-2 gate, brokkr-smithy-dev, 2026-08-26.)
|
||||||
|
|
||||||
#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null
|
#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null
|
||||||
|
|
||||||
`0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output
|
`0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output
|
||||||
|
|||||||
Reference in New Issue
Block a user