docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect
Run 2's reasoning battery reported zero truncations and zero degenerates on both arms across four passes. The same tune, measured on long-form generation in the same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real output-stability regression, structurally invisible to that gate because its answers are short. Not a bug in the battery -- a coverage property. An instrument measures the regime it samples, and output length is a regime. Generalises to context length, conversation depth, and any axis where the gate's operating point is narrower than production's. The actionable form: enumerate the regimes your gate set spans, name the ones it does not, and decide deliberately rather than discovering the gap downstream. Corollary on sequencing -- put a long-form generation in the gate and put it early, because a length-dependent regression is exactly the one you want found before four clean short-task passes make everyone comfortable.
This commit is contained in:
@@ -861,6 +861,37 @@ that, because the instrument was dark.
|
||||
|
||||
If breaking it changes nothing, you have documentation, not a test.
|
||||
|
||||
#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect
|
||||
|
||||
Run 2's reasoning battery reported **zero truncations and zero degenerates on
|
||||
both arms, across all four passes.** The same tune, measured on long-form
|
||||
generation in the same session:
|
||||
|
||||
truncated base 0/384 -> tuned 38/384 (9.9%)
|
||||
degenerate base 0/384 -> tuned 19/384 (4.9%)
|
||||
|
||||
**A real output-stability regression, invisible to the reasoning gate**, because
|
||||
that gate's answers are short and the defect only appears past a length its
|
||||
tasks never reach. Had the gate been composed only of short-answer tasks, the
|
||||
tune would have passed it cleanly.
|
||||
|
||||
This is not a bug in the battery. It is a **coverage** property: an instrument
|
||||
measures the regime it samples, and output length is a regime. The same shape
|
||||
applies to context length, conversation depth, batch composition, and any other
|
||||
axis where your gate's operating point is narrower than production's.
|
||||
|
||||
**When you assemble a gate set, enumerate the regimes your tasks actually span**
|
||||
and name the ones they do not — then decide deliberately whether the gap is
|
||||
acceptable, rather than discovering it from a downstream complaint. Ours was
|
||||
found only because a separate long-form instrument ran in the same window.
|
||||
|
||||
⚠ The corollary is about *sequencing*, and it is cheap: put at least one
|
||||
long-form generation in the gate, and put it **early**. A regression that only
|
||||
manifests at length is exactly the one you want to find before the short tasks
|
||||
give you four passes of clean numbers to feel good about.
|
||||
|
||||
(Run-2 gate, brokkr-smithy-dev, 2026-08-26.)
|
||||
|
||||
#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null
|
||||
|
||||
`0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output
|
||||
|
||||
Reference in New Issue
Block a user