docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect

Run 2's reasoning battery reported zero truncations and zero degenerates on both
arms across four passes. The same tune, measured on long-form generation in the
same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real
output-stability regression, structurally invisible to that gate because its
answers are short.

Not a bug in the battery -- a coverage property. An instrument measures the
regime it samples, and output length is a regime. Generalises to context length,
conversation depth, and any axis where the gate's operating point is narrower
than production's.

The actionable form: enumerate the regimes your gate set spans, name the ones it
does not, and decide deliberately rather than discovering the gap downstream.
Corollary on sequencing -- put a long-form generation in the gate and put it
early, because a length-dependent regression is exactly the one you want found
before four clean short-task passes make everyone comfortable.
This commit is contained in:
2026-08-26 02:24:42 -07:00
parent 3ae32ddc7f
commit 0bb9ee7777
+31
View File
@@ -861,6 +861,37 @@ that, because the instrument was dark.
If breaking it changes nothing, you have documentation, not a test.
#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect
Run 2's reasoning battery reported **zero truncations and zero degenerates on
both arms, across all four passes.** The same tune, measured on long-form
generation in the same session:
truncated base 0/384 -> tuned 38/384 (9.9%)
degenerate base 0/384 -> tuned 19/384 (4.9%)
**A real output-stability regression, invisible to the reasoning gate**, because
that gate's answers are short and the defect only appears past a length its
tasks never reach. Had the gate been composed only of short-answer tasks, the
tune would have passed it cleanly.
This is not a bug in the battery. It is a **coverage** property: an instrument
measures the regime it samples, and output length is a regime. The same shape
applies to context length, conversation depth, batch composition, and any other
axis where your gate's operating point is narrower than production's.
**When you assemble a gate set, enumerate the regimes your tasks actually span**
and name the ones they do not — then decide deliberately whether the gap is
acceptable, rather than discovering it from a downstream complaint. Ours was
found only because a separate long-form instrument ran in the same window.
⚠ The corollary is about *sequencing*, and it is cheap: put at least one
long-form generation in the gate, and put it **early**. A regression that only
manifests at length is exactly the one you want to find before the short tasks
give you four passes of clean numbers to feel good about.
(Run-2 gate, brokkr-smithy-dev, 2026-08-26.)
#### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null
`0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output