diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 51f4479..486c81a 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -861,6 +861,37 @@ that, because the instrument was dark. If breaking it changes nothing, you have documentation, not a test. +#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect + +Run 2's reasoning battery reported **zero truncations and zero degenerates on +both arms, across all four passes.** The same tune, measured on long-form +generation in the same session: + + truncated base 0/384 -> tuned 38/384 (9.9%) + degenerate base 0/384 -> tuned 19/384 (4.9%) + +**A real output-stability regression, invisible to the reasoning gate**, because +that gate's answers are short and the defect only appears past a length its +tasks never reach. Had the gate been composed only of short-answer tasks, the +tune would have passed it cleanly. + +This is not a bug in the battery. It is a **coverage** property: an instrument +measures the regime it samples, and output length is a regime. The same shape +applies to context length, conversation depth, batch composition, and any other +axis where your gate's operating point is narrower than production's. + +**When you assemble a gate set, enumerate the regimes your tasks actually span** +and name the ones they do not — then decide deliberately whether the gap is +acceptable, rather than discovering it from a downstream complaint. Ours was +found only because a separate long-form instrument ran in the same window. + +⚠ The corollary is about *sequencing*, and it is cheap: put at least one +long-form generation in the gate, and put it **early**. A regression that only +manifests at length is exactly the one you want to find before the short tasks +give you four passes of clean numbers to feel good about. + +(Run-2 gate, brokkr-smithy-dev, 2026-08-26.) + #### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null `0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output