From 0bb9ee7777e7d459101812cae2acbca16b514cf7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 02:24:42 -0700 Subject: [PATCH] =?UTF-8?q?docs(training-playbook):=204.6.3=20=E2=80=94=20?= =?UTF-8?q?a=20short-answer=20gate=20cannot=20see=20a=20long-form=20defect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 2's reasoning battery reported zero truncations and zero degenerates on both arms across four passes. The same tune, measured on long-form generation in the same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real output-stability regression, structurally invisible to that gate because its answers are short. Not a bug in the battery -- a coverage property. An instrument measures the regime it samples, and output length is a regime. Generalises to context length, conversation depth, and any axis where the gate's operating point is narrower than production's. The actionable form: enumerate the regimes your gate set spans, name the ones it does not, and decide deliberately rather than discovering the gap downstream. Corollary on sequencing -- put a long-form generation in the gate and put it early, because a length-dependent regression is exactly the one you want found before four clean short-task passes make everyone comfortable. --- docs/pfi/training-throughput-playbook.md | 31 ++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/pfi/training-throughput-playbook.md b/docs/pfi/training-throughput-playbook.md index 51f4479..486c81a 100644 --- a/docs/pfi/training-throughput-playbook.md +++ b/docs/pfi/training-throughput-playbook.md @@ -861,6 +861,37 @@ that, because the instrument was dark. If breaking it changes nothing, you have documentation, not a test. +#### ⚠ 4.6.3 ⭐⭐ A gate of short-answer tasks cannot see a long-form defect + +Run 2's reasoning battery reported **zero truncations and zero degenerates on +both arms, across all four passes.** The same tune, measured on long-form +generation in the same session: + + truncated base 0/384 -> tuned 38/384 (9.9%) + degenerate base 0/384 -> tuned 19/384 (4.9%) + +**A real output-stability regression, invisible to the reasoning gate**, because +that gate's answers are short and the defect only appears past a length its +tasks never reach. Had the gate been composed only of short-answer tasks, the +tune would have passed it cleanly. + +This is not a bug in the battery. It is a **coverage** property: an instrument +measures the regime it samples, and output length is a regime. The same shape +applies to context length, conversation depth, batch composition, and any other +axis where your gate's operating point is narrower than production's. + +**When you assemble a gate set, enumerate the regimes your tasks actually span** +and name the ones they do not — then decide deliberately whether the gap is +acceptable, rather than discovering it from a downstream complaint. Ours was +found only because a separate long-form instrument ran in the same window. + +⚠ The corollary is about *sequencing*, and it is cheap: put at least one +long-form generation in the gate, and put it **early**. A regression that only +manifests at length is exactly the one you want to find before the short tasks +give you four passes of clean numbers to feel good about. + +(Run-2 gate, brokkr-smithy-dev, 2026-08-26.) + #### ⚠ 4.6.2 ⭐⭐ A NULL RESULT needs a positive control before it counts as a null `0.00% / floor 0.00% / max_item 0.0%` across all 72 items is the correct output