af5582c96db20beed8fa0160151855a186dade01
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
1a4ef5c7a1 |
docs(training-playbook): 4.6.3 was wrong twice — correct it, and keep the retraction visible
The entry reported an 'output-stability regression' as a novel run-2 finding. Both halves were false and the corrections are more instructive than the original conclusion, so they stay in-line rather than being edited over. Not new: run 1's own gate record already carried the same effect with a caveat attached and unresolved. Two runs across two different base models makes it a property of the RECIPE, not of the base swap -- which also means a third run that changes the base again will not fix it. Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns of 3-14 words, and the two collapse guards fired ZERO times on any run. It is the left tail of a length distribution that had been measured and reported in the same message. Truncation is the same mechanism mirrored on the story side. Both are thresholds calibrated on the base's output shape applied to a model with a different one -- 4.6.1, which both parties had written down and neither applied. The surviving lesson is sharper: a short-answer gate cannot see length behaviour AT ALL, and because it could not, the effect went two full runs before anyone named it. The cost of a gate-set blind spot is measured in runs. Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is non-deterministic' but 'the trip point sits inside the jitter', because the fix follows from the precise statement. Includes the split-design rule for measuring such a rate, and the rule that a measured rate must carry its corpus in its name. |
||
|
vh
|
0bb9ee7777 |
docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect
Run 2's reasoning battery reported zero truncations and zero degenerates on both arms across four passes. The same tune, measured on long-form generation in the same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real output-stability regression, structurally invisible to that gate because its answers are short. Not a bug in the battery -- a coverage property. An instrument measures the regime it samples, and output length is a regime. Generalises to context length, conversation depth, and any axis where the gate's operating point is narrower than production's. The actionable form: enumerate the regimes your gate set spans, name the ones it does not, and decide deliberately rather than discovering the gap downstream. Corollary on sequencing -- put a long-form generation in the gate and put it early, because a length-dependent regression is exactly the one you want found before four clean short-task passes make everyone comfortable. |
||
|
vh
|
a0f59d2778 |
docs(training-playbook): 4.6.2 — a null result needs a positive control
From the run-2 gate. A memorisation probe reporting 0.00% across all 72 items is the correct output for a model that has not seen the corpus, and is also the exact output of a probe that is not firing. Nothing in the number distinguishes them. brokkr-smithy-dev drove the overlap function with known-answer inputs (identical 100%, half-verbatim 65.38%, unrelated 0%, empty 0%) before trusting the null, which is what converts a suspicious zero into evidence. This is 4.5's inert gate wearing a different face: there a check that could not return 'fail', here a measurement that cannot return non-zero. A clean null is the most reassuring output an instrument produces and the least self-evidencing. Same section records the identical-on-both-arms variant: the diversity battery's rp family froze zero markers, so its attractor hit rate read 0.0 on base AND tuned. That reads as a clean result and means the instrument cannot discriminate on that family. Report as a bounded limitation, never as a delta of zero -- a check returning the same value for every input is not measuring. Checklist gains the line. |
||
|
vh
|
d54f25605f |
docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2 launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on the same checkout during its seven hours and _git_commit() was called at save time. Commit AHEAD of the code that ran, naming changes it never executed -- including the provenance fields this section prompted. Same defect as run 1's BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at launch, carry it, and record the dirty flag beside the commit rather than instead of it. Generalises to every run-scoped identity: anything read at save time describes the world at save time. 4.6.1 -- the inverse of the inert gate, and it costs trust rather than correctness. A coherence gate false-rejected 'The capital of Portugal is Lisbon' as degenerate against a global 15-word floor. The floor was calibrated against the wrong reference, not set too strict. Lowering it globally would blunt the check where short output genuinely is degeneration; the fix is a floor per prompt. Write the positive test alongside the negative one. |
||
|
vh
|
dbca9a3c66 |
docs(training-playbook): audit the whole manifest against the pairing rule
§4.3's generalisation was stated and then not applied to the manifest that prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and the one pairing that would have caught the §4.1 cache failure -- the mask's sha against the loss-token delta -- existed by accident, because someone had asked for an encode report for unrelated reasons. Adds the audit table, and the rider that matters more than the table: put the observed check where it can actually FAIL. chat_template_sha256's pair is the sha of the string the tokenizer carries, but asserting that in the parent one line after assigning the file to the tokenizer compares a value to itself. It belongs in the encode worker -- a different process, across a pickle boundary, where an unset config key silently leaves every worker rendering through the checkpoint's own template. |
||
|
vh
|
c1db188e6a |
docs(training-playbook): §4.3 records an OBSERVED consequence, not just a config string
brokkr-smithy-dev pointed §4.5's own test at §4.3's remedy: recording `attn_implementation_resolved` is a check that cannot fail on the axis the failure lives on. A silent Dynamo fallback to uncompiled flex leaves `config._attn_implementation == "flex_attention"` untouched while the run computes at ~20x the cost and, per torch's own docs, does not work correctly through the backward pass. The field records the request's RESOLUTION, not its SURVIVAL. On the failure mode that matters it reports success either way. So the section now requires the step-time distribution beside it -- n, min, p50, p99, max -- which is the check that can actually fail. Compiled sits at p50 ~20 s; a fallback at ~400 s. One perf_counter() in on_step_end buys it. Distribution rather than a mean, because a mean hides exactly the bimodality a PARTIAL fallback produces. Generalised past this instance: any provenance field recording a CONFIGURED value is a claim about intent. If the failure you fear is the configuration silently not taking effect, you need a second field recording an OBSERVED consequence, and the pairing is the check. A settings dump alone is decorative. Two implementation details are called out because both were wrong in the first draft -- percentiles nearest-rank so every reported value is a real observation, and exclude the FIRST step rather than the slowest, since step 1 carries compilation but is not reliably the maximum on a variable-width run. New §4.7.1: rotate the log on relaunch. Run 2's first attempt died on the warmup_ratio TypeError and the relaunch appended, so the traceback sat at line 15 of a file whose live run began at line 39 -- and a `tail -n +1 -F` monitor replayed the dead traceback as a fresh event. One file describes one run. Checklist gains both lines. |
||
|
vh
|
dae6ede8e2 |
docs(training-playbook): §4 — when the artifact lies about itself
The playbook covered why a run is SLOW. It did not cover the more expensive
failure: a run that COMPLETES, reports plausible numbers, and is wrong about
itself. Seven of those turned up on the Gemma-4 ERP/RP tune between 08-24 and
08-26 and not one raised an error.
New §4, seven landmines plus a pre-launch checklist:
4.1 a cache key must cover the MEANING of the cached thing. The encode
cache missed the impersonation mask; run 2 would have reused run 1's
unmasked encodings and written impersonation_mask_sha256 into its own
manifest while doing it. No error, no count change, normal loss curve.
4.2 validating a VALUE is not validating the PARAMETER. warmup_ratio was
in range and deleted from transformers 5. Build kwargs as data and
diff the NAMES against the installed signature -- you cannot check the
argument list of a call you have already made.
4.3 record what the run RESOLVED to, never what it requested. Run 1
recorded no attention backend, so an MFU panel profiled the serving
seat under sdpa and recommended adopting flex_attention for a run that
was already using it.
4.4 never train from a dirty tree; harness_commit will name a commit that
does not describe the run. Annotate afterwards, never edit the shipped
artifact -- and state what is NOT wrong, or the note casts doubt on
every field it omits.
4.5 a watchdog whose pgrep pattern appears in its own argv can only ever
return "alive". The inert-gate shape in a liveness check.
4.6 an instrument nobody runs is not an instrument. Mutation-check any
test guarding a property that fails silently.
4.7 fix a stale measurement at the source. "~4.3 HOURS to rebuild the
encode cache" (really 145.5 s) was copied into a new launcher by the
same person who had just measured the real number.
4.8 the pre-launch honesty checklist, ten minutes.
Also:
- Header and framing widened. The file is now a training playbook with a
throughput half and an integrity half; the filename stays for inbound links.
- Sections 4-7 renumbered to 5-8. External refs are all to §1.1 and §3.4 and
are unaffected.
- Four rows added to the superseded-claims table, including the kernel table /
68% quadratic / 8.6% MFU set, which describe the serving seat rather than
the training run.
- gemma4-erp-tune-sizing.md §6 carries a correction banner with the explicit
falls/survives split, because that is the doc someone actually reads before
a run.
|
||
|
vh
|
64bf9d313f |
docs(training-playbook): measure refusal retention on the abliteration's OWN axis
§3.13, plus the probe that produced it. Two lessons, both about measuring the wrong thing confidently. First: a tune applied AFTER an abliteration can walk it back, and a reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's preregistered gate measured none of it — a tune that gains 41 items of contradiction detection and quietly restores refusals passes every check. The compliance axis has to be added explicitly. Second, and this is the trap: measure the axis the abliteration was actually FOR. Ours was run so the model engages explicit fiction. The probe reached for mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and carried a recorded baseline. Different refusal surface entirely, and a model moves on them independently. 29/100 general-harm refusals on a tune whose prose the operator was praising at the time is not obviously a defect and may be desirable: general-harm refusals returning while domain compliance holds is close to the ideal shape for an internal creative seat. The measurement was real; its relevance was assumed. Also recorded, because both were nearly missed: - Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the load-bearing number is 71. Stock refused 100/100; near that would mean the abliteration was undone. 71 complying means partially walked back on one axis — a different finding, and only one of the two threatens the seat. - A baseline from a different harness is not a baseline. The recorded 3/100 came from the abliteration tool's scorer, which reads first-token probability distributions; a probe that generates and regexes is a different instrument. Run your own against both arms on the same seat or report the number alone. - A refusal regex undercounts, so classify hard/deflect/comply — and the free discriminator: if both arms return zero deflections the model is binary; if only one does, the regex is fine. An artifact does not care which arm it runs against. |
||
|
vh
|
a696b49e2a |
docs(training-playbook): merging a tune back toward stock can UNDO an abliteration
§3.12. brokkr-smithy-dev caught and retracted his own recommendation mid-thread; recording it before it reads back later as advice. A common remedy for an overfit tune is a partial merge back toward the base to recover general capability. The published recipes that recommend it merge into the STOCK instruct checkpoint. On an abliterated base, following that literally re-introduces the exact refusal directions the abliteration was run to remove — and it is silent, because the merged model looks healthier on general benchmarks while the property the seat exists for quietly returns. Rule: any merge-back targets the SAME base the LoRA was trained against, never the upstream stock weights however similar the name. The wider lesson is about recipe-card provenance. Community cards are per-checkpoint artifacts and do not transfer across dense-vs-MoE, stock-vs-abliterated, or size variants. The worked example: a recommendation carried from a card for a DENSE STOCK 31B onto a MoE ABLITERATED 26B-A4B on the strength of a shared family name. The overfitting warning on that card happened to come from the right architecture; the pipeline, reward stacks and merge-back came from the wrong one. Same family, three axes apart. So: before quoting a recipe card at a decision, state which checkpoint it was written for and which axes differ. "Same family" is not an answer. |
||
|
vh
|
2ec8f42297 |
docs(training-playbook): base-viability pre-flight, three greps before you pick
§3.11. Three consecutive "what about X as a base?" questions in one session,
each answerable in minutes, none of which had been asked before a 7-hour
training window was committed. Writing the check down so it runs first.
1. does it fit for TRAINING - BF16 weights against the real measured peak,
not the weight figure (Gemma-4 is 48.1 GiB of weights and peaks at
79.7 GiB at mb2/seq-16k). Model-line names lie: "Mistral Small 4" is 119 B,
238 GB in BF16, more than both cards combined. QLoRA is not an escape
hatch for MoE - bitsandbytes walks nn.Linear and fused 3-D experts are not
that.
2. if MoE - does the serving engine implement get_expert_mapping. Zero means
LoRA cannot be served at all. gemma4*.py -> 0; deepseek_v2, mixtral,
glm4_moe, ernie45_moe -> present.
3. does the model class support LoRA - and GREP THE CLASS, NOT THE FILE.
Point 3 has teeth and I nearly got it wrong twice in one turn. mistral.py greps
as SupportsLoRA=0 and is fully LoRA-capable via LlamaForCausalLM.
mistral_large_3.py greps as 0 for both and inherits get_expert_mapping from
DeepseekV3ForCausalLM. Capability is inherited; a file-level grep misses it and
only MRO resolution answers it. Same class of error as asserting a substring
instead of an effective value.
Worked results recorded for the three candidates evaluated:
Gemma-4 26B-A4B fits, no expert mapping -> trainable, MERGE-ONLY
Mistral Small 4 119B 238 GB, has mapping -> servable, NOT trainable here
Ministral 3 14B ~28 GB, dense, inherited -> passes all three
Adds a fourth glance at architecture shape, since it predicts how much of this
playbook applies at all: uniform head_dim <= 128 with no sliding window keeps
both flash and cuDNN reachable and makes §3.1/§3.3 moot, while mixed head dims
plus a sliding window is exactly what forces dense O(n^2) attention onto
Ampere-generation kernels for 65% of the step.
|
||
|
vh
|
96731bb090 |
docs(training-playbook): prove the serving path before spending the window
§3.10. The quantization playbook already says prove your targets before spending GPU time; this is the same rule one step later, and easier to skip. A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a quantized base at serve time. The sizing doc flagged serving as unsettled and said the requirement was needed "while he is early, not after the run" — the concern was identified correctly and then the check was deferred. Tested afterwards, vLLM refuses outright: gemma4's model class implements zero occurrences of get_expert_mapping, which process_packed_modules_mapping requires for any MoE model. One grep, available months earlier. Two generalisations recorded: - Feature support is per-architecture, not per-family. LoRA works for the DENSE sibling of this same model family and not the MoE one, so "model X is supported" says nothing about X's variants. - A capability gap in the serving engine cannot be worked around from the training side. The adapter here never touched experts and was refused anyway, because the refusal keys on the model being MoE, not on what the adapter targets. Includes the mechanical check: grep the engine's model class for the capability, then start the engine with the feature flag alone — no adapter required, since --enable-lora forces the machinery to initialise and that is where it fails. The recovery is cheap here (merge, ~35 min per tune). The cost of finding out late is that it forecloses an architecture choice after the training window has already been spent. |
||
|
vh
|
7b5fd91d3c |
docs(gemma4-erp-tune): root-cause the 8.6% MFU — attention on Ampere kernels, 29.9% padding
Run-01 was killed at step 19 by operator instruction to root-cause before spending a ~13.9h window. Two independent methods now agree on where the step time went, and neither was the hypothesis the consult panel converged on. Scaling fit (3 points, 2 params, residuals <3ms over an 8x range): A = 6.87e-4 s/token, B = 8.85e-8 s/token^2 quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384 No fixed term was needed, which refutes launch-bound outright. Profiler kernel table (device rows only): attention 22,835.8 ms 65.2% fmha_cutlass*_sm80 dense GEMM 2,774.0 ms 7.9% other 5,739.0 ms 16.4% The attention kernels are sm80 — Ampere-generation CUTLASS running on an sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS. Correctness cleared separately: the sliding mask asserts at max 1024 allowed/row, so the 25 windowed layers were genuinely windowed. The same probe found that right-padding is what pins the 5 global layers to an explicit 4D mask and off the is_causal fast path — measured at 9.4% slower for 24% less loss work at fixed width. The largest available win is not the attention kernel. The corpus is 29.9% padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall clock, no new dependency, unchanged peak memory. Bucket size turned out not to be a diversity knob — roots per accumulation window are flat across a 256x range, so the global micro-batch shuffle does that work alone and the bucket should be tight. Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic home (sibling to the quantization playbook), the four probes under scripts/training-probes/ with raw output kept for re-derivation, and a §6 to the sizing doc carrying the Gemma-4-specific numbers and round-2 restart parameters. Measured negatives recorded so they are not re-chased: grouped_mm (0.9% slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever), FA4 on sm_120. Round-1 state preserved: 609MB encode cache, order manifest, truncation report, resume script. No checkpoints — it died at step 19 and the first was due at 100, so the lora_B inert-adapter gate never ran and moves to the restart. |