vh
64bf9d313f
§3.13, plus the probe that produced it. Two lessons, both about measuring the wrong thing confidently. First: a tune applied AFTER an abliteration can walk it back, and a reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's preregistered gate measured none of it — a tune that gains 41 items of contradiction detection and quietly restores refusals passes every check. The compliance axis has to be added explicitly. Second, and this is the trap: measure the axis the abliteration was actually FOR. Ours was run so the model engages explicit fiction. The probe reached for mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and carried a recorded baseline. Different refusal surface entirely, and a model moves on them independently. 29/100 general-harm refusals on a tune whose prose the operator was praising at the time is not obviously a defect and may be desirable: general-harm refusals returning while domain compliance holds is close to the ideal shape for an internal creative seat. The measurement was real; its relevance was assumed. Also recorded, because both were nearly missed: - Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the load-bearing number is 71. Stock refused 100/100; near that would mean the abliteration was undone. 71 complying means partially walked back on one axis — a different finding, and only one of the two threatens the seat. - A baseline from a different harness is not a baseline. The recorded 3/100 came from the abliteration tool's scorer, which reads first-token probability distributions; a probe that generates and regexes is a different instrument. Run your own against both arms on the same seat or report the number alone. - A refusal regex undercounts, so classify hard/deflect/comply — and the free discriminator: if both arms return zero deflections the model is binary; if only one does, the regex is fine. An artifact does not care which arm it runs against.
Training throughput probes
Instruments for finding where a training step's time actually went. Written
2026-08-24 during the Gemma-4 26B-A4B ERP/RP tune investigation; the lessons
they produced live in
docs/pfi/training-throughput-playbook.md.
These are diagnostic instruments, not production code. They hard-code paths for that run. Adapt the constants at the top; keep the measurement design.
The probes
| script | settles | GPU | runtime |
|---|---|---|---|
step0_mask.py |
mask band structure; which layers keep the is_causal fast path |
no | ~30 s |
step2_padding.py |
padding waste, length distribution, CE chunk sizing | no | ~2 min |
step_bucket.py |
bucketing gain, bucket-size sweep, source diversity | no | ~3 min |
step1_profile.py |
scaling fit, padding penalty, CE wall clock, kernel table | yes | ~15 min |
Run in that order. Only the last needs the real checkpoint, and it wants an
idle card — it loads ~48 GiB and peaks near 77 GiB at 2 × 16,384.
Design rules worth preserving when you adapt these
step1_profile.py reuses the harness's own discover_target_modules and
replicates its compute_loss byte-for-byte rather than re-implementing the
step. A probe that reimplements the training step measures the probe. If you
port this, keep the import from the real harness.
step0_mask.py needs no weights and no GPU — SDPA backend selection and
mask construction depend on shapes, dtype and mask presence, not on weight
values. That is what makes the correctness assertion cheap enough to run before
every job.
The scaling test takes three points, not two. Two points over three plausible terms (quadratic, linear, fixed-per-batch) is underdetermined; see playbook §1.1 for the hour that cost.
step_bucket.py sweeps bucket size deliberately. The first version
re-sorted within each bucket, which silently collapsed every bucket size to a
full global sort and made the sweep a no-op. If you change the pairing logic,
check that the sweep still varies something.
Raw evidence
step1-profile-output-2026-08-24.txt is the unedited output of the run the
playbook's numbers come from — scaling points, padding penalty, CE timing, and
the full key_averages() kernel table. Kept so the claims can be re-derived
rather than taken on faith.
⚠ That table double-counts: key_averages() lists both the ATen op and the
CUDA kernel it launched, each carrying the same self device time. Sum device
kernel rows only. See playbook §3.4.