vh
2656196f47
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
93 lines
4.3 KiB
Markdown
93 lines
4.3 KiB
Markdown
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
|
|
|
`[2026-08-25]`
|
|
|
|
## The run
|
|
|
|
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
|
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
|
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
|
|
|
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
|
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
|
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
|
*entirely padding*, and removing padding removed the variance rather than just
|
|
the mean.
|
|
|
|
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
|
rolling average.** The first two were instantaneous tqdm readings off a number
|
|
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
|
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
|
the one metric I kept reporting.
|
|
|
|
## lora_B gate — PASSED, twice
|
|
|
|
checkpoint-100 205/205 non-zero, median norm 0.829
|
|
final adapter 205/205 non-zero, median norm 1.708
|
|
vision_tower tensors: 0 on both
|
|
|
|
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
|
rather than saturating early. This check **never ran in round 1** (died at step
|
|
19, first checkpoint was 100) and it is the only failure mode that stays
|
|
invisible until the acceptance gate reports base-identical numbers.
|
|
|
|
## The gate — brokkr-smithy-dev
|
|
|
|
**It did the thing it was built to do:**
|
|
|
|
metric base A/B tuned delta floor
|
|
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
|
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
|
|
|
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
|
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
|
|
|
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
|
memorisation: 0.0000 on BOTH arms, all three corpora
|
|
|
|
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
|
than argument.**
|
|
|
|
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
|
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
|
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
|
the number carries an asterisk.
|
|
|
|
## The noise-floor near-miss — the methodology lesson
|
|
|
|
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
|
twice his instrument's own variance.**
|
|
|
|
--per-type 32 max swing across tasks: 9 points
|
|
--per-type 128 max swing across tasks: 1 point
|
|
|
|
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
|
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
|
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
|
running a control he did not strictly need. Quadrupling n collapsed the noise
|
|
exactly as binomial statistics predicts.
|
|
|
|
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
|
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
|
default of 256. Continuous batching changes reduction order and borderline items
|
|
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
|
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
|
reason: a floor measured on a seat serving one request at a time is not the floor
|
|
that applies to the seat we ship.
|
|
|
|
## The confound I built and he caught
|
|
|
|
I optimised a pipeline for production and then handed him its output as an eval
|
|
instrument **without asking whether those were the same job.** The tuned arm
|
|
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
|
regression would have been tuning-damage OR quantization-damage with no way to
|
|
separate them, and the gate's whole question is "did the tune cost us
|
|
capability."
|
|
|
|
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
|
lines** (weights path, served name), template sha256 identical
|
|
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
|
three launches). Quantization moved *downstream* of the gate.
|
|
|
|
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|