memory: snapshot — the tune is trained, gated, and serving
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.
Five new detail files carry the substance:
erp-tune-run2-complete the run, the gate, the noise-floor near-miss
(brokkr was one step from reporting a 13-point
T6 regression sitting inside twice his
instrument's own variance)
mfu-root-caused-attention 8.6% MFU was an accounting artifact; real
utilisation 17-20%, cost was attention on
AMPERE kernels. Two independent methods agreed
to 2.6 points.
nvfp4-serving-pipeline merged weights are MANDATORY — vLLM cannot
serve a LoRA on ANY Gemma-4 — plus the recipe
that silently misses all 11,520 expert tensors
refusal-retention-probe measured base 0/100 -> tuned 29/100, then had
to accept it was the wrong axis
worldtree-b188-b189-and-selene three arcs closed, and a #411 diagnosis I got
wrong twice before a directory probe settled it
Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.
The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.
Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.
Index stays at 291 lines, under the soft cap. No archival this run.
This commit is contained in:
@@ -0,0 +1,92 @@
|
||||
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
|
||||
|
||||
`[2026-08-25]`
|
||||
|
||||
## The run
|
||||
|
||||
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
|
||||
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
|
||||
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
|
||||
|
||||
**47% faster than the round-1 projection of 13.85h**, from two changes: the
|
||||
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
|
||||
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
|
||||
*entirely padding*, and removing padding removed the variance rather than just
|
||||
the mean.
|
||||
|
||||
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
|
||||
rolling average.** The first two were instantaneous tqdm readings off a number
|
||||
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
|
||||
measure-don't-sample discipline I wrote into the throughput playbook, violated on
|
||||
the one metric I kept reporting.
|
||||
|
||||
## lora_B gate — PASSED, twice
|
||||
|
||||
checkpoint-100 205/205 non-zero, median norm 0.829
|
||||
final adapter 205/205 non-zero, median norm 1.708
|
||||
vision_tower tensors: 0 on both
|
||||
|
||||
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
|
||||
rather than saturating early. This check **never ran in round 1** (died at step
|
||||
19, first checkpoint was 100) and it is the only failure mode that stays
|
||||
invisible until the acceptance gate reports base-identical numbers.
|
||||
|
||||
## The gate — brokkr-smithy-dev
|
||||
|
||||
**It did the thing it was built to do:**
|
||||
|
||||
metric base A/B tuned delta floor
|
||||
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
|
||||
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
|
||||
|
||||
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
|
||||
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
|
||||
|
||||
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
|
||||
memorisation: 0.0000 on BOTH arms, all three corpora
|
||||
|
||||
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
|
||||
than argument.**
|
||||
|
||||
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
|
||||
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
|
||||
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
|
||||
the number carries an asterisk.
|
||||
|
||||
## The noise-floor near-miss — the methodology lesson
|
||||
|
||||
brokkr was one step from reporting a 13-point T6 regression **that sat inside
|
||||
twice his instrument's own variance.**
|
||||
|
||||
--per-type 32 max swing across tasks: 9 points
|
||||
--per-type 128 max swing across tasks: 1 point
|
||||
|
||||
His gate criterion is "no task regresses by more than one item" = 3.1 points at
|
||||
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
|
||||
preregistered gate at 4x finer resolution than it could resolve, and caught it by
|
||||
running a control he did not strictly need. Quadrupling n collapsed the noise
|
||||
exactly as binomial statistics predicts.
|
||||
|
||||
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
|
||||
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
|
||||
default of 256. Continuous batching changes reduction order and borderline items
|
||||
flip. Temperature 0 buys deterministic *sampling*, not deterministic
|
||||
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
|
||||
reason: a floor measured on a seat serving one request at a time is not the floor
|
||||
that applies to the seat we ship.
|
||||
|
||||
## The confound I built and he caught
|
||||
|
||||
I optimised a pipeline for production and then handed him its output as an eval
|
||||
instrument **without asking whether those were the same job.** The tuned arm
|
||||
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
|
||||
regression would have been tuning-damage OR quantization-damage with no way to
|
||||
separate them, and the gate's whole question is "did the tune cost us
|
||||
capability."
|
||||
|
||||
**Both arms now bf16, same seat, same port, argv differing in exactly two
|
||||
lines** (weights path, served name), template sha256 identical
|
||||
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
|
||||
three launches). Quantization moved *downstream* of the gate.
|
||||
|
||||
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
|
||||
Reference in New Issue
Block a user