vh
62f01a02da
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).
Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.
Records three things a future session would otherwise get wrong:
- the mask is proven by the loss-token delta, NOT by the matching p50 step
times -- step time is insensitive to which positions carry loss, so that
check cannot go red on the axis I originally cited it for
- two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
resourcing call, not a scheduling one
- erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
fix needs a config edit plus a reload that interrupts fleet traffic, so it
is batched for morning rather than done at 2am
6.3 KiB
[2026-08-26] ERP run 2 — complete, merged, coherence-gated, and serving as erp-tune-v2
Run 2 is the SFT on the official instruct base. It trained 1312/1312 in
7:22:44, train_loss 2.839, and is serving bf16 on ana-ml2 GPU0.
endpoint http://10.250.50.54:8098/v1 model erp-tune-v2
container erp-eval-v2 (restart unless-stopped, 84,272 MiB)
weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB)
adapter /tank/erp-tune/run-02/adapter/
harness eitri-smithy 1909d86 at launch (see the provenance caveat below)
The two operator decisions that produced it
Both taken 2026-08-25 evening, both my recommendations:
- Base = stock
google/gemma-4-26B-A4B-it, not an abliteration. My prior session had ranked this option 3 of 4 on attributability grounds; that objection did not survive contact, because the mask and the hygiene fixes land under every option and therefore do not discriminate between bases. The real trade is scope: run 2 stops being the deliverable and becomes stage 1 of 2, because a stock-instruct SFT refuses at near-stock rates and needs a targeted DPO stage after it. The operator accepted that to replace "inherit a stranger's judgement about which refusals to strip" with "strip the ones we choose." - Stop
erp-evalto free GPU0. There was no free card — GPU0 held run 1's seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat had served one request in 24 hours and its merged weights are on disk, so it re-serves in minutes.
Gates — all passed
lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573
checkpoint-100 min was 0.1977 -> growth, not a plateau
vision_tower 0 tensors
merge 410 adapter tensors applied; 1/1 sampled target changed
chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge
coherence 5/5 prompts, all distinct, no looping, no marker leakage
step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75)
⚠ What the p50 match does and does NOT show
Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the impersonation mask applied. Masking changes which positions contribute to the loss reduction — not tensor shapes, not either pass. Step time is therefore insensitive to whether the mask applied, so a silent fallback to unmasked encodings produces the same p50. The check cannot go red on that axis. (brokkr-smithy-dev caught this after I had asserted it; withdrawn.)
The evidence the mask applied is the loss-token delta: 45,899,397 → 45,677,685, −221,712, against byte-identical context tokens, unchanged record count, and the same nine unfittable drops. It could have returned zero.
Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently derived 1.142% of bot words — two measurements from opposite ends of the pipeline agreeing within 4%.
What the p50 match does show: step timing was stable across a base swap.
⚠ provenance.json records harness_commit 460f372 — AHEAD of the code that ran
The run launched when HEAD was 1909d86. Three commits landed on the same
checkout during the seven hours it trained, and _git_commit() was called at
save time, so it read HEAD seven hours late. The recorded commit names
step_seconds, the resolved-backend fields and the worker template assertion —
none of which were in the running process.
Exact mirror of run 1, whose commit was BEHIND its code because the tree was
dirty. Same defect, opposite sign. Fixed from run 3 by 9d27b4f.
Annotated on the artifact at run-02/adapter/PROVENANCE-NOTE.md, never edited
into provenance.json. Attention backend supplied there as reconstructed
(requested + resolved both flex_attention, confirmed by a config-only probe on
the same library versions), with the step-timing distribution in
step-timing-reconstructed.json.
Harness work this run — all local, UNPUSHED
5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34)
1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call
908aadb record the attention backend the run RESOLVED to
a80a5a7 record the step-time distribution beside the resolved backend
460f372 pair chat_template_sha256 with the sha of what actually RENDERED
9d27b4f sample the harness identity at LAUNCH, not at save time
242 tests passing, up from a suite that was 10 passed / 4 failed with no pytest installed at all — nothing had ever run it.
The through-line, and it is the reason for playbook §4: three of these six
commits fix a check that could not fail. The encode cache key omitted the
mask; attn_implementation_resolved cannot detect a Dynamo fallback;
chat_template_sha256 records a file rather than what rendered. Each was found
after shipping the previous one, and 4.5's inert-gate reading is what makes
them one defect rather than three.
Open, with surfaces
- Base arm cannot be concurrent. Two bf16 26B arms = 98 GB of weights on a
97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both
arms in one window because the diversity battery mines its frozen marker list
from base. Either sequential-with-everything-pinned, or displace GPU1 seats —
an operator call. Thread
01M0WQ8W5574KMEVCHCEKEXNS5. erp-tune-v1still registered in the LiteLLM gateway, returning HTTP 500. I stopped its container and left the route. Config-file-defined (db_model: false), so removal needs an edit plus a gateway reload, and a reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the worse trade at 1 request/24h. Batch it with the v2 registration decision.erp-tune-v2deliberately NOT in the gateway. Served on a direct endpoint. Adding an un-gated ERP tune to the shared gateway reaches every agent holding the shared key — broader than "serve it" requires, and trivial once the battery reads out.- DPO stage — survey at
docs/pfi/erp-dpo-stage-prep.md. Gating item is not technical: which refusal axes are in scope and which are explicitly kept. No preference data for refusal axes exists; the two Gutenberg DPO sets on disk are prose-quality.trlis not installed.