vh
62f01a02da
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).
Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.
Records three things a future session would otherwise get wrong:
- the mask is proven by the loss-token delta, NOT by the matching p50 step
times -- step time is insensitive to which positions carry loss, so that
check cannot go red on the axis I originally cited it for
- two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
resourcing call, not a scheduling one
- erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
fix needs a config edit plus a reload that interrupts fleet traffic, so it
is batched for morning rather than done at 2am
115 lines
6.3 KiB
Markdown
115 lines
6.3 KiB
Markdown
# `[2026-08-26]` ERP run 2 — complete, merged, coherence-gated, and serving as `erp-tune-v2`
|
||
|
||
Run 2 is the SFT on the **official instruct** base. It trained 1312/1312 in
|
||
**7:22:44**, `train_loss` **2.839**, and is serving bf16 on ana-ml2 GPU0.
|
||
|
||
endpoint http://10.250.50.54:8098/v1 model erp-tune-v2
|
||
container erp-eval-v2 (restart unless-stopped, 84,272 MiB)
|
||
weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB)
|
||
adapter /tank/erp-tune/run-02/adapter/
|
||
harness eitri-smithy 1909d86 at launch (see the provenance caveat below)
|
||
|
||
## The two operator decisions that produced it
|
||
|
||
Both taken 2026-08-25 evening, both my recommendations:
|
||
|
||
1. **Base = stock `google/gemma-4-26B-A4B-it`**, not an abliteration. My prior
|
||
session had ranked this option 3 of 4 on attributability grounds; that
|
||
objection did not survive contact, because the mask and the hygiene fixes
|
||
land under *every* option and therefore do not discriminate between bases.
|
||
The real trade is scope: **run 2 stops being the deliverable and becomes
|
||
stage 1 of 2**, because a stock-instruct SFT refuses at near-stock rates and
|
||
needs a targeted DPO stage after it. The operator accepted that to replace
|
||
"inherit a stranger's judgement about which refusals to strip" with "strip
|
||
the ones we choose."
|
||
2. **Stop `erp-eval` to free GPU0.** There was no free card — GPU0 held run 1's
|
||
seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat
|
||
had served **one request in 24 hours** and its merged weights are on disk, so
|
||
it re-serves in minutes.
|
||
|
||
## Gates — all passed
|
||
|
||
lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573
|
||
checkpoint-100 min was 0.1977 -> growth, not a plateau
|
||
vision_tower 0 tensors
|
||
merge 410 adapter tensors applied; 1/1 sampled target changed
|
||
chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge
|
||
coherence 5/5 prompts, all distinct, no looping, no marker leakage
|
||
step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75)
|
||
|
||
## ⚠ What the p50 match does and does NOT show
|
||
|
||
Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the
|
||
impersonation mask applied. **Masking changes which positions contribute to the
|
||
loss reduction — not tensor shapes, not either pass.** Step time is therefore
|
||
insensitive to whether the mask applied, so a silent fallback to unmasked
|
||
encodings produces the same p50. The check cannot go red on that axis.
|
||
(brokkr-smithy-dev caught this after I had asserted it; withdrawn.)
|
||
|
||
**The evidence the mask applied is the loss-token delta**: 45,899,397 →
|
||
45,677,685, −221,712, against **byte-identical context tokens**, unchanged
|
||
record count, and the same nine unfittable drops. It could have returned zero.
|
||
|
||
Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently
|
||
derived 1.142% of bot words — two measurements from opposite ends of the
|
||
pipeline agreeing within 4%.
|
||
|
||
What the p50 match *does* show: step timing was stable across a base swap.
|
||
|
||
## ⚠ `provenance.json` records `harness_commit 460f372` — AHEAD of the code that ran
|
||
|
||
The run launched when HEAD was `1909d86`. Three commits landed on the same
|
||
checkout during the seven hours it trained, and `_git_commit()` was called at
|
||
save time, so it read HEAD seven hours late. The recorded commit names
|
||
`step_seconds`, the resolved-backend fields and the worker template assertion —
|
||
**none of which were in the running process.**
|
||
|
||
Exact mirror of run 1, whose commit was BEHIND its code because the tree was
|
||
dirty. Same defect, opposite sign. Fixed from run 3 by `9d27b4f`.
|
||
|
||
Annotated on the artifact at `run-02/adapter/PROVENANCE-NOTE.md`, never edited
|
||
into `provenance.json`. Attention backend supplied there as reconstructed
|
||
(requested + resolved both `flex_attention`, confirmed by a config-only probe on
|
||
the same library versions), with the step-timing distribution in
|
||
`step-timing-reconstructed.json`.
|
||
|
||
## Harness work this run — all local, UNPUSHED
|
||
|
||
5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34)
|
||
1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call
|
||
908aadb record the attention backend the run RESOLVED to
|
||
a80a5a7 record the step-time distribution beside the resolved backend
|
||
460f372 pair chat_template_sha256 with the sha of what actually RENDERED
|
||
9d27b4f sample the harness identity at LAUNCH, not at save time
|
||
|
||
242 tests passing, up from a suite that was **10 passed / 4 failed with no
|
||
pytest installed at all** — nothing had ever run it.
|
||
|
||
The through-line, and it is the reason for playbook §4: **three of these six
|
||
commits fix a check that could not fail.** The encode cache key omitted the
|
||
mask; `attn_implementation_resolved` cannot detect a Dynamo fallback;
|
||
`chat_template_sha256` records a file rather than what rendered. Each was found
|
||
*after* shipping the previous one, and 4.5's inert-gate reading is what makes
|
||
them one defect rather than three.
|
||
|
||
## Open, with surfaces
|
||
|
||
- **Base arm cannot be concurrent.** Two bf16 26B arms = 98 GB of weights on a
|
||
97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both
|
||
arms in one window because the diversity battery mines its frozen marker list
|
||
from base. Either sequential-with-everything-pinned, or displace GPU1 seats —
|
||
an operator call. Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
|
||
- **`erp-tune-v1` still registered in the LiteLLM gateway, returning HTTP 500.**
|
||
I stopped its container and left the route. Config-file-defined
|
||
(`db_model: false`), so removal needs an edit plus a gateway reload, and a
|
||
reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the
|
||
worse trade at 1 request/24h. Batch it with the v2 registration decision.
|
||
- **`erp-tune-v2` deliberately NOT in the gateway.** Served on a direct endpoint.
|
||
Adding an un-gated ERP tune to the shared gateway reaches every agent holding
|
||
the shared key — broader than "serve it" requires, and trivial once the
|
||
battery reads out.
|
||
- **DPO stage** — survey at `docs/pfi/erp-dpo-stage-prep.md`. Gating item is not
|
||
technical: **which refusal axes are in scope and which are explicitly kept.**
|
||
No preference data for refusal axes exists; the two Gutenberg DPO sets on disk
|
||
are prose-quality. `trl` is not installed.
|