memory: snapshot — run 2 trained, merged, coherence-gated and serving as erp-tune-v2

Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).

Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.

Records three things a future session would otherwise get wrong:

  - the mask is proven by the loss-token delta, NOT by the matching p50 step
    times -- step time is insensitive to which positions carry loss, so that
    check cannot go red on the axis I originally cited it for
  - two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
    any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
    resourcing call, not a scheduling one
  - erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
    fix needs a config edit plus a reload that interrupts fleet traffic, so it
    is batched for morning rather than done at 2am
This commit is contained in:
vh
2026-08-26 01:37:53 -07:00
parent d54f25605f
commit 62f01a02da
2 changed files with 130 additions and 12 deletions
@@ -0,0 +1,114 @@
# `[2026-08-26]` ERP run 2 — complete, merged, coherence-gated, and serving as `erp-tune-v2`
Run 2 is the SFT on the **official instruct** base. It trained 1312/1312 in
**7:22:44**, `train_loss` **2.839**, and is serving bf16 on ana-ml2 GPU0.
endpoint http://10.250.50.54:8098/v1 model erp-tune-v2
container erp-eval-v2 (restart unless-stopped, 84,272 MiB)
weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB)
adapter /tank/erp-tune/run-02/adapter/
harness eitri-smithy 1909d86 at launch (see the provenance caveat below)
## The two operator decisions that produced it
Both taken 2026-08-25 evening, both my recommendations:
1. **Base = stock `google/gemma-4-26B-A4B-it`**, not an abliteration. My prior
session had ranked this option 3 of 4 on attributability grounds; that
objection did not survive contact, because the mask and the hygiene fixes
land under *every* option and therefore do not discriminate between bases.
The real trade is scope: **run 2 stops being the deliverable and becomes
stage 1 of 2**, because a stock-instruct SFT refuses at near-stock rates and
needs a targeted DPO stage after it. The operator accepted that to replace
"inherit a stranger's judgement about which refusals to strip" with "strip
the ones we choose."
2. **Stop `erp-eval` to free GPU0.** There was no free card — GPU0 held run 1's
seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat
had served **one request in 24 hours** and its merged weights are on disk, so
it re-serves in minutes.
## Gates — all passed
lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573
checkpoint-100 min was 0.1977 -> growth, not a plateau
vision_tower 0 tensors
merge 410 adapter tensors applied; 1/1 sampled target changed
chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge
coherence 5/5 prompts, all distinct, no looping, no marker leakage
step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75)
## ⚠ What the p50 match does and does NOT show
Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the
impersonation mask applied. **Masking changes which positions contribute to the
loss reduction — not tensor shapes, not either pass.** Step time is therefore
insensitive to whether the mask applied, so a silent fallback to unmasked
encodings produces the same p50. The check cannot go red on that axis.
(brokkr-smithy-dev caught this after I had asserted it; withdrawn.)
**The evidence the mask applied is the loss-token delta**: 45,899,397 →
45,677,685, −221,712, against **byte-identical context tokens**, unchanged
record count, and the same nine unfittable drops. It could have returned zero.
Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently
derived 1.142% of bot words — two measurements from opposite ends of the
pipeline agreeing within 4%.
What the p50 match *does* show: step timing was stable across a base swap.
## ⚠ `provenance.json` records `harness_commit 460f372` — AHEAD of the code that ran
The run launched when HEAD was `1909d86`. Three commits landed on the same
checkout during the seven hours it trained, and `_git_commit()` was called at
save time, so it read HEAD seven hours late. The recorded commit names
`step_seconds`, the resolved-backend fields and the worker template assertion —
**none of which were in the running process.**
Exact mirror of run 1, whose commit was BEHIND its code because the tree was
dirty. Same defect, opposite sign. Fixed from run 3 by `9d27b4f`.
Annotated on the artifact at `run-02/adapter/PROVENANCE-NOTE.md`, never edited
into `provenance.json`. Attention backend supplied there as reconstructed
(requested + resolved both `flex_attention`, confirmed by a config-only probe on
the same library versions), with the step-timing distribution in
`step-timing-reconstructed.json`.
## Harness work this run — all local, UNPUSHED
5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34)
1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call
908aadb record the attention backend the run RESOLVED to
a80a5a7 record the step-time distribution beside the resolved backend
460f372 pair chat_template_sha256 with the sha of what actually RENDERED
9d27b4f sample the harness identity at LAUNCH, not at save time
242 tests passing, up from a suite that was **10 passed / 4 failed with no
pytest installed at all** — nothing had ever run it.
The through-line, and it is the reason for playbook §4: **three of these six
commits fix a check that could not fail.** The encode cache key omitted the
mask; `attn_implementation_resolved` cannot detect a Dynamo fallback;
`chat_template_sha256` records a file rather than what rendered. Each was found
*after* shipping the previous one, and 4.5's inert-gate reading is what makes
them one defect rather than three.
## Open, with surfaces
- **Base arm cannot be concurrent.** Two bf16 26B arms = 98 GB of weights on a
97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both
arms in one window because the diversity battery mines its frozen marker list
from base. Either sequential-with-everything-pinned, or displace GPU1 seats —
an operator call. Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
- **`erp-tune-v1` still registered in the LiteLLM gateway, returning HTTP 500.**
I stopped its container and left the route. Config-file-defined
(`db_model: false`), so removal needs an edit plus a gateway reload, and a
reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the
worse trade at 1 request/24h. Batch it with the v2 registration decision.
- **`erp-tune-v2` deliberately NOT in the gateway.** Served on a direct endpoint.
Adding an un-gated ERP tune to the shared gateway reaches every agent holding
the shared key — broader than "serve it" requires, and trivial once the
battery reads out.
- **DPO stage** — survey at `docs/pfi/erp-dpo-stage-prep.md`. Gating item is not
technical: **which refusal axes are in scope and which are explicitly kept.**
No preference data for refusal axes exists; the two Gutenberg DPO sets on disk
are prose-quality. `trl` is not installed.