memory: snapshot — run 2 trained, merged, coherence-gated and serving as erp-tune-v2
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).
Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.
Records three things a future session would otherwise get wrong:
- the mask is proven by the loss-token delta, NOT by the matching p50 step
times -- step time is insensitive to which positions carry loss, so that
check cannot go red on the axis I originally cited it for
- two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
resourcing call, not a scheduling one
- erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
fix needs a config edit plus a reload that interrupts fleet traffic, so it
is batched for morning rather than done at 2am
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# `[2026-08-26]` ERP run 2 — complete, merged, coherence-gated, and serving as `erp-tune-v2`
|
||||
|
||||
Run 2 is the SFT on the **official instruct** base. It trained 1312/1312 in
|
||||
**7:22:44**, `train_loss` **2.839**, and is serving bf16 on ana-ml2 GPU0.
|
||||
|
||||
endpoint http://10.250.50.54:8098/v1 model erp-tune-v2
|
||||
container erp-eval-v2 (restart unless-stopped, 84,272 MiB)
|
||||
weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB)
|
||||
adapter /tank/erp-tune/run-02/adapter/
|
||||
harness eitri-smithy 1909d86 at launch (see the provenance caveat below)
|
||||
|
||||
## The two operator decisions that produced it
|
||||
|
||||
Both taken 2026-08-25 evening, both my recommendations:
|
||||
|
||||
1. **Base = stock `google/gemma-4-26B-A4B-it`**, not an abliteration. My prior
|
||||
session had ranked this option 3 of 4 on attributability grounds; that
|
||||
objection did not survive contact, because the mask and the hygiene fixes
|
||||
land under *every* option and therefore do not discriminate between bases.
|
||||
The real trade is scope: **run 2 stops being the deliverable and becomes
|
||||
stage 1 of 2**, because a stock-instruct SFT refuses at near-stock rates and
|
||||
needs a targeted DPO stage after it. The operator accepted that to replace
|
||||
"inherit a stranger's judgement about which refusals to strip" with "strip
|
||||
the ones we choose."
|
||||
2. **Stop `erp-eval` to free GPU0.** There was no free card — GPU0 held run 1's
|
||||
seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat
|
||||
had served **one request in 24 hours** and its merged weights are on disk, so
|
||||
it re-serves in minutes.
|
||||
|
||||
## Gates — all passed
|
||||
|
||||
lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573
|
||||
checkpoint-100 min was 0.1977 -> growth, not a plateau
|
||||
vision_tower 0 tensors
|
||||
merge 410 adapter tensors applied; 1/1 sampled target changed
|
||||
chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge
|
||||
coherence 5/5 prompts, all distinct, no looping, no marker leakage
|
||||
step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75)
|
||||
|
||||
## ⚠ What the p50 match does and does NOT show
|
||||
|
||||
Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the
|
||||
impersonation mask applied. **Masking changes which positions contribute to the
|
||||
loss reduction — not tensor shapes, not either pass.** Step time is therefore
|
||||
insensitive to whether the mask applied, so a silent fallback to unmasked
|
||||
encodings produces the same p50. The check cannot go red on that axis.
|
||||
(brokkr-smithy-dev caught this after I had asserted it; withdrawn.)
|
||||
|
||||
**The evidence the mask applied is the loss-token delta**: 45,899,397 →
|
||||
45,677,685, −221,712, against **byte-identical context tokens**, unchanged
|
||||
record count, and the same nine unfittable drops. It could have returned zero.
|
||||
|
||||
Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently
|
||||
derived 1.142% of bot words — two measurements from opposite ends of the
|
||||
pipeline agreeing within 4%.
|
||||
|
||||
What the p50 match *does* show: step timing was stable across a base swap.
|
||||
|
||||
## ⚠ `provenance.json` records `harness_commit 460f372` — AHEAD of the code that ran
|
||||
|
||||
The run launched when HEAD was `1909d86`. Three commits landed on the same
|
||||
checkout during the seven hours it trained, and `_git_commit()` was called at
|
||||
save time, so it read HEAD seven hours late. The recorded commit names
|
||||
`step_seconds`, the resolved-backend fields and the worker template assertion —
|
||||
**none of which were in the running process.**
|
||||
|
||||
Exact mirror of run 1, whose commit was BEHIND its code because the tree was
|
||||
dirty. Same defect, opposite sign. Fixed from run 3 by `9d27b4f`.
|
||||
|
||||
Annotated on the artifact at `run-02/adapter/PROVENANCE-NOTE.md`, never edited
|
||||
into `provenance.json`. Attention backend supplied there as reconstructed
|
||||
(requested + resolved both `flex_attention`, confirmed by a config-only probe on
|
||||
the same library versions), with the step-timing distribution in
|
||||
`step-timing-reconstructed.json`.
|
||||
|
||||
## Harness work this run — all local, UNPUSHED
|
||||
|
||||
5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34)
|
||||
1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call
|
||||
908aadb record the attention backend the run RESOLVED to
|
||||
a80a5a7 record the step-time distribution beside the resolved backend
|
||||
460f372 pair chat_template_sha256 with the sha of what actually RENDERED
|
||||
9d27b4f sample the harness identity at LAUNCH, not at save time
|
||||
|
||||
242 tests passing, up from a suite that was **10 passed / 4 failed with no
|
||||
pytest installed at all** — nothing had ever run it.
|
||||
|
||||
The through-line, and it is the reason for playbook §4: **three of these six
|
||||
commits fix a check that could not fail.** The encode cache key omitted the
|
||||
mask; `attn_implementation_resolved` cannot detect a Dynamo fallback;
|
||||
`chat_template_sha256` records a file rather than what rendered. Each was found
|
||||
*after* shipping the previous one, and 4.5's inert-gate reading is what makes
|
||||
them one defect rather than three.
|
||||
|
||||
## Open, with surfaces
|
||||
|
||||
- **Base arm cannot be concurrent.** Two bf16 26B arms = 98 GB of weights on a
|
||||
97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both
|
||||
arms in one window because the diversity battery mines its frozen marker list
|
||||
from base. Either sequential-with-everything-pinned, or displace GPU1 seats —
|
||||
an operator call. Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
|
||||
- **`erp-tune-v1` still registered in the LiteLLM gateway, returning HTTP 500.**
|
||||
I stopped its container and left the route. Config-file-defined
|
||||
(`db_model: false`), so removal needs an edit plus a gateway reload, and a
|
||||
reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the
|
||||
worse trade at 1 request/24h. Batch it with the v2 registration decision.
|
||||
- **`erp-tune-v2` deliberately NOT in the gateway.** Served on a direct endpoint.
|
||||
Adding an un-gated ERP tune to the shared gateway reaches every agent holding
|
||||
the shared key — broader than "serve it" requires, and trivial once the
|
||||
battery reads out.
|
||||
- **DPO stage** — survey at `docs/pfi/erp-dpo-stage-prep.md`. Gating item is not
|
||||
technical: **which refusal axes are in scope and which are explicitly kept.**
|
||||
No preference data for refusal axes exists; the two Gutenberg DPO sets on disk
|
||||
are prose-quality. `trl` is not installed.
|
||||
Reference in New Issue
Block a user