From 62f01a02da5f3180e822525e696e851e043364ee Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 26 Aug 2026 01:37:53 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20run=202=20trai?= =?UTF-8?q?ned,=20merged,=20coherence-gated=20and=20serving=20as=20erp-tun?= =?UTF-8?q?e-v2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and the base decision the previous snapshot recorded as OPEN is resolved (stock instruct, operator 2026-08-25). Four new decisions, and the detail file carries the arc: the two operator calls that produced run 2, all five gates, the harness commit chain, and the caveat that its own provenance names a commit AHEAD of the code that ran. Records three things a future session would otherwise get wrong: - the mask is proven by the loss-token delta, NOT by the matching p50 step times -- step time is insensitive to which positions carry loss, so that check cannot go red on the axis I originally cited it for - two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU resourcing call, not a scheduling one - erp-tune-v1 is still registered in the gateway and returns HTTP 500; the fix needs a config edit plus a reload that interrupts fleet traffic, so it is batched for morning rather than done at 2am --- ...2026-08-26-erp-run2-complete-and-served.md | 114 ++++++++++++++++++ persistent-memory.md | 28 +++-- 2 files changed, 130 insertions(+), 12 deletions(-) create mode 100644 persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md diff --git a/persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md b/persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md new file mode 100644 index 0000000..e8e6dc6 --- /dev/null +++ b/persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md @@ -0,0 +1,114 @@ +# `[2026-08-26]` ERP run 2 — complete, merged, coherence-gated, and serving as `erp-tune-v2` + +Run 2 is the SFT on the **official instruct** base. It trained 1312/1312 in +**7:22:44**, `train_loss` **2.839**, and is serving bf16 on ana-ml2 GPU0. + + endpoint http://10.250.50.54:8098/v1 model erp-tune-v2 + container erp-eval-v2 (restart unless-stopped, 84,272 MiB) + weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB) + adapter /tank/erp-tune/run-02/adapter/ + harness eitri-smithy 1909d86 at launch (see the provenance caveat below) + +## The two operator decisions that produced it + +Both taken 2026-08-25 evening, both my recommendations: + +1. **Base = stock `google/gemma-4-26B-A4B-it`**, not an abliteration. My prior + session had ranked this option 3 of 4 on attributability grounds; that + objection did not survive contact, because the mask and the hygiene fixes + land under *every* option and therefore do not discriminate between bases. + The real trade is scope: **run 2 stops being the deliverable and becomes + stage 1 of 2**, because a stock-instruct SFT refuses at near-stock rates and + needs a targeted DPO stage after it. The operator accepted that to replace + "inherit a stranger's judgement about which refusals to strip" with "strip + the ones we choose." +2. **Stop `erp-eval` to free GPU0.** There was no free card — GPU0 held run 1's + seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat + had served **one request in 24 hours** and its merged weights are on disk, so + it re-serves in minutes. + +## Gates — all passed + + lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573 + checkpoint-100 min was 0.1977 -> growth, not a plateau + vision_tower 0 tensors + merge 410 adapter tensors applied; 1/1 sampled target changed + chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge + coherence 5/5 prompts, all distinct, no looping, no marker leakage + step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75) + +## ⚠ What the p50 match does and does NOT show + +Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the +impersonation mask applied. **Masking changes which positions contribute to the +loss reduction — not tensor shapes, not either pass.** Step time is therefore +insensitive to whether the mask applied, so a silent fallback to unmasked +encodings produces the same p50. The check cannot go red on that axis. +(brokkr-smithy-dev caught this after I had asserted it; withdrawn.) + +**The evidence the mask applied is the loss-token delta**: 45,899,397 → +45,677,685, −221,712, against **byte-identical context tokens**, unchanged +record count, and the same nine unfittable drops. It could have returned zero. + +Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently +derived 1.142% of bot words — two measurements from opposite ends of the +pipeline agreeing within 4%. + +What the p50 match *does* show: step timing was stable across a base swap. + +## ⚠ `provenance.json` records `harness_commit 460f372` — AHEAD of the code that ran + +The run launched when HEAD was `1909d86`. Three commits landed on the same +checkout during the seven hours it trained, and `_git_commit()` was called at +save time, so it read HEAD seven hours late. The recorded commit names +`step_seconds`, the resolved-backend fields and the worker template assertion — +**none of which were in the running process.** + +Exact mirror of run 1, whose commit was BEHIND its code because the tree was +dirty. Same defect, opposite sign. Fixed from run 3 by `9d27b4f`. + +Annotated on the artifact at `run-02/adapter/PROVENANCE-NOTE.md`, never edited +into `provenance.json`. Attention backend supplied there as reconstructed +(requested + resolved both `flex_attention`, confirmed by a config-only probe on +the same library versions), with the step-timing distribution in +`step-timing-reconstructed.json`. + +## Harness work this run — all local, UNPUSHED + + 5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34) + 1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call + 908aadb record the attention backend the run RESOLVED to + a80a5a7 record the step-time distribution beside the resolved backend + 460f372 pair chat_template_sha256 with the sha of what actually RENDERED + 9d27b4f sample the harness identity at LAUNCH, not at save time + +242 tests passing, up from a suite that was **10 passed / 4 failed with no +pytest installed at all** — nothing had ever run it. + +The through-line, and it is the reason for playbook §4: **three of these six +commits fix a check that could not fail.** The encode cache key omitted the +mask; `attn_implementation_resolved` cannot detect a Dynamo fallback; +`chat_template_sha256` records a file rather than what rendered. Each was found +*after* shipping the previous one, and 4.5's inert-gate reading is what makes +them one defect rather than three. + +## Open, with surfaces + +- **Base arm cannot be concurrent.** Two bf16 26B arms = 98 GB of weights on a + 97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both + arms in one window because the diversity battery mines its frozen marker list + from base. Either sequential-with-everything-pinned, or displace GPU1 seats — + an operator call. Thread `01M0WQ8W5574KMEVCHCEKEXNS5`. +- **`erp-tune-v1` still registered in the LiteLLM gateway, returning HTTP 500.** + I stopped its container and left the route. Config-file-defined + (`db_model: false`), so removal needs an edit plus a gateway reload, and a + reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the + worse trade at 1 request/24h. Batch it with the v2 registration decision. +- **`erp-tune-v2` deliberately NOT in the gateway.** Served on a direct endpoint. + Adding an un-gated ERP tune to the shared gateway reaches every agent holding + the shared key — broader than "serve it" requires, and trivial once the + battery reads out. +- **DPO stage** — survey at `docs/pfi/erp-dpo-stage-prep.md`. Gating item is not + technical: **which refusal axes are in scope and which are explicitly kept.** + No preference data for refusal axes exists; the two Gutenberg DPO sets on disk + are prose-quality. `trl` is not installed. diff --git a/persistent-memory.md b/persistent-memory.md index aba4afe..a7783a5 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-08-25_ +_Last updated: 2026-08-26_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,22 +108,26 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-08-25 ~21:00Z — **the ERP/RP tune is DONE and SERVING.** The live threads are brokkr's remaining gate work and an operator decision on run 2's base._ +_As of 2026-08-26 ~02:10 PDT — **run 2 is trained, merged, coherence-gated and SERVING.** The live threads are brokkr's behavioural battery, a GPU-resourcing call for the base arm, and two gateway-routing decisions I deliberately left for the morning._ -- **🟢 THE TUNE IS TRAINED, MERGED, AND SERVING.** 1312/1312 in **7:21:52** (47% faster than round 1's 13.85h projection), train_loss 2.793, `lora_B` gate **205/205 non-zero** on the final adapter. Serving bf16 as **`erp-tune-v1`** on ana-ml2 GPU0 `:8097`, routed through LiteLLM at `10.250.50.70:4000`. **Operator wants it left UP.** ⚠ **It is a bare `docker run` named `erp-eval`, NOT a compose stack — it will NOT survive an ana-ml2 reboot.** Promote to `/opt/docker/compose/erp-tune/` before relying on it. Artifacts: adapter `/tank/erp-tune/run-01/adapter/`, merged bf16 `/tank/erp-tune/serve/merged-final` (49 GB), base arm `/tank/erp-tune/serve/base-arm` (symlinked weights + upstream template), NVFP4A16 dry-run build `/tank/erp-tune/serve/nvfp4a16-test` (17 GB). → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md` -- **🟢 GATE RESULT: the tune did the thing it was built to do.** Diversity **+0.178 against a 0.008 floor (22x)**, attractor hit rate **−11.3pt against a 2.0pt floor**, memorisation **0.0000 on both arms** (closes the R20 licensed-prose exposure on measurement). T6 81-82 vs base's recorded 88 is the open regression question. ⚠ Tuned lost **18 of 192 generations** to truncation/degeneracy vs base's 1-2 — lopsided exclusions plausibly flatter the diversity magnitude. -- **⏳ REMAINING GATE WORK (brokkr's, seat is his on request):** the **domain-compliance probe** — the RIGHT refusal axis (explicit fiction, from R19's track-2 map), which my `harmful_behaviors` probe did not measure. Scaffold supplied at `scripts/training-probes/counted_classifier.py`. Swap on his word; the tuned arm goes back up afterwards. -- **🔴 OPERATOR DECISION OPEN — RUN 2's BASE.** brokkr listed `base -> stock google/gemma-4-26B-A4B-it` in a change table alongside `weight_decay`, which framed a reversal of the trainee-selection decision as a knob. Four options, ranked: **(1) TrevorJS lower-damage abliteration** (KL 0.09 vs 0.1237, manually audited) — raises the safety floor AND is the clean single-variable ablation; **(2) hold llmfan46, test the mask alone**; **(3) stock instruct** — two-variable, regression unattributable; **(4) pretrained base** — ⚠ **voids T1-T6 entirely** (the instrument assumes a model that answers questions), ships no chat template, and **removes the last non-lexical floor** on the CSAM axis: stage-2 detector INERT + contamination scan ABSENT are both already overridden, leaving only an 829-record lexical quarantine. Nothing staged. **I have pulled nothing and configured nothing.** -- **⏳ RUN 2 CHANGE LIST, wired on the word** (spec read-back confirmed correct by brokkr): impersonation loss-mask (813 turns staged at `/mnt/smithy/datasets/derived/_recipes/erp-seat-sft-r1/impersonation-mask-v1.jsonl`, **1.142% of bot words**), warmup_ratio 0.1, cosine, weight_decay 0.01, lora_dropout 0, and **baseline ALL instruments BEFORE training** — run 1 had no pre-training baseline on any axis, which is why 08-25 involved so much reconstruction. ⚠ **Mask at the SOURCE-message level BEFORE `_merge_consecutive_assistant`**, mask the ENTIRE merged run if any message in it is flagged, and **assert `content_sha_prefix` before masking** or a shifted index silently masks the wrong turn. ⚠ **Loss-mask, NOT delete** — deleting a bot turn leaves the next human turn answering a message that is not there, which trains discontinuity to remove impersonation. -- **🟢 WORLDTREE b188 + b189 SHIPPED, #404/#406/#407/#409/#410 CLOSED.** matrix.yaml pre-sync deployed to both instances; bridge extracted to `pfi/wt-matrix-bridge` and pinned to `f3f8ec902267`; Synapse alias namespace added on ana-docker (healthy in 40s, aliased create worked first try). → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md` -- **⏳ WORLDTREE RIDERS QUEUED (mine):** #411 orphan cleanup (**17 rooms** created by `@aipa-debug`, unaliased, cosmetic — after the sink is confirmed stable); providers.yaml comment parity; **`:latest` local-tag cleanup on the VMs LAST** (only irreversible item in the set). **The selene metadata fix `a77639d` is committed but NOT deployed** — needs the operator's yes under the managed-box rule. -- **⚠ TWO GAPS I OPENED AND HAVE NOT CLOSED (no tracker):** the **synapse stack has a compose.yaml on ana-docker but is NOT mirrored into `stacks/`** unlike every other stack; and the **trainee checkpoint dir carries no README/model-card/`base_model` field** — its entire lineage survives only in `/tank/aimodels/gemma4-26b-heretic-dl.py` and the adapter's `base_model_revision` free-text field. -- **🟢 SEAT MAP.** GPU0: `erp-tune-v1` (the tune, bf16, :8097) — `gen` and `char-rp` were displaced for the training window and **`char-rp` is still DOWN**. GPU1: `gen` (:8015, restored healthy after a 15-min stop for the quant), Scriberr, rerank/coder/reward/embed. **`sec`/mog-sec still DOWN** from the training window. ⚠ ana-ml2 runs a vLLM **version spread**, not one version. -- **⏳ OPEN ELSEWHERE:** MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift; `/tank` DEGRADED **70+ days**; Lobe retirement; brokkr's MeroMero bake-off (⚠ **three axes differ, not two** — corpus, pipeline, AND operation order: ours is tune(abliterate(stock)), MeroMero is abliterate(tune(stock)) — it is a preference test between shipped artifacts, **not an ablation**). **Commits are local and unpushed** — push is the operator's call. +- **🟢 RUN 2 COMPLETE AND SERVING as `erp-tune-v2`.** 1312/1312 in **7:22:44**, `train_loss` **2.839**, on the **stock `google/gemma-4-26B-A4B-it`** base (operator's call, 08-25). Endpoint `http://10.250.50.54:8098/v1`, container `erp-eval-v2` on GPU0 (84,272 MiB, `restart unless-stopped`), weights `/tank/erp-tune/serve/merged-run02` (bf16 merged, 51.6 GB), adapter `/tank/erp-tune/run-02/adapter/`. Gates all passed: `lora_B` **205/205** (min 0.6826), vision_tower 0, merge verified, upstream template shipped, **coherence 5/5**. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` +- **⚠ THE MASK IS PROVEN BY THE LOSS-TOKEN DELTA, NOT BY THE p50 MATCH.** −221,712 loss tokens against **byte-identical context tokens**, unchanged records, same nine drops; cross-checks to brokkr's independent corpus-side figure within 4%. The p50 match (19.79 vs run 1's 19.75) is evidence about *timing stability across a base swap* and **cannot** evidence masking — step time is insensitive to which positions carry loss. Do not re-assert it; I did, and brokkr was right. +- **⏳ BROKKR'S BATTERY IS NEXT AND THE BASE ARM IS BLOCKED ON GPU.** He needs **both arms reachable in one window** (the diversity battery mines its frozen marker list from base). **Two bf16 26B arms = 98 GB of weights on a 97.9 GB card**; GPU1 has ~30 GB free under six shared seats. Reduced `gpu-memory-utilization` does not help — weights are the floor. Options put to him: sequential-with-every-nuisance-variable-pinned, or displace GPU1 seats (**operator call, affects other consumers**). Thread `01M0WQ8W5574KMEVCHCEKEXNS5`. +- **🔴 `erp-tune-v1` IS STILL IN THE LITELLM GATEWAY AND RETURNS HTTP 500.** I stopped its container to free GPU0 and left the route dangling. It is **config-file-defined (`db_model: false`)**, so removal needs a config edit **plus a gateway reload**, and a reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the worse trade at 1 request/24h. **Batch the removal with the v2-registration decision.** +- **⏳ `erp-tune-v2` is DELIBERATELY NOT in the gateway.** Direct endpoint only. Registering an un-gated ERP tune makes it reachable by every agent holding the shared key — broader than "serve it" requires. Trivial to add once the battery reads out; operator's call. +- **⏳ DPO STAGE — survey at `docs/pfi/erp-dpo-stage-prep.md`.** Run 2 will refuse at near-stock rates **by design**; DPO is where refusals get pruned on chosen axes. ⚠ **The gating item is not technical: WHICH refusal axes are in scope and which are explicitly kept.** Data spec, pair counts, held-out split and success probe are all functions of that list. **No preference data for refusal axes exists** (the two Gutenberg DPO sets on disk are prose-quality); **`trl` is not installed**. Measure the domain-compliance probe against run 2 BEFORE pruning — the pre-number is the only baseline that will ever exist. +- **⚠ HARNESS COMMITS ARE LOCAL AND UNPUSHED** on ana-ml2 at `/tank/erp-tune/eitri-smithy` (`5349ef0` → `9d27b4f`, six commits, 242 tests green). Push is the operator's call. The repo is `pfi/eitri-smithy` and its working tree also carries unrelated untracked dirs I did not touch. +- **🟢 SEAT MAP.** GPU0: `erp-tune-v2` (:8098). GPU1: `gen` (:8015), scriberr, rerank-a3, coder, reward, embed. **`char-rp` and `sec`/mog-sec still DOWN** from the training window. Run 1's `erp-eval` container is **stopped but not removed**; its weights are kept at `/tank/erp-tune/serve/merged-final`. +- **⏳ OPEN ELSEWHERE (unchanged):** Worldtree #411 orphan cleanup + providers.yaml parity + `:latest` tag cleanup (selene fix `a77639d` committed NOT deployed, needs operator yes under the managed-box rule); synapse stack not mirrored into `stacks/`; trainee checkpoint dir has no model card; MTP-k3 isolating experiment; upstream vLLM issue to file; Cold-Fusion NVFP4 quants delete/keep; `/tank` DEGRADED **70+ days**; Lobe retirement; brokkr's MeroMero bake-off (⚠ three axes differ — it is a preference test, not an ablation). - **⚠️ STANDING: NO FLEET NOTIFICATIONS unless the operator asks** (2026-08-24). Direct task correspondence with a counterparty is fine. ## Recent decisions +- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` +- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name). +- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before. +- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path. + - `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md` - `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md` - `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`