diff --git a/persistent-memory.md b/persistent-memory.md index 57eaba4..f651a1f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -112,7 +112,8 @@ _As of 2026-08-26 ~02:10 PDT — **run 2 is trained, merged, coherence-gated and - **🟢 RUN 2 COMPLETE AND SERVING as `erp-tune-v2`.** 1312/1312 in **7:22:44**, `train_loss` **2.839**, on the **stock `google/gemma-4-26B-A4B-it`** base (operator's call, 08-25). Endpoint `http://10.250.50.54:8098/v1`, container `erp-eval-v2` on GPU0 (84,272 MiB, `restart unless-stopped`), weights `/tank/erp-tune/serve/merged-run02` (bf16 merged, 51.6 GB), adapter `/tank/erp-tune/run-02/adapter/`. Gates all passed: `lora_B` **205/205** (min 0.6826), vision_tower 0, merge verified, upstream template shipped, **coherence 5/5**. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md` - **⚠ THE MASK IS PROVEN BY THE LOSS-TOKEN DELTA, NOT BY THE p50 MATCH.** āˆ’221,712 loss tokens against **byte-identical context tokens**, unchanged records, same nine drops; cross-checks to brokkr's independent corpus-side figure within 4%. The p50 match (19.79 vs run 1's 19.75) is evidence about *timing stability across a base swap* and **cannot** evidence masking — step time is insensitive to which positions carry loss. Do not re-assert it; I did, and brokkr was right. -- **šŸ”„ BATTERY IS RUNNING — MID-SWAP, BASE ARM LIVE.** brokkr **withdrew** his both-arms-concurrent requirement (his diversity battery emits the frozen marker list to a *file*, so the arms were never a live dependency; the real requirement is all of ONE arm's passes on ONE instance before the swap). So: **sequential, no fleet seats displaced, operator not woken.** Live now: `erp-eval-base` on `:8099` (`erp-base-stock`, stock bf16 via symlinks into `/tank/aimodels`). **`erp-eval-v2` is DOWN and its container REMOVED**; weights are on disk so it is a cheap restart. Swap back with `/tank/erp-tune/serve-arm.sh tuned` when he reports the base set complete. ⚠ **Both arms go through `serve-arm.sh`**, which pins the vLLM image **by digest** (`sha256:251eba5cc7c1…`) — a version change between arms is a base swap nobody would see — and mounts `/tank/aimodels` for BOTH arms even though only base needs it, because a mount that differs between arms is a difference between arms. +- **šŸ”„ BATTERY IS RUNNING — TUNED ARM LIVE, BASE SET COMPLETE.** Base arm ran 08:54:52Z→09:03:43Z, **zero errors/truncations/empties/degenerates across 1,728 generations**. Swapped to `erp-tune-v2` on `:8098`; **image digest `sha256:4091d5593f77…` VERIFIED IDENTICAL across both arms** (brokkr's requirement — a differing digest voids the run). brokkr is now running the tuned set with `--freeze-from` the base's marker list. **The floors the tuned deltas must clear:** reasoning core **0.5 pt**, diversity overall **0.0125** (rp 0.0153, story 0.0096), story attractor **0.0000**, memorisation signal 0.00%. ⚠ **The rp family froze ZERO markers**, so its attractor metric is structurally 0.0 on both arms and cannot discriminate — rp is measured on the distance axis only; do not quote an rp attractor delta. ⚠ Base's slop attractor: **"Elias" in 92/96 bare-prompt stories** (published figure on the same family is 102/144 — ours is an independent replication at a higher rate, not a novel finding). +- **ā³ SUPERSEDED — the mid-swap state.** brokkr **withdrew** his both-arms-concurrent requirement (his diversity battery emits the frozen marker list to a *file*, so the arms were never a live dependency; the real requirement is all of ONE arm's passes on ONE instance before the swap). So: **sequential, no fleet seats displaced, operator not woken.** Live now: `erp-eval-base` on `:8099` (`erp-base-stock`, stock bf16 via symlinks into `/tank/aimodels`). **`erp-eval-v2` is DOWN and its container REMOVED**; weights are on disk so it is a cheap restart. Swap back with `/tank/erp-tune/serve-arm.sh tuned` when he reports the base set complete. ⚠ **Both arms go through `serve-arm.sh`**, which pins the vLLM image **by digest** (`sha256:251eba5cc7c1…`) — a version change between arms is a base swap nobody would see — and mounts `/tank/aimodels` for BOTH arms even though only base needs it, because a mount that differs between arms is a difference between arms. - **ā³ SUPERSEDED — the GPU arithmetic that forced sequential.** He needs **both arms reachable in one window** (the diversity battery mines its frozen marker list from base). **Two bf16 26B arms = 98 GB of weights on a 97.9 GB card**; GPU1 has ~30 GB free under six shared seats. Reduced `gpu-memory-utilization` does not help — weights are the floor. Options put to him: sequential-with-every-nuisance-variable-pinned, or displace GPU1 seats (**operator call, affects other consumers**). Thread `01M0WQ8W5574KMEVCHCEKEXNS5`. - **šŸ”“ `erp-tune-v1` IS STILL IN THE LITELLM GATEWAY AND RETURNS HTTP 500.** I stopped its container to free GPU0 and left the route dangling. It is **config-file-defined (`db_model: false`)**, so removal needs a config edit **plus a gateway reload**, and a reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the worse trade at 1 request/24h. **Batch the removal with the v2-registration decision.** - **ā³ `erp-tune-v2` is DELIBERATELY NOT in the gateway.** Direct endpoint only. Registering an un-gated ERP tune makes it reachable by every agent holding the shared key — broader than "serve it" requires. Trivial to add once the battery reads out; operator's call.