memory: base set complete, tuned arm live with digests verified identical
Records the floors the tuned deltas have to clear, since they are the whole
point of the base pass and are not recoverable from anywhere else: reasoning
core 0.5 pt, diversity overall 0.0125, story attractor 0.0000.
Two caveats that would otherwise be misread:
- the rp family froze ZERO markers, so its attractor hit rate is structurally
0.0 on both arms. That reads as a clean result and means the instrument
cannot discriminate on that family; rp is measured on the distance axis
only.
- 'Elias' in 92/96 base stories is an independent replication of a published
102/144 on the same family, at a higher rate -- not a novel finding.
Image digest sha256:4091d5593f77 verified identical across both arms, which was
brokkr's stated void condition.
This commit is contained in:
@@ -112,7 +112,8 @@ _As of 2026-08-26 ~02:10 PDT — **run 2 is trained, merged, coherence-gated and
|
||||
|
||||
- **🟢 RUN 2 COMPLETE AND SERVING as `erp-tune-v2`.** 1312/1312 in **7:22:44**, `train_loss` **2.839**, on the **stock `google/gemma-4-26B-A4B-it`** base (operator's call, 08-25). Endpoint `http://10.250.50.54:8098/v1`, container `erp-eval-v2` on GPU0 (84,272 MiB, `restart unless-stopped`), weights `/tank/erp-tune/serve/merged-run02` (bf16 merged, 51.6 GB), adapter `/tank/erp-tune/run-02/adapter/`. Gates all passed: `lora_B` **205/205** (min 0.6826), vision_tower 0, merge verified, upstream template shipped, **coherence 5/5**. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
|
||||
- **⚠ THE MASK IS PROVEN BY THE LOSS-TOKEN DELTA, NOT BY THE p50 MATCH.** −221,712 loss tokens against **byte-identical context tokens**, unchanged records, same nine drops; cross-checks to brokkr's independent corpus-side figure within 4%. The p50 match (19.79 vs run 1's 19.75) is evidence about *timing stability across a base swap* and **cannot** evidence masking — step time is insensitive to which positions carry loss. Do not re-assert it; I did, and brokkr was right.
|
||||
- **🔄 BATTERY IS RUNNING — MID-SWAP, BASE ARM LIVE.** brokkr **withdrew** his both-arms-concurrent requirement (his diversity battery emits the frozen marker list to a *file*, so the arms were never a live dependency; the real requirement is all of ONE arm's passes on ONE instance before the swap). So: **sequential, no fleet seats displaced, operator not woken.** Live now: `erp-eval-base` on `:8099` (`erp-base-stock`, stock bf16 via symlinks into `/tank/aimodels`). **`erp-eval-v2` is DOWN and its container REMOVED**; weights are on disk so it is a cheap restart. Swap back with `/tank/erp-tune/serve-arm.sh tuned` when he reports the base set complete. ⚠ **Both arms go through `serve-arm.sh`**, which pins the vLLM image **by digest** (`sha256:251eba5cc7c1…`) — a version change between arms is a base swap nobody would see — and mounts `/tank/aimodels` for BOTH arms even though only base needs it, because a mount that differs between arms is a difference between arms.
|
||||
- **🔄 BATTERY IS RUNNING — TUNED ARM LIVE, BASE SET COMPLETE.** Base arm ran 08:54:52Z→09:03:43Z, **zero errors/truncations/empties/degenerates across 1,728 generations**. Swapped to `erp-tune-v2` on `:8098`; **image digest `sha256:4091d5593f77…` VERIFIED IDENTICAL across both arms** (brokkr's requirement — a differing digest voids the run). brokkr is now running the tuned set with `--freeze-from` the base's marker list. **The floors the tuned deltas must clear:** reasoning core **0.5 pt**, diversity overall **0.0125** (rp 0.0153, story 0.0096), story attractor **0.0000**, memorisation signal 0.00%. ⚠ **The rp family froze ZERO markers**, so its attractor metric is structurally 0.0 on both arms and cannot discriminate — rp is measured on the distance axis only; do not quote an rp attractor delta. ⚠ Base's slop attractor: **"Elias" in 92/96 bare-prompt stories** (published figure on the same family is 102/144 — ours is an independent replication at a higher rate, not a novel finding).
|
||||
- **⏳ SUPERSEDED — the mid-swap state.** brokkr **withdrew** his both-arms-concurrent requirement (his diversity battery emits the frozen marker list to a *file*, so the arms were never a live dependency; the real requirement is all of ONE arm's passes on ONE instance before the swap). So: **sequential, no fleet seats displaced, operator not woken.** Live now: `erp-eval-base` on `:8099` (`erp-base-stock`, stock bf16 via symlinks into `/tank/aimodels`). **`erp-eval-v2` is DOWN and its container REMOVED**; weights are on disk so it is a cheap restart. Swap back with `/tank/erp-tune/serve-arm.sh tuned` when he reports the base set complete. ⚠ **Both arms go through `serve-arm.sh`**, which pins the vLLM image **by digest** (`sha256:251eba5cc7c1…`) — a version change between arms is a base swap nobody would see — and mounts `/tank/aimodels` for BOTH arms even though only base needs it, because a mount that differs between arms is a difference between arms.
|
||||
- **⏳ SUPERSEDED — the GPU arithmetic that forced sequential.** He needs **both arms reachable in one window** (the diversity battery mines its frozen marker list from base). **Two bf16 26B arms = 98 GB of weights on a 97.9 GB card**; GPU1 has ~30 GB free under six shared seats. Reduced `gpu-memory-utilization` does not help — weights are the floor. Options put to him: sequential-with-every-nuisance-variable-pinned, or displace GPU1 seats (**operator call, affects other consumers**). Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
|
||||
- **🔴 `erp-tune-v1` IS STILL IN THE LITELLM GATEWAY AND RETURNS HTTP 500.** I stopped its container to free GPU0 and left the route dangling. It is **config-file-defined (`db_model: false`)**, so removal needs a config edit **plus a gateway reload**, and a reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the worse trade at 1 request/24h. **Batch the removal with the v2-registration decision.**
|
||||
- **⏳ `erp-tune-v2` is DELIBERATELY NOT in the gateway.** Direct endpoint only. Registering an un-gated ERP tune makes it reachable by every agent holding the shared key — broader than "serve it" requires. Trivial to add once the battery reads out; operator's call.
|
||||
|
||||
Reference in New Issue
Block a user