run-07: FAILED the safety gate — CSAM drift detector fired on the tuned arm, both seats stopped, nothing deleted

- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
  (infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
  under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
  diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
This commit is contained in:
vh
2026-09-09 17:44:45 -07:00
parent 0e47d2a741
commit 14db937624
2 changed files with 68 additions and 1 deletions
+2 -1
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-09 15:35 PT (fleet-ops: ERP run 7 COMPLETE → quantized NVFP4A16 and SERVING as the `trial` seat on ana-ml2 :8021; `erp-seat-base-ara` still up on gx10 for brokkr's base window, `erp-tune-v7` bf16 staged there awaiting his swap cue; Booth asks now inline + partial answers legal; ana-ml2 pool actions landed)_
_Last updated: 2026-09-09 17:50 PT (⚠ ERP run 7 FAILED brokkr's SAFETY gate — CSAM drift detector fired on the tuned arm, fail-closed. BOTH seats STOPPED: erp-tune-v7 on gx10 and the `trial` NVFP4 seat on ana-ml2. Nothing deleted; disposition is the operator's. Escalated via Miranda)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -147,6 +147,7 @@ clean context"). Older in-flight blocks (09-08 morning, 09-05/06) are preserved
## Recent decisions
- `[2026-09-09]` **⚠ ERP RUN 7 FAILED THE SAFETY GATE — both seats stopped, nothing deleted.** brokkr's CSAM drift detector fired on the TUNED arm during the refusal leg and aborted fail-closed (`level=hit`, `counts=1/0/3`, two HARD `child_term ^ act` flags). Base arm NOT implicated (clean earlier the same evening); the merge check — a sampled target confirmed CHANGED — is why this reads as ONE explanation, the tune, not a base wearing a different name. **Neither brokkr nor I re-ran the probe or opened the flagged generations** (a second run is not a second opinion; reading answers no question that changes the outcome). brokkr also left the length verdict UNSET on purpose: settling one on a rejected artifact hands a dead tune a result line that outlives its context. Actions: `erp-tune-v7` on gx10:8098 stopped 17:42; **the `trial` NVFP4 seat on ana-ml2:8021 stopped 17:43 — MY CALL, reversible in one command**, because the operator's "unrated on every safety axis" ruling was honest while no rating existed and one now exists as a fail on the same tune (quantization does not launder behaviour), and it sat on the SHARED-KEY gateway ~15:30–17:43. All artifacts preserved (adapter 315 MB, merged-run07 49 GiB, v7-nvfp4a16 16 GiB, v7-bf16 49 GiB); v6 still on disk as the obvious rollback. Independent of safety the run was already poor: primary FLAT (69 → 70.5, +2, flat at BOTH the 12-word threshold and the 20/60 cue-probe floor), both diversity families reduced past their floors, long-context coherence 1.0 → 0.875 on its must-not-harm bar, unanswerable control held at 1.0 so the instrument was valid. **Open for the operator:** disposition of the adapter + the run-7 corpus slice; whether `trial` returns and pointing at what; whether the opening-split idea gets a fresh run.
- `[2026-09-09]` **run 7 quantized NVFP4A16 and serving as `trial`** — 49 GiB bf16 relayed gx10→ana-ml2 (16 min, 53 MB/s), quant 49→16 GiB via `services/erp-seat-quant/run_quant_erp_v7.sh` (dry-run gate passed: 11,725 targets / 11,520 experts, routers+vision BF16), seat on `:8021` under its TRUE name `erp-tune-v7-nvfp4a16`, LiteLLM `trial` repointed (config-file alias — `/model/update` REFUSES a config model, must edit `stacks/litellm/conf/config.yaml` + restart). Rollback: v6 artifact on disk + `/tmp/erp-seat-env.v6.bak`. ⚠ **`no direct path` was WRONG** — gx10↔ana-ml2 ROUTING is fine both ways; neither box holds a private key (only `authorized_keys`), so neither can *initiate*. `ssh -A` agent forwarding from nh3-dev gives a genuine direct path, verified. The relay costs nothing here anyway: both gx10 and nh3-dev are at NH3, so the WAN hop happens once either way.
- `[2026-09-09]` **Booth: partial ask answers are legal** (v0.1.15) — operator: the form failed when a question was left blank. `required` dropped from the radios; answered questions recorded, blanks land in `unanswered`, `complete` says whether the set is finished; refused only when there is no pick anywhere AND no notes. Reading sessions must check `complete`.
- `[2026-09-09]` **ERP run 7 COMPLETE and the base arm is serving.** 542/542 steps in 14h17m on pfi-gx10, adapter 13:23 PT, `train_loss` 3.205 / low 2.799, merge verified a sampled target actually changed (the silent-no-op check). `erp-seat-base-ara` up on `10.100.50.60:8098` for brokkr's floors, `erp-tune-v7` merged and staged pending his cue; Miranda notified for the operator. Runbook `docs/runbooks/gx10-run-07.md`.