memory: snapshot — semif 0.1.3 live (averaging + fast kernels + bug-hunt folds), hermes-gateway restart for SVOS seat_up, overnight backups green; next: SemIf spike (scope to confirm with Prime)

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-27 ~0255 PT (augaman v0.1.3 live on esh-ml1 with a verified off-site gallery backup; restic creds out of systemd units on all 8 hosts + vaulted; infra-ops on vm-esh-nas; SemIf live on fv-ml1 GPU1, order-averaging spike done, 0.1.3 build + fast-kernel trial next; corviduo/svoperatingsystem created.)_
+_Last updated: 2026-09-27 ~0900 PT (semif-serve 0.1.3 live: order averaging + fast kernels + heid bug-hunt folds; hermes-gateway restarted for High Seat v2.1.12; overnight backups all green; NEXT: a SemIf spike, scope to confirm with Prime.)_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -115,7 +115,7 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-09-27 ~0255 PT._
+_As of 2026-09-27 ~0900 PT._
 
 ### nh3-pve + nh3-ml1: post-visit, all live (2026-09-25/26)
 
@@ -155,6 +155,11 @@ _As of 2026-09-27 ~0255 PT._
   as LAN + auth.
 - NVFP4 is not worth it (see the SemIf detail files). Prime's probes (2AM/2PM, dragon/lottery) are
   recorded in [[2026-09-27-semif-order-averaging]].
+- **NEXT (Prime, 0859): "a spike of the SemIf system" after this /clear. Its scope was NOT specified,
+  so ask Prime what it should cover before starting.** Candidates to offer: a real-workload trial
+  (pick a fleet decision and label ~150 rows, then fit a calibration T); paraphrase robustness
+  (the 2AM case in several wordings, which the ordering spike never covered); a catch-all wording
+  study; or a load and concurrency run against the 429 and VRAM envelope.
 
 ### restic: credential leak fixed (2026-09-27, Prime)
 
@@ -165,6 +170,8 @@ _As of 2026-09-27 ~0255 PT._
 - **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
   and vaulted.**
 - The passwords were readable until today, so the rotation (Prime's) remains the real fix.
+- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
+  (done ~0140–0200).
 - **Check:** 2026-09-28 0100 PT is the first nightly run on `repository-file` for seven of the hosts
   (nh3-docker was proven end to end by a manual run, snapshot `a29b889d`). The 0800 freshness check
   should be all-green.
@@ -200,14 +207,15 @@ _As of 2026-09-27 ~0255 PT._
 
 ### Live threads
 
-- git: `main` is ahead of origin by 3 at `739aa03`, plus this snapshot. Pushing
+- git: `main` is ahead of origin by 5 at `77b8cb4`, plus this snapshot. Pushing
   is Prime's call.
 - ESH has a single outside route (esh-scale on esh-pve). Noted, untracked.
 
 ## Recent decisions
 
+- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
 - `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
-- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md`
+- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
 - `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
 - `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
 - `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
@@ -424,6 +432,8 @@ _106 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
+- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
 - `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
 - `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
 - `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.