From 3048e4194f8794544b476219624accd7217a1f88 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 27 Sep 2026 09:00:51 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20semif=200.1.3?= =?UTF-8?q?=20live=20(averaging=20+=20fast=20kernels=20+=20bug-hunt=20fold?= =?UTF-8?q?s),=20hermes-gateway=20restart=20for=20SVOS=20seat=5Fup,=20over?= =?UTF-8?q?night=20backups=20green;=20next:=20SemIf=20spike=20(scope=20to?= =?UTF-8?q?=20confirm=20with=20Prime)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-09-27-semif-order-averaging.md | 8 ++++++++ persistent-memory.md | 18 ++++++++++++++---- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/persistent-memory.d/2026-09-27-semif-order-averaging.md b/persistent-memory.d/2026-09-27-semif-order-averaging.md index 14e41dc..6f8b157 100644 --- a/persistent-memory.d/2026-09-27-semif-order-averaging.md +++ b/persistent-memory.d/2026-09-27-semif-order-averaging.md @@ -33,3 +33,11 @@ The service is deterministic, so the CI measures item sampling, not run noise. T reference PyTorch paths. Adopt them only if the authored144 parity re-check holds. Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]]. + +**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows, +accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels +won the A/B on the empty GPU 3: +- parity with upstream 144/144 (was 142/144); +- a ~2k-token `/decide` 169 → 92 ms; +- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13). +All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`). diff --git a/persistent-memory.md b/persistent-memory.md index ed827ef..52db5fd 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-27 ~0255 PT (augaman v0.1.3 live on esh-ml1 with a verified off-site gallery backup; restic creds out of systemd units on all 8 hosts + vaulted; infra-ops on vm-esh-nas; SemIf live on fv-ml1 GPU1, order-averaging spike done, 0.1.3 build + fast-kernel trial next; corviduo/svoperatingsystem created.)_ +_Last updated: 2026-09-27 ~0900 PT (semif-serve 0.1.3 live: order averaging + fast kernels + heid bug-hunt folds; hermes-gateway restarted for High Seat v2.1.12; overnight backups all green; NEXT: a SemIf spike, scope to confirm with Prime.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,7 +115,7 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-27 ~0255 PT._ +_As of 2026-09-27 ~0900 PT._ ### nh3-pve + nh3-ml1: post-visit, all live (2026-09-25/26) @@ -155,6 +155,11 @@ _As of 2026-09-27 ~0255 PT._ as LAN + auth. - NVFP4 is not worth it (see the SemIf detail files). Prime's probes (2AM/2PM, dragon/lottery) are recorded in [[2026-09-27-semif-order-averaging]]. +- **NEXT (Prime, 0859): "a spike of the SemIf system" after this /clear. Its scope was NOT specified, + so ask Prime what it should cover before starting.** Candidates to offer: a real-workload trial + (pick a fleet decision and label ~150 rows, then fit a calibration T); paraphrase robustness + (the 2AM case in several wordings, which the ordering spike never covered); a catch-all wording + study; or a load and concurrency run against the 429 and VRAM envelope. ### restic: credential leak fixed (2026-09-27, Prime) @@ -165,6 +170,8 @@ _As of 2026-09-27 ~0255 PT._ - **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean and vaulted.** - The passwords were readable until today, so the rotation (Prime's) remains the real fix. +- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move + (done ~0140–0200). - **Check:** 2026-09-28 0100 PT is the first nightly run on `repository-file` for seven of the hosts (nh3-docker was proven end to end by a manual run, snapshot `a29b889d`). The 0800 freshness check should be all-green. @@ -200,14 +207,15 @@ _As of 2026-09-27 ~0255 PT._ ### Live threads -- git: `main` is ahead of origin by 3 at `739aa03`, plus this snapshot. Pushing +- git: `main` is ahead of origin by 5 at `77b8cb4`, plus this snapshot. Pushing is Prime's call. - ESH has a single outside route (esh-scale on esh-pve). Noted, untracked. ## Recent decisions +- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call. - `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md` -- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` +- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`). - `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md` - `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`). - `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`). @@ -424,6 +432,8 @@ _106 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`). +- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra. - `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`). - `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`. - `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.