memory: snapshot — semif 0.1.3 live (averaging + fast kernels + bug-hunt folds), hermes-gateway restart for SVOS seat_up, overnight backups green; next: SemIf spike (scope to confirm with Prime)
This commit is contained in:
@@ -33,3 +33,11 @@ The service is deterministic, so the CI measures item sampling, not run noise. T
|
||||
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
|
||||
|
||||
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
|
||||
|
||||
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
|
||||
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
|
||||
won the A/B on the empty GPU 3:
|
||||
- parity with upstream 144/144 (was 142/144);
|
||||
- a ~2k-token `/decide` 169 → 92 ms;
|
||||
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
|
||||
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
|
||||
|
||||
+14
-4
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-27 ~0255 PT (augaman v0.1.3 live on esh-ml1 with a verified off-site gallery backup; restic creds out of systemd units on all 8 hosts + vaulted; infra-ops on vm-esh-nas; SemIf live on fv-ml1 GPU1, order-averaging spike done, 0.1.3 build + fast-kernel trial next; corviduo/svoperatingsystem created.)_
|
||||
_Last updated: 2026-09-27 ~0900 PT (semif-serve 0.1.3 live: order averaging + fast kernels + heid bug-hunt folds; hermes-gateway restarted for High Seat v2.1.12; overnight backups all green; NEXT: a SemIf spike, scope to confirm with Prime.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,7 +115,7 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-27 ~0255 PT._
|
||||
_As of 2026-09-27 ~0900 PT._
|
||||
|
||||
### nh3-pve + nh3-ml1: post-visit, all live (2026-09-25/26)
|
||||
|
||||
@@ -155,6 +155,11 @@ _As of 2026-09-27 ~0255 PT._
|
||||
as LAN + auth.
|
||||
- NVFP4 is not worth it (see the SemIf detail files). Prime's probes (2AM/2PM, dragon/lottery) are
|
||||
recorded in [[2026-09-27-semif-order-averaging]].
|
||||
- **NEXT (Prime, 0859): "a spike of the SemIf system" after this /clear. Its scope was NOT specified,
|
||||
so ask Prime what it should cover before starting.** Candidates to offer: a real-workload trial
|
||||
(pick a fleet decision and label ~150 rows, then fit a calibration T); paraphrase robustness
|
||||
(the 2AM case in several wordings, which the ordering spike never covered); a catch-all wording
|
||||
study; or a load and concurrency run against the 429 and VRAM envelope.
|
||||
|
||||
### restic: credential leak fixed (2026-09-27, Prime)
|
||||
|
||||
@@ -165,6 +170,8 @@ _As of 2026-09-27 ~0255 PT._
|
||||
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
|
||||
and vaulted.**
|
||||
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
|
||||
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
|
||||
(done ~0140–0200).
|
||||
- **Check:** 2026-09-28 0100 PT is the first nightly run on `repository-file` for seven of the hosts
|
||||
(nh3-docker was proven end to end by a manual run, snapshot `a29b889d`). The 0800 freshness check
|
||||
should be all-green.
|
||||
@@ -200,14 +207,15 @@ _As of 2026-09-27 ~0255 PT._
|
||||
|
||||
### Live threads
|
||||
|
||||
- git: `main` is ahead of origin by 3 at `739aa03`, plus this snapshot. Pushing
|
||||
- git: `main` is ahead of origin by 5 at `77b8cb4`, plus this snapshot. Pushing
|
||||
is Prime's call.
|
||||
- ESH has a single outside route (esh-scale on esh-pve). Noted, untracked.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
|
||||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md`
|
||||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
|
||||
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
|
||||
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
|
||||
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
|
||||
@@ -424,6 +432,8 @@ _106 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
|
||||
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
|
||||
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
|
||||
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
|
||||
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
|
||||
|
||||
Reference in New Issue
Block a user