feat(diariz): pilot live on fv-ml1 GPU 3 — smoke test passed (8 speakers, gen summary); GPU 3 2 MiB idle / 9.9 GB peak / 5.1 GB resident
This commit is contained in:
@@ -124,6 +124,7 @@ _As of 2026-10-18 ~2125 PT. The newest subsection is first; older subsections ca
|
||||
- **SVOS v0.0.17 (15c2d04) LIVE since 10-18 2255** (Prime's "deploy now" 2235 via svos-dev, thread `01M595YP8R62Y665C38MHPH6YX`): kernel 0.0.17 + UI 0.0.21 on nh3-svos, config gen `29bc36bf074b8245` (the six runtimes' prompt/working/window), voice `utterance_seconds = 60`; nh3-dev `current → 15c2d04`, only svos-terminal 0.0.6 restarted. No card: terminal revision 6 changes only `version`, so it pinned silently. Verified: 13 `seat/activity_changed` events at 2255:13 (ops-infra working = the positive control, 12 waiting; dev-ldp stays unknown, its 14-day-old claude matches neither mark). Not verified by infra-ops: GET /seats (needs a principal) and the hero "Fleet · working · N" (Prime's eyes). Rollback: servers/nh3-svos/README.md § Release 15c2d04.
|
||||
- **SVOS task launches were broken since v0.0.11 by OUR front door, FIXED 10-19 0118:** the kernel-origin Caddy (`svos-kernel.phasefinal.com`, path allowlist from U13 B2) never admitted the seat host's U15 reports (POST /tasks/start|progress|checking|exit), so every launch died `start_unreported` ("seat launch failed"). Prime's first two tasks (0110/0111) failed that way. Paths added (`configs/svos-front/Caddyfile`, `playbooks/svos-front-caddy-task-reports.yaml`); probe 401 from the kernel, `/auth/nope` still 404. ⚠ **When SVOS adds a host->kernel endpoint, this allowlist must grow with it.** **VERIFIED 0116:** Prime's relaunch `task:f0e9cc59…` ran hermes/gen-large 28 s, exit 0, check exit 0, settled ok, and produced a 1-page PDF. The result read then 403'd for lack of `prime → app:app-terminal read` (artifact resolve dispatches through app-terminal's read verb); grant 267 was approved 0118:44 and also covers the /tasks scan step.
|
||||
- **[2026-10-19] NH3 WAN outage ~0351–0825 (hosts did NOT reboot; WAN IP unchanged 70.230.226.88).** After it, these were re-run and came back clean: headscale-ddns, nemi-vault-pull, claude-memory-sync, ha-backup. backup-freshness reported DEGRADED, which led to: ⚠ **nh3-svos 10-18 21:30 fs-freeze FAILED because the v0.0.16 deploy left `50-svos.pre-judge` (0755) IN `/etc/qemu/fsfreeze-hook.d/`**. qemu's hook runs EVERY executable there (only .bak/.orig/~/.dpkg-* are skipped; a dir passes `-x` too), so the copy ran as a second freeze and refused. Moved to `/root/fsfreeze-hook-retired/`, __pycache__ removed at 0838; infra-hermes' live freeze at 0838:38 ran clean (only 50-svos). **Never back a hook up inside a hook dir.** Tonight's 21:30 is the real check.
|
||||
- **Diariz pilot LIVE since 10-19 1052 on fv-ml1 GPU 3** (Prime 1035: GPU 3 OK as an exception to the no-resident rule; nightly worker restart ~0300, idle-gated). URL https://diariz.nh3.phasefinal.com; login lkraven@lkraven.com / `secret get diariz/seed-password`. GPU 3: 2 MiB before the first job, 9.9 GB peak (cold 30-min job), 5.1 GB resident after (n=1). Alongside Scriberr, untouched. LiteLLM key alias `diariz` (gen + whisper-1). → `stacks/diariz/README.md`. Brokkr thread `01M5AH9QCVW7ZQ6VVXGWQGFPRS`. Open: Prime's browser live session + mp4 upload; not in restic.
|
||||
- **althing-core 3.7.1 on nh3-dev since 10-19 1004** (infra-hermes' herald stop condition: a re-poke rung >= 1 asks the post office and is cancelled ONLY on an answered unread 0; fails open on any other answer). That fixes bil-smithy-dev's phantom pokes. Installed with `uv pip --no-deps` into the uv tool env, so the deps are unchanged, but ⚠ the tool receipt still names the 3.7.0 /tmp wheel; the next `uv tool install` should use `~/.local/share/althing-releases/<ver>/`. Rollback wheel: `~/.local/share/althing-releases/3.7.0/`. althing `master` + tag v3.7.1 are LOCAL; the push is Prime's call. nh3-extdev's system wheel is still 3.7.0.
|
||||
- **SVOS v0.0.28 (d6298f7) UI 0.0.33 LIVE since 10-19 1032** (UI-only: U24 the 3-D tree). Before: v0.0.27 (d5e15c3) since 0954, whose notes follow. ⟶ (interface 0.0.9 restarted, no card; UI 0.0.32). Processes: kernel/knowledge/voice 2f29d0d code, judge d14dc94, interface d5e15c3. Before: v0.0.26 (a83c223) UI 0.0.30 since 0903 (UI-only: the flows' cadence). Before: v0.0.25 (f40bf6d) UI 0.0.29 since 0855. Processes on 2f29d0d (kernel/knowledge/voice) + d14dc94 (judge/interface) code. Before: v0.0.24 (d14dc94) since 10-19 0841 (U22 card summaries: judge 0.0.2 explain, interface 0.0.8 summaries, UI 0.0.28; 4 cards approved by Prime 0839–0841; grants 294/295). → servers/nh3-svos/README.md § Release d14dc94. Before: v0.0.22 (2f29d0d) UI 0.0.26 since 10-19 0125 (task result files by path). Before: v0.0.21 (45d3a48) UI 0.0.25 since 10-19 0104 (UI-only: the /tasks draft survives a hidden tab). Processes still on 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep those dirs. Before it: v0.0.20 (81cc5a2) UI 0.0.24 since 10-19 0055 (UI-only again: `stronger` look by default; the /tasks draft waits 125 s, after a 10 s browser abort discarded four good drafts). Nothing restarted. → servers/nh3-svos/README.md § Release 81cc5a2. Before it: (UI-only: U21 working look behind `?look=strong|stronger`, default unchanged; Prime picks, a later patch sets the default). Nothing restarted: the processes run 15c2d04 (kernel/knowledge/judge/voice) and 6d4cb1c (interface) code — keep both dirs. → servers/nh3-svos/README.md § Release 91e9f2d.
|
||||
- **SVOS v0.0.18 (6d4cb1c, U20 settings) LIVE since 10-18 2352** (thread `01M59D7NTZNN75SRP0M4ZQF91A`): infra-ops pushed svoperatingsystem main + tag on Prime's in-session "push it" (his own push never landed). Interface 0.0.7 + UI 0.0.22; only svos-interface restarted; card approved by Prime 2352:16 (via Matrix), interface down 30 s. ⚠ The kernel process still runs 15c2d04's code and reports 0.0.17 until its next restart (identical code). Config gen `d750257258f9fb68` since 10-19 0101 (`[[seats.repos]]` + repo-albok 0016, + repo-dread [path dreadnaught; the session handle is dread-dev, and the repo name must match the handle] and repo-svtc 0101, Prime via svos-dev; backups `instance.toml.pre-repo-albok`, `.pre-repo-dread-svtc`). Both `ns:interface-settings` grants are IN (issued by infra-ops as Prime on his word, approved by him 2356; grant ids 207/208). **Open, Prime's:** the /settings and panel checks plus his listening test. Details + rollback: servers/nh3-svos/README.md § Release 6d4cb1c.
|
||||
|
||||
@@ -40,6 +40,23 @@ So `diariz-nightly-restart.timer` (03:00) restarts both workers to drop the mode
|
||||
20 s, and any sample above 10% (a Diariz or a Scriberr job) means wait 15 min, retrying until 0500, then skip.
|
||||
Playbook: `playbooks/fv-ml1-diariz-nightly-restart.yaml`. Log: `journalctl -u diariz-nightly-restart`.
|
||||
|
||||
## Live since 2026-10-19 1052 — first measurements (n=1 each; cold first run, so not a speed figure)
|
||||
|
||||
| | GPU 3 memory |
|
||||
|---|---|
|
||||
| stack up, no job yet (models lazy) | **2 MiB** |
|
||||
| peak during the first upload: 30 min public SCOTUS audio, sampled every ~2 s, includes the first model loads | **9,890 MiB** |
|
||||
| idle after it (main worker's models resident) | **5,098 MiB** |
|
||||
| live worker | not loaded until a live session; expect about another 5 GB (estimate) |
|
||||
|
||||
Smoke test (public audio, deleted afterwards):
|
||||
- The upload went Queued → Summarized in 234 s, including the first-time model downloads. The worker posted 347 segments and 8 speakers
|
||||
(Scriberr's pyannote found 8 on the same file).
|
||||
- The `gen` summary was correct (Loper Bright v. Raimondo, 715 chars).
|
||||
- Embedding is off as intended: each recording logs ONE 403 from the gateway (`nomic-embed-text` is outside the `diariz` key).
|
||||
Harmless, and search stays lexical.
|
||||
- Not yet verified: a browser-mic live session and an mp4 upload (Prime's, in the browser).
|
||||
|
||||
## Operate
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user