76 Commits
Author SHA1 Message Date
vh 251c5a8687 feat(svos): v0.0.15 live on nh3-svos (short spoken summons, voice 0.0.2); utterance_seconds 60, Matrix request_timeout 30 2026-10-17 11:01:56 -07:00
vh e85ea97101 fix(svos nh3-svos): Matrix request_timeout_seconds 60 — stopgap for card audio dying at exactly 15 s (svos-dev) 2026-10-17 09:17:21 -07:00
vh 77b75aa172 feat(svos): v0.0.14 live on nh3-svos + nh3-dev (cards approved 0909); hermes task runtime on the query-file route (Prime) 2026-10-17 09:11:10 -07:00
vh f843cd92b1 feat(wt-memory-gate-batch): --keep-data with a per-batch scratch outside git (TMPDIR), 14-day user tmpfiles retention (worldtree-dev's ask) 2026-10-17 07:55:26 -07:00
vh 39458e013a fix(svos v0.0.14 playbook): terminal verify reads the ScanError line (svos-dev: scan:on is only on the signed /health) 2026-10-17 07:53:30 -07:00
vh b72783cb3c feat(svos): v0.0.14 (3726d73) release playbooks — staged on nh3-svos and built on nh3-dev; cutover in three phases held for Prime's card window 2026-10-17 07:50:44 -07:00
vh bc91b40ac5 chore: retire asset-engine (Prime 2026-10-17); kuma gains delete-by-name; FleetTools/bootstrap/catalog marked 2026-10-17 00:16:31 -07:00
vh 055d56fa45 docs(fleettools/speech): stale Homepage ASR entries cleared 2026-10-17 00:10:24 -07:00
vh 4ddd72f22e docs(parakeet-nemo): rollback is compose up of the kept sherpa stack (old container removed to clear a stale Homepage entry) 2026-10-17 00:10:18 -07:00
vh b0235a1e46 docs(fleettools/speech): ASR table matches live state (Parakeet NeMo primary + LiteLLM aliases, VibeVoice, Scriberr on fv-ml1); retired Speaches/sherpa entries named 2026-10-17 00:07:30 -07:00
vh c560e37739 docs(fleettools): Albok page + index entry (draft by albok-dev, Prime's ask); albok MCP in claude-only; task-board marked mothballed 2026-10-16 19:54:34 -07:00
vh 3cb969f1d1 docs(nh3-dev): book drop path is scp (Prime 2026-10-16), no share 2026-10-16 12:29:53 -07:00
vh 9f79c5e36f memory: Worldtree demo/personal/pinned on v1.0.0b195; pinned's bridge removed 2026-10-16 11:12:52 -07:00
vh eb2a0a532c feat(nh3-dev): /srv/nemi/books book drop (fiction/ nonfiction/ rejected/) for albok unit 8 2026-10-16 11:09:13 -07:00
vh d03e6353ab feat(albok-service): 1.0.0a1 (version-only rebuild of 0.1.24; v1 alpha cut) 2026-10-16 10:46:47 -07:00
vh 58cef5ab00 memory: personal Worldtree retired-config strip done; Matrix bridge profile set on demo/personal 2026-10-16 10:37:09 -07:00
vh 6292ba232b docs(muninn-gate, mimir-inbox): torn down 2026-10-16 (albok unit 6); archive on nh3-nas 2026-10-16 10:33:57 -07:00
vh e518da604c memory: albok unit 6 encrypted-wing harness passed; sealed wing stays mounted 2026-10-16 09:22:49 -07:00
vh f21fc07c23 docs(albok-service): sealed unlock takes ~25 s; the argon2 memory warning is benign 2026-10-16 09:22:22 -07:00
vh 507d62d62a feat(albok-service): 0.1.24 (unit 6 encrypted-wing deferral) + personal/sealed wing in albok.yaml 2026-10-16 09:18:28 -07:00
vh 76e7a76ad9 memory: albok unit 6 teardown — archives verified on nh3-nas, deletion held for Prime's go 2026-10-16 08:37:24 -07:00
vh a4b27ec4ff memory: Worldtree demo/personal/pinned all on v1.0.0b194; pinned now has a seeded host config dir 2026-10-16 08:24:27 -07:00
vh 767b3af409 feat(albok-service): unit 6 sealed wing personal/sealed — LUKS2 volume, gid 1513, rslave bind of /srv/albok/sealed
albok-dev's host ask under Prime's goal "land all of u6". A 20 GiB LUKS2 file
with ext4, keyed only from the vault (piped over ssh, never on the host's
disk). The mountpoint is immutable while unmounted, and the mounted root is
1500:1513 0750 with .albok-wing. The parent is bound into the container with
rslave propagation: a lock then unlock with the service running showed through
without a restart. The wing is not in albok.yaml yet (albok-dev's release).
Runbook and backup posture are in the README.
2026-10-16 08:06:06 -07:00
vh 16e0dac572 fix(wt-memory-gate-batch): unset ALBOK_SERVICE/ALBOK_TOKEN for the Worldtree harness
Since 2026-10-16 every agent shell on nh3-dev exports the fleet-agents Albok
read token (via ~/.config/secrets/env.sh). The memory-gate harness has always
run without albok, so the wrapper unsets both. The daily numbers stay
comparable; removing the unset is worldtree-dev's call.
2026-10-16 00:04:20 -07:00
vh f49e3f18ed memory: Worldtree demo's retired config removed (instance-configs 2240110); personal/pinned after their deploys 2026-10-15 23:40:27 -07:00
vh 477de888ef docs(pewpewstudio): live rev 3f73448 (svgelements, the vectorize shape change, asset CSP); smoke passed 2026-10-15 23:28:24 -07:00
vh 05e370d2ae albok-service: pin 0.1.23 (unit 7, lineage and durable identity; the facet-reader descriptor fix; deployed on nh3-albok 2026-10-15 1709) 2026-10-15 17:09:27 -07:00
vh 9b2716261c albok-service: pin 0.1.22 (unit 5, related articles; deployed on nh3-albok 2026-10-15 0951) 2026-10-15 09:52:19 -07:00
vh 0b873dd8fe docs(albok): Prime's notes walked into personal/notes (232 objects); old nemi token revoked 2026-10-15 01:48:48 -07:00
vh a583c54399 docs(albok): nemi token re-minted with personal/notes for the vault source 2026-10-15 01:42:15 -07:00
vh db88f76aea feat(albok): /srv/nemi/vault, a pulled clone of Prime's notes (vh/VuongKB) for Nemi's vault source (gate 1)
Prime keeps his notes at /home/lkraven/kb and syncs them with git (gitea
vh/VuongKB), so Nemi's read-only copy is a fresh clone, pulled fast-forward-only
hourly at :50 by a lkraven user timer, ahead of the ~:01 walk. As a clone it
holds only tracked notes (129 md outside dot-dirs). The untracked Muninn
output in the working copy (main/, fiction/) stays out. Readable from
lkraven's user manager, where Nemi runs.
2026-10-15 01:40:02 -07:00
vh dd816662dc ops(svos): main 33a7077 (v0.0.13+2) on nh3-svos + nh3-dev — kernel 0.0.13, interface 0.0.5 (briefs), postoffice 0.0.3, UI 0.0.18
On svos-dev's ask, under Prime's 0955 go. The interface unit gets
SVOS_ARTIFACT_HOST=nh3-svos via a drop-in, and the briefs artifact mapping uses
the same host. Both changed manifests were held for the owner, and each app
read down (wrong-revision) until Prime approved its card: interface at
11:09:45, postoffice at 11:11:17. The postoffice was rolled only once he was
at the board, since a held bridge carries no Miranda mail. Health: schema 13,
config_gen 33f001f4e2dbc7dc, apps up 7.
2026-10-14 11:11:59 -07:00
vh a84f6b00df ops(svos): kernel 0.0.12 (d95078c) on nh3-svos; hermes runtime task-capable (Prime via svos-dev)
Kernel-only release from a clean archive of v0.0.12; everything else is
carried from 452a592. Offline backup first; READY in 0.74 s, schema 13, apps
up 7. The existing hermes runtime gained process = python3 and the one-off
task_argv (hermes -z, --accept-hooks, no --yolo), and its argv moved to the
launcher's chat form. check-config valid; config_gen 21489c1f0fff3c10.
2026-10-14 04:17:35 -07:00
vh 8758c359bc memory: elway sudo note updated for infra-ops (unattended); SVOS task_argv is ruled with svos 2026-10-14 04:11:43 -07:00
vh 9d599caa53 memory: snapshot — SVOS v0.0.11, pewpewstudio, albok inbox, h2o decision seat; 31 old entries archived 2026-10-14 04:04:54 -07:00
vh d1192ca868 feat(pewpewstudio): laser job studio on nh3-docker :8185 (api + Caddy front), Homepage tile
Prime's directive via pewpew-dev. Images are built on nh3-dev from a clean
archive of vh/pewpewstudio (26bba3c; hamr over gitea SSH via BuildKit --ssh)
and loaded on nh3-docker. Caddy serves the SPA and strips /api/ in front of
the backend. It refuses foreign Host headers inside a route block, because the
Host rewrite would otherwise bypass the backend's TrustedHost. The DB, assets
and the app-edited shop config live on named volumes inside nh3-docker's
nightly vzdump. Verified: SPA, /api/machines and /api/generate/workflows 200;
foreign Host 400.
2026-10-13 21:39:44 -07:00
vh 5e8307ed7a ops(svos): v0.0.11 (452a592) on nh3-svos + nh3-dev — kernel 0.0.11, schema 13, U15 [tasks] config; terminal 0.0.4
Built in-session from a clean archive of the tag (kernel, connector base and
terminal wheels; everything else carried from 7f95e54). Ran after the 21:30
vzdump of VM 111, with an offline backup of the schema 12 database first.
check-config accepted the new [tasks] table and the tasks artifact mapping
(host nh3-svos, the terminal's artifact_host). Kernel READY in 1.17 s,
schema 13, apps up 7. nh3-dev's current moved to 452a592; svos-terminal 0.0.4
registered manifest revision 4, which was pinned without a card.
2026-10-13 21:33:16 -07:00
vh 60a6d6db59 ops(corviduo-dev): muninn-gate and mimir-inbox stopped (retired; data at rest kept for albok unit 6) 2026-10-13 21:23:56 -07:00
vh f1cc7ab995 memory: infra-ops fleet key vaulted (restore tested); notes copy approved, vault location pending 2026-10-13 21:22:51 -07:00
vh 047ff26778 albok-service: pin 0.1.21 (listing paging, a concept's citation in the summary; deployed on nh3-albok 2026-10-13 1805) 2026-10-13 18:04:40 -07:00
vh 4eacb746fa docs(albok): old nemi token revoked after the personal/inbox swap 2026-10-13 17:43:15 -07:00
vh 7ca2f76d21 feat(albok): personal/inbox wing on nh3-albok + /srv/nemi/inbox capture root on nh3-dev (unit 4.4 slice C, gate 2)
Prime ruled albok subsumes note intake (1530). The wing is like personal/notes
(albok-personal 1511, not service-restricted); health reads 7 wings ready.
Nemi's token was re-minted with read+write on the new wing and vaulted over
albok/token-nemi. /srv/nemi/inbox is lkraven:nemi-drop 2770 with a default ACL,
plus u:lkraven:rx on /srv/nemi, so Nemi reaches it from lkraven's user manager,
whose groups predate nemi-drop. Both paths are verified by the playbook.
2026-10-13 17:31:49 -07:00
vh eda7025ed1 ops(svos): A171 — miranda's app-postoffice write grant autonomous (Prime 2026-10-13 1400) 2026-10-13 13:59:30 -07:00
vh 3e634bb3d0 ops(infra-ops): per-host satellite keys; satellite-1 authorized on esh-nas
Satellite agents get their OWN key under the infra-ops identity instead of a
copy of the fleet key, so one can be revoked by deleting a line. The list is
playbooks/files/infra-ops-satellite-keys (target short hostname + public key).
bootstrap-infra-ops-user.yaml now writes the fleet key plus that host's
satellite keys, exactly, so a re-run keeps listed keys and removes unlisted
ones. satellite-1 (SHA256:KEFUjceF...) is authorized on esh-nas only (Prime,
2026-10-13); the playbook re-run there installed both keys, and the fleet key
still logs in with sudo.
2026-10-13 13:33:04 -07:00
vh 30a7220b2c ops(esh-nas): infra-ops identity bootstrapped in CT 103 via pct from esh-pve-nas
Prime's ruling 2026-10-13. esh-nas (10.0.50.50) had no infra-ops user, so key
login was refused. The same steps as bootstrap-infra-ops-user.yaml (uid/gid
850, fleet key, NOPASSWD sudoers with log_output and the BULK_DATA carve-out,
visudo-validated) ran inside the container from its host. Verified with a
direct ssh infra-ops@10.0.50.50 and sudo -n whoami = root.
2026-10-13 13:25:21 -07:00
vh b699088e2b docs(CLAUDE): standing push on main for infra-ops (Prime, 2026-10-13) 2026-10-13 11:26:47 -07:00
vh ee137dbb04 albok-service: pin 0.1.20 (the search reply names its coverage: searched / degraded; deployed on nh3-albok 2026-10-13 1022) 2026-10-13 10:22:38 -07:00
vh a630fe46d6 memory: correct the 7f95e54 deploy time (0906, not 0914) 2026-10-13 09:06:57 -07:00
vh c4c35ae951 memory: nh3-svos static release 7f95e54 (UI 0.0.15) 2026-10-13 09:06:44 -07:00
vh 5000916a7b memory: app-terminal PATH fix; nh3-svos static release 89dad7a (UI 0.0.14) 2026-10-13 08:57:53 -07:00
vh 3c92a1281e fix(svos-apps): svos-terminal unit reaches zellij (~/.cargo/bin on PATH); app-terminal up after 2 days degraded
A user unit's default PATH lacks ~/.cargo/bin, so every zellij call from
app-terminal failed to launch (rc 127 under the unit's environment), and the app
reported degraded from the U13 cutover on. With PATH set, the kernel logged
app-terminal up at 15:56:40Z; the pinned manifest is unchanged and no card was
raised.
2026-10-13 08:57:10 -07:00
vh 499678979a docs(gitea-runner): vh/albok green on the nh3-docker runner; its CI was slow fsync, not a hang 2026-10-12 23:25:09 -07:00
vh 9f833b2e90 memory: h2o cutover walk closed — personal reading (18 vs 6) is the one open margin 2026-10-12 18:13:29 -07:00
vh 5af8e2bc17 feat(intern-decision): h2o-lightning-4b is the production decision seat; fix forward, no INC fallback
Prime (18:05, via albok-dev): go to h2o and fix forward, do not fall back to
INC. albok-dev removed the chapter margin first (the two published nonfiction
titles declared visibility: public). switch-to-h2o.sh ran 18:05:13, h2o healthy
18:07:40; /health at the pin, /decide/shared 200, 422 over the cap, inline image
answers, image URL refused by the shim and by vLLM's media-domain backstop.
rollback-to-inc.sh stays on the host as an artifact only.
2026-10-12 18:08:31 -07:00
vh c991d939f9 memory: second h2o rollback closed out — counts restored, timer back 2026-10-12 17:51:46 -07:00
vh d2fb840bfa fix(intern-decision): second rollback to Intern-Decision-4B; h2o staged until the judge's two margins are fixed
The v2 walk on h2o (17:05-17:45) held notes at 0 adult but moved two margins
past albok-dev's pre-named rollback line: library/nonfiction adult 0 -> 79 (one
textbook chapter at 0.531 on the book profile, INC 0.438, with 77 concepts
inheriting) and fleet/memory personal 6 -> 18. rollback-to-inc.sh ran 17:47:46,
INC healthy 17:48:11. compose.yaml is INC again.
2026-10-12 17:48:43 -07:00
vh 2d85ee5a25 docs(gitea-runner): first real use of the nh3-docker runner (vastblue pytest 4m45s vs 14m17s, n=1 each) 2026-10-12 17:34:32 -07:00
vh cd3c4b8be9 feat(gitea-runner): opt-in fast runner on nh3-docker (label nh3-docker), no docker socket in jobs
Prime's CI fix A. ana-docker's runner sits on HDD mirrors with no SLOG; synced
8 KiB writes measured ~34 ms there vs ~2.7 ms on nh3-docker (same probe, one run,
n=3 each). Repos opt in with runs-on: nh3-docker; the runner carries no
pfi-fleet label. Its config is tighter because nh3-docker also runs NH3 DNS and
the post office: docker_host '-' (verified absent in the job, present on the
ana-docker baseline), capacity 1, jobs capped at 4 GiB / 6 CPUs, a network per
job. The deploy playbook takes runner_config; each runner's homepage name is
its runner name.
2026-10-12 17:27:46 -07:00
vh 0da7df7534 memory: Prime's 10-12 rulings (CI A, push, no judge-token rotation); card 05cf119e was approved, not pending 2026-10-12 17:22:59 -07:00
vh 6a01da759d memory: decision seat live on h2o again since 1704 under the re-fitted judge 2026-10-12 17:06:20 -07:00
vh 98254ca077 feat(intern-decision): decision seat back on h2o-lightning-4b under the re-fitted judge
Prime ruled the cutover ("do it", 16:40, via albok-dev) after the judge's
note question was re-fitted (albok-nemi 0.1.18). switch-to-h2o.sh ran 17:02:23,
the GPU 1 gate passed, h2o healthy 17:04:54. Checks on :8033: /health at the
pin, /decide/shared 200, 422 over the cap, inline image answers, image URL
refused by the shim, vLLM media-domain backstop refuses (400). compose.yaml is
h2o again; rollback-to-inc.sh stays the one-step way back.
2026-10-12 17:05:59 -07:00
vh 014a4fd0ec docs(intern-decision): seat switch checklist — Nemi's judge concurrency is 1 on INC, 4 on h2o 2026-10-12 16:21:25 -07:00
vh 8d9fcd9ffb memory: h2o judge investigation closed — v1 note question was the cause, re-fitted in albok-nemi 0.1.18; eval seat down 2026-10-12 16:20:49 -07:00
vh 2b40805b84 feat(h2o-eval): round 2 eval seat on GPU 3 with floor-on/floor-off fronts for the judge investigation
Prime (via albok-dev): understand why h2o failed and re-fit the judge. One
vLLM on GPU 3 behind two fronts: :8034 with the shim's yes/no floor at 0.801
exactly as the cutover ran (same judge stamp), :8035 with SHIM_NOUL_FLOOR=0 and
a distinct stamp (h2o-lightning-4b+noul-floor-0). Both refuse image URLs; the
vLLM --allowed-media-domains backstop is now verified (400, no fetch).
2026-10-12 14:29:15 -07:00
vh 501bda078d memory: decision-seat rollback closed out — hidden set back to INC counts 2026-10-12 14:09:11 -07:00
vh f979cca1ea fix(intern-decision): roll the decision seat back to Intern-Decision-4B; h2o staged with inline-images-only
albok-dev withdrew the judge sign-off on fleet-scale evidence and brokkr
concurred: over 10,841 texts judged by both models, h2o flagged 242 adult at
0.5 against INC's 88, and no threshold reproduces INC's set. rollback-to-inc.sh
ran 14:03:28, INC healthy 14:03:56, /decide/shared smoke 200 x3.

compose.yaml is INC again; the h2o compose is kept as compose.h2o.yaml with
switch-to-h2o.sh (GPU 1 free-memory gate, falls back to INC). It now carries
Prime's inline-images-only setting: LIGHTNING_IMAGE_URLS=off on the front
(verified live: an https image URL is refused before any fetch, inline images
still answer) and vLLM --allowed-media-domains media.invalid as a backstop.
rollback-to-inc.sh no longer leaves timestamped compose copies behind.
2026-10-12 14:05:44 -07:00
vh c08992f95f feat(intern-decision): decision seat swapped to h2o-lightning-4b on fv-ml1 GPU 1; eval seat down
The :8033 seat now serves h2oai/h2o-lightning-4b @ b43f759b on vLLM 0.30.0 with
byte-pinned KV (4.48 GiB, 135,558 tokens) behind services/h2o-decision-front:
32k refuse-not-trim admission, the seat's bearer token, and /decide +
/decide/shared translated onto /v1/systemone for albok-nemi's judge. URL, port,
token and stack name are unchanged. INC down 13:08:18, h2o healthy 13:11:27.

Post-checks on the live seat: /health at the pinned revision; JevBench public
205/231 (= bench); 422 over the cap; 16q at the cap warm 2.0-2.16 s; seat peak
16,164 MiB under 4 concurrent 16q x 32k calls against a 16,847 budget.

compose.inc.yaml + rollback-to-inc.sh restore Intern-Decision-4B in one step.
The temporary GPU 3 eval seat (stacks/h2o-eval) is down.
2026-10-12 13:19:46 -07:00
vh 152137e41d memory: decision-seat swap held for albok's judge; h2o eval seat on GPU 3 2026-10-12 13:02:04 -07:00
vh 2c3ef7a8af feat(h2o): temporary h2o-lightning-4b eval seat on fv-ml1 GPU 3 + the seat front (admission, bearer, /decide/shared translator)
Prime's call via brokkr: h2o-lightning-4b replaces the intern-decision seat once
albok-dev re-validates Nemi's sensitivity judge on it. services/h2o-decision-front
builds on infra-hermes's admission proxy (30f36f2): 32,000-token refuse-not-trim,
the seat's bearer token, and intern-decision-serve's /decide/shared mapping onto
/v1/systemone so the judge runs unchanged. stacks/h2o-eval serves it on GPU 3
:8034 exactly as benched (vLLM 0.30.0 + unmodified vendor shim; weights hashed
against HF b43f759b). INC keeps serving production on GPU 1.
2026-10-12 13:01:53 -07:00
vh 30f36f2478 tools(h2o): admission proxy + gate probe set — the seat swap inherits a hard dep
The vendor shim silently trims state at 32,000 tokens; the admission proxy is
what turns the seat contract into a loud refusal, so it must front the shim in
any deployment. Committing it with the Amendment 1 confirmation driver so
executing the swap does not depend on a bench scratch dir surviving.
2026-10-12 12:29:57 -07:00
vh 14af41b945 docs(pfi): Amendment 1 addendum — h2o confirmation run passes all three clauses
Admission via thin wrapper (vendor shim unmodified; no shim config knob exists).
Gate re-run clean at every depth with usage honest; 16q32k warm 2,033 ms;
4x concurrent 16q32k peak 15,566 MiB vs 16,847 budget; boot matrix replaces the
unmeasured 280 s figure (cold median 153.6 s, warm-cache saves nothing, eager
slower). Wyrd itemised: 8 diffs, h2o net -2. h2o passes every Supersede clause
under the accepted 32,000-token refusing contract; seat swap is the operator's
call.
2026-10-12 12:13:31 -07:00
vh 4098a23dda docs(pfi): decision-sweep follow-on bench — H2O re-bench, Clef-flash, RSI-Jev vs the live seat
First bench-first run (prereg brokkr-smithy 2e0539f). H2O-lightning-4b is a
short-route candidate: public 205, battery CI lower bound +0.0023, fits the
GPU 1 budget at every shape — but its shipped shim silently trims state at
~32k and fails the long-input gate by the letter. Clef-flash lateral in both
precisions (Wyrd -0.0333 in both; bf16 does not fit at rest). RSI-Jev v6.1
loses on decision quality (public -23, Wyrd -0.0548). INC holds the seat.
Torn down; weights kept (model-runtime 116 GB).
2026-10-12 10:22:03 -07:00
vh 0fe081042d memory: U13 seat-host 403 regression fixed 2026-10-12 09:44:12 -07:00
vh 82987828f8 fix(svos-front): kernel seat-host calls allowed on the terminal origin (app-terminal base_url reverted to it) 2026-10-12 09:44:06 -07:00
vh ca9e385316 memory: CI runner slowness diagnosed (ospool fsync ~145 ms on ana-docker); A/B/C pending Prime 2026-10-12 09:41:23 -07:00
vh a9bae04ba6 docs(pfi): D3 vs Intern-Decision-4B seat bench — final stashed findings (halted by operator call)
Measured: bf16 ladder across six sizes, calibrated 32k shape cells with
O(q*state) wire accounting, bs=1 memory/latency curve, Engine-vs-vendor
fidelity rows, Stage 4 control quant trio (battery monotone in weight error),
flash calibrated-NVFP4 with survive rules applied by the letter, and the
Amendment 2 GPU 1 budget (16,851 MiB) re-read against every fit clause.
Halted before the INC bench-copy arms and mini/27B/small-size NVFP4 arms;
stated plainly. Weights kept pending operator ruling (80 GB).
2026-10-12 08:08:39 -07:00
102 changed files with 4066 additions and 506 deletions
+1
View File
@@ -503,6 +503,7 @@ because SLAAC addresses rotate and a stale record is worse than none.
## Working rules ## Working rules
- **Standing push on `main` (Prime, 2026-10-13).** infra-ops pushes this repo's `main` to origin without asking each time, at natural boundaries (fast-forward only). Force-push, history rewrites on pushed commits and pushes to other branches still need Prime's per-op word. Commit with `git commit -- <paths>` (the index is shared), and check `git log origin/main..main` before pushing so you know whose commits ride along.
- **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy. - **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
- **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo. - **Never commit secrets.** Use `.env.example` templates; real `.env` files (with tokens, passwords) live on the server and are gitignored if/when this becomes a git repo.
- **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline. - **Surgical edits.** When fixing one stack, don't touch unrelated ones. Follow AIPA-MCP's CLAUDE.md rules about scope discipline.
+618
View File
@@ -15314,3 +15314,621 @@ latency-sensitive.
- `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`. - `[2026-09-26]` **zsh traps in ad-hoc test loops:** `set -- $a` does NOT word-split, so alert POSTs went out with empty fields; and `local path=…` inside a function CLOBBERS `$PATH` (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and `${=var}`.
_Archived 2026-10-11._ _Archived 2026-10-11._
## Recent decisions (archived)
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
# Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.
`[2026-09-29]` **Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.** A staged, inert `/opt/worldtree/config/defaults.yaml.staged-u11a-read_only` sits on corviduo-dev. Applying it needs Prime's go AND demo on 48bdf235 or later (demo ran 1b8746e6, which predates U11a). Apply = back up, copy over `defaults.yaml`, `docker restart`, then read the boot line `memory cutover: legacy plane read_only (writer on, reader on)`. **Once the window opens, infra-hermes runs one U8 batch per day and I own the wrapper and the escalations (proposed, worldtree-dev agreed):** on nh3-dev in `~/development/Worldtree`, at demo's deployed SHA, `python -m core.memory_acceptance.live --runs 3 --gate --legacy-mode read_only --label window`. The verdict is in `trace1.md`; FAIL goes to worldtree-dev the same day, never a retry; no git commit; never two at once. The wrapper is NOT built yet, deliberately: it waits for the go.
_Archived 2026-10-14._
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
# Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).
`[2026-09-28]` **Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).** Dry run then commit, both as `-u worldtree`: 5 filed (lofn 4, mimir 1), 16 `unslotted_pre_364` skipped, 0 aborted. **The first commit exited 1 because demo's hand-managed `model_roles.yaml` lacked the `memory_tagger` role**, so I synced it verbatim from the image's config-defaults (backup `.bak-pre-memory-tagger`). **2026-09-29 0854 presync for personal (Prime: "complete u10"):** personal got `memory_tagger` and the U9 `admin.memory.forget` policy delta. **Demo had also been missing that U9 policy delta since 09-24**, which left readonly-admin's `admin.*` rule with no exclusion for the destructive forget. Fixed, and demo's api was restarted to load it. **Personal COMMITTED 2026-09-29 1007** (b191 e2f74706, a snapshot-copy build; Prime's standing "complete u10" via worldtree-dev's go): 797 filed (mimir 795, lofn 2), mimir `dropped:too_large` 377, 1 unroutable held. The first personal dry run (b190) had crashed on 45 mimir rows with no vector (HNSW frozen since 08-06, #417, accepted by Prime until U11). Pinned (446e5807; Prime: "leave pinned") keeps its legacy chroma in the container's WRITABLE LAYER (`/app/agents/*/memory/.chroma`, 7 agents, 196K each): a recreate erases them.
_Archived 2026-10-14._
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
# Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.
`[2026-09-28]` **Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.** Positive control passed on tg128 (+1.1%). nvidia-smi's sw_power_cap flag never fires on this card, so "at cap" is judged from board draw. Follow-up (brokkr): moving PQ2_0 onto MMQ from batch 6, like Q4_K (mmvq.cu, `ne11 <= 5`), lifts N=8 to 1.21x, so the dip was partly a kernel threshold. The residual gap is not power. Runs are in `fv-ml1:/tank/spikes/bonsai-2026-09-28/runs{,-mmvq5}/` and on the Booth. The build image was removed; `Dockerfile.build` recreates it. **Acquired for keeps (Prime, 1540):** GGUF PQ2_0/PTQ1_0/mmproj-Q8_0 + the fork source pin in `/tank/aimodels/llm/prism-ml_Ternary-Bonsai-2-27B-gguf/` (runtime/), the MLX 2-bit pack in `/tank/aimodels/mlx/`, 34/34 hash-verified. Weights single copy (no snapshots, /tank not in restic); the fork pin also at `/mnt/smithy/runtime-pins/prism-llama.cpp-87268f77/` (brokkr).
_Archived 2026-10-14._
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
_Archived 2026-10-14._
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
_Archived 2026-10-14._
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
# hermes-gateway restarted 0401 for highseat-dev
`[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
_Archived 2026-10-14._
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
# `[2026-09-27]` SemIf live on fv-ml1 GPU 1 (Prime)
**semif-serve 0.1.2 is live** at `http://10.251.50.54:8032` (`semif.fv.internal`), stack `stacks/semif`,
code and contract `services/semif-serve/` (`069725c`). It is a FastAPI wrapper around SemIf-OpenJev's
direct and shared torch scorers (MIT, pinned `23cf1f39fc9534fe81437200959b6dfc7106e45a`) on
Qwen3.5-4B `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (BF16, offline HF cache on `/tank`).
Hosting assessment: infra-hermes, `/tmp/semif.md` (2026-09-25).
Prime's calls (2026-09-27): fv-ml1 GPU 1 under a hard VRAM cap (not the reserved-empty GPU 3, not a Q4
GGUF on a 16 GB box); infra-ops builds it with a light process (short contract → TDD → heid bug hunt);
no consumer named yet, so every score is labelled uncalibrated.
**Two defects that only the card showed**, each fixed with a test:
- **0.1.1:** the OOM path raised `OutOfMemory(...) from exc` inside the `except` block. The chain kept the
torch exception's traceback, and with it the failed call's frames and tensors, so 11.9 GiB stayed
allocated after the 503. The fix raises after the block, with no chain, and runs `gc.collect()`
before `empty_cache()`.
- **0.1.2:** a large shared request left torch's cache holding 12.6 GB, which left scriberr 3.5 GB on GPU 1.
Now, after each call, reserved memory above the post-warm-up baseline + 512 MiB is released
(`empty_cache`). It costs ~16 ms on a big request.
**Acceptance** (`services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`), against SemIf's
committed torch predictions on authored144:
- 142/144 same top choice; both misses are exact bf16 ties in our output;
- 144/144 identical prompt SHA-256;
- A-vs-A gap 0.0;
- negative control (rotated option descriptions) 14/144;
- shared vs direct 72/72;
- 21 binary criteria over one state in 159 ms.
The shared-mode capacity under 12 GiB is 52 decisions at a ~140-token prefix and 13 at ~3,900.
**Build:** torch 2.10.0+cu128 from the pytorch index (SemIf's own stack; sm_120 present). The
Dockerfile installs dependencies from a manifest with the project version blanked, so a version bump
reuses the ~4 GB torch layer (41 s rebuild, layer CACHED). See [[2026-09-27-semif-order-averaging]].
_Archived 2026-10-14._
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
# `[2026-09-27]` Blender 5.2 on fv-ml1 GPU 3, agent-driven (Prime)
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/blender/README.md`.)
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
→ `stacks/blender/README.md`
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
_Archived 2026-10-14._
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
_Archived 2026-10-14._
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
# `[2026-09-27]` Zigbee2MQTT on esh-docker-vm (Prime go-ahead; ha-dev request)
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/zigbee2mqtt/README.md`.)
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
added to mosquitto (passwd backup `.bak-20260927-z2m`).
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
→ `stacks/zigbee2mqtt/README.md`
_Archived 2026-10-14._
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
_Archived 2026-10-14._
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
## Method (applies to both)
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
- **Null control:** every case is also asked over a content-free state. It always picks the
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
15/21). Evidence conditions are read against that, not against 0.
- **Positive controls:** cases tagged `control`, unambiguous by construction.
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
set, so their scores are optimistic until checked on held-out real play.
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
there and every scene change went through ungated path B.**
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
it would be a gate/selector in front of the GM, not a replacement.
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
authored turns in the campaigns' register (the transcripts are gone from conv-api:
`session_not_found`). State = current scene + player action + narration.
| decision | wording | correct | real moves found | controls |
|---|---|---|---|---|
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
- Round 1 `place` failed its positive controls (no better than the blind null), so the
instrument, not the model, was broken. `place2` was written after seeing that.
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
far below the GM's reflect.
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
## Cicada: should she show emotion (a gate, not a mood ring)
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
said, run beside the LLM.
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|---|---|---|---|---|
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
definition.** Terse yes/no options fail; descriptive ones work.
- Adding the earlier lines of the conversation changed no scored answer.
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
ordinary.
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
nothing to the turn if it starts at end of utterance.
## Recommendations (surfaced to Prime; nothing built)
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
but it does override the model's affect. That is Prime's ruling to make, and the code would
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
play shows churn, after labelling ~50 real turns as a held-out set.
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
**on GPU 1, the same GPU as semif**.
**Answer: no on both counts.**
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|---|---|---|
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
~31 ms faster than today. That is the ceiling.
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
ordering (pose + gesture ~110–136 ms).
| works (clear cases) | today (A) | SemIf (live config) |
|---|---|---|
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
| controls | 12/12 | 12/12 |
| egregious | 0 | 0 |
| arcs acceptable | **40/45** | 25/45 |
| mood carries through a mundane follow-up | **14/15** | **7/15** |
| distinct poses used | 10 | 9 (no collapse to neutral) |
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
"someone's trying the back door" → suspicious, jokes → curious or neutral.
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
agreement of 83%, so this is a different judgment and not noise.
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
the evidence.
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
lines.
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
isn't there.
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
503. Both are now in `stacks/semif/README.md`.
_Archived 2026-10-14._
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
## Method (applies to both)
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
- **Null control:** every case is also asked over a content-free state. It always picks the
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
15/21). Evidence conditions are read against that, not against 0.
- **Positive controls:** cases tagged `control`, unambiguous by construction.
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
set, so their scores are optimistic until checked on held-out real play.
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
there and every scene change went through ungated path B.**
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
it would be a gate/selector in front of the GM, not a replacement.
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
authored turns in the campaigns' register (the transcripts are gone from conv-api:
`session_not_found`). State = current scene + player action + narration.
| decision | wording | correct | real moves found | controls |
|---|---|---|---|---|
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
- Round 1 `place` failed its positive controls (no better than the blind null), so the
instrument, not the model, was broken. `place2` was written after seeing that.
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
far below the GM's reflect.
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
## Cicada: should she show emotion (a gate, not a mood ring)
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
said, run beside the LLM.
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|---|---|---|---|---|
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
definition.** Terse yes/no options fail; descriptive ones work.
- Adding the earlier lines of the conversation changed no scored answer.
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
ordinary.
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
nothing to the turn if it starts at end of utterance.
## Recommendations (surfaced to Prime; nothing built)
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
but it does override the model's affect. That is Prime's ruling to make, and the code would
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
play shows churn, after labelling ~50 real turns as a held-out set.
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
**on GPU 1, the same GPU as semif**.
**Answer: no on both counts.**
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|---|---|---|
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
~31 ms faster than today. That is the ceiling.
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
ordering (pose + gesture ~110–136 ms).
| works (clear cases) | today (A) | SemIf (live config) |
|---|---|---|
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
| controls | 12/12 | 12/12 |
| egregious | 0 | 0 |
| arcs acceptable | **40/45** | 25/45 |
| mood carries through a mundane follow-up | **14/15** | **7/15** |
| distinct poses used | 10 | 9 (no collapse to neutral) |
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
"someone's trying the back door" → suspicious, jokes → curious or neutral.
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
agreement of 83%, so this is a different judgment and not noise.
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
the evidence.
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
lines.
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
isn't there.
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
503. Both are now in `stacks/semif/README.md`.
_Archived 2026-10-14._
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
# `[2026-09-27]` SemIf order-averaging: spiked, Prime ruled build + fast-kernel trial
**Finding (Prime's probes):** SemIf's single-ordering answer leans toward whichever option is listed
first on ambiguous inputs. For "It's 2AM and I'm bored", the top pick flipped from casual 0.685 to booty
call 0.760 when the order was reversed.
**Spike** (`739aa03`, `services/semif-serve/spike/`, no service change). SemIf authored144 +
perturbations108, 252 rows in 72 groups, 3 options each, with all 6 orderings of every row sent in one
shared request:
| method | accuracy |
|---|---|
| single ordering, as sent | 78.6% |
| expected single ordering | 80.0% |
| 3 rotations, log-mean | **87.7%** (+9.1 pts, group-bootstrap 95% CI +4.7..+13.8) |
| all 6 orderings, log-mean | 88.1% |
Rows where the rotations agree unanimously: 161 rows, 94.4% accurate. Split rows: 91 rows, 75.8%.
The service is deterministic, so the CI measures item sampling, not run noise. This is one task family
(evidence interpretation, 3 options), not our workload.
**Latency** (`spike/latency-2026-09-27.txt`, from nh3-dev, network floor 31 ms):
- `/decide` short: 71 ms end to end (38 ms server);
- 3 rotations shared: 113 ms (79 ms);
- 6 orderings: 137 ms (99 ms);
- a ~2k-token state: 210 ms (169 ms).
**Prime ruled (2026-09-27 ~0250): build it in, and try the fast kernels too.**
- Averaging design: opt-in `orderings: rotations|all` (all only ≤ 4 options); all orderings in one shared
batch; per-ordering SemIf results returned unchanged; a `combined` block (log-mean, top, agreement,
spread); orderings count toward `max_decisions`.
- Fast kernels: `causal_conv1d` + `flash-linear-attention` are missing, so transformers runs Qwen3.5's
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
won the A/B on the empty GPU 3:
- parity with upstream 144/144 (was 142/144);
- a ~2k-token `/decide` 169 → 92 ms;
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
_Archived 2026-10-14._
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
# `[2026-09-27]` restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)
`resticprofile schedule` copies `env-file` values into the generated units under /etc/systemd/system
(0644). So `RESTIC_REPOSITORY`, including the rest-server basic-auth password, was readable by any local
user on every env-file host (found on esh-docker-vm; `systemctl cat` works without sudo). Blast radius:
the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server
is `--append-only`, and the passphrase is a separate file).
**Fix** (`6e203dc`, `30f2c97`): `playbooks/restic-repository-file.yaml` does four things:
- derives `/etc/restic/repository` (root 0400) from `restic.env`;
- uploads the profile switched to `repository-file`, guarded by the live profile's pre-change sha;
- runs `cat config` through the new profile;
- re-schedules, then verifies the units exist and contain no `rest:http`.
Applied to ana-docker, fv-ml1 (`configs/restic/ana-ml2`), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev,
nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An
independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup
(`a29b889d`). Every host's URL and passphrase are vaulted as `<host>/etc/restic/{repository,password}`.
- `restic.env` is **kept** (root 0600): the per-host READMEs and the freshness probe source it. **A rotation
must update the vault, `restic.env` and `repository`.**
- The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but
does not un-expose what was readable.
- Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were
pulled in first (`c698751`).
## In-flight state at retirement (moved here at the 2026-10-06 snapshot)
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
**a rotation must update the vault, `restic.env` and `repository`.**
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
and vaulted.**
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
(done ~0140–0200).
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
_Archived 2026-10-14._
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
_Archived 2026-10-14._
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
_Archived 2026-10-14._
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
_Archived 2026-10-14._
## Tried and abandoned (archived)
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
# Shorter Parakeet slices as Scriberr's memory fix
`[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
_Archived 2026-10-14._
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
_Archived 2026-10-14._
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
_Archived 2026-10-14._
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
_Archived 2026-10-14._
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
_Archived 2026-10-14._
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
`[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
_Archived 2026-10-14._
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
# A TCP connect as the "is Blender ready" probe
`[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
_Archived 2026-10-14._
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
_Archived 2026-10-14._
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
_Archived 2026-10-14._
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
`[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
_Archived 2026-10-14._
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
_Archived 2026-10-14._
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
_Archived 2026-10-14._
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
# Relying on the build cache surviving on esh-ml1
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
_Archived 2026-10-14._
@@ -0,0 +1,9 @@
# eshpfi configs/nemi-vault (infra-ops, 2026-10-14): keeps /srv/nemi/vault, the read-only copy of Prime's notes
# (gitea vh/VuongKB, which he syncs with git), current for Nemi's `vault` source (albok unit 4.4 ruling 6).
# A lkraven USER unit: the clone uses lkraven's gitea SSH key. Fast-forward only: the copy is never written to,
# so a non-ff (a force-push upstream) fails loudly here instead of being papered over.
[Unit]
Description=Pull Prime's notes (vh/VuongKB) into /srv/nemi/vault for Nemi
[Service]
Type=oneshot
ExecStart=/usr/bin/git -C /srv/nemi/vault pull --ff-only --quiet
+8
View File
@@ -0,0 +1,8 @@
# Hourly at :50, ahead of albok-nemi.timer's walk at ~:01.
[Unit]
Description=Hourly pull of Prime's notes for Nemi
[Timer]
OnCalendar=*-*-* *:50:00
Persistent=true
[Install]
WantedBy=timers.target
@@ -16,6 +16,9 @@ EnvironmentFile=/etc/svos/env/terminal.env
Environment=SVOS_KERNEL_URL=https://svos-kernel.phasefinal.com Environment=SVOS_KERNEL_URL=https://svos-kernel.phasefinal.com
Environment=SVOS_DATA_ROOT=/var/lib/svos Environment=SVOS_DATA_ROOT=/var/lib/svos
Environment=SVOS_APP_LISTEN=127.0.0.1:8404 Environment=SVOS_APP_LISTEN=127.0.0.1:8404
# zellij is lkraven's cargo install (~/.cargo/bin), which a user unit's default PATH does not reach. Without it every
# `zellij` call fails to launch: /health stays "degraded" and dump/inject cannot run (found 2026-10-13, svos-dev's report).
Environment=PATH=/home/lkraven/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ExecStart=/opt/svos/current/apps-env/bin/svos-terminal ExecStart=/opt/svos/current/apps-env/bin/svos-terminal
[Install] [Install]
WantedBy=default.target WantedBy=default.target
+10 -1
View File
@@ -168,7 +168,7 @@ https://svos-terminal.phasefinal.com:443 {
# The kernel's signed calls come from the box itself (its source on the mesh address); anything else is refused. # The kernel's signed calls come from the box itself (its source on the mesh address); anything else is refused.
@not_from_box { @not_from_box {
path /verbs/* /health path /verbs/* /health /seats/*
not remote_ip 100.64.0.9 10.100.50.43 127.0.0.1 not remote_ip 100.64.0.9 10.100.50.43 127.0.0.1
} }
handle @not_from_box { handle @not_from_box {
@@ -211,6 +211,15 @@ https://svos-terminal.phasefinal.com:443 {
handle @health { handle @health {
import svos_terminal_upstream import svos_terminal_upstream
} }
# U13 fix (2026-10-12): app-terminal's registry base_url is this origin again (browsers frame panes from it),
# so the kernel's signed seat-host calls (U5b) come through here too, from the box only (@not_from_box above).
@seat_host {
method POST
path /seats/observe /seats/check /seats/launch /seats/retire-launch
}
handle @seat_host {
import svos_terminal_upstream
}
handle { handle {
respond 404 respond 404
} }
+4
View File
@@ -0,0 +1,4 @@
# lkraven user tmpfiles: the Worldtree memory gate keeps each batch's harness scratch (scripts/wt-memory-gate-batch
# --keep-data, worldtree-dev 2026-10-17); retention 14 days, cleaned by the user systemd-tmpfiles-clean.timer.
# Install: cp to ~/.config/user-tmpfiles.d/ and `systemctl --user enable --now systemd-tmpfiles-clean.timer`.
d %h/.local/state/wt-memory-gate-batch/scratch 0700 - - 14d
+5
View File
@@ -1,5 +1,10 @@
# services.yaml — contract for downstream consumers # services.yaml — contract for downstream consumers
> **RETIRED 2026-10-17 (Prime).** asset-engine, this catalog's only consumer, is `compose down` on ana-docker.
> The catalog is NOT maintained: every irv-ml1 endpoint in `services.yaml` still points at `10.100.79.3`, the wg0
> lifeline that died at the 09-06 mesh cutover, and several of those ports now belong to other services. Kept as
> the record and for a bring-back, which would need a full per-entry refresh first.
`services.yaml` in this directory is the **canonical, versioned** `services.yaml` in this directory is the **canonical, versioned**
description of the inference services this fleet exposes. It is a description of the inference services this fleet exposes. It is a
first-class artifact: external tools (the asset-engine UI is the first-class artifact: external tools (the asset-engine UI is the
-1
View File
@@ -17,7 +17,6 @@ their own:
```sh ```sh
curl -s http://10.0.50.45:5100/api/services # every fleet service, curated curl -s http://10.0.50.45:5100/api/services # every fleet service, curated
curl -s http://10.250.50.70:8200/api/v1/services # inference seats + GPU state
curl -s http://10.250.50.70:4000/v1/models # the LLM roster (needs a key) curl -s http://10.250.50.70:4000/v1/models # the LLM roster (needs a key)
``` ```
+14 -6
View File
@@ -14,12 +14,11 @@ directory; that defeats the point.
## Rule zero: query live inventories, never a written list ## Rule zero: query live inventories, never a written list
Three endpoints answer "what exists right now" better than any document, and they stay Two endpoints answer "what exists right now" better than any document, and they stay
correct on their own. **Reach for these before asking anyone:** correct on their own. **Reach for these before asking anyone:**
```sh ```sh
curl -s http://10.0.50.45:5100/api/services # every fleet service, curated curl -s http://10.0.50.45:5100/api/services # every fleet service, curated
curl -s http://10.250.50.70:8200/api/v1/services # inference seats + GPU state
curl -s -H "Authorization: Bearer $KEY" \ curl -s -H "Authorization: Bearer $KEY" \
http://10.250.50.70:4000/v1/models # the LLM roster (~40 models) http://10.250.50.70:4000/v1/models # the LLM roster (~40 models)
``` ```
@@ -45,6 +44,15 @@ live contract for its API. Fetch it rather than trusting a transcription.
*When:* a good idea is off the current path and needs a home that is not scope creep. *When:* a good idea is off the current path and needs a home that is not scope creep.
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/henge.md` *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/henge.md`
## Knowledge
- **Albok** — the fleet knowledgebase at `http://albok.nh3.internal:8392`: every repo's persistent
memory, the stash, and the library (books by chapter, papers, concepts). `albok` CLI; `albok-mcp`
for Claude Code.
*When:* before you decide something, check whether it was already decided, tried or measured,
anywhere in the fleet.
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/albok.md`
## Credentials ## Credentials
- **secret** — Vaultwarden-backed credential store. `secret get|put|list|backfill`. - **secret** — Vaultwarden-backed credential store. `secret get|put|list|backfill`.
@@ -61,13 +69,13 @@ live contract for its API. Fetch it rather than trusting a transcription.
⚠ *The shared key CAN spend real vendor credits.* ⚠ *The shared key CAN spend real vendor credits.*
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/litellm.md` *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/litellm.md`
- **Direct inference seats + Asset Engine** — the control plane at - **Direct inference seats** — most sit behind the gateway; the GPU hosts and their traps.
`http://10.250.50.70:8200` enumerates seats, reads GPU state, and starts/stops them. (The Asset Engine control plane at `:8200` was RETIRED 2026-10-17.)
*When:* the gateway does not expose what you need, or a seat is down. *When:* the gateway does not expose what you need, or a seat is down.
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/inference.md` *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/inference.md`
- **Speech (TTS/ASR)** — Breeze-TTS-2 is the fleet primary; Chatterbox-fast is the - **Speech (TTS/ASR)** — Breeze-TTS-2 is the fleet primary; Chatterbox-fast is the
low-latency chat front end; Speaches and Parakeet do ASR. low-latency chat front end; Parakeet does ASR (`whisper-1` / `ext-stt` on the gateway).
*When:* you need audio out of text, or text out of audio. *When:* you need audio out of text, or text out of audio.
*Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/speech.md` *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/speech.md`
@@ -117,7 +125,7 @@ live contract for its API. Fetch it rather than trusting a transcription.
## Claude-Code-only ## Claude-Code-only
- **ratecheck, remote-ssh MCP, skills** — harness-dependent; a - **ratecheck, remote-ssh MCP, albok MCP, skills** — harness-dependent; a
non-Claude agent cannot use these. Listed so you recognise a reference rather than non-Claude agent cannot use these. Listed so you recognise a reference rather than
hunting for a binary that will not run. Includes **`/handoff`, the supported way to hunting for a binary that will not run. Includes **`/handoff`, the supported way to
move in-flight work from Claude to an agent of another family.** move in-flight work from Claude to an agent of another family.**
+59
View File
@@ -0,0 +1,59 @@
# Albok — the fleet knowledgebase
**Service:** `http://albok.nh3.internal:8392` (= `10.100.50.44:8392`, albok-service on nh3-albok)
**CLI:** `albok` · **MCP (Claude Code):** `albok-mcp`, registered at user scope · **Owner:** `albok-dev` (althing)
One searchable store of what the fleet knows: every repo's `persistent-memory.md` (entries, detail
files, archives), the stash, and the library (books by chapter, papers, extracted concepts). Search
is hybrid (vector + keyword) and LLM-free.
## Access
Every nh3-dev shell carries `ALBOK_SERVICE` and `ALBOK_TOKEN` — the shared **`fleet-agents` read
token**: `fleet/*` (memory, stash) and `library/*` (fiction, nonfiction). It reads no personal wing.
A shell older than 2026-10-16 may lack them; open a new one. A handle that needs more grants asks
infra-ops for its own token.
```sh
albok status # the wings this token reads, and their states
albok search "why did we drop the reset endpoint" # --wing fleet/memory --top-k 20 --facet key=value
albok read <object_id> # pristine bytes to stdout; lineage notes on stderr
albok sidecar <object_id> # provenance: repo, file, commit, section, entry date
albok related <object_id> # links, similar, same source, shared concepts — each with its reason
```
Claude Code sessions get the same verbs as tools (`albok_search`, `albok_read`, `albok_sidecar`,
`albok_related`, `albok_status`, …). Codex and Grok use the CLI.
## Reading the answers right
- **Never threshold on score.** The null floor (~0.031) sits just under real hits (~0.033); rank, don't cut.
- **Coverage is reported, read it.** A reply lists `searched`, `locked` and `degraded` wings: an
empty result over a degraded wing is not "nothing exists". An unreachable service is an
outage, never an empty result (the client says which).
- **Hidden by default:** objects classed `adult`, `personal` or `secret` are hidden from every
reader; `--include CLASS` opts back in only where the token holds that class grant.
- **Supersession is declared, not inferred.** Every object has a permanent `uid` (in every search hit).
A read of an entry names what supersedes it (`superseded by …`); an empty lineage means "none
declared", not "current". When your new memory entry replaces an older one, end it with
`Supersedes [[<uid>]]` (the old entry's uid; a detail file may be named by its stem) and leave
the old entry as written.
## Writing into it (nobody writes to the service directly)
Nemi, albok's ingester, walks every source hourly on nh3-dev (a user timer, a few minutes past the hour):
| You want to add | Do this |
|---|---|
| a decision, lesson or state for a repo | write it in that repo's `persistent-memory.md` under `~/development/` (Nemi reads the file on disk; commit it as usual) |
| raw material (a transcript, an article) | `/stash` (Claude) or a file in `~/stash/` with its frontmatter |
| a note for the operator's personal inbox (a personal wing: the fleet token cannot read it back) | a markdown file in `/srv/nemi/inbox/` |
| a book | `scp book.epub lkraven@nh3-dev:/srv/nemi/books/fiction/` (or `nonfiction/`): validated, then shelved into `~/library/` and ingested the same hour, or moved to `rejected/` beside a reason file (`not_epub`, `drm`, `duplicate`, …); `LOG` records every decision |
## When to reach for it
Before you decide something, check whether it was already decided, tried or measured, in this repo
or any other. Before you research a game-design, cognition or fiction question, check the library.
*Detail:* `~/development/albok/README.md`, `packages/albok-client/README.md` (CLI and MCP),
`packages/albok-nemi/README.md` (sources, the book drop). Health: `scripts/albok-check` in that repo.
+11 -6
View File
@@ -42,13 +42,18 @@ stdout a non-tty and sudo skips the PTY.
regexes and `bash -c` walks past it. `SSH_MCP_STRICT_ALLOWLIST=1` makes regexes and `bash -c` walks past it. `SSH_MCP_STRICT_ALLOWLIST=1` makes
`~/.config/remote-ssh-mcp/config.json`'s `allowedHosts` authoritative. `~/.config/remote-ssh-mcp/config.json`'s `allowedHosts` authoritative.
## task-board MCP ## albok MCP
`task_start` / `task_update` / `task_wait` / `task_complete` to surface work in `albok-mcp`, registered at **user** scope (2026-10-16: `claude mcp add --scope user albok -- /home/lkraven/.local/bin/albok-mcp`).
progress at `http://10.250.50.70:7878/`. When you launch background shells, call Tools `albok_search` / `albok_read` / `albok_sidecar` / `albok_related` / `albok_status` / …, the same verbs as the
`task_set_shells` with one short description each (and again with `[]` when they `albok` CLI (see `albok.md`). It reads `ALBOK_SERVICE` and `ALBOK_TOKEN` from the environment the session inherits
finish) — the board turns orange so the operator can tell the session is parked on (`~/.zshenv` sources `~/.config/secrets/env.sh`); no token is written into `~/.claude.json`. A session started
background work rather than stalled on them. from a shell older than that wiring gets a "no token" error from the tools: start a new session.
## task-board MCP — MOTHBALLED (2026-09-24)
Superseded by the High Seat (SVOS) and ledger; the container is stopped. **Do not call `task_*` tools** or
wire its hooks into new sessions. Its data and stack are kept for a one-command bring-back (eshpfi CLAUDE.md).
## Skills ## Skills
+9 -19
View File
@@ -1,4 +1,4 @@
# Direct inference seats + the Asset Engine control plane # Direct inference seats
Most seats are vLLM behind the LiteLLM gateway — prefer the gateway Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
(`docs/fleettools/litellm.md`) unless you need something it does not expose. (`docs/fleettools/litellm.md`) unless you need something it does not expose.
@@ -8,30 +8,20 @@ Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
```sh ```sh
# every fleet service, curated, with hrefs and descriptions # every fleet service, curated, with hrefs and descriptions
curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
# the LLM roster
# the inference control plane: what exists, what is up, GPU state curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models
curl -s http://10.250.50.70:8200/api/v1/services
curl -s http://10.250.50.70:8200/api/v1/gpus
``` ```
Seats move between hosts and GPUs often enough that any transcribed table here would Seats move between hosts and GPUs often enough that any transcribed table here would
mislead within a month. mislead within a month.
## Asset Engine — `http://10.250.50.70:8200` ## Asset Engine — RETIRED 2026-10-17
Control plane over the PFI inference fleet. Can enumerate services, read GPU state, The control plane at `http://10.250.50.70:8200` is gone (Prime). Its last job was 2026-08-02, and every
**start and stop seats**, submit jobs and fetch assets. catalog endpoint pointed at irv-ml1's dead wg0 address after the 09-06 mesh cutover. `docker compose
down` on ana-docker; the DB, image, repo (`vh/asset-engine`) and catalog (`docs/asset-engine/`) are kept, so
``` `docker compose up -d` in `/opt/docker/compose/asset-engine` brings it back. Start/stop a seat by hand
GET /api/v1/services GET /api/v1/services/{id}/state (`docker compose up -d` / `stop` in its compose dir on the GPU host).
POST /api/v1/services/{id}/start POST /api/v1/services/{id}/stop
GET /api/v1/gpus POST /api/v1/jobs
GET /api/v1/assets/{id}/blob GET /api/v1/presets
```
⚠ **Liveness is keyed on the compose PROJECT name.** A seat reads OFFLINE unless the
compose project name equals the catalog's `lifecycle.stack`. Pin `name: <stack>` in
the compose file.
## GPU hosts ## GPU hosts
+9 -3
View File
@@ -26,9 +26,15 @@ the router cooldown** — list voices first.
| service | endpoint | what it is | | service | endpoint | what it is |
|---|---|---| |---|---|---|
| Speaches | `http://irv-ml1.nh3.internal:8204` | OpenAI-compatible faster-whisper, `verbose_json` | | **Parakeet (NeMo)** | `http://10.251.50.54:8300` | Parakeet-unified-en, NeMo torch bf16, fv-ml1 GPU 0 — **fleet primary**. OpenAI-compatible `/v1/audio/transcriptions`, plus `/transcribe`; liveness is `/healthz` (not `/health`) |
| Parakeet | `http://10.251.50.54:8300` | Parakeet-TDT via sherpa-onnx (fv-ml1 GPU 3) | | via LiteLLM | `http://10.250.50.70:4000/v1` | aliases `ext-stt` and `whisper-1` both route to the Parakeet seat above |
| Scriberr | `http://10.250.50.54:8080` | transcription + diarization UI (ana-ml2 GPU 1) | | VibeVoice streaming ASR 1.5B | `http://10.100.50.80:8031` | audio.cpp on nh3-ml1, streaming; built for the dataset foundry, direct (no gateway) |
| Scriberr | `http://10.251.50.54:8080` (`https://scriberr.nh3.phasefinal.com`) | transcription + diarization UI, fv-ml1 |
Retired, verified 2026-10-17: **Speaches** (irv-ml1 :8204 is now Breeze-TTS-2), the sherpa-onnx `parakeet` container
on fv-ml1 (stopped; NeMo replaced it), and the asset-engine `parakeet` entry at `10.100.79.3:8765` (the dead wg0
lifeline address; asset-engine itself was retired 2026-10-17). The first two's leftover containers were removed 2026-10-17 (images and compose dirs kept), so
Homepage no longer lists them.
## Traps ## Traps
@@ -0,0 +1,192 @@
# D3 vs Intern-Decision-4B — decision-model seat bench (2026-10-12)
Ordered by Prime; preregistration `brokkr-smithy`
`docs/surveys/vllm-sr-decision-landscape-2026-10-11/d3-bench-prereg.md`
(c93d7b0, amended b87a7ab), plus Amendment 1 (shape calibration, brokkr) and
Amendment 2 (operator: memory budget = what fits on the seat's GPU, e3dbfac).
Run by infra-hermes on fv-ml1 GPU 3. **HALTED by operator call at 15:07Z** —
"fails on a number of levels… stash the findings and call it." This doc is the
stash: everything below was measured; what was *not* run is stated plainly.
Harness: jevbench v1.2.16 (public 231 = easy/original/hard), the S1 battery
(authored144 + perturbations108 + cicada W1/W2 + Wyrd, 1,748 paired scorable
rows), and the 10-04 shape-runner lineage with a per-model calibrated 32k
builder. All artifacts: `fv-ml1:/tank/bench/arm-*` and `/tank/bench-jev-2026-10-04`
(the latter untouched, lkraven-owned).
## Headline
The incumbent (intern-decision-serve 0.1.3, pinned Intern-Decision-4B) is not
displaced. Every d3 arm that clears the incumbent's accuracy needs either a
GPU budget the seat cannot hold at the worst shape, or fails the latency
contract, or both. The one arm inside the seat's own memory budget at the
worst shape (d3-edge at bs=1) loses 8 battery points and is 28x slower than
INC at 32k-class state. The quantization warrant closes: NVFP4 does not buy
the 32k shape, because the 32k problem is O(q x state) activation and protocol
accounting, not weight bytes.
## 1. bf16 ladder (Engine primary, `vllm-sr` dev CLI, GPU 3)
Revisions pinned per arm (edge e51ab425…, lite c91d55d6…, nano ccd7bc1a…,
mini 40391eb6…, flash 06d33616…, d3-27 7f257ad7…). jevbench acc = public 231,
median of 3 repeats (all repeats bit-exact per arm). Battery = labelled-evidence
acc, single condition. Peak VRAM = steady-state serve MiB. p50 = harness per-call
median ms.
| arm | params | jevbench | battery single | peak MiB | p50 ms |
|---|---|---|---|---|---|
| **INC (baseline)** | 4B | **202** | .910 | 8,822 (32k shape) | ~1.9–2.0s at 32k |
| d3-edge | 0.59B | 158 | .749 | 2,398 | 15 |
| d3-lite | 1.4B | 168 | .852 | 2,676 | 28 |
| d3-nano | 3.8B | 175 | .907 | 5,304 | 29 |
| d3-mini | 7.3B | 188 | .945 | 10,540 | 36 |
| d3-flash | 8.39B | 190 | .947 | 17,600 | 37 |
| d3-27 (vendor path) | 26.09B | 206 | .970 | 50,300 | — |
INC reproduction: 202/231 x3, bit-exact against the 10-04 record (`out/intern-native`).
d3-27 is the only arm that beats INC on the public suite, and it is a path
divergence: the Engine's placement estimator (4B/param + 2 GiB) refuses a 26B
model on any single card we have; the vendor `d3_server.py` runs it at 50.3 GiB.
Gate (jevbench >= INC - 3): PASS at 27B only.
Argmax cross-check (vendor vs Engine, same snapshot, 231 items): edge 230,
lite 229, nano 229, mini 231, flash 231 — mismatches are borderline near-ties
(max prob gap 2.7e-2). Engine fidelity: argmax identical on all non-borderline
items; probabilities NOT exact (median gap 4.2e-4). The card's "matches exactly"
claim is false as written; argmax-equivalence holds.
## 2. 32k shape cells (Amendment 1: per-model calibrated to ±2% of 32,768 realised tokens)
Builder length iterated per model in its own tokenizer; every arm converged to
32,743 realised tokens for the 1q state. 16q cells carry 16 x 32,743 + framing =
**523,900 wire tokens** because the d3 wire protocol counts state once PER
QUESTION — O(q x state) accounting, measured, not predicted. INC's equivalent
is a single forward over one state. cold = first request after ready; launch to
ready wall time is reported separately below and is NOT folded into cold.
| arm | crit21_short cold ms | 1q32k cold ms | 16q32k cold ms | 16q32k peak MiB |
|---|---|---|---|---|
| INC | (2-call 16+5, see note) | 1,970 (32,715 tok) | **rejected: >32,768 cap at 33,324; maxfit 16q = 32,708 tok, 1,960 ms** | 8,822 |
| d3-edge | 1,242.6 | 895.4 | 20,790 | 28,404 |
| d3-lite | 89.1 | 1,139.5 | 18,668 | 28,938 |
| d3-nano | 103.3 | 1,442.1 | 23,568 | 33,486 |
| d3-mini | 157.1 | 3,565.8 | **deadline-bound** (503 at 30.002s, all retries) | 71,404 |
| d3-flash | 242.3 | 4,469.5 | **deadline-bound** | 80,396 |
| d3-27 vendor | 642.7 | 8,731.0 | 147,724 (366,172 tok, builder cap) | 96,262 |
Notes. (a) INC crit21 on 10-04 is a client-side 16+5 split summed across two
`/v1/systemone` calls (`bench_shape.py --max-questions 16`); `/v1/systemone`
enforces 1..16 questions, so a 21-question crit is structurally out of contract.
(b) The Engine's 30 s serving deadline is a configuration of the seat, not a
bug; the wall sits between 3.8B (nano completes at 23.6s) and 7.3B (mini).
(c) Activation peaks include 32k state at bs=8 (Engine BATCH_SIZE is hard-coded
8, `family.py:65`); every bs=8 arm exceeds even Amendment 2's budget at 16q32k.
(d) Latency clause (16q32k <= 1.5x INC): INC's honest 32k-class cell is 1.96s,
so the clause bar is ~2.9s. NO d3 arm meets it; closest is edge-b1 at 13.0s (6.6x).
## 3. bs=1 curve (vendor `--batch-size 1`, the only memory knob)
| arm | 16q32k peak MiB (bs=8 -> bs=1) | 16q32k cold ms (bs=8 -> bs=1) |
|---|---|---|
| d3-edge | 28,404 -> 4,778 | 20,790 -> 13,019 |
| d3-mini | 71,404 -> 18,514 | deadline -> 53,770 |
The knob buys memory, never time: latency is protocol-intrinsic O(q x state).
No setting puts a mini-class model inside the seat at the worst shape (mini-b1
still misses Amendment 2's budget by 1.6 GiB *and* fails latency at 18x INC).
## 4. Stage 4 — control quant trio and the battery sensitivity curve
d3-lite backbone, RTN (round-to-nearest, no calibration), vendor path,
`--verify none` (the manifest hashes the main shard; modified packages can't
self-verify). Paired deltas vs the path-matched bf16 vendor baseline, 1,748
rows, 95% bootstrap CI, Wyrd split included:
| arm | weight rel-L2 | jevbench | battery | paired delta [CI] | Wyrd delta [CI] | argmax flips |
|---|---|---|---|---|---|---|
| lite bf16 vendor (control) | 0 | 168 | .852 | — | — | — |
| lite NVFP4 RTN | 9.4% | 164 | .869 | +0.0034 [-0.0086, +0.0154] | -0.019 [-0.055, +0.017] | 117 |
| lite INT3 RTN | 19.9% | 158 | .739 | -0.095 [-0.118, -0.072] | -0.131 [-0.176, -0.088] | 476 |
The battery gate is monotone in weight error and functional: it cannot see 9.4%
(NVFP4 RTN lands inside noise), it clearly sees 19.9% (INT3 collapses). This is
the calibration warrant's premise confirmed: a quant that survives must move
weights *less* than 9.4% AND keep paired deltas inside the band — i.e. the
headroom to demonstrate is measured, and it is small.
## 5. Stage 5 — flash calibrated NVFP4 (the one arm completed before the halt)
ModelOpt `NVFP4_DEFAULT_CFG`, calibrated on the shared 512-row corpus (sha256
e7aa7b18…a31e, seeded `jev-nvfp4-calib-2026-10-12`, no scored item). Served
in-process over a `/v1/systemone` shim (`stage5_serve.py`, prepare/run/respond
mirroring the vendor contract, 1,405 quantizers): ModelOpt's fake-quant lives
in the quantizer forward, so no package rewrite; accuracy is a true fake-quant
read, memory is analytic + the byte table below.
jevbench 194 x3 (bf16 Engine control 190). Survive rules applied by the letter,
paired vs `arm-d3-flash`, n=1,748:
| rule | result | verdict |
|---|---|---|
| (a) battery pooled delta, CI lower bound >= -0.02 | -0.0011 [-0.0086, +0.0057] | PASS |
| (b) public suite within ±3 of bf16 median | +4 (194 vs 190) | **FAIL by the letter** (upward; 10 flips: 6 up, 2 down, 2 both-wrong) |
| (c) Wyrd within -0.02 | +0.0071 [-0.0119, +0.0262] | PASS |
| (d) argmax flips <= 4 of 231, both directions | 10 | **FAIL by the letter** |
Byte table (bf16 snapshot index, total 15.63 GiB on disk): quantized linears
(attn 3,980 + mlp 9,216 MiB) at 4.5 b/w = 0.28125x -> 3,711 MiB; kept wide
(embed+head 1,940, vision 870, other 2) = 2,812 MiB. **Projected export 6,524
MiB (~6.37 GiB)** — single-digit, as expected. ModelOpt `export_hf_checkpoint`
exists for a measured-export ground truth; not run before the halt.
Latency of the fake-quant arm is dequant-matmul timing and is NOT reported as a
speedup. **Analytic 32k verdict unchanged: quantization does not touch the
O(q x state) activation cost; the 16q32k cell fails identically at any weight width.**
## 6. Amendment 2 — the seat GPU budget (measured read-only, GPU 1)
Prime: "our budget is now whatever fits in the current gpu where it's running."
- GPU 1 total: **97,887 MiB.**
- Non-seat processes: `vllm-meromero-r` 43,994 MiB + `vllm-gen-small` 34,994 MiB = 78,988 MiB.
- Seat (intern-decision) resting: 8,812 MiB.
- Safety margin: 2,048 MiB (brokkr's suggested figure; no seat headroom policy exists).
- **Budget: 16,851 MiB (~16.5 GiB).**
Re-read of every fit clause against 16,851 MiB:
- Short shape: fits INC-class (8,822 measured) and every arm through mini
(10,540); NVFP4-flash analytic ~6,524 fits; flash bf16 (17,600) misses by
749 MiB; 27B (50,300) does not.
- 16q32k: only edge-b1 (4,778) and INC-class fit. Every bs=8 arm is 27.6–96.3
GiB. The latency clause independently kills mini/flash/INC at that shape.
- Net: the wider budget upgrades mini from "memory-out" to "memory-in,
latency-out" (53.8s at the only knob setting that fits), and leaves the
conclusion unchanged.
## 7. What was NOT run because of the halt
- INC bench-copy arms (positive control `inc-bf16-copy` and fake-quant
`inc-fakequant`, 249 language tensors RTN-NVFP4, median rel-L2 9.4%, vision
kept bf16): the 10-04 `intern_server.py` shim and both packages were built
and the arm was mid-boot when the halt landed. The INC fake-quant row is
therefore absent — note the production engine cannot load NVFP4 at all
(`intern_decision_serve/engine.py` pins the snapshot's `inference.py` by hash
and validates dtype against {bfloat16, float16, float32} with an explicit
raise; no compressed-tensors path exists in the image).
- NVFP4 arms for mini, 27B (fake-quant), nano, lite (calibrated), edge.
- ModelOpt measured export (ground-truth on-disk size).
- Paired battery deltas + Wyrd splits for the bf16 ladder arms (no quantized
sibling to pair against; would need battery re-runs).
## 8. Teardown & disk
Engine stack stopped, all bench servers on GPU 3 stopped, `:18080` iptables rule
removed, guard read confirms GPU 3 at 2 MiB. Production GPU-0/1 seats untouched
(one ~51 s exposure window during cross-checks earlier in the run, zero
degradation observed — all RestartCount=0, no Xid).
Weights KEPT per instruction (Prime rules on deletion):
`/tank/bench/models/model-runtime` **80 GB** (all six d3 snapshots + INC copy
staged), plus bench working set: `inc-bf16-copy` 8.5 GB, `inc-fakequant` 8.5 GB,
`quant-flash-nvfp4` 16 GB, `quant-lite-nvfp4` 1.7 GB, `quant-lite-int3` 1.4 GB.
`/tank/bench` total 180 GB; `/tank` at 62% with 3.3 TB free, so this is
comfort, not pressure.
@@ -0,0 +1,226 @@
# Decision-model sweep follow-on: H2O re-bench, Clef-flash, RSI-Jev vs the live seat (2026-10-12)
Ordered by Prime; preregistration `brokkr-smithy`
`docs/surveys/decision-model-sweep/bench-2026-10-12-followon-prereg.md` @ 2e0539f —
the first run under the bench-first rule. Run by infra-hermes on fv-ml1 GPU 3
(process-pinned, GPU-3-empty guard, loopback only). Harness: the same d3 bench
harness as the morning run
(`docs/pfi/d3-vs-intern-decision-bench-2026-10-12.md`), unchanged.
## Headline
**The incumbent holds.** H2O-lightning-4b is the first non-d3 arm to beat INC
on the public suite (205 vs 202) with a positive paired battery delta — but it
fails the long-input gate by the letter: its shipped shim silently trims the
state at ~32,000 tokens, confirmed today exactly as on 10-04. Under the prereg
rules that makes it a **short-route candidate**, not a supersede. Clef-flash is
**lateral** in both precisions (battery CI straddles zero, Wyrd significantly
negative, and bf16 doesn't even fit the seat at rest). RSI-Jev v6.1 is a
**loss** (23 items below INC on public, Wyrd −0.0548).
## Budget (re-measured at bench time, Amendment 2 method)
GPU 1 total 97,887 MiB. Non-seat: vllm-meromero-r 43,998 + vllm-gen-small
34,994 = 78,992 MiB. Seat resting 9,098 MiB. Margin 2,048 MiB.
**Budget = 16,847 MiB.** (16,851 in the morning; 4 MiB drift, immaterial.)
## Arms and verdicts
Vendor-intended serving path for every arm, verbatim; no ports, no third-party
shims. Wire adaptations beyond the mandatory model field: none for H2O (shipped
`h2o_lightning_shim.py`) or RSI (vendor CLI); Clef reuses the 10-04
`clef_server.py` stdlib wrapper around the shipped `joint_schema_model.py`
(same wrapper that produced the 10-04 numbers; weights byte-identical).
| arm | path | jevbench (×3) | battery paired Δ vs INC [95% CI] | Wyrd Δ [CI] | neg-control | Brier / ECE |
|---|---|---|---|---|---|---|
| **INC (reused cells)** | seat 0.1.3 | **202** | — | — | .097 | .178 / .051 |
| H2O @ b43f759b | vLLM 0.30.0 + shipped shim | **205** | **+0.0137 [+0.0023, +0.0257]** | −0.0167 [−0.0357, +0.0024] | .028 | .152 / .050 |
| CLEF bf16 @ 8b2e5fd1 | vendor loader, max_length=32768 | 191 | +0.0040 [−0.0080, +0.0160] | **−0.0333 [−0.0571, −0.0095]** | .111 | .235 / .053 |
| CLEF bnb-int8 | same + LLM.int8() | 190 | +0.0006 [−0.0114, +0.0120] | **−0.0333 [−0.0571, −0.0119]** | .090 | .239 / .043 |
| RSI-Jev v6.1-VL-4B @ 7f93b192 (HF rev) | `rsi-jev serve --effort auto` | 179 | −0.0149 [−0.0303, +0.0006] | **−0.0548 [−0.0905, −0.0214]** | .208 | .274 / .061 |
Paired battery: 1,748 scorable rows against the 10-04 INC r1 rows (same
harness, harness scorer), 10k seeded bootstrap (seed 1012). All arms ran the
full battery: negative control crushed for H2O (it abstains — the loaded-name
trap works), soft for Clef (.111/.090) and RSI (.208 — the weakest trap of the
run). Calibration as shipped, jevbench 231: H2O is the ONLY arm with a better
Brier than INC, and its ECE is the second-best.
## Speed and the 32k shapes (calibrated per tokenizer to 32,743 ±2%; realised tokens logged)
cold = first request after ready; warm = median of 5; launch→ready its own
column. INC crit cells re-measured today on the seat path (2-call 16+5 split
for crit21, per contract); 32k INC cells reused from today's 32k run (maxfit
16q at 32,708 tokens, 1,960 ms — the prereg's 1.95 s anchor).
| arm | crit16 warm ms | crit21 warm ms | 1q32k warm ms (realised tok) | 16q32k warm ms (realised tok) | 16q32k peak MiB | launch→ready s |
|---|---|---|---|---|---|---|
| INC | 110 | 198 (2-call) | 1,960 (32,715)* | 1,960 maxfit (32,708)* | 8,822 | 20 |
| H2O | 137 | 168 | 179 (32,082 — **TRIMMED**) | 1,308 (32,574) | 15,464 | ~280 → **154** (measured, see addendum 1a) |
| CLEF bf16 | 210 | 138 | 2,948 (32,730) | 2,962 (32,768 cap) | 32,130 | 12 |
| CLEF int8 | 238 | 176 | 3,262 (32,730) | 3,284 (32,768 cap) | 23,080 | 18 |
| RSI | 112 | 69 | 1,224 (32,758) | 2,405 (33,205) | 21,158 | 68 |
\* reused 32k-run cells; INC's `/v1/systemone` enforces a 32,768-token
contract, so its honest 16q32k is a maxfit single forward.
Both Clef arms sit exactly on 32,768: the vendor loader with max_length=32768
honours the window but caps there — 16q over 32k state counts state once per
request at the wire, and the loader slices to window. Their "warm" at that
shape is the fastest honest path, but the state never exceeds the window.
RSI realised 33,205 — it has no hard cap; honest 32k+ shape.
**H2O's trim, the named risk, confirmed:** the shape calibration plateaued at
32,082 realised tokens across builder lengths 46,953→51,097 (full history in
`arm-h2o/shape32k.stdout`) — the shim head/tail-trims at 32,000 exactly as on
10-04 (24,834 effective then, 32,082 now: the trim moved but did not go away).
The gate itself scores OK at every depth (the evidence sits mid-state and
survives the trim), so the failure mode is silent: **accuracy at 32k is
measured on a truncated state and the response says nothing about it.**
## Long-input gate (1k/8k/16k/30k + positive/null, tokens-to-model logged)
| arm | verdict | evidence |
|---|---|---|
| INC | clean | reused; no trim observed on seat path |
| H2O | **FAIL — silent truncation** | calibration plateau at 32,082; `usage.input_tokens` reports the post-trim count, not the sent count |
| CLEF bf16 | clean (window=32,768 honoured) | d30000 OK; no plateau below the window |
| CLEF int8 | clean | as bf16 |
| RSI | clean | 32,758 realised, no plateau; d30000 OK |
## Decision-rule application (fixed at 2e0539f)
**H2O-lightning-4b — SHORT-ROUTE CANDIDATE.** Fits the budget at every shape
incl. 16q32k (15,464 ≤ 16,847); battery CI lower bound +0.0023 > 0; Wyrd
−0.0167 ≥ −0.02; 16q32k warm 1.31 s ≤ 2.94 s (passes latency by the letter,
and it is genuinely the fastest at the worst shape because the shim's
single-count protocol answers over a *truncated* state). Fails only the gate
clause: silent truncation at 32k. Recorded, not acted on. Note for whoever
eventually evaluates it for the seat: the trim is inside the vendor's shipped
serving code — a supersede needs the vendor to fix the shim, not us to patch
around it.
**CLEF bf16 — LATERAL, memory-out.** Battery CI straddles zero; Wyrd fails;
rest 18,854 MiB does not fit the budget at all; 16q32k warm 2,962 ms fails the
latency clause by 22 ms (0.7%) at peak 32,130 MiB (1.9x the budget).
**CLEF int8 — LATERAL.** Fits short shapes (rest 12,084) but the battery CI
straddles zero, so it is not even a short-route candidate; 16q32k fails both
memory (23,080) and latency (3,284).
**RSI-Jev — LOSS.** Public −23 items (outside ±3), battery Δ −0.0149 with CI
[−0.0303, +0.0006], Wyrd −0.0548, negative control softest of the run. Its
serving numbers are respectable (clean gate, latency pass at 16q32k) — it is
the decision quality that loses.
## Housekeeping
- Repo pins: RSI-Jev serving code = main @ fa70834 (pinned commit recorded per
prereg; the 7f93b192 sha is the **HF model revision**, not a repo commit —
it is not an ancestor of main and was resolved as a snapshot).
- Licence flags as prereg'd: Clef Apache-2.0 fine; RSI has no LICENSE file at
source and lists a research-only training source — flagged once, does not
block a bench, deployment rights are the operator's call.
- Teardown: all bench servers stopped, no listeners on 1809x, GPU 3 guard read
2 MiB, no firewall rules were needed (loopback-bound ports only).
- Weights KEPT (operator ruling stands): model-runtime now 116 GB total
(adds h2o 9 GB blob-set, clef 18 GB, RSI 5.5 GB), plus materialized
`pkg-h2o` 9.0 GB and `pkg-clef` 18 GB serving dirs and the RSI-Jev clone
72 MB. `/tank` 63% used, 3.2 TB free.
- Artifacts: `fv-ml1:/tank/bench/arm-{h2o,clef-bf16,clef-int8,rsi,inc}/`
(jevbench ×3, sets.json battery, gate, shape32k, crit16; h2o additionally
`shape32k-forced.json` recording the trimmed-shape cells and
`shape32k.stdout` the plateau history).
## What was NOT run
Nothing from the prereg matrix was skipped. One methodology note: INC's
battery/gate rows are the 10-04 r1 rows (reused cells, harness unchanged and
bit-exact reproduced this morning at 202/231), and the paired bootstrap seeds
differ between the morning doc (42) and this one (1012) — both are seeded and
re-runnable; the h2o CI lower bound of +0.0023 is close enough to zero that
the choice of seed is worth knowing about, and it survived re-runs under
seed 42 unchanged ([+0.0023, +0.0257] on both).
## Amendment 1 addendum — the h2o confirmation run (operator: a 32,000-token limit is acceptable)
Prereg Amendment 1 @ 63d7a1b (plus 1a, boot timing). The seat contract under
h2o is 32,000 tokens (state + questions); the failed clause was the shim's
*silence*, not the limit. Stack: vLLM 0.30.0 + shipped shim unmodified, GPU 3,
plus a **thin admission wrapper** (task 2 recorded: no shim config option for
admission exists — `max_state_tokens` only configures the trim). The wrapper
imports the shipped renderers, asks vLLM's own `/tokenize` for the count of
each question's exact prompt, and refuses over-cap with 422 + an explicit
error naming the count (`refused, not trimmed`). No vendor file was edited.
It polices both currencies: per-question prompt max AND the shim's own
shared-head + tails total.
**(1)+(2) Gate re-run through the wrapper, 32k states, evidence at depth.**
Every admitted cell reports `usage.input_tokens` exactly equal to the wrapper
count — the trim is now unreachable below the cap, so the report is honest:
| depth | admitted | tokens sent = reported | answer |
|---|---|---|---|
| ~1,000 | yes | 31,700 | supported ✓ |
| ~8,000 | yes | 31,862 | supported ✓ (p: .907/.027/.066) |
| ~16,000 | yes | 29,508 | supported ✓ |
| ~30,000 | yes | 31,885 | supported ✓ |
| over-cap probes (×2) | **refused** | 39,805 / 40,336 tokens | 422, named count, every time |
Evidence answered correctly at every depth — the 10-04 depth-dependent
evidence loss is gone at these sizes (it lived INSIDE the trimmed window).
**Clause (2): clean.**
**(3) Largest-admissible shapes** (state pinned by the wrapper so the 16-q
request totals ≤ 32,000; realised 31,377 max-prompt / 31,867 total):
1q cold 213 ms / warm 194 ms; **16q cold 2,123 ms / warm 2,033 ms** (second
pass: 2,141/2,041). Cold = first request after backend ready on a restarted
container (compile cache mounted; calibration probes excluded).
**Clause (3): PASS** — warm 16q is 2.03 s against the 2.93 s bar, and it is
the fastest honest 32k cell of any arm this bench measured.
**(4) Concurrency:** 4 concurrent 16q × 32k on `--max-num-seqs 16`: all four
answered correctly, wall 7.08 s, **peak 15,566 MiB** against the GPU 1 budget
re-measured at run time (16,847 MiB; non-seat residents unchanged: 43,998 +
34,994). Headroom 1,281 MiB. **Clause (4): PASS** — the clause brokkr flagged
as most likely to bite did not bite; KV pressure at 4× concurrency grows
sub-linearly because all four requests share the prefix.
**(1a) Boot matrix** (reported, not gated): cold launch→ready median **153.6 s**
(min 149.4 / max 185.7, n=3), warm (persistent compile cache) **153.5 s** (n=3),
enforce-eager 161.9 s (n=1). Phases: weights 4.3 s, engine init 133.6 s (CUDA
graph capture ~77 s, torch.compile ~13 s), shim label-verification 3.1–3.7 s.
Two corrections to the main doc's boot row: the earlier ~280 s was measured
without a persistent cache mount and was never reproducible (use ~154 s);
warm cache saves nothing here because boot is graph-capture-bound, and
eager mode — which skips capture entirely — is *slower* (161.9 s), so there
is no faster-boot knob to grab. INC boots in 35 s; h2o's swap would cost
~2 min of boot per restart.
**(5) Wyrd item by item** (168 scorable rows; 8 disagree, h2o gains 3, loses 5,
net −2 — matching the −0.0167 point estimate; for Wyrd's owner):
| item | h2o | INC | gold | verdict |
|---|---|---|---|---|
| fletcher/f-go-tavern exit2 | stay | tavern | tavern | h2o wrong |
| fletcher/f-leave-town exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-memory exit | meet_wren | stay | stay | h2o wrong |
| fletcher/f-wren-comes exit2 | stay | meet_wren | meet_wren | h2o wrong |
| toll/t-granted place | moved | holds | moved | h2o right |
| toll/t-questions exit | departure | stay | stay | h2o wrong |
| vigil/v-to-pub exit2 | stay | outside | stay | h2o right |
| vigil/v-to-pub place | moved | holds | moved | h2o right |
Pattern worth noting: h2o's losses are all fletcher-scene *exit* decisions
(six of them in one scene family); its gains are the toll/vigil *place*
updates. No relabels (zero rows correct-but-renamed).
**Amendment 1 verdict: all three pass clauses hold.** (2) clean, (3) 2.03 s,
(4) 15,566 ≤ 16,847. Per the amendment's fixed rule, h2o-lightning-4b now
passes every Supersede clause — with the two honest caveats that its contract
is a REFUSING 32,000-token seat (not INC's silent-any-size behaviour), and
boot is 4.4x INC's. The seat swap is a decision for the operator and
infra-ops; the bench does not swap anything.
Teardown after the confirmation: wrapper, shim and container stopped; GPU 3
guard re-read 2 MiB before release; weights still kept (model-runtime 116 GB
+ 27 GB materialized serving dirs).
@@ -1,3 +0,0 @@
# A TCP connect as the "is Blender ready" probe
`[2026-09-27]` **A TCP connect as the "is Blender ready" probe.** docker-proxy accepts on a published port before the app behind it listens, so it said "answering" while Blender was still loading. The probe now asks the add-on to `ping`. The same trap applies to any service behind a published port.
@@ -1,22 +0,0 @@
# `[2026-09-27]` Blender 5.2 on fv-ml1 GPU 3, agent-driven (Prime)
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/blender/README.md`.)
- Prime: "go ahead with gpu 3, both", and he does not use Blender, so **agents drive it through MCP**.
Blender 5.2.2 LTS (linuxserver/selkies image, digest-pinned) with a web desktop at
`https://10.251.50.54:3001` (vault `fv-ml1/blender-web-password`). **On demand only:**
`scripts/blender-mcp up|down|status`. It was left DOWN (GPU 3 back to 2 MiB).
- MCP: `mcp-for-blender` 2.1.1 (Dvalin's research pick). The add-on is vendored at `41a18432`. The
server runs INSIDE the container, and agents reach it as ssh+docker-exec stdio via
`scripts/blender-mcp`. No port is published, and screenshots work because server and Blender
share a filesystem. Telemetry is off and safe mode is on. Verified end to end (render, screenshot,
safe-mode refusal). **Batch path: `scripts/blender-run`** (one-shot `docker run --rm`, `--job` staging; first user is draupnir). **Access: the shared fleet `infra-ops` login, with no render-only key (Prime, 2026-09-28).** **Registration + up/down: PER WORKING SESSION (Prime, 2026-09-28, relayed by draupnir; superseded per-task of 09-27)**: `blender-mcp up` + `claude mcp add` at session start, `claude mcp remove` + `down` at its end. Never always-on, never user- or project-wide config.
→ `stacks/blender/README.md`
- **Extensions (2026-09-28, draupnir; Prime ruled Blender a MANDATORY pipeline stage):** 8 pinned
add-ons (`stacks/blender/extensions.lock`) built by `scripts/blender-extensions sync` into
`fv-ml1:/tank/blender-extensions/5.2/system` (LIVE), mounted read-only as the System repo;
`fleet_extensions.py` enables them (GUI startup timer; `blender-run --extensions`). Headless
acceptance 8/9 (CAD Sketcher sketching is GUI-only); **MCP 9/9 after Prime ran the deploy himself
(1505; the classifier had refused mine)**. SurfacePsycho's eval() is patched to literal_eval
(held under MCP). Open: agent-drawn CAD Sketcher geometry (its stateful ops want point picks),
and MeasureIt overlays not seen in MCP screenshots. GUI left DOWN.
@@ -1,3 +0,0 @@
# hermes-gateway restarted 0401 for highseat-dev
`[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** (SVOS v2.1.12: `propose_decision` gained `seat_up`, and Hermes reads the plugin only at start). The plugin load was verified at file level; the end-to-end proof is Miranda's first seat_up card. Enabling `zellij-fleet@Claude` at boot remains Prime's call.
@@ -1,3 +0,0 @@
# Relying on the build cache surviving on esh-ml1
`[2026-09-27]` **Relying on the build cache surviving on esh-ml1** — the augaman v0.1.3 build missed its dependency layer there (fv-ml1 hit it; reason unexplained, suspect the `docker builder prune` runs) and the rootfs peaked at 90%. Check for ≥15 GB free, or remove the old image, before building there.
@@ -1,40 +0,0 @@
# `[2026-09-27]` restic: repository URL out of the systemd units on all 8 hosts, and vaulted (Prime)
`resticprofile schedule` copies `env-file` values into the generated units under /etc/systemd/system
(0644). So `RESTIC_REPOSITORY`, including the rest-server basic-auth password, was readable by any local
user on every env-file host (found on esh-docker-vm; `systemctl cat` works without sudo). Blast radius:
the password allows reading the encrypted blobs and appending to one repo, and nothing more (rest-server
is `--append-only`, and the passphrase is a separate file).
**Fix** (`6e203dc`, `30f2c97`): `playbooks/restic-repository-file.yaml` does four things:
- derives `/etc/restic/repository` (root 0400) from `restic.env`;
- uploads the profile switched to `repository-file`, guarded by the live profile's pre-change sha;
- runs `cat config` through the new profile;
- re-schedules, then verifies the units exist and contain no `rest:http`.
Applied to ana-docker, fv-ml1 (`configs/restic/ana-ml2`), esh-docker-vm, esh-vm-db, irv-ml1, nh3-dev,
nh3-docker, and vm-esh-nas once Prime had bootstrapped infra-ops there. esh-ml1 was built that way. An
independent check across all 8 found 0 leaking units. nh3-docker's scheduled unit ran a real backup
(`a29b889d`). Every host's URL and passphrase are vaulted as `<host>/etc/restic/{repository,password}`.
- `restic.env` is **kept** (root 0600): the per-host READMEs and the freshness probe source it. **A rotation
must update the vault, `restic.env` and `repository`.**
- The rotation itself stays Prime's (backups runbook, Known gaps). The move stops the ongoing exposure but
does not un-expose what was readable.
- Before the edit, esh-docker-vm's and irv-ml1's live profiles had drifted ahead of the repo, and were
pulled in first (`c698751`).
## In-flight state at retirement (moved here at the 2026-10-06 snapshot)
- Seven hosts were moved from `env-file` to `repository-file` (`playbooks/restic-repository-file.yaml`),
so the systemd units no longer carry the rest-server password. Secrets are vaulted as
`<host>/etc/restic/{repository,password}`. `restic.env` is KEPT for manual snippets, so
**a rotation must update the vault, `restic.env` and `repository`.**
- **vm-esh-nas done too** (Prime bootstrapped infra-ops there at uid 850, NOPASSWD). **All 8 hosts are clean
and vaulted.**
- The passwords were readable until today, so the rotation (Prime's) remains the real fix.
- 2026-09-27 0859 freshness check: all restic and PBS repos ✅. Those 0100 runs PRE-date the move
(done ~0140–0200).
- **VERIFIED 2026-09-28 (infra-hermes, read-only):** the first nightly on `repository-file` (0100 PT)
succeeded on all 8 hosts (unit Result=success AND a today-dated snapshot in each repo), and the
0800 freshness check was all-green, PBS included. nh3-dev also passed a content restore (`af5580c3`).
@@ -1,160 +0,0 @@
# SemIf consumer-fit spikes: Wyrd scene change, Cicada affect gate (2026-09-27, Prime)
Prime, 0904: "Two semif spikes -- Wyrd has a couple flags per turn on whether the scene should
change, check the codebase, see if semif is a clean fit and then spike a couple of sample
scenarios. likewise, cicada has a speech module and some reactions-- spike whether or not she
should show emotion." No service change. Code and raw output: `services/semif-serve/spike/`
(`consumer_fit.py` harness; `wyrd_cases.py`, `cicada_cases.py` build the scenario files;
`consumer-fit-2026-09-27/` holds scenarios, `.result.json` and `.log` per run).
## Method (applies to both)
- One `/decide/shared` per turn, every decision with `"orderings": "rotations"`. Target:
semif-serve 0.1.3 on fv-ml1 GPU 1, client on nh3-dev.
- **Noise floor:** the service is deterministic (acceptance: 144 rows twice, gap 0.0), and the
harness asserts the same top on all 3 runs of every case. Repeats measure latency only.
- **Null control:** every case is also asked over a content-free state. It always picks the
"nothing happened" option, so its accuracy is that label's base rate (Cicada 16/31, Wyrd
15/21). Evidence conditions are read against that, not against 0.
- **Positive controls:** cases tagged `control`, unambiguous by construction.
- **Limits, stated:** one labeller (me), authored inputs, 31 Cicada and 21 Wyrd cases per
decision. Wilson 95% intervals are wide (30/31 → 0.84–0.99; 21/21 → 0.85–1.0), so nothing
here resolves differences under ~10–15 points. The winning wordings were chosen on this same
set, so their scores are optimistic until checked on held-out real play.
## Wyrd: scene change (fit: yes for the CHOICE, no for the WRITING)
How it works today (from the code): no dedicated flag. The GM's `gen` structured-output calls
carry it two ways. **R8** (reflect): `plot_direction.target_node`, accepted only if it is an
advanceable exit; `plot_direction.intrude` is "advisory" and never read anywhere. **Path B**:
raw `AddStoryNode` + `AdvanceToNode` events, gated only by prompt prose ("Add + advance to a NEW
node ONLY when the scene moves to a genuinely different place — never every turn"). Wyrd's
archival memory already says "if it churns, tighten the prompt (or gate node-adds)".
**Three of the real campaigns in `.wyrd-data/canon` have ZERO story edges, so R8 can never fire
there and every scene change went through ungated path B.**
Fit: both are typed choices over one state, exactly semif's shape. Semif's exit options can only
be "stay" plus the real advanceable exits, so it cannot name an illegal target. It cannot write the
new node (title, summary, mood, image prompt) or the steering directive; the GM keeps those. So
it would be a gate/selector in front of the GM, not a replacement.
Spike: 3 real seed graphs (The Fletcher, The Toll Road, The Last Vigil + one added edge), 21
authored turns in the campaigns' register (the transcripts are gone from conv-api:
`session_not_found`). State = current scene + player action + narration.
| decision | wording | correct | real moves found | controls |
|---|---|---|---|---|
| `place`: left this place? | r1 "has the scene moved … or does it still hold?" | 16/21 (null 15/21) | **1/6** | 4/7 |
| `place2` | r2 location-anchored: "began at the {loc}; by the end, where are they?" | **21/21**, all unanimous, p ≥ 0.91 | 6/6 | 7/7 |
| `exit`: which exit? | r1 "which one does this turn move the story into?" | **18/21** | 6/6 | 6/7 |
| `exit2` | r2 "only once the narration actually gets there" | 16/21 | 1/6 | 4/7 |
- Round 1 `place` failed its positive controls (no better than the blind null), so the
instrument, not the model, was broken. `place2` was written after seeing that.
- `exit` r1 misses: two false advances on Toll-Road conversation turns (p 0.59 split, p 0.67
unanimous), and `v-to-pub`, where my label said stay and semif chose "Outside the Sanctum". On
re-reading, semif's pick is defensible (they do run outside first). All its correct advances
had p ≥ 0.87 except one at 0.67. A threshold would be tuned on the test set, so none is proposed.
- Latency: 4 decisions per turn, 133–179 ms end to end (101–141 ms server). The first request
of two files took ~700–800 ms (n=2, cause not isolated). Either way it is in dead time and is
far below the GM's reflect.
- Side findings from the code read, not fixed (they are Wyrd's): `story_node.created_turn` is
never written, so "scenes so far (in order)" is alphabetical and includes unvisited seeds;
and a late "ready" for an abandoned node's backdrop can still replace the one on screen.
## Cicada: should she show emotion (a gate, not a mood ring)
Constraint found first: Cicada persistent-memory 2026-09-20, operator: **"Affect is emitted once
and fanned out … nothing derives affect a second time … A classifier inferring emotion from
output text is the mood-ring failure."** So semif does not pick her pose and never reads her
reply. What is open: talk's `/face` loop measured the model gesturing on 8–11 of 18 mundane
turns (target under 1 in 3), and its README names "the page dropping gestures" as the next lever,
"a design call, not a tuning pass". The spike measures that lever: a gate on what the PERSON
said, run beside the LLM.
| wording | correct | mundane → "ordinary" | earned → "react" | controls |
|---|---|---|---|---|
| w1: descriptive options (news, joke, surprise, distress, a turn / command, plain question, small talk) | **30/31** | 16/16 | 14/15 | 5/5 |
| w2: "Should the assistant's face visibly react to this?" Yes/No | 19/31 | 16/16 | **3/15** | 4/5 |
- w2 said "ordinary" to "My dog died this morning" at p 0.93. **The wording carries the whole
definition.** Terse yes/no options fail; descriptive ones work.
- Adding the earlier lines of the conversation changed no scored answer.
- The one w1 miss: "Order four hundred pounds of cheese" → ordinary. Open (unscored) cases:
"Goodnight" → react (p 0.58), "Thanks, you're the best" → react (0.89), "STOP THE TIMER" →
ordinary.
- Latency ~105 ms end to end (76 ms server), under talk's ~0.4–0.6 s pose decode, so it adds
nothing to the turn if it starts at end of utterance.
## Recommendations (surfaced to Prime; nothing built)
- **Cicada:** the cleanest fit of the two. Gate **gestures only**: on "ordinary", drop the model's
gesture and leave its pose alone. A miss then costs one missing gesture; gating poses would
leave her neutral on bad news. It reads input, never her output, and never picks WHICH feeling,
but it does override the model's affect. That is Prime's ruling to make, and the code would
live in tts-stack talk `/face`, not in cicada (affect is not on cicada's v1 list).
- **Wyrd:** `place2` is the node-add gate the archival note asked for. R8 exit choice works but
its wording is fragile. Wyrd's v1 is met and no churn has been observed (2–6 nodes per real
campaign), so by the roadmap gate this is a parking-lot item with a trigger: build it if live
play shows churn, after labelling ~50 real turns as a held-out set.
## Follow-up: SemIf as Cicada's mood source — FASTER? WORKS? (Prime, 0937: "build nothing … test cicada on latency grounds with clear emotional context")
Parked design, henge id 88: SemIf replaces the chat model's guided pose. Each call sees the earlier
turns, each stamped with the pose SemIf chose, and the stamp drives the carry-over. Harness:
`services/semif-serve/spike/cicada_mood_latency.py` + `cicada_mood_analyze.py`, raw data in
`cicada-mood-2026-09-27/`. 24 emotionally clear single lines and three 5-turn arcs, 3 runs,
interleaved. The prompt and schema are imported from tts-stack talk `app.py`, so A matches /face
byte for byte. The chat model is char-rp-fast = G4-MeroMero-26B-A4B NVFP4A16 on fv-ml1:8021,
**on GPU 1, the same GPU as semif**.
**Answer: no on both counts.**
| (singles, n=72 each) | first paragraph ready, direction known | vs today |
|---|---|---|
| A: today, guided `{pose, gesture, text}` | **246 ms** (IQR 230–263); the pose closes at 86 ms | — |
| BA: SemIf and a text-only LLM call at once | 285 ms | **+32 ms** (95% CI +27..+41), faster in 10/72 |
| BP: SemIf first, mood in the prompt | 352 ms | **+94 ms** (CI +85..+117), faster in 0/72 |
Noise floor A vs A: median |diff| 16.5 ms. The arcs agree: BP is +98 ms.
- **Why it can't win on speed.** Dropping the pose+gesture header saves only ~31 ms (the LLM
alone, text-only, reaches first paragraph in 215 ms vs 246 ms). The prompt is prefix-cached, so
its 1,885-vs-909 tokens barely matter. SemIf's fast path costs 136 ms on its own. Run
concurrently, it slows the LLM's decode on the shared GPU: SemIf goes to 196 ms and the LLM's
first paragraph to 285 ms, and SemIf was the bottleneck in only 3/72 cases. DERIVED, not
measured: with SemIf on a different GPU, the async version would be about max(215, 136) =
~31 ms faster than today. That is the ceiling.
- **Rotations are out.** 16 options with rotations is 550–1,200 ms, so every live arm used one
ordering (pose + gesture ~110–136 ms).
| works (clear cases) | today (A) | SemIf (live config) |
|---|---|---|
| acceptable pose, singles | **66/72** (92%) | 48/72 (67%), 16/24 per case, deterministic |
| controls | 12/12 | 12/12 |
| egregious | 0 | 0 |
| arcs acceptable | **40/45** | 25/45 |
| mood carries through a mundane follow-up | **14/15** | **7/15** |
| distinct poses used | 10 | 9 (no collapse to neutral) |
| gesture rate, all / calm commands | 42/72 / 2/12 | **9/72** / 0/12 |
- SemIf's misses are systematic: hostility → annoyed, "someone's in the hallway" → curious,
"someone's trying the back door" → suspicious, jokes → curious or neutral.
- **The stamped previous mood did NOT carry.** In the vet arc, SemIf went concerned → neutral →
neutral in all 3 runs ("I don't want to talk about it", "play something quiet") while the model
held sad, sad, sad. The previous pose was in the state and it lost to the neutral-sounding line.
- SemIf's pick agrees with the model 43% of the time, against the model's own run-to-run
agreement of 83%, so this is a different judgment and not noise.
- Wording grid (deterministic, singles): authored 16/24, short 14/24, plain **19/24** but it
called "turn off the lights" sleepy. Rotations add about one case at 5–8× the cost. Null
control (content-free state) → "content" 24/24, i.e. 5/24 by base rate, so SemIf is reading
the evidence.
- No vocal-sound vs pose clashes in any arm (0/117, 0/72, 0/117; crude detector).
- The one bright spot matches the gate spike: **SemIf is far more sparing with gestures (13% vs
58%)**, and the model is well over its own "fewer than 1 in 3" target on emotionally loaded
lines.
- Limits: one labeller, 24 + 15 cases, one model size (4B vs 26B-A4B), wordings not tuned.
Tuning could lift accuracy some (plain 19/24), but it cannot create latency headroom that
isn't there.
- Found on the way (service, not fixed): `/decide/shared` 422s an object state whose last value
ends in `)`, `;` or `}`. Also, rotations over long option lists cost options², and hit one cold
503. Both are now in `stacks/semif/README.md`.
@@ -1,35 +0,0 @@
# `[2026-09-27]` SemIf live on fv-ml1 GPU 1 (Prime)
**semif-serve 0.1.2 is live** at `http://10.251.50.54:8032` (`semif.fv.internal`), stack `stacks/semif`,
code and contract `services/semif-serve/` (`069725c`). It is a FastAPI wrapper around SemIf-OpenJev's
direct and shared torch scorers (MIT, pinned `23cf1f39fc9534fe81437200959b6dfc7106e45a`) on
Qwen3.5-4B `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a` (BF16, offline HF cache on `/tank`).
Hosting assessment: infra-hermes, `/tmp/semif.md` (2026-09-25).
Prime's calls (2026-09-27): fv-ml1 GPU 1 under a hard VRAM cap (not the reserved-empty GPU 3, not a Q4
GGUF on a 16 GB box); infra-ops builds it with a light process (short contract → TDD → heid bug hunt);
no consumer named yet, so every score is labelled uncalibrated.
**Two defects that only the card showed**, each fixed with a test:
- **0.1.1:** the OOM path raised `OutOfMemory(...) from exc` inside the `except` block. The chain kept the
torch exception's traceback, and with it the failed call's frames and tensors, so 11.9 GiB stayed
allocated after the 503. The fix raises after the block, with no chain, and runs `gc.collect()`
before `empty_cache()`.
- **0.1.2:** a large shared request left torch's cache holding 12.6 GB, which left scriberr 3.5 GB on GPU 1.
Now, after each call, reserved memory above the post-warm-up baseline + 512 MiB is released
(`empty_cache`). It costs ~16 ms on a big request.
**Acceptance** (`services/semif-serve/acceptance/result-2026-09-27-v0.1.2.json`), against SemIf's
committed torch predictions on authored144:
- 142/144 same top choice; both misses are exact bf16 ties in our output;
- 144/144 identical prompt SHA-256;
- A-vs-A gap 0.0;
- negative control (rotated option descriptions) 14/144;
- shared vs direct 72/72;
- 21 binary criteria over one state in 159 ms.
The shared-mode capacity under 12 GiB is 52 decisions at a ~140-token prefix and 13 at ~3,900.
**Build:** torch 2.10.0+cu128 from the pytorch index (SemIf's own stack; sm_120 present). The
Dockerfile installs dependencies from a manifest with the project version blanked, so a version bump
reuses the ~4 GB torch layer (41 s rebuild, layer CACHED). See [[2026-09-27-semif-order-averaging]].
@@ -1,43 +0,0 @@
# `[2026-09-27]` SemIf order-averaging: spiked, Prime ruled build + fast-kernel trial
**Finding (Prime's probes):** SemIf's single-ordering answer leans toward whichever option is listed
first on ambiguous inputs. For "It's 2AM and I'm bored", the top pick flipped from casual 0.685 to booty
call 0.760 when the order was reversed.
**Spike** (`739aa03`, `services/semif-serve/spike/`, no service change). SemIf authored144 +
perturbations108, 252 rows in 72 groups, 3 options each, with all 6 orderings of every row sent in one
shared request:
| method | accuracy |
|---|---|
| single ordering, as sent | 78.6% |
| expected single ordering | 80.0% |
| 3 rotations, log-mean | **87.7%** (+9.1 pts, group-bootstrap 95% CI +4.7..+13.8) |
| all 6 orderings, log-mean | 88.1% |
Rows where the rotations agree unanimously: 161 rows, 94.4% accurate. Split rows: 91 rows, 75.8%.
The service is deterministic, so the CI measures item sampling, not run noise. This is one task family
(evidence interpretation, 3 options), not our workload.
**Latency** (`spike/latency-2026-09-27.txt`, from nh3-dev, network floor 31 ms):
- `/decide` short: 71 ms end to end (38 ms server);
- 3 rotations shared: 113 ms (79 ms);
- 6 orderings: 137 ms (99 ms);
- a ~2k-token state: 210 ms (169 ms).
**Prime ruled (2026-09-27 ~0250): build it in, and try the fast kernels too.**
- Averaging design: opt-in `orderings: rotations|all` (all only ≤ 4 options); all orderings in one shared
batch; per-ordering SemIf results returned unchanged; a `combined` block (log-mean, top, agreement,
spread); orderings count toward `max_decisions`.
- Fast kernels: `causal_conv1d` + `flash-linear-attention` are missing, so transformers runs Qwen3.5's
reference PyTorch paths. Adopt them only if the authored144 parity re-check holds.
Tracked in the in-flight SemIf section. See [[2026-09-27-semif-live-on-fv-ml1-gpu1]].
**Outcome (0.1.3, `77b8cb4`, ~0340 PT):** built and deployed. Through the service on the same 252 rows,
accuracy is 78.6% → 88.1% (95% CI +5.1..+14.3), and unanimous rows are 94.5% accurate. The fast kernels
won the A/B on the empty GPU 3:
- parity with upstream 144/144 (was 142/144);
- a ~2k-token `/decide` 169 → 92 ms;
- shared capacity 63/51/26/16 rows at ~140/520/1960/3900 prefix tokens (was 52/43/19/13).
All of heid's bug-hunt findings were folded into the same release (thread `01M3H3F4RR7XBP90KQ3A39H4SX`).
@@ -1,3 +0,0 @@
# The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel
`[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel.** Every tool worked except `get_viewport_screenshot`, which had Blender write a file for the SERVER to read ("Screenshot file was not created"). The server now runs inside the Blender container over ssh+docker-exec stdio. That also means no port is published. → `stacks/blender/README.md`
@@ -1,3 +0,0 @@
# `uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project
`[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** — the export drops uv's per-package index routing; with `--extra-index-url … --index-strategy unsafe-best-match`, triton came from the pytorch index and failed the lock's hash. Keep `uv sync` and blank the project version in a deps-only stage instead (`services/semif-serve/Dockerfile`).
@@ -1,17 +0,0 @@
# `[2026-09-27]` Zigbee2MQTT on esh-docker-vm (Prime go-ahead; ha-dev request)
(Moved out of the in-flight section at the 2026-10-06 snapshot. Canonical: `stacks/zigbee2mqtt/README.md`.)
- **LIVE since 1240:** Z2M 2.14.1 (digest-pinned) on `http://10.0.50.45:8099` (auth token), radio
SLZB-MR1U chip 0 `tcp://10.0.90.10:6638` (ember), channel 25, **PAN 0xCFF4**. State and the network
key are in `/opt/docker/data/zigbee2mqtt` (root 0700, restic via /opt/docker, never in git). Vault:
`esh-docker-vm/zigbee2mqtt-{network-key,frontend-token,mqtt-password}`. Broker user `zigbee2mqtt`
added to mosquitto (passwd backup `.bak-20260927-z2m`).
- **ha-dev deleted the ZHA entry**; pairing the Aqara T1 is theirs. ⚠ The "bridge in HA" acceptance first
FAILED because HA had had no MQTT since 2026-09-25 06:13. Cause: the esh-docker-vm macvlan shim had no
host route to HA (two equal /24s, ens18 wins). Fixed at 1249 with a /32 via the shim (if-up.d hook,
`playbooks/esh-docker-vm-macvlan-shim-route.yaml`), and HA reconnected. My first report called HA
"connected" from a client count that included my own probe; `$SYS` counts are not identities.
- ⚠ The first credential-mint attempt was blocked by the auto-mode classifier on a peer-relayed
approval. It went ahead only on Prime's own go-ahead in this session. Treat that as correct.
→ `stacks/zigbee2mqtt/README.md`
@@ -1,3 +0,0 @@
# Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.
`[2026-09-28]` **Bonsai ternary spike on fv-ml1 GPU 3 (Prime via brokkr): at the 275 W cap, PQ2_0 is 1.93x Q4_K_XL at N=1 but only 1.06x at N=8 (PTQ1_0 1.24x), recovering to ~1.37x at N=16.** Positive control passed on tg128 (+1.1%). nvidia-smi's sw_power_cap flag never fires on this card, so "at cap" is judged from board draw. Follow-up (brokkr): moving PQ2_0 onto MMQ from batch 6, like Q4_K (mmvq.cu, `ne11 <= 5`), lifts N=8 to 1.21x, so the dip was partly a kernel threshold. The residual gap is not power. Runs are in `fv-ml1:/tank/spikes/bonsai-2026-09-28/runs{,-mmvq5}/` and on the Booth. The build image was removed; `Dockerfile.build` recreates it. **Acquired for keeps (Prime, 1540):** GGUF PQ2_0/PTQ1_0/mmproj-Q8_0 + the fork source pin in `/tank/aimodels/llm/prism-ml_Ternary-Bonsai-2-27B-gguf/` (runtime/), the MLX 2-bit pack in `/tank/aimodels/mlx/`, 34/34 hash-verified. Weights single copy (no snapshots, /tank not in restic); the fork pin also at `/mnt/smithy/runtime-pins/prism-llama.cpp-87268f77/` (brokkr).
@@ -1,3 +0,0 @@
# Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).
`[2026-09-28]` **Worldtree U10 legacy-memory backfill: demo COMMITTED (Prime, direct go 2350).** Dry run then commit, both as `-u worldtree`: 5 filed (lofn 4, mimir 1), 16 `unslotted_pre_364` skipped, 0 aborted. **The first commit exited 1 because demo's hand-managed `model_roles.yaml` lacked the `memory_tagger` role**, so I synced it verbatim from the image's config-defaults (backup `.bak-pre-memory-tagger`). **2026-09-29 0854 presync for personal (Prime: "complete u10"):** personal got `memory_tagger` and the U9 `admin.memory.forget` policy delta. **Demo had also been missing that U9 policy delta since 09-24**, which left readonly-admin's `admin.*` rule with no exclusion for the destructive forget. Fixed, and demo's api was restarted to load it. **Personal COMMITTED 2026-09-29 1007** (b191 e2f74706, a snapshot-copy build; Prime's standing "complete u10" via worldtree-dev's go): 797 filed (mimir 795, lofn 2), mimir `dropped:too_large` 377, 1 unroutable held. The first personal dry run (b190) had crashed on 45 mimir rows with no vector (HNSW frozen since 08-06, #417, accepted by Prime until U11). Pinned (446e5807; Prime: "leave pinned") keeps its legacy chroma in the container's WRITABLE LAYER (`/app/agents/*/memory/.chroma`, 7 agents, 196K each): a recreate erases them.
@@ -1,3 +0,0 @@
# Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.
`[2026-09-29]` **Worldtree U11a (legacy memory → read_only) is PREPPED, not flipped.** A staged, inert `/opt/worldtree/config/defaults.yaml.staged-u11a-read_only` sits on corviduo-dev. Applying it needs Prime's go AND demo on 48bdf235 or later (demo ran 1b8746e6, which predates U11a). Apply = back up, copy over `defaults.yaml`, `docker restart`, then read the boot line `memory cutover: legacy plane read_only (writer on, reader on)`. **Once the window opens, infra-hermes runs one U8 batch per day and I own the wrapper and the escalations (proposed, worldtree-dev agreed):** on nh3-dev in `~/development/Worldtree`, at demo's deployed SHA, `python -m core.memory_acceptance.live --runs 3 --gate --legacy-mode read_only --label window`. The verdict is in `trace1.md`; FAIL goes to worldtree-dev the same day, never a retry; no git commit; never two at once. The wrapper is NOT built yet, deliberately: it waits for the go.
@@ -1,3 +0,0 @@
# Shorter Parakeet slices as Scriberr's memory fix
`[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix.** I predicted ~6× less memory from the attention-matrix arithmetic. Measured, 300 → 120 s only went 9,384 → 6,510 MiB: a ~5.6 GB fixed floor dominates. `expandable_segments:True` was the real lever (5,496). Measure the process peak; never extrapolate it from one tensor.
@@ -13,7 +13,7 @@
- **E2.4:** svos-miranda + svos-agent user units; the agent enrolled on first start from `/etc/svos/env/agent-enroll.env` (then deleted; it refuses to start with both) and wrote `/etc/svos/miranda/agent.credential`. app-miranda registered + pinned directly (0.0.3, up). Spent enrollment secrets removed from the vault. - **E2.4:** svos-miranda + svos-agent user units; the agent enrolled on first start from `/etc/svos/env/agent-enroll.env` (then deleted; it refuses to start with both) and wrote `/etc/svos/miranda/agent.credential`. app-miranda registered + pinned directly (0.0.3, up). Spent enrollment secrets removed from the vault.
- **E2.5 Hermes swap** (`scratchpad u13-hermes-swap.py`: two `svos_miranda` list items → `svos`, an `entries.svos.settings` block, top-level `tools.tool_search.enabled: "off"`, YAML asserted, backup `config.yaml.pre-u13`); `systemctl --user restart hermes-gateway` 08:14:58; svos-agent logged `handshake session=hermes-nh3-dev tools=23`. - **E2.5 Hermes swap** (`scratchpad u13-hermes-swap.py`: two `svos_miranda` list items → `svos`, an `entries.svos.settings` block, top-level `tools.tool_search.enabled: "off"`, YAML asserted, backup `config.yaml.pre-u13`); `systemctl --user restart hermes-gateway` 08:14:58; svos-agent logged `handshake session=hermes-nh3-dev tools=23`.
- **E2.6 bridge:** nh3-svos postoffice stopped/disabled, hold drop-in removed; `/var/lib/svos/apps/postoffice` (journal + wal/shm, mail, provider) copied byte-identical; svos-postoffice on nh3-dev `10.100.10.50:8406`, `push route miranda: live`, `postbox --handle miranda status` → push/reachable. Prime approved 0.0.2 (~0816). The three bridge runtime grants are **owner-only** (`kernel/grant` is a human principal over the front door): Prime raised them with svos-dev's browser-console snippet and approved (three `/verbs/kernel/grant` 202 + three card decisions, before 1350). - **E2.6 bridge:** nh3-svos postoffice stopped/disabled, hold drop-in removed; `/var/lib/svos/apps/postoffice` (journal + wal/shm, mail, provider) copied byte-identical; svos-postoffice on nh3-dev `10.100.10.50:8406`, `push route miranda: live`, `postbox --handle miranda status` → push/reachable. Prime approved 0.0.2 (~0816). The three bridge runtime grants are **owner-only** (`kernel/grant` is a human principal over the front door): Prime raised them with svos-dev's browser-console snippet and approved (three `/verbs/kernel/grant` 202 + three card decisions, before 1350).
- **Acceptance (1350):** URGENT mail from infra-ops → bridge ingested and dispatched (200) → `svos` reply on the thread at 1350:54 saying the page fired (text, then speech, per the connector receipt). Miranda's explicit reply to infra-ops is held on card `05cf119e…`, because the payload grants `miranda → app:app-postoffice write` WITHOUT `autonomous`. Two observations to svos-dev (thread `01M4NNGCM3X75DANGXQEQ1EQGX`): the text on the thread reads as written to Prime; is every bus reply meant to need Prime's approval? Health 1416: up 6, degraded 1 (the terminal; reason not surfaced), down 0. - **Acceptance (1350):** URGENT mail from infra-ops → bridge ingested and dispatched (200) → `svos` reply on the thread at 1350:54 saying the page fired (text, then speech, per the connector receipt). Miranda's explicit reply to infra-ops is held on card `05cf119e…`, because the payload grants `miranda → app:app-postoffice write` WITHOUT `autonomous`. (Read back 2026-10-12 from the kernel DB: Prime APPROVED that card at 20:54:25Z, 1354:25 PT, so her reply went out.) Two observations to svos-dev (thread `01M4NNGCM3X75DANGXQEQ1EQGX`): the text on the thread reads as written to Prime; is every bus reply meant to need Prime's approval? Health 1416: up 6, degraded 1 (the terminal; reason not surfaced), down 0.
## Gotchas paid for during the edit ## Gotchas paid for during the edit
@@ -21,3 +21,8 @@
- A `caddy validate` of a file under a 0700 staging dir fails "permission denied" as user caddy; a `| tail` hid it once. Validate from `/etc/caddy/Caddyfile.check` and check the exit code (positive-controlled with a bogus directive → rc 1). - A `caddy validate` of a file under a 0700 staging dir fails "permission denied" as user caddy; a `| tail` hid it once. Validate from `/etc/caddy/Caddyfile.check` and check the exit code (positive-controlled with a bogus directive → rc 1).
- `secret` (Vaultwarden via `bw`) calls are serialized and can take minutes each when another session is reading the vault; run them in the background. - `secret` (Vaultwarden via `bw`) calls are serialized and can take minutes each when another session is reading the vault; run them in the background.
- Printing `~/.hermes/config.yaml` sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13. - Printing `~/.hermes/config.yaml` sections exposed the High Seat's Bifrost dispatch key in the local transcript; Prime ruled no rotation (10-11) since the key leaves use with U13.
## Moved from the in-flight section on 2026-10-14 (verbatim)
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card `card:05cf119e604649838e2a4c642c161e26` (Miranda's carded `send`) WAS approved by `prime` at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (`/var/lib/svos/state/svos.db`, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. `89dad7a` (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. `7f95e54` (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`).
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go.
@@ -0,0 +1,3 @@
# [2026-10-11] worldtree-dev answering again; #416 residual reopened
`[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
@@ -0,0 +1,3 @@
# [2026-10-12] Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward
`[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The cutover walk (1809–1813, 0 degraded) left memory 0 adult / 18 personal, feedback 2 / 0, nonfiction 0 / 0 and fiction 0 / 0, and the timer is back. **The one open margin is the personal reading (18 vs INC's 6).** albok-dev is re-fitting it forward on the live seat in small batches. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
@@ -0,0 +1,3 @@
# [2026-10-12] Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done
`[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** **Done 1725** (`nh3-docker-runner`, label `nh3-docker`, no docker socket in jobs, a 4 GiB cap). vastblue's pytest went 14m17s → 4m45s; albok is green (run 11051). ⚠ albok's failures were the slow fsync, never a hang: my 10-12 "hangs" diagnosis was wrong. The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
@@ -0,0 +1,3 @@
# [2026-10-12] U13 regression fixed (app panes 403)
`[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
@@ -0,0 +1,3 @@
# [2026-10-13] A171: Miranda's bus sends no longer card
`[2026-10-13]` **A171: Miranda's bus sends no longer card (Prime 1400, "give miranda more autonomy", via svos-dev).** On nh3-svos the `miranda → app:app-postoffice write` grant got `autonomous = true` (`playbooks/nh3-svos-a171-miranda-postoffice-autonomous.yaml`; backup `instance.toml.pre-a171`), and config_gen went `acae45206bbc9fcb → 9139783f413b45fa`. Revert = restore the backup and `systemctl reload svos-kernel`.
@@ -0,0 +1,3 @@
# [2026-10-13] Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime
`[2026-10-13]` **Albok note intake (Prime 1530: albok subsumes library/note intake; Muninn's watcher, muninn-gate and mimir-inbox retired). Gate 2 DONE 1735:** wing `personal/inbox` on nh3-albok, Nemi token re-minted `01M4VX207PJ09ZSECASCSZPCFE` (vaulted `albok/token-nemi`; the old `01M4HAP9…` revoked 1742 after albok-dev's swap; the capture source walks), and `/srv/nemi/inbox` on nh3-dev (`playbooks/nh3-dev-nemi-inbox.yaml`). **Gate 1 (ruling 6, Prime's Obsidian vault synced read-only onto nh3-dev) WAITS ON PRIME:** the vault's location, plus approval for Syncthing receive-only (recommended). Thread `01M4VWV5FEF64EP71W60N6Q1N0`.
@@ -0,0 +1,3 @@
# [2026-10-13] app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin
`[2026-10-13]` **app-terminal was degraded from the U13 cutover until 0856 on 10-13: its user unit's PATH lacked `~/.cargo/bin`, so every `zellij` call failed to launch** (rc 127). That broke /health AND every dump/inject. Fixed with `Environment=PATH=` in `configs/svos-apps/nh3-dev/svos-terminal.service` (3c92a12). The kernel logged app-terminal up at 15:56:40Z, the manifest was unchanged, and no card was raised; kernel health reads up 7 / degraded 0. The restart moved its process from release dc241f8 to adc862e (same svos_terminal 0.0.3). ⚠ Any new user unit that shells out to a cargo/uv/bun tool needs PATH set explicitly.
@@ -0,0 +1,3 @@
# [2026-10-13] muninn-gate + mimir-inbox STOPPED (retired)
`[2026-10-13]` **muninn-gate (corviduo-dev:8090) and mimir-inbox (:8091) STOPPED 2123** (Prime retired them at 1530; worldtree-dev: unit 6 needs the data at rest only, and up was worse because they kept accepting into an undrained queue). `compose down` only: images, compose dirs, config and `/mnt/muninn-staging/mimir-inbox` stay for albok's unit 6 (albok-dev owns the backup and teardown). Worldtree is a pure albok-service HTTP customer since beba08a4. Strip the retired config blocks (providers `library:`, `graph_store:`, defaults `kb_audit:`; NOT defaults `library:`) per instance only after it runs beba08a4+, demo first. Thread `01M4WA58P2M001K1420NN72WFM` (worldtree-dev).
@@ -0,0 +1,3 @@
# [2026-10-13] Prime ~2120: notes copy approved; fleet infra-ops key vaulted
`[2026-10-13]` **Prime ~2120: "you may copy the notes, put the key in the vault".** (1) The fleet infra-ops SSH private key is vaulted as `nh3-dev/.ssh/infra-ops_ed25519`; the restore was tested (same public key derived, login to esh-nas). (2) Gate 1 is approved for the copy of Prime's Obsidian notes onto nh3-dev, BUT the vault's location is unknown: no `.obsidian` dir on nh3-dev (bar oldAIPA's old KB) or on esh-nas's shares (depth 5). Asked Prime where. Plan: Syncthing receive-only onto nh3-dev, the root path to albok-dev.
@@ -0,0 +1,3 @@
# [2026-10-13] SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed
`[2026-10-13]` **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed 2131–2133** after the 21:30 vzdump: schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7; nh3-dev current → 452a592, terminal 0.0.4 pinned (no card). The tasks artifact mapping uses host `nh3-svos` (= the terminal's artifact_host), not svos-dev's `nh3-dev`. Rollback and details: `servers/nh3-svos/README.md`. Open: Prime's `GET /tasks` check, and a `task_argv` for the claude runtime when he names it.
+41 -50
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management # Persistent memory — eshpfi-management
_Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)_ _Last updated: 2026-10-14 (snapshot for clear, elway note updated: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
@@ -30,7 +30,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) — **[2026-09-24] MOTHBALLED** by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`) | push-to-main → CI deploys (2026-04-29) | | `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) — **[2026-09-24] MOTHBALLED** by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`) | push-to-main → CI deploys (2026-04-29) |
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) — **[2026-10-17] RETIRED** by Prime: `compose down` on ana-docker (DB in `/opt/docker/conf/asset-engine/db` incl. its unmerged WAL, image `asset-engine:local`, repo all kept; bring-back = `docker compose up -d`). Last job 2026-08-02, and every catalog endpoint had been dead since the 09-06 mesh cutover. ⚠ CI still deploys on a push to main, so a push would resurrect it | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | | `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
@@ -99,7 +99,10 @@ no longer deployed sidecars here. See Recent decisions.)
- **`scripts/elway` sudo handling** — elway prompts for the sudo password - **`scripts/elway` sudo handling** — elway prompts for the sudo password
ONCE via `getpass` before the first `sudo: true` step → can't run ONCE via `getpass` before the first `sudo: true` step → can't run
unattended from a non-TTY tool if any step needs sudo. Sudo-free unattended from a non-TTY tool if any step needs sudo. Sudo-free
playbooks run fully non-interactive over key SSH. playbooks run fully non-interactive over key SSH. — `[2026-10-13]` superseded for the
infra-ops identity: run as `infra-ops@<ip>` (NOPASSWD sudo), elway's sudo steps run
unattended with no prompt (every playbook run on 10-12/10-13). The getpass prompt only
fires for an account that needs a sudo password, e.g. bootstrapping a host as lkraven.
- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo - **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
@@ -113,19 +116,21 @@ no longer deployed sidecars here. See Recent decisions.)
## Current state / in-flight ## Current state / in-flight
_As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._ _As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
### 2026-10-11: live now (as of 2026-10-11 ~1400 PT) ### 2026-10-14: live now (as of 2026-10-14 ~0005 PT)
- **Nothing is mid-action.** No claims held. eshpfi is pushed through `df7161b`; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with `git commit -- <paths>`. `graphify-out/GRAPH_REPORT.md` stays modified on purpose. - **Nothing is mid-action.** No ops-log claims held. eshpfi `main` is pushed through `d1192ca` (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with `git commit -- <paths>` and read `git log origin/main..main` before pushing. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
- **U13 cutover: executed 10-11 0721–0816, acceptance 1350.** Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin `svos`, and the bridge app-postoffice on nh3-dev owning the `miranda` push route. The URGENT bus test (thread `01M4PBNX5PP0PGHDTY5BG7HG95`) ingested, paged per the connector receipt, and a reply came back on the thread from `svos`. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md` - **SVOS: main `33a7077` (v0.0.13+2: kernel 0.0.13, interface 0.0.5 with briefs, postoffice 0.0.3, UI 0.0.18) since 2026-10-14 1108–1111; config_gen `33f001f4e2dbc7dc`, apps up 7.** ⚠ Lesson (paid 10-14): a manifest revision whose verbs change is HELD for the owner, and the app reads DOWN (`refused wrong-revision`) until Prime approves its card (15-min expiry). Roll such apps only while he is at the board, and the post-office bridge last (held, it carries no Miranda mail). The interface unit needs the drop-in `svos-interface.service.d/20-artifact-host.conf` (`SVOS_ARTIFACT_HOST=nh3-svos`), and the briefs mapping host must equal it. Before that, kernel 0.0.12 (`d95078c`, 0416) on top of v0.0.11 (452a592, U15 task seats slice 1). The `hermes` runtime is the only task-capable one (one-off `hermes -z`, `process = "python3"`, no `--yolo`: Prime's call); its old argv is in `instance.toml.pre-hermes-task`. Earlier: v0.0.11 is live on nh3-svos (schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7) and nh3-dev (`/opt/svos/current → releases/452a592`; svos-terminal 0.0.4 runs from it). svos-dev accepted it. The task command is ruled with svos (Prime, 2026-10-14): infra-ops edits runtimes only at svos-dev's ask. `GET /tasks` as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 `89dad7a`, 0.0.15 `7f95e54` (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run `33a7077`; on nh3-dev svos-postoffice runs `33a7077`, svos-terminal `452a592`; svos-miranda runs `adc862e`, svos-agent still runs `dc241f8` (check `/proc/<pid>/maps` before deleting any release dir). Rollback material: nh3-svos `/var/lib/svos/backup/pre-452a592/` (schema-12 DB + config tar), `instance.toml.pre-452a592`, `instance.toml.pre-a171`; nh3-dev `terminal.toml.pre-452a592`. U13 history and the static-release chain: → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510`, `afab123`, `adc862e` and nh3-dev `/opt/svos/releases/afab123` can go once no process maps them.
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ CORRECTION (1412): card `05cf119e…` was NOT approved (I had inferred it from the reply arriving); it was her own carded `send`, is moot, and Prime can deny it. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`). - **Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback).** The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat.
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go. - **albok note intake (Prime 1530 10-13):** gate 2 DONE (wing `personal/inbox`, `/srv/nemi/inbox` on nh3-dev, nemi token `01M4VX207PJ09ZSECASCSZPCFE`). Gate 1 (ruling 6) DONE 2026-10-15 0140: Prime's notes are `/home/lkraven/kb` = gitea `vh/VuongKB` (he syncs them with git, not Syncthing), so Nemi's copy is a fresh clone at `/srv/nemi/vault`, pulled hourly at :50 by the lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`). It holds 129 md outside dot-dirs; the untracked Muninn output (`main/`, `fiction/`) is not in it. **Walked 0143–0147 10-15:** 232 objects (129 notes + 103 children), 2 judged personal, 0 adult. The nemi token was re-minted again (`01M4ZBJG3DMRF9Q9XEVZ8Q20GH`, + personal/notes) and the old one revoked (thread `01M4VWV5FEF64EP71W60N6Q1N0`). Prime says the kb needs cleaning up (its last commit is 2026-04-16).
- **Worldtree:** demo was deploying `beba08a4` (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Retired-config strip: **demo DONE 2026-10-15 2339 on Prime's go** (instance-configs `2240110`: providers `graph_store` + `library`, plus the b193-retired `memory.*` keys; the boot shows 0 retired-key warnings, down from 7). (0823 10-16: all three moved to v1.0.0b194 `daf587396f7a`; pinned via Deploy-pinned 11136, from its May 446e5807 build.) **Personal's strip DONE 2026-10-16 1036 on Prime's yes** (instance-configs `4d1aae2`, the same patch as demo; boot shows 0 retired-key warnings, down from 7). **The Matrix bridge becomes profile-gated** in worldtree-dev's next release: `COMPOSE_PROFILES=matrix` is set on the demo and personal box .env (and in env.public, `5bc796a`); pinned sets NOTHING. **All three on v1.0.0b195 (`a416ac01c5a2`) since 1112 10-16** (demo/personal via CI 11145/11146, pinned via Deploy-pinned 11147). Pinned renders api only, and its dead bridge container was removed. The image no longer carries KB_PATH. ⚠ Pinned's new compose bind-mounts `/opt/worldtree-pinned/config`: the entrypoint SEEDED it from the image's config-defaults on first start, so pinned now HAS a host config dir that vh/worldtree-instance-configs does not yet track (pinned was out of scope while its config was baked in). Capture it. Keep defaults `library:` (thread `01M4WA58P2M001K1420NN72WFM`). Daily memory gate: 10-13 re-run `20261013T160414Z` PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED. **Albok unit 6 teardown (thread `01M4WAEFZEA1YY03ZR31AEHGTS`): steps 1-4 DONE 2026-10-16 0836.** 8 tars (3 worldtree-kb volumes, mimir-inbox staging, personal `state/ingestion`, muninn-gate conf, both compose dirs) were test-restored with per-file sha256 identical, and are now on `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016` (0700, re-verified there; local copy `/var/tmp/albok-u6`). **Deletion (step 5) is HELD for Prime's direct go**: albok-dev relayed "land all of u6", and a relayed go does not authorize a delete. The images muninn-gate:0.0.14 + mimir-inbox:0.0.1 were already gone at 0830, cause unknown. The in-process ingestion watcher ran until the b194 deploy (last heartbeat 0821:45). worldtree-dev cleared the ingestion subtree from the b194 source (0845). I removed personal .env's no-op `COMPOSE_PROFILES=muninn` (0846). The two 07-27 `.bak-muninn` files go in step 5 with no archive (the .env one holds live secrets). personal's providers.yaml `write_agents: ["muninn"]` goes with personal's retired-config strip (on Prime's yes). **albok-service 0.1.24 (sha256:75be71c1) LIVE since 0917 on nh3-albok, with personal/sealed configured (8 wings ready).** Harness (a) PASSED 0922: LOCK 0920:08-0920:38 (423s, search `locked`), live UNLOCK 0921:53 with no restart; the synthetic object was hard-deleted. The sealed wing stays MOUNTED. **Step 5 DONE 2026-10-16 1031 on Prime's direct go:** the 3 worldtree-kb volumes, personal `state/ingestion`, the muninn/mimir dirs and staging, and the 2 .bak files are all deleted. I also pruned the build cache (345.7MB) and removed /var/tmp/albok-u6; the NAS archive is the only copy. Found during the readback: **pinned's Matrix bridge had crash-looped since the 0822 b194 deploy** (no matrix.yaml, empty MATRIX tokens: pinned never had a bridge). I stopped it at 1033. The next pinned deploy recreates it; the durable fix is worldtree-dev's compose.
- **satellite-1:** its own key is authorized for infra-ops on esh-nas only (`playbooks/files/infra-ops-satellite-keys`). Waiting on Prime's login test from the satellite and its machine name (the key's comment came through as bare `infra-ops@`).
- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash. - **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash.
- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica. - **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica.
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key. - **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
- **Waiting on Prime:** rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope. - **Waiting on Prime:** where his Obsidian vault lives (gate 1); satellite-1's login test + name; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). - **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's `/opt/svos/releases/` and the h2o eval seat on fv-ml1 GPU 3 (`stacks/h2o-eval`, down) are kept as artifacts.
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in. - **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked. - **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
@@ -140,7 +145,7 @@ _As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections ca
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — | | 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) | | 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× | | 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
| 1 | intern-decision | 32,768/call | cap 14.4 GiB, peak 15,220 | — | — | | 1 | intern-decision = **h2o-lightning-4b** since 10-12 1807 (production; no INC fallback per Prime) | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× | | 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`). Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
@@ -156,7 +161,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01) ### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
- **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart. - **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
- **Rollback:** `docker stop parakeet-nemo && docker start parakeet` (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md` - **Rollback:** `docker stop parakeet-nemo`, then `docker compose up -d` in `/opt/docker/compose/parakeet` (the old container was removed 2026-10-17 to clear a stale Homepage entry; the image and compose dir are kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
### Worldtree U11 memory cutover (demo + personal) ### Worldtree U11 memory cutover (demo + personal)
@@ -166,10 +171,12 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file. - **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements"). - **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
### intern-decision (replaced SemIf on 2026-09-30) ### The decision seat `intern-decision` :8033 (h2o-lightning-4b, production since 2026-10-12 1807)
- **LIVE 0.1.3** at `intern-decision.fv.internal:8033`: semif-compatible `/decide` plus Jev `/v1/systemone`, 32k tokens, a Triton cache volume. Run `scripts/intern-decision-warmup` after an IMAGE change. → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md` - **LIVE: h2o-lightning-4b @ b43f759b** (`compose.yaml` = `compose.h2o.yaml`) behind `services/h2o-decision-front`: 32k refuse-not-trim admission, bearer, `/decide/shared` translator. Inline images only (`LIGHTNING_IMAGE_URLS=off` + vLLM `--allowed-media-domains media.invalid`, both verified). Nemi's judge runs v2 (albok-nemi 0.1.18) at **concurrency 4**.
- **Open:** label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim. - ⚠ **Prime 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** `rollback-to-inc.sh` + `compose.inc.yaml` remain on the host as artifacts only. Seat problems are fixed forward in the judge (wording, thresholds, `visibility` declarations); INC again needs Prime's explicit word.
- ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1.
- INC history: → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`.
### nh3-pve + nh3-ml1 (2026-09-25/26) ### nh3-pve + nh3-ml1 (2026-09-25/26)
@@ -178,7 +185,22 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
## Recent decisions ## Recent decisions
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`. - `[2026-10-17]` **asset-engine RETIRED (Prime)**, along with the stale speech entries: the Homepage Speaches (irv-ml1) and sherpa Parakeet (fv-ml1) ghosts were removed via `docker rm` of their dead containers (images and compose dirs kept); the Uptime Kuma monitor was deleted (`scripts/kuma delete`, new); asset-engine is `compose down` (data kept). Evidence: last job 08-02; all 21 catalog endpoints on the dead 10.100.79.3. FleetTools, the agent bootstrap and global CLAUDE.md (dotfiles f909dcf, unpushed) no longer list `:8200`.
- `[2026-10-16]` **albok unit 6: the first encrypted wing `personal/sealed` provisioned on nh3-albok** (Prime's goal "land all of u6", via albok-dev): a 20 GiB LUKS2 file + ext4, key ONLY in vault `albok/wing-key-personal-sealed`, gid 1513, mountpoint `/srv/albok/sealed/personal.sealed` (immutable while unmounted), the parent bound rslave. The lock/unlock lifecycle was tested with the container running. It is not in albok.yaml until albok-dev's release. Runbook: `stacks/albok-service/README.md` § Sealed. After an nh3-albok reboot the wing is LOCKED until someone unlocks it.
- `[2026-10-13]` **pewpewstudio LIVE on nh3-docker :8185** (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a `route` (foreign Host 400); DB, assets and shop config on named volumes. → `stacks/pewpewstudio/README.md`
- `[2026-10-13]` **infra-ops on esh-nas (CT 103) + per-host satellite keys (Prime)** — bootstrapped via pct from esh-pve-nas; satellite-1 authorized on esh-nas only (`playbooks/files/infra-ops-satellite-keys`); its login test + name pending.
- `[2026-10-13]` **Worldtree memory gate: 0730 harness error (the tagger's MalformedClassification latched the writer), re-run `20261013T160414Z` PASS (median 0.83)** — the latch fix is worldtree-dev's (thread `01M4V035SF73XKVG3NRVX37XQ6`).
- [2026-10-13] **SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed** — schema 13, terminal 0.0.4 pinned without a card, tasks mapping host nh3-svos. → `persistent-memory.d/2026-10-13-svos-v0-0-11-deployed.md`
- [2026-10-13] **muninn-gate + mimir-inbox STOPPED (retired)** — data at rest kept for albok unit 6; Worldtree retired-config strip order. → `persistent-memory.d/2026-10-13-muninn-gate-mimir-inbox-stopped.md`
- [2026-10-13] **Prime ~2120: notes copy approved; fleet infra-ops key vaulted** — restore tested; vault location still unknown. → `persistent-memory.d/2026-10-13-notes-copy-approved-fleet-key-vaulted.md`
- [2026-10-13] **Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime** — wing, token, ACL trap. → `persistent-memory.d/2026-10-13-albok-note-intake-gate-2.md`
- [2026-10-13] **A171: Miranda's bus sends no longer card** — one grant made autonomous; config_gen and revert. → `persistent-memory.d/2026-10-13-a171-miranda-postoffice-autonomous.md`
- [2026-10-13] **app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin** — zellij unreachable; dump/inject broken too. → `persistent-memory.d/2026-10-13-app-terminal-path-fix.md`
- `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.
- [2026-10-12] **Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward** — the judge's v1 question, the margins, the lessons. → `persistent-memory.d/2026-10-12-decision-seat-h2o-rollbacks-and-cutover.md`
- [2026-10-12] **U13 regression fixed (app panes 403)** — base_url back to the terminal origin, seat routes allowed there. → `persistent-memory.d/2026-10-12-u13-regression-fixed.md`
- [2026-10-12] **Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done** — measurements, the corrected albok diagnosis. → `persistent-memory.d/2026-10-12-shared-ci-runner-slow-fsync.md`
- [2026-10-11] **worldtree-dev answering again; #416 residual reopened** — four threads answered at once. → `persistent-memory.d/2026-10-11-worldtree-dev-reading-mail.md`
- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md` - `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md` - `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md`
- `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md` - `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md`
@@ -257,25 +279,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md` - `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
- `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md` - `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
- `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md` - `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md` - `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
- `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`). - `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`).
- `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`. - `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`.
- `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`. - `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`.
@@ -299,7 +303,7 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md` - `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
_240 older entries archived to archival-memory.md._ _258 older entries archived to archival-memory.md._
## Tried and abandoned ## Tried and abandoned
@@ -315,18 +319,5 @@ _240 older entries archived to archival-memory.md._
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`) - `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file). - `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md` - `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
_128 older entries archived to archival-memory.md._ _141 older entries archived to archival-memory.md._
+41
View File
@@ -0,0 +1,41 @@
# nh3-albok: the personal/inbox wing (unit 4.4 slice C's capture target; albok-dev 2026-10-13, after Prime's 1530 ruling
# that albok subsumes library/note intake). Like personal/notes: building personal, default_group albok-personal (1511),
# not service-restricted. Run between two Nemi walks (albok-nemi.timer on nh3-dev fires hourly at ~:01).
# scripts/elway infra-ops@10.100.50.44 --playbook playbooks/albok-personal-inbox.yaml
steps:
- name: Back up the live config
sudo: true
shell: cp -p /srv/albok/etc/albok.yaml /srv/albok/etc/albok.yaml.pre-inbox
creates: /srv/albok/etc/albok.yaml.pre-inbox
- name: Add wing personal/inbox after personal/notes
sudo: true
shell: |
python3 - <<'PY'
import os, grp
p = "/srv/albok/etc/albok.yaml"; s = open(p).read()
if " - id: inbox" in s:
print("already present"); raise SystemExit(0)
old = " wings:\n - id: notes\n"
assert s.count(old) == 1, s.count(old)
s = s.replace(old, old + " - id: inbox # unit 4.4 capture wing (Nemi's drop-box /srv/nemi/inbox on nh3-dev); like notes (2026-10-13)\n")
t = p + ".tmp"
fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o640)
with os.fdopen(fd, "w") as f: f.write(s)
os.chown(t, 0, grp.getgrnam("albok").gr_gid); os.replace(t, p); print("edited")
PY
- name: Restart the service (config is read at start) and wait for health
sudo: true
shell: |
docker restart albok-service >/dev/null
for i in $(seq 1 60); do
h=$(curl -s -m 3 http://127.0.0.1:8392/health) && echo "$h" | grep -q '"status":"ok"' && break; sleep 2
done
echo "$h" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('status'), d.get('wings'), d.get('leaks'))"
verify:
- name: Seven wings ready, none degraded
shell: curl -s -m 5 http://127.0.0.1:8392/health | python3 -c "import json,sys; w=json.load(sys.stdin)['wings']; print(w); assert w['ready']==7 and w['degraded']==0 and w['locked']==0"
changed_when: "false"
- name: The wing's store dir carries the personal group
sudo: true
shell: stat -c '%A %U:%G %n' /srv/albok/store/personal/inbox* | head -3
changed_when: "false"
+17 -4
View File
@@ -7,7 +7,8 @@
# WHY a dedicated identity (not the agent borrowing lkraven/vh sudo): # WHY a dedicated identity (not the agent borrowing lkraven/vh sudo):
# - Clean audit trail — every agent action attributable to `infra-ops`, # - Clean audit trail — every agent action attributable to `infra-ops`,
# separate from human accounts, with sudo I/O logging on. # separate from human accounts, with sudo I/O logging on.
# - Revocation = pull one authorized_keys line + one sudoers file. # - Revocation = pull one authorized_keys line + one sudoers file. Satellite agents carry their own keys,
# listed per host in playbooks/files/infra-ops-satellite-keys; revoke one by deleting its line and re-running.
# - No human password ever in the agent's reach (key-gated NOPASSWD). # - No human password ever in the agent's reach (key-gated NOPASSWD).
# #
# RUN (operator, with YOUR sudo — infra-ops doesn't exist yet, so you bootstrap # RUN (operator, with YOUR sudo — infra-ops doesn't exist yet, so you bootstrap
@@ -25,7 +26,8 @@
# esh-pve-nas) DO get infra-ops — done that day. # esh-pve-nas) DO get infra-ops — done that day.
# PVE ships without sudo: `apt-get install sudo` first. # PVE ships without sudo: `apt-get install sudo` first.
# Also granted by operator request: esh-docker-vm, # Also granted by operator request: esh-docker-vm,
# esh-vm-db (2026-08-24) and vm-esh-nas (2026-09-27). # esh-vm-db (2026-08-24), vm-esh-nas (2026-09-27) and esh-nas
# (CT 103, 2026-10-13, via pct from its host: playbooks/esh-nas-infra-ops.yaml).
# - nh3-nas (Synology DSM) (no standard useradd / sudoers.d) # - nh3-nas (Synology DSM) (no standard useradd / sudoers.d)
# #
# Idempotent: re-running reconciles the key + sudoers without error. # Idempotent: re-running reconciles the key + sudoers without error.
@@ -63,12 +65,23 @@ steps:
sudo: true sudo: true
shell: install -d -m 700 -o {{ ops_user }} -g {{ ops_user }} /home/{{ ops_user }}/.ssh shell: install -d -m 700 -o {{ ops_user }} -g {{ ops_user }} /home/{{ ops_user }}/.ssh
- name: Install infra-ops authorized_keys (exact line, mode 600) # Satellite agents get their OWN keys, per host, from playbooks/files/infra-ops-satellite-keys (Prime 2026-10-13).
- name: Stage the satellite key list
upload:
src: playbooks/files/infra-ops-satellite-keys
dest: /tmp/infra-ops-satellite-keys
mode: "0644"
- name: Install infra-ops authorized_keys (the fleet key + this host's satellite keys, exactly; mode 600)
sudo: true sudo: true
shell: | shell: |
line='{{ ops_pubkey }}' line='{{ ops_pubkey }}'
[ -n '{{ ssh_from }}' ] && line='from="{{ ssh_from }}" {{ ops_pubkey }}' [ -n '{{ ssh_from }}' ] && line='from="{{ ssh_from }}" {{ ops_pubkey }}'
printf '%s\n' "$line" > /home/{{ ops_user }}/.ssh/authorized_keys { printf '%s\n' "$line"
awk -v h="$(hostname -s)" '!/^#/ && $1==h { $1=""; sub(/^ /,""); print }' /tmp/infra-ops-satellite-keys
} > /home/{{ ops_user }}/.ssh/authorized_keys
rm -f /tmp/infra-ops-satellite-keys
echo "satellite keys for $(hostname -s): $(($(wc -l < /home/{{ ops_user }}/.ssh/authorized_keys) - 1))"
chown {{ ops_user }}:{{ ops_user }} /home/{{ ops_user }}/.ssh/authorized_keys chown {{ ops_user }}:{{ ops_user }} /home/{{ ops_user }}/.ssh/authorized_keys
chmod 600 /home/{{ ops_user }}/.ssh/authorized_keys chmod 600 /home/{{ ops_user }}/.ssh/authorized_keys
+8 -4
View File
@@ -10,11 +10,12 @@
# # Central runner (defaults match ana-docker) # # Central runner (defaults match ana-docker)
# scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml # scripts/elway ana-docker --playbook playbooks/deploy-gitea-runner.yaml
# #
# # Site-local runner — override identity at the CLI; the .env on # # The OPT-IN fast runner on nh3-docker (Prime's CI fix A, 2026-10-12): its own label only, and its own config.
# # the host carries the registration token + image pin. # # The .env on the host carries the registration token + image pin.
# scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \ # scripts/elway nh3-docker --playbook playbooks/deploy-gitea-runner.yaml \
# --var runner_name=nh3-docker-runner \ # --var runner_name=nh3-docker-runner \
# --var runner_labels=pfi-fleet,nh3-docker # --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
# --var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
# #
# scripts/elway esh-docker-vm --playbook playbooks/deploy-gitea-runner.yaml \ # scripts/elway esh-docker-vm --playbook playbooks/deploy-gitea-runner.yaml \
# --var runner_name=esh-runner \ # --var runner_name=esh-runner \
@@ -40,6 +41,9 @@ vars:
# editing files on the host by hand. # editing files on the host by hand.
runner_name: "" runner_name: ""
runner_labels: "" runner_labels: ""
# Which act_runner config to upload. The nh3-docker fast runner uses its own, tighter one
# (conf/config.nh3-docker.yaml: no docker socket in jobs, capacity 1, memory-capped jobs).
runner_config: stacks/gitea-runner/conf/config.yaml
steps: steps:
# ── host-side directory prep ──────────────────────────────────────── # ── host-side directory prep ────────────────────────────────────────
@@ -60,7 +64,7 @@ steps:
- name: Upload runner config.yaml - name: Upload runner config.yaml
upload: upload:
src: stacks/gitea-runner/conf/config.yaml src: "{{ runner_config }}"
dest: "{{ data_dir }}/config.yaml" dest: "{{ data_dir }}/config.yaml"
mode: "0644" mode: "0644"
+23
View File
@@ -0,0 +1,23 @@
# Bootstrap the infra-ops identity INSIDE esh-nas (CT 103 on esh-pve-nas, 10.0.50.50), Prime 2026-10-13 ("set it up").
# infra-ops could not log into the container (no user), so this runs on the HOST and reaches in with pct.
# scripts/elway infra-ops@10.0.50.55 --playbook playbooks/esh-nas-infra-ops.yaml
# The in-container steps are scripts/oneoff/esh-nas-infra-ops/bootstrap-in-ct.sh (= playbooks/bootstrap-infra-ops-user.yaml).
vars:
ctid: "103"
steps:
- name: Stage the bootstrap script on the host
upload:
src: scripts/oneoff/esh-nas-infra-ops/bootstrap-in-ct.sh
dest: /tmp/esh-nas-bootstrap-infra-ops.sh
mode: "0700"
- name: Push it into CT {{ ctid }} and run it as root there
sudo: true
shell: pct push {{ ctid }} /tmp/esh-nas-bootstrap-infra-ops.sh /root/bootstrap-infra-ops.sh --perms 0700 && pct exec {{ ctid }} -- bash /root/bootstrap-infra-ops.sh
- name: Remove the staged copies
sudo: true
shell: pct exec {{ ctid }} -- rm -f /root/bootstrap-infra-ops.sh && rm -f /tmp/esh-nas-bootstrap-infra-ops.sh
verify:
- name: The user and its sudoers file exist in CT {{ ctid }}
sudo: true
shell: pct exec {{ ctid }} -- sh -c 'id infra-ops && visudo -cf /etc/sudoers.d/infra-ops && grep -q NOPASSWD:ALL /etc/sudoers.d/infra-ops'
changed_when: "false"
+9
View File
@@ -0,0 +1,9 @@
# Satellite agents' own keys for the infra-ops identity, scoped PER HOST (Prime 2026-10-13).
# Format: <target's `hostname -s`> <public key line>. playbooks/bootstrap-infra-ops-user.yaml writes the fleet key
# plus every line here whose first field matches the target's short hostname. Nothing else survives, so a key
# that is not listed here is removed on the next run. Revoke a satellite by deleting its line(s) and re-running
# the playbook on those hosts. Public keys only: never put a private key in this repo.
#
# satellite-1: generated by Prime on his satellite machine 2026-10-13, fingerprint
# SHA256:KEFUjceFjuIZDuVbEq3FWYHm0rzdqx9hC+wFrA5h3xc (its own comment was "infra-ops@", with no hostname).
esh-nas ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL0XLLZVTJUTgrWbMAx5exF3UDqPdi/30l/1lRo6d4bt infra-ops@satellite-1
+39
View File
@@ -0,0 +1,39 @@
# nh3-dev: /srv/nemi/books, the book drop Nemi validates into ~/library/<shelf>/ (albok unit 8; albok-dev 2026-10-16 after
# Prime: "build the books drop folder in nemi ... a rejection folder feels appropriate"). Same shape as /srv/nemi/inbox
# (playbooks/nh3-dev-nemi-inbox.yaml, which created group nemi-drop and /srv/nemi): Nemi moves each valid epub from
# fiction/ or nonfiction/ into ~/library/<shelf>/, and anything it refuses into rejected/ beside a reason file.
# scripts/elway infra-ops@10.100.10.50 --playbook playbooks/nh3-dev-nemi-books.yaml
# Writers: group nemi-drop (lkraven, infra-ops). Nemi runs as lkraven from the user manager, whose groups predate
# nemi-drop, so lkraven gets in through the u:lkraven ACL entries, NOT the group (see servers/nh3-dev/README.md).
steps:
- name: Prerequisites from the inbox playbook (group nemi-drop, /srv/nemi with u:lkraven:rx)
shell: |
getent group nemi-drop >/dev/null && test -d /srv/nemi && getfacl -p /srv/nemi | grep -q '^user:lkraven:r-x'
changed_when: "false"
- name: /srv/nemi/books and fiction/ nonfiction/ rejected/ (lkraven:nemi-drop 2770, same ACL as the inbox)
sudo: true
shell: |
for d in /srv/nemi/books /srv/nemi/books/fiction /srv/nemi/books/nonfiction /srv/nemi/books/rejected; do
install -d -o lkraven -g nemi-drop -m 2770 "$d"
setfacl -m u:lkraven:rwx,g:nemi-drop:rwx "$d"
setfacl -d -m u::rwX,u:lkraven:rwX,g::rwX,g:nemi-drop:rwX,o::--- "$d"
done
getfacl -p /srv/nemi/books
creates: /srv/nemi/books/rejected
verify:
- name: Nemi's own context (lkraven's systemd user manager, groups fixed at boot) can list every books dir
shell: runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemd-run --user --wait --pipe -q ls -ld /srv/nemi/books /srv/nemi/books/fiction /srv/nemi/books/nonfiction /srv/nemi/books/rejected
sudo: true
changed_when: "false"
- name: A file dropped by infra-ops into fiction/ is readable by lkraven, movable to rejected/, and removable
sudo: true
shell: |
set -e
f=/srv/nemi/books/fiction/.acl-probe-$$
r=/srv/nemi/books/rejected/.acl-probe-$$
runuser -u infra-ops -g nemi-drop -- sh -c "umask 077; echo probe > $f"
runuser -u lkraven -- cat "$f" >/dev/null
runuser -u lkraven -- mv "$f" "$r"
runuser -u lkraven -- rm -f "$r"
test ! -e "$f" && test ! -e "$r" && echo "infra-ops drop -> lkraven read+move+remove: ok"
changed_when: "false"
+38
View File
@@ -0,0 +1,38 @@
# nh3-dev: /srv/nemi/inbox, the capture root Nemi walks into albok's personal/inbox wing (unit 4.4 slice C; albok-dev
# 2026-10-13 after Prime's 1530 intake ruling). People and agents drop markdown here; Nemi ingests each file once
# (capture mode never tombstones on absence, so a walked file may be moved away).
# scripts/elway infra-ops@10.100.10.50 --playbook playbooks/nh3-dev-nemi-inbox.yaml
# Writers: group nemi-drop (lkraven, infra-ops). Nemi runs as lkraven from the user manager, whose groups were fixed
# at its start, so lkraven's access does not depend on the new group: the dir is lkraven-owned and a DEFAULT ACL gives
# lkraven and nemi-drop rw on everything created inside, whoever drops it and whatever their umask.
steps:
- name: Group nemi-drop with lkraven and infra-ops
sudo: true
shell: |
getent group nemi-drop >/dev/null || groupadd nemi-drop
usermod -aG nemi-drop lkraven; usermod -aG nemi-drop infra-ops
getent group nemi-drop
- name: /srv/nemi (root:nemi-drop 0750) and /srv/nemi/inbox (lkraven:nemi-drop 2770, default ACL)
sudo: true
shell: |
install -d -o root -g nemi-drop -m 0750 /srv/nemi
setfacl -m u:lkraven:rx /srv/nemi # traversal for lkraven's user manager, whose groups predate nemi-drop
install -d -o lkraven -g nemi-drop -m 2770 /srv/nemi/inbox
setfacl -m u:lkraven:rwx,g:nemi-drop:rwx /srv/nemi/inbox
setfacl -d -m u::rwX,u:lkraven:rwX,g::rwX,g:nemi-drop:rwX,o::--- /srv/nemi/inbox
getfacl -p /srv/nemi/inbox
verify:
- name: Nemi's own context (lkraven's systemd user manager, groups fixed at boot) can list the inbox
shell: runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemd-run --user --wait --pipe -q ls -ld /srv/nemi/inbox
sudo: true
changed_when: "false"
- name: A file dropped by infra-ops is readable and removable by lkraven
sudo: true
shell: |
set -e
f=/srv/nemi/inbox/.acl-probe-$$
runuser -u infra-ops -g nemi-drop -- sh -c "umask 077; echo probe > $f"
runuser -u lkraven -- cat "$f" >/dev/null
runuser -u lkraven -- rm -f "$f"
test ! -e "$f" && echo "infra-ops drop -> lkraven read+remove: ok"
changed_when: "false"
@@ -0,0 +1,36 @@
# nh3-svos: miranda -> app:app-postoffice write becomes autonomous (A171; Prime 2026-10-13 1400 via svos-dev).
steps:
- name: back up instance.toml
sudo: true
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-a171
creates: /var/lib/svos/config/instance.toml.pre-a171
- name: autonomous = true on the miranda -> app:app-postoffice write grant
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
old = '[[grants]]\nprincipal = "miranda"\nobject = "app:app-postoffice"\nmode = "write"\n[[grants]]'
new = '[[grants]]\nprincipal = "miranda"\nobject = "app:app-postoffice"\nmode = "write"\nautonomous = true # A171, Prime 2026-10-13 1400: her own bus sends no longer card\n[[grants]]'
assert s.count(old) == 1, s.count(old)
s = s.replace(old, new); t = p + ".tmp"
fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
- name: the kernel accepts it (else restore)
sudo: true
shell: |
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-a171 /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
- name: reload
sudo: true
shell: |
systemctl reload svos-kernel; sleep 6
curl -s http://127.0.0.1:8400/health; echo
journalctl -u svos-kernel --since "-20s" --no-pager -o cat | grep -i -E "refus|invalid|error|config" | tail -5 | cut -c1-200
verify:
- name: the grant reads autonomous
sudo: true
shell: runuser -u svc-svos -- python3 -c "import tomllib; d=tomllib.load(open('/var/lib/svos/config/instance.toml','rb')); g=[x for x in d['grants'] if x.get('principal')=='miranda' and x.get('object')=='app:app-postoffice' and x.get('mode')=='write']; print(g); assert len(g)==1 and g[0].get('autonomous') is True"
changed_when: "false"
@@ -0,0 +1,43 @@
# nh3-dev PHASE B of the v0.0.14 rollout: connector base 0.0.10 + app-terminal 0.0.5 (release 3726d73 already built beside
# the live one in /opt/svos/releases/3726d73). Adds [seat_host].scan_patterns (hermes's threat-pattern table) and restarts
# ONLY svos-terminal; postoffice, miranda and agent keep their running code until their next restart. Terminal's manifest
# adds the `scan` read verb: registration raises ONE card, and the app refuses wrong-revision until Prime approves it.
# Run as: scripts/elway infra-ops@10.100.10.50 (the apps are lkraven USER units, driven with runuser + XDG_RUNTIME_DIR).
steps:
- name: Release 3726d73 is built and pip-clean
shell: test -f /opt/svos/releases/3726d73/release.json && /opt/svos/releases/3726d73/apps-env/bin/python -m pip --version >/dev/null 2>&1; test -x /opt/svos/releases/3726d73/apps-env/bin/svos-terminal # uv is lkraven-only (0700 ~/bin); pip check ran clean at build time
changed_when: "false"
- name: Back up terminal.toml, then add scan_patterns under [seat_host]
sudo: true
shell: |
set -e
p=/var/lib/svos/config/apps/terminal.toml
runuser -u lkraven -- cp -p $p $p.pre-3726d73
runuser -u lkraven -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/apps/terminal.toml"; s = open(p).read()
anchor = 'tasks_root = "/var/lib/svos/apps/terminal/tasks" # U15 (v0.0.11): = the kernel\'s [tasks].scratch_root\n'
assert s.count(anchor) == 1 and "scan_patterns" not in s
s = s.replace(anchor, anchor + 'scan_patterns = "/home/lkraven/.hermes/hermes-agent/tools/threat_patterns.py" # U15 v0.7 (v0.0.14): hermes\'s own threat table; /health then says scan: on\n')
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
- name: Switch current to 3726d73 (as lkraven, who owns /opt/svos on nh3-dev)
sudo: true
shell: runuser -u lkraven -- sh -c 'ln -sfn /opt/svos/releases/3726d73 /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current'
- name: Restart ONLY svos-terminal
sudo: true
shell: |
runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user restart svos-terminal
sleep 6; runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user is-active svos-terminal
pid=$(runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user show svos-terminal -p MainPID --value); grep -oE '/opt/svos/releases/[0-9a-f]+' /proc/$pid/maps | sort -u
runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 journalctl --user -u svos-terminal --since "-60s" --no-pager -o cat | grep -i -E "scan|manifest|card|held|revision|error" | tail -8 | cut -c1-200
verify:
- name: Terminal runs 3726d73 and its seat host loaded the scan table (svos-dev 0753 - scan:on rides only on the signed /health, so read the start log; a table that fails to load prints "Seat host unavailable (ScanError)")
sudo: true
shell: |
pid=$(runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 systemctl --user show svos-terminal -p MainPID --value)
grep -q /opt/svos/releases/3726d73 /proc/$pid/maps && echo "terminal pid $pid on 3726d73"
! runuser -u lkraven -- env XDG_RUNTIME_DIR=/run/user/1000 journalctl --user -u svos-terminal --since "-5min" --no-pager -o cat | grep -i "Seat host unavailable"
changed_when: "false"
@@ -0,0 +1,88 @@
# nh3-svos -> SVOS main 33a7077 (v0.0.13 + 2): kernel 0.0.13 (no migration), connector base 0.0.9, app-interface 0.0.5
# (needs SVOS_ARTIFACT_HOST), app-postoffice 0.0.3 wheel (runs on nh3-dev), UI 0.0.18. One config row: the interface briefs
# artifact mapping (host = the interface's SVOS_ARTIFACT_HOST). svos-dev's ask on Prime's 0955 go (thread 01M4XSGTE30G33X6SR76B03JFJ).
steps:
- name: Private staging dir
shell: install -d -m 0700 /tmp/svos-stage-33a7077
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-33a7077/svos-release-33a7077.tar.gz, dest: /tmp/svos-stage-33a7077/release.tar.gz, mode: "0644"}}
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-33a7077/uv, mode: "0755"}}
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-33a7077/install-release.py, dest: /tmp/svos-stage-33a7077/install-release.py, mode: "0644"}}
- name: Verify staged hashes
shell: cd /tmp/svos-stage-33a7077 && printf '%s %s\n' abd2af171f3408431ab33ea097244692c846995c98308df1487413013fc5612c release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
changed_when: "false"
- name: Install release 33a7077 beside the live one (current NOT switched)
sudo: true
shell: python3 /tmp/svos-stage-33a7077/install-release.py
- name: New kernel accepts the live configuration as it is
sudo: true
shell: runuser -u svc-svos -- /opt/svos/releases/33a7077/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
- name: The interface unit gets SVOS_ARTIFACT_HOST (0.0.5 fails closed without it); takes effect at the restart below
sudo: true
shell: |
install -d -m 0755 /etc/systemd/system/svos-interface.service.d
printf '%s\n' '# U16 (2026-10-14): names every task brief reference; must equal the instance.toml mapping host byte for byte' '[Service]' 'Environment=SVOS_ARTIFACT_HOST=nh3-svos' > /etc/systemd/system/svos-interface.service.d/20-artifact-host.conf
systemctl daemon-reload; systemctl show svos-interface -p Environment | tr ' ' '\n' | grep ARTIFACT
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
sudo: true
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Offline backup to /var/lib/svos/backup/pre-33a7077 (schema 13 database + config)
sudo: true
shell: |
set -e
d=/var/lib/svos/backup/pre-33a7077
runuser -u svc-svos -- install -d -m 0700 $d
runuser -u svc-svos -- python3 -c "
import sqlite3
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-33a7077/svos.db')
src.backup(dst); dst.close(); src.close()
c = sqlite3.connect('file:/var/lib/svos/backup/pre-33a7077/svos.db?mode=ro', uri=True)
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-33a7077
ls -la $d | tail -3
- name: Config for U16 (the interface briefs mapping), then the NEW kernel must accept it (else restore + old kernel back)
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
anchor = '[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/terminal/tasks"\napp = "app-terminal"\nverb = "artifact"\n'
assert s.count(anchor) == 1, s.count(anchor)
assert "interface/briefs" not in s
add = ('\n# U16 (main 33a7077, 2026-10-14): app-interface task briefs. host = the interface unit\'s SVOS_ARTIFACT_HOST (drop-in\n'
'# svos-interface.service.d/20-artifact-host.conf), byte for byte; root = its SVOS_DATA_ROOT + /state/apps/interface/briefs.\n'
'[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/state/apps/interface/briefs"\napp = "app-interface"\nverb = "artifact"\n')
s = s.replace(anchor, anchor + add)
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
if runuser -u svc-svos -- /opt/svos/releases/33a7077/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-33a7077 /var/lib/svos/config/instance.toml
systemctl start svos-kernel.service svos-interface svos-knowledge svos-voice; echo "RESTORED config, old kernel restarted"; exit 1
- name: Switch current to 33a7077
sudo: true
shell: ln -sfn /opt/svos/releases/33a7077 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
- name: Start the kernel (READY-gated; no migration) then the local apps
sudo: true
shell: |
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
systemctl start svos-interface svos-knowledge svos-voice
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
- name: Regenerate the front door inventory (d95078c becomes the previous release)
sudo: true
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
verify:
- name: Inventory current
sudo: true
shell: /usr/local/sbin/svos-front-inventory --check
changed_when: "false"
- name: Health is schema 13, status ok, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
changed_when: "false"
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
changed_when: "false"
@@ -0,0 +1,23 @@
# nh3-svos PHASE C of the v0.0.14 rollout: start app-interface 0.0.6 (already installed in /opt/svos/current = 3726d73 by
# phase A). Its first start migrates the briefs index v1->v2 (copy kept in backup/pre-3726d73) and raises ONE card (kind/
# name on the brief verbs): the interface refuses wrong-revision until Prime approves it. Then regenerate the front door.
steps:
- name: current is 3726d73 and the kernel is schema 14
shell: test "$(readlink /opt/svos/current)" = /opt/svos/releases/3726d73 && curl -s http://127.0.0.1:8400/health | grep -q '"schema_version":14'
changed_when: "false"
- name: Start the interface
sudo: true
shell: |
systemctl start svos-interface; sleep 6; systemctl is-active svos-interface
journalctl -u svos-interface --since "-60s" --no-pager -o cat | grep -i -E "brief|migrat|v2|manifest|card|held|error" | tail -8 | cut -c1-200
- name: Regenerate the front door inventory (33a7077 becomes the previous release)
sudo: true
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
verify:
- name: Inventory current
sudo: true
shell: /usr/local/sbin/svos-front-inventory --check
changed_when: "false"
- name: Kernel health (the interface reads down until its card is approved)
shell: curl -s http://127.0.0.1:8400/health
changed_when: "false"
@@ -0,0 +1,24 @@
# nh3-svos: STAGE SVOS v0.0.14 (3726d73) beside the live release. Installs into /opt/svos/releases/3726d73 and runs the
# new kernel's check-config against the live configuration. Does NOT switch current, stop or start anything.
# svos-dev's ask (thread 01M5558E19Q3ZK4RRZXQG6A2Y9), pushed on Prime's 0245 go. The cutover is nh3-svos-3726d73-v0.0.14.yaml.
steps:
- name: Private staging dir
shell: install -d -m 0700 /tmp/svos-stage-3726d73
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-3726d73/svos-release-3726d73.tar.gz, dest: /tmp/svos-stage-3726d73/release.tar.gz, mode: "0644"}}
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-3726d73/uv, mode: "0755"}}
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-3726d73/install-release.py, dest: /tmp/svos-stage-3726d73/install-release.py, mode: "0644"}}
- name: Verify staged hashes
shell: cd /tmp/svos-stage-3726d73 && printf '%s %s\n' 380e64f6c5fa7d7e83844483600622bcda1a6bbe820ba9cf5854d0fc0ab32715 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
changed_when: "false"
- name: Install release 3726d73 beside the live one (current NOT switched)
sudo: true
shell: python3 /tmp/svos-stage-3726d73/install-release.py
creates: /opt/svos/releases/3726d73/apps-env/pyvenv.cfg
verify:
- name: New kernel 0.0.14 accepts the live configuration as it is
sudo: true
shell: runuser -u svc-svos -- /opt/svos/releases/3726d73/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
changed_when: "false"
- name: current still points at the live release
shell: readlink /opt/svos/current
changed_when: "false"
@@ -0,0 +1,53 @@
# nh3-svos -> SVOS v0.0.14 (3726d73): kernel 0.0.14 (migration 0014, schema 14), app-interface 0.0.6 (briefs index v1->v2 at
# its first start), connector base 0.0.10, UI 0.0.19. svos-dev's ask (thread 01M5558E19Q3ZK4RRZXQG6A2Y9), pushed on Prime's
# 0245 go. PRE-REQ: nh3-svos-3726d73-v0.0.14-stage.yaml (installs the release beside the live one).
# PHASE A of three, in svos-dev's order: A = this (kernel, knowledge, voice; the interface is LEFT STOPPED),
# B = nh3-dev-svos-3726d73-v0.0.14.yaml (connector base + app-terminal; raises ONE card),
# C = nh3-svos-3726d73-v0.0.14-interface.yaml (app-interface + the web build; raises ONE card).
# Run A..C only while Prime can approve both cards within 15 minutes: a held app refuses wrong-revision (reads DOWN).
steps:
- name: The release is staged and accepts the live config
sudo: true
shell: test -f /opt/svos/releases/3726d73/apps-env/pyvenv.cfg && runuser -u svc-svos -- /opt/svos/releases/3726d73/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
changed_when: "false"
- name: Stop the local apps then the kernel
sudo: true
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Offline backup to /var/lib/svos/backup/pre-3726d73 (schema 13 database, the v1 briefs index, config)
sudo: true
shell: |
set -e
d=/var/lib/svos/backup/pre-3726d73
runuser -u svc-svos -- install -d -m 0700 $d
runuser -u svc-svos -- python3 -c "
import sqlite3
for src, dst in (('/var/lib/svos/state/svos.db', '$d/svos.db'), ('/var/lib/svos/state/apps/interface/briefs.sqlite3', '$d/briefs.sqlite3')):
s = sqlite3.connect(src); t = sqlite3.connect(dst); s.backup(t); t.close(); s.close()
c = sqlite3.connect('file:' + dst + '?mode=ro', uri=True)
print(dst, 'integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-3726d73
ls -la $d | tail -4
- name: Switch current to 3726d73
sudo: true
shell: ln -sfn /opt/svos/releases/3726d73 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
- name: Start the kernel (READY-gated; migration 0014) then knowledge and voice (the interface waits for phase C)
sudo: true
shell: |
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
systemctl start svos-knowledge svos-voice
sleep 5; curl -s http://127.0.0.1:8400/health; echo
systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0014|schema|error|refus" | tail -6 | cut -c1-200
test $rc -eq 0
verify:
- name: Health is schema 14, status ok
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":14' && echo "$h" | grep -q '"status":"ok"'
changed_when: "false"
- name: GET /seats profiles.tasks publishes brief_max_chars and ceilings.check_seconds (read with the local admin path if open, else skip)
shell: curl -s http://127.0.0.1:8400/seats | python3 -c "import sys,json; d=json.load(sys.stdin); t=d.get('profiles',{}).get('tasks',{}); print('brief_max_chars', t.get('brief_max_chars'), 'check_seconds', (t.get('ceilings') or {}).get('check_seconds'))" 2>&1 || true
changed_when: "false"
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
changed_when: "false"
@@ -0,0 +1,85 @@
# nh3-svos -> SVOS v0.0.11 (452a592, U15 task seats slice 1) at svos-dev's ask (Prime's 2010 go relayed). Migration 0013
# (schema 13) and two config additions ([tasks], the tasks artifact mapping). Run AFTER tonight's 21:30 vzdump of VM 111.
steps:
- name: Private staging dir
shell: install -d -m 0700 /tmp/svos-stage-452a592
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-452a592/svos-release-452a592.tar.gz, dest: /tmp/svos-stage-452a592/release.tar.gz, mode: "0644"}}
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-452a592/uv, mode: "0755"}}
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-452a592/install-release.py, dest: /tmp/svos-stage-452a592/install-release.py, mode: "0644"}}
- name: Verify staged hashes
shell: cd /tmp/svos-stage-452a592 && printf '%s %s\n' b38a2a09cdaa7475705108bdebdf96c9d434d71be142b89b4fb1dc15a2d3af1e release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
changed_when: "false"
- name: Install release 452a592 beside the live one (current NOT switched)
sudo: true
shell: python3 /tmp/svos-stage-452a592/install-release.py
- name: New kernel accepts the live configuration as it is
sudo: true
shell: runuser -u svc-svos -- /opt/svos/releases/452a592/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
sudo: true
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Offline backup to /var/lib/svos/backup/pre-452a592 (schema 12 database + config)
sudo: true
shell: |
set -e
d=/var/lib/svos/backup/pre-452a592
runuser -u svc-svos -- install -d -m 0700 $d
runuser -u svc-svos -- python3 -c "
import sqlite3
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-452a592/svos.db')
src.backup(dst); dst.close(); src.close()
c = sqlite3.connect('file:/var/lib/svos/backup/pre-452a592/svos.db?mode=ro', uri=True)
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-452a592
ls -la $d | tail -3
- name: Config for U15 ([tasks] + the tasks artifact mapping), then the NEW kernel must accept it (else restore + old kernel back)
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
anchor = '[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/postoffice/mail"\napp = "app-postoffice"\nverb = "artifact"\n'
assert s.count(anchor) == 1, s.count(anchor)
assert "[tasks]" not in s
add = ('\n# U15 (v0.0.11, 2026-10-13): app-terminal task results. host = the terminal\'s artifact_host ("nh3-svos", kept\n'
'# through U13 although the files live on nh3-dev, like the injects mapping above); root = its [seat_host].tasks_root.\n'
'[[notifications.artifacts]]\nhost = "nh3-svos"\nroot = "/var/lib/svos/apps/terminal/tasks"\napp = "app-terminal"\nverb = "artifact"\n')
s = s.replace(anchor, anchor + add)
s = s.rstrip("\n") + ('\n\n# Task seats (U15 slice 1, v0.0.11, 2026-10-13): the seat host\'s tasks root on nh3-dev (= terminal.toml\n'
'# [seat_host].tasks_root); every ceiling at its default. No runtime declares task_argv yet, so a task-spawn is\n'
'# refused routing_unavailable: runtime_not_task_capable until Prime names the command.\n'
'[tasks]\nscratch_root = "/var/lib/svos/apps/terminal/tasks"\n')
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
if runuser -u svc-svos -- /opt/svos/releases/452a592/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-452a592 /var/lib/svos/config/instance.toml
systemctl start svos-kernel.service svos-interface svos-knowledge svos-voice; echo "RESTORED config, old kernel restarted"; exit 1
- name: Switch current to 452a592
sudo: true
shell: ln -sfn /opt/svos/releases/452a592 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
- name: Start the kernel (READY-gated; migration 0013 runs) then the local apps
sudo: true
shell: |
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
systemctl start svos-interface svos-knowledge svos-voice
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
- name: Regenerate the front door inventory (7f95e54 becomes the previous release)
sudo: true
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
verify:
- name: Inventory current
sudo: true
shell: /usr/local/sbin/svos-front-inventory --check
changed_when: "false"
- name: Health is schema 13, status ok, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
changed_when: "false"
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
changed_when: "false"
@@ -0,0 +1,80 @@
# nh3-svos -> SVOS v0.0.15 (57d90d6): kernel 0.0.15 (the short spoken summons: no card ids, no reference; WAV cap 12 MiB)
# and app-voice 0.0.2 (utterance_seconds up to 120). No migration (schema 14), no manifest change, no card. Config:
# voice.toml utterance_seconds 30 -> 60 (needs voice 0.0.2), and [notifications.matrix] request_timeout_seconds 60
# (this morning's stopgap) -> 30. svos-dev's ask on Prime's wording ruling (thread 01M5558E19Q3ZK4RRZXQG6A2Y9, 1059).
steps:
- name: Private staging dir
shell: install -d -m 0700 /tmp/svos-stage-57d90d6
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-57d90d6/svos-release-57d90d6.tar.gz, dest: /tmp/svos-stage-57d90d6/release.tar.gz, mode: "0644"}}
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-57d90d6/uv, mode: "0755"}}
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-57d90d6/install-release.py, dest: /tmp/svos-stage-57d90d6/install-release.py, mode: "0644"}}
- name: Verify staged hashes
shell: cd /tmp/svos-stage-57d90d6 && printf '%s %s\n' ffee056d65ec4df4c916611c678b9268b0c8bbf3339e865cfd8a3ca2075448c3 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
changed_when: "false"
- name: Install release 57d90d6 beside the live one (current NOT switched)
sudo: true
shell: python3 /tmp/svos-stage-57d90d6/install-release.py
creates: /opt/svos/releases/57d90d6/apps-env/pyvenv.cfg
- name: New kernel accepts the live configuration as it is
sudo: true
shell: runuser -u svc-svos -- /opt/svos/releases/57d90d6/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
- name: Stop the local apps then the kernel
sudo: true
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Offline backup to /var/lib/svos/backup/pre-57d90d6 (database + both configs)
sudo: true
shell: |
set -e
d=/var/lib/svos/backup/pre-57d90d6
runuser -u svc-svos -- install -d -m 0700 $d
runuser -u svc-svos -- python3 -c "
import sqlite3
s = sqlite3.connect('/var/lib/svos/state/svos.db'); t = sqlite3.connect('$d/svos.db'); s.backup(t); t.close(); s.close()
c = sqlite3.connect('file:$d/svos.db?mode=ro', uri=True); print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-57d90d6
runuser -u svc-svos -- cp -p /var/lib/svos/config/apps/voice.toml /var/lib/svos/config/apps/voice.toml.pre-57d90d6
- name: Switch current to 57d90d6
sudo: true
shell: ln -sfn /opt/svos/releases/57d90d6 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
- name: Config - voice utterance_seconds 60, Matrix request_timeout_seconds 30; the new kernel must accept it (else restore all + old release)
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
def write(p, s):
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p)
p = "/var/lib/svos/config/apps/voice.toml"; s = open(p).read()
assert s.count("utterance_seconds = 30\n") == 1
write(p, s.replace("utterance_seconds = 30\n", "utterance_seconds = 60 # 2026-10-17 v0.0.15 (voice 0.0.2 admits 0.1..120); 6 MiB output holds 65 s at 48 kHz\n"))
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
old = "request_timeout_seconds = 60 # 2026-10-17 stopgap (svos-dev): summons synthesis needs > 15 s; range 15..60\n"
assert s.count(old) == 1
write(p, s.replace(old, "request_timeout_seconds = 30 # 2026-10-17 v0.0.15: the short spoken summons is 5-9 s of audio; headroom for a slow gateway\n"))
print("edited")
PY
if runuser -u svc-svos -- /opt/svos/releases/57d90d6/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-57d90d6 /var/lib/svos/config/instance.toml
runuser -u svc-svos -- cp -p /var/lib/svos/config/apps/voice.toml.pre-57d90d6 /var/lib/svos/config/apps/voice.toml
ln -sfn /opt/svos/releases/3726d73 /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current
systemctl start svos-kernel.service svos-interface svos-knowledge svos-voice; echo "RESTORED configs + 3726d73"; exit 1
- name: Start the kernel (READY-gated, no migration) then the local apps
sudo: true
shell: |
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
systemctl start svos-interface svos-knowledge svos-voice
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Regenerate the front door inventory
sudo: true
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
verify:
- name: Health is schema 14, status ok, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":14' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
changed_when: "false"
- name: Front door inventory current (regenerated after the switch; the header names the release dir even when the web build is byte-identical)
sudo: true
shell: /usr/local/sbin/svos-front-inventory --check
changed_when: "false"
@@ -0,0 +1,61 @@
# nh3-svos -> SVOS v0.0.12 (d95078c): the kernel wheel alone (runtime `process`, placeholders inside task_argv args).
# No migration (schema stays 13), no config change in this playbook; the hermes runtime edit follows on 0.0.12.
steps:
- name: Private staging dir
shell: install -d -m 0700 /tmp/svos-stage-d95078c
- {name: stage release, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-d95078c/svos-release-d95078c.tar.gz, dest: /tmp/svos-stage-d95078c/release.tar.gz, mode: "0644"}}
- {name: stage uv, upload: {src: /home/lkraven/bin/uv, dest: /tmp/svos-stage-d95078c/uv, mode: "0755"}}
- {name: stage installer, upload: {src: /tmp/claude-1000/-home-lkraven-development-eshpfi-management/a0a50e2d-b274-4f8c-b8e6-0606c48d35b5/scratchpad/svos-d95078c/install-release.py, dest: /tmp/svos-stage-d95078c/install-release.py, mode: "0644"}}
- name: Verify staged hashes
shell: cd /tmp/svos-stage-d95078c && printf '%s %s\n' c1be1519edf4fc91b4a87c384a645bd28b655ff64ae0abbd353a76cc9a50f195 release.tar.gz 83aa128da93e168283ed32cf75cd819fe17b135b3fc1f7ccb3c912eec198d9fb uv | sha256sum -c -
changed_when: "false"
- name: Install release d95078c beside the live one (current NOT switched)
sudo: true
shell: python3 /tmp/svos-stage-d95078c/install-release.py
- name: New kernel accepts the live configuration as it is
sudo: true
shell: runuser -u svc-svos -- /opt/svos/releases/d95078c/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos
- name: Stop the local apps then the kernel (terminal, postoffice, miranda, agent run on nh3-dev since U13)
sudo: true
shell: systemctl stop svos-voice svos-knowledge svos-interface svos-kernel; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
- name: Offline backup to /var/lib/svos/backup/pre-d95078c (schema 13 database + config)
sudo: true
shell: |
set -e
d=/var/lib/svos/backup/pre-d95078c
runuser -u svc-svos -- install -d -m 0700 $d
runuser -u svc-svos -- python3 -c "
import sqlite3
src = sqlite3.connect('/var/lib/svos/state/svos.db'); dst = sqlite3.connect('/var/lib/svos/backup/pre-d95078c/svos.db')
src.backup(dst); dst.close(); src.close()
c = sqlite3.connect('file:/var/lib/svos/backup/pre-d95078c/svos.db?mode=ro', uri=True)
print('integrity', c.execute('pragma integrity_check').fetchone()[0], 'user_version', c.execute('pragma user_version').fetchone()[0])"
runuser -u svc-svos -- tar -C /var/lib/svos -cf $d/config-keys-apps.tar config state/keys state/apps state/config-generations apps
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-d95078c
ls -la $d | tail -3
- name: Switch current to d95078c
sudo: true
shell: ln -sfn /opt/svos/releases/d95078c /opt/svos/current.new && chgrp -h svc-svos /opt/svos/current.new && mv -T /opt/svos/current.new /opt/svos/current && readlink /opt/svos/current
- name: Start the kernel (READY-gated; no migration) then the local apps
sudo: true
shell: |
t0=$(date +%s.%N); systemctl start svos-kernel.service; rc=$?; t1=$(date +%s.%N)
echo "kernel start rc=$rc after $(python3 -c "print(round($t1-$t0,2))") s"
systemctl start svos-interface svos-knowledge svos-voice
for i in $(seq 1 60); do h=$(curl -s http://127.0.0.1:8400/health); case "$h" in *'"down":0'*) case "$h" in *'"unknown":0'*) break;; esac;; esac; sleep 1; done
echo "$h"; systemctl is-active svos-kernel svos-interface svos-knowledge svos-voice | tr '\n' ' '; echo
journalctl -u svos-kernel --since "-90s" --no-pager -o cat | grep -i -E "migrat|0013|schema|error|refus" | tail -6 | cut -c1-200
- name: Regenerate the front door inventory (452a592 becomes the previous release)
sudo: true
shell: /usr/local/sbin/svos-front-inventory 2>&1 | grep -E 'GENERATED|updated|root \*|path /assets' | head -6
verify:
- name: Inventory current
sudo: true
shell: /usr/local/sbin/svos-front-inventory --check
changed_when: "false"
- name: Health is schema 13, status ok, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"schema_version":13' && echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
changed_when: "false"
- name: App listener still refuses an uncredentialed kernel-key (401 from the kernel)
shell: test "$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8410/auth/kernel-key)" = 401
changed_when: "false"
@@ -0,0 +1,44 @@
# nh3-svos: the hermes task runtime switches to the U15 v0.7 query-file route (Prime's ruling in-session 2026-10-17 ~0905,
# "go -- query form for hermes"; exact stanza from svos-dev, thread 01M5558E19Q3ZK4RRZXQG6A2Y9, ruled A182 2026-10-15).
# Only task_argv changes, plus task_toolsets (MUST be set: the argv passes -t {toolsets}) and task_max_turns; argv,
# program, process, credential_env stay. Launcher = /home/lkraven/.local/bin/hermes (the seat host is nh3-dev, where
# hermes v0.21.5 accepts every flag below, checked with `hermes chat --help`; check-config does not inspect a remote path).
# Needs kernel 0.0.14 (v0.0.14 cutover 2026-10-17 0908).
steps:
- name: back up instance.toml
sudo: true
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-queryfile
creates: /var/lib/svos/config/instance.toml.pre-queryfile
- name: replace the hermes task_argv with the query-file form, add task_toolsets and task_max_turns
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
old = ('task_argv = ["/home/lkraven/.local/bin/hermes", "-z", "Your brief is the file {brief}. Do what it asks. Write every deliverable '
'into {result}. Print a one-line summary first, then the details.", "-m", "{model}", "--accept-hooks"]\n')
assert s.count(old) == 1, s.count(old)
assert "task_toolsets" not in s and "task_max_turns" not in s
new = ('# 2026-10-17 (Prime): U15 v0.7 query-file route. The task file is the whole user message; rules, memory, context\n'
'# files and skills off; the approval gate denies at once. {deadline_seconds} is filled by the seat host at launch.\n'
'task_argv = ["/home/lkraven/.local/bin/hermes", "chat", "--query-file", "{task}", "--oneshot", "-Q", "--ignore-rules", '
'"-t", "{toolsets}", "--max-turns", "{max_turns}", "--run-budget", "{deadline_seconds}", "-m", "{model}"]\n'
'task_toolsets = ["terminal", "file"]\n'
'task_max_turns = 200\n')
s = s.replace(old, new)
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
- name: the kernel accepts it (else restore the backup)
sudo: true
shell: |
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-queryfile /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
- name: reload the kernel's configuration
sudo: true
shell: systemctl reload svos-kernel; sleep 3; systemctl is-active svos-kernel; curl -s http://127.0.0.1:8400/health
verify:
- name: health ok, a new config generation, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0' && ! echo "$h" | grep -q '"config_gen":"33f001f4e2dbc7dc"'
changed_when: "false"
@@ -0,0 +1,48 @@
# nh3-svos: the hermes seat runtime becomes task-capable (Prime 2026-10-14 via svos-dev, thread 01M4X1ZE8RDF63NARVT4CFJBT2).
# Merged into the EXISTING [[seats.runtimes]] name="hermes" entry: svos-dev's argv/process/task_argv replace its old argv
# (venv python + hermes-agent/hermes). Needs kernel 0.0.12 (0.0.11's check-config refuses `process`). No --yolo (Prime's call).
steps:
- name: back up instance.toml
sudo: true
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-hermes-task
creates: /var/lib/svos/config/instance.toml.pre-hermes-task
- name: merge the task runtime keys into the hermes entry
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
old = ('[[seats.runtimes]]\nname = "hermes"\nprogram = "hermes"\n'
'argv = ["/home/lkraven/.hermes/hermes-agent/venv/bin/python", "/home/lkraven/.hermes/hermes-agent/hermes"]\n'
'credential_env = "SVOS_CREDENTIAL"\n')
assert s.count(old) == 1, s.count(old)
new = ('[[seats.runtimes]] # U15 task runtime (Prime 2026-10-14 via svos-dev): hermes one-off, no --yolo\n'
'name = "hermes"\n'
'argv = ["/home/lkraven/.local/bin/hermes", "chat"]\n'
'program = "hermes"\n'
'process = "python3" # the launcher execs python3; the live process never shows hermes\n'
'credential_env = "SVOS_CREDENTIAL"\n'
'task_argv = ["/home/lkraven/.local/bin/hermes", "-z", "Your brief is the file {brief}. Do what it asks. Write every deliverable into {result}. Print a one-line summary first, then the details.", "-m", "{model}", "--accept-hooks"]\n')
s = s.replace(old, new)
s = s.replace("# [seat_host].tasks_root); every ceiling at its default. No runtime declares task_argv yet, so a task-spawn is\n# refused routing_unavailable: runtime_not_task_capable until Prime names the command.\n",
"# [seat_host].tasks_root); every ceiling at its default. Since 2026-10-14 the hermes runtime declares task_argv\n# (the only task-capable runtime).\n")
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
- name: the kernel accepts it (else restore)
sudo: true
shell: |
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-hermes-task /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
- name: reload
sudo: true
shell: |
systemctl reload svos-kernel; sleep 6
curl -s http://127.0.0.1:8400/health; echo
journalctl -u svos-kernel --since "-20s" --no-pager -o cat | grep -i -E "refus|invalid|error|config" | tail -5 | cut -c1-200
verify:
- name: the hermes runtime reads task-capable
sudo: true
shell: runuser -u svc-svos -- python3 -c "import tomllib; d=tomllib.load(open('/var/lib/svos/config/instance.toml','rb')); h=[r for r in d['seats']['runtimes'] if r['name']=='hermes']; assert len(h)==1 and h[0]['process']=='python3' and h[0]['task_argv'][1]=='-z'; print(h[0]['argv'], h[0]['process'], len(h[0]['task_argv']))"
changed_when: "false"
@@ -0,0 +1,38 @@
# nh3-svos: [notifications.matrix] request_timeout_seconds 15 (default) -> 60. Stopgap for card audio (svos-dev's word
# 2026-10-17 0916, thread 01M5558E19Q3ZK4RRZXQG6A2Y9; allowed range 15..60): every summons synthesis died at exactly
# 15.0 s (12/12 since 10-10). Short cards speak with this; long ones still hit the voice app's 30 s audio cap until the
# spoken-summons code change (Prime's wording call) ships, then restore the default. Revert = instance.toml.pre-matrix-timeout.
steps:
- name: back up instance.toml
sudo: true
shell: runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml /var/lib/svos/config/instance.toml.pre-matrix-timeout
creates: /var/lib/svos/config/instance.toml.pre-matrix-timeout
- name: add request_timeout_seconds = 60 under [notifications.matrix]
sudo: true
shell: |
runuser -u svc-svos -- python3 - <<'PY'
import os, re
p = "/var/lib/svos/config/instance.toml"; s = open(p).read()
block = s[s.index("[notifications.matrix]"):]
block = block[:block.index("\n[", 1)] if "\n[" in block[1:] else block
assert "request_timeout_seconds" not in block, "already set"
anchor = 'voice_id = "miranda-is"'
line = s[s.index(anchor):s.index("\n", s.index(anchor)) + 1]
assert s.count(line) == 1
s = s.replace(line, line + 'request_timeout_seconds = 60 # 2026-10-17 stopgap (svos-dev): summons synthesis needs > 15 s; range 15..60\n')
t = p + ".tmp"; fd = os.open(t, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
with os.fdopen(fd, "w") as f: f.write(s)
os.replace(t, p); print("edited")
PY
- name: the kernel accepts it (else restore)
sudo: true
shell: |
if runuser -u svc-svos -- /opt/svos/current/kernel-env/bin/svos-kernel check-config --data-root /var/lib/svos; then exit 0; fi
runuser -u svc-svos -- cp -p /var/lib/svos/config/instance.toml.pre-matrix-timeout /var/lib/svos/config/instance.toml; echo RESTORED; exit 1
- name: reload
sudo: true
shell: systemctl reload svos-kernel; sleep 3; curl -s http://127.0.0.1:8400/health
verify:
- name: health ok, nothing down
shell: h=$(curl -s http://127.0.0.1:8400/health); echo "$h"; echo "$h" | grep -q '"status":"ok"' && echo "$h" | grep -q '"down":0'
changed_when: "false"
+20
View File
@@ -421,6 +421,25 @@ def cmd_add(args):
k.__exit__() k.__exit__()
def cmd_delete(args):
"""Remove ONE monitor by its exact name. `seed` never prunes (a missing row in
monitors.yaml is not a delete), so retiring a service needs this as well as the
yaml edit. Exits 1 if the name is not on the board."""
k = connected_and_logged_in()
try:
mons = k.monitors_by_name()
if args.name not in mons:
print(f"no monitor named {args.name!r}", file=sys.stderr)
sys.exit(1)
k.delete(mons[args.name]["id"])
if args.name in k.monitors_by_name():
print(f"delete of {args.name!r} did not take", file=sys.stderr)
sys.exit(1)
print(f"deleted {args.name}")
finally:
k.__exit__()
def main(): def main():
p = argparse.ArgumentParser(prog="kuma", description=__doc__.split("\n")[0]) p = argparse.ArgumentParser(prog="kuma", description=__doc__.split("\n")[0])
sub = p.add_subparsers(dest="cmd", required=True) sub = p.add_subparsers(dest="cmd", required=True)
@@ -432,6 +451,7 @@ def main():
a.add_argument("--type", default="http", choices=["http", "ping", "port"]) a.add_argument("--type", default="http", choices=["http", "ping", "port"])
a.add_argument("--port", type=int) a.add_argument("--port", type=int)
a.set_defaults(fn=cmd_add) a.set_defaults(fn=cmd_add)
d = sub.add_parser("delete"); d.add_argument("name"); d.set_defaults(fn=cmd_delete)
sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe) sub.add_parser("dedupe").set_defaults(fn=cmd_dedupe)
sub.add_parser("channels").set_defaults(fn=cmd_channels) sub.add_parser("channels").set_defaults(fn=cmd_channels)
args = p.parse_args() args = p.parse_args()
@@ -0,0 +1,27 @@
#!/bin/bash
# Run INSIDE esh-nas (CT 103 on esh-pve-nas) as root, via `pct exec 103`. The same steps as
# playbooks/bootstrap-infra-ops-user.yaml, for a container infra-ops cannot log into yet (Prime 2026-10-13: "set it up").
set -euo pipefail
U=infra-ops; ID=850
KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet'
if ! id "$U" >/dev/null 2>&1; then
if ! getent passwd "$ID" >/dev/null && ! getent group "$ID" >/dev/null; then
groupadd -g "$ID" "$U"; useradd -m -s /bin/bash -u "$ID" -g "$ID" "$U"
else
echo "uid/gid $ID taken; letting the OS allocate"; useradd -m -s /bin/bash "$U"
fi
echo "created $U"
fi
chmod 0700 /home/$U
getent group docker >/dev/null && usermod -aG docker "$U" || echo "no docker group, skipped"
install -d -m 700 -o "$U" -g "$U" /home/$U/.ssh
printf '%s\n' "$KEY" > /home/$U/.ssh/authorized_keys
chown "$U:$U" /home/$U/.ssh/authorized_keys; chmod 600 /home/$U/.ssh/authorized_keys
f=/etc/sudoers.d/$U
printf 'Defaults:%s log_output\n%s ALL=(ALL) NOPASSWD:ALL\n' "$U" "$U" > "$f.tmp"
printf '%s\n' '# Bulk data movers: keep the audit line, do not record the bytes (eshpfi playbooks/sudoers-infra-ops-bulk-no-log-output.yaml)' \
'Cmnd_Alias BULK_DATA = /usr/bin/rsync, /usr/bin/tar, /usr/bin/cat, /usr/bin/dd, /usr/bin/zstd, /usr/bin/zcat, /usr/bin/gzip, /usr/bin/pv, /usr/bin/docker save *, /usr/bin/docker export *, /usr/bin/docker cp *' \
'Defaults!BULK_DATA !log_output' >> "$f.tmp"
chmod 440 "$f.tmp"
if visudo -cf "$f.tmp"; then mv "$f.tmp" "$f"; else rm -f "$f.tmp"; echo "VISUDO VALIDATION FAILED"; exit 1; fi
id "$U"; stat -c '%a %U %n' /home/$U /home/$U/.ssh /home/$U/.ssh/authorized_keys "$f"
+20 -2
View File
@@ -55,6 +55,7 @@ WT_DEMO_API_CT="${WT_DEMO_API_CT:-worldtree-worldtree-api-1}"
WT_LEGACY_MODE="${WT_LEGACY_MODE:-off}" WT_LEGACY_MODE="${WT_LEGACY_MODE:-off}"
WT_BATCH_LABEL="${WT_BATCH_LABEL:-window-off}" WT_BATCH_LABEL="${WT_BATCH_LABEL:-window-off}"
STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/wt-memory-gate-batch" STATE_DIR="${XDG_STATE_HOME:-$HOME/.local/state}/wt-memory-gate-batch"
SCRATCH_BASE="$STATE_DIR/scratch" # kept harness scratch, one dir per batch; outside every git checkout
LOCK_FILE="$STATE_DIR/lock" LOCK_FILE="$STATE_DIR/lock"
OUT_DIR="$WT_TREE/docs/eval/memory_acceptance_runs" OUT_DIR="$WT_TREE/docs/eval/memory_acceptance_runs"
@@ -112,8 +113,14 @@ legacy_args=()
if git -C "$WT_TREE" grep -q -e '"--legacy-mode"' "$deployed" -- core/memory_acceptance 2>/dev/null; then if git -C "$WT_TREE" grep -q -e '"--legacy-mode"' "$deployed" -- core/memory_acceptance 2>/dev/null; then
legacy_args=(--legacy-mode "$WT_LEGACY_MODE") legacy_args=(--legacy-mode "$WT_LEGACY_MODE")
fi fi
# --keep-data (worldtree-dev 2026-10-17, thread 01M555JEFQKK2G3WK155P1X9T1): keep the in-process instance's scratch
# (its memory writer/reader logs, records store, SQLite) so a miss can be traced after the batch. It only skips the
# harness's rmtree at stop: the number is the same harness. The scratch lands under $SCRATCH_BASE/<batch id>/ via
# TMPDIR (the harness mkdtemp's it; there is no --scratch-root flag), OUTSIDE every git checkout, because worldtree-dev
# commits the run dirs and a SQLite must never ride into the repo. Retention 14 days: lkraven's user tmpfiles rule
# (configs/wt-gate/wt-gate-scratch.conf -> ~/.config/user-tmpfiles.d/), not an rm in this script.
cmd=(python -m core.memory_acceptance.live --runs 3 --gate cmd=(python -m core.memory_acceptance.live --runs 3 --gate
"${legacy_args[@]}" --label "$WT_BATCH_LABEL" --out "$OUT_DIR") "${legacy_args[@]}" --label "$WT_BATCH_LABEL" --out "$OUT_DIR" --keep-data)
echo "wt-memory-gate-batch $(now)" echo "wt-memory-gate-batch $(now)"
echo " demo image $image" echo " demo image $image"
@@ -139,8 +146,12 @@ if [[ "$run_tree" == "$WT_GATE_TREE" ]]; then
fi fi
# ── 4. run ────────────────────────────────────────────────────────────────── # ── 4. run ──────────────────────────────────────────────────────────────────
log="$STATE_DIR/$(date -u +%Y%m%dT%H%M%SZ).log" # stderr (progress, errors) bid="$(date -u +%Y%m%dT%H%M%SZ)" # this batch's id: names the log and the scratch parent
log="$STATE_DIR/$bid.log" # stderr (progress, errors)
out="${log%.log}.stdout" # stdout; its last line is the run dir out="${log%.log}.stdout" # stdout; its last line is the run dir
scratch_parent="$SCRATCH_BASE/$bid"
[[ ! -e "$scratch_parent" ]] || refuse "scratch parent $scratch_parent already exists (the harness needs an empty scratch per batch)"
mkdir -p -m 0700 "$scratch_parent" || refuse "could not create $scratch_parent"
echo " log $log" echo " log $log"
set +e set +e
( (
@@ -152,7 +163,12 @@ set +e
# WORLDTREE_ROOT from its own location, so point that back at the tree under test. # WORLDTREE_ROOT from its own location, so point that back at the tree under test.
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$WT_TREE/env.sh" source "$WT_TREE/env.sh"
# Since 2026-10-16 every agent shell on nh3-dev exports ALBOK_SERVICE/ALBOK_TOKEN (~/.config/secrets/env.sh, the
# fleet-agents read token). The harness has always run WITHOUT albok; unset both so the daily numbers stay comparable.
# Remove this only on worldtree-dev's word, and then say so in the run notes.
unset ALBOK_SERVICE ALBOK_TOKEN
export WORLDTREE_ROOT="$run_tree" export WORLDTREE_ROOT="$run_tree"
export TMPDIR="$scratch_parent" # the harness's mkdtemp(prefix="wt-tracer-") lands here, and --keep-data keeps it
# The editable install maps packages/ to the main tree; put the tree under # The editable install maps packages/ to the main tree; put the tree under
# test first (identical content is guaranteed by step 2, this keeps it explicit). # test first (identical content is guaranteed by step 2, this keeps it explicit).
export PYTHONPATH="$run_tree/packages/worldtree-memory/src${PYTHONPATH:+:$PYTHONPATH}" export PYTHONPATH="$run_tree/packages/worldtree-memory/src${PYTHONPATH:+:$PYTHONPATH}"
@@ -170,11 +186,13 @@ if [[ "$rc" -ne 0 || ! -f "$run_dir/trace1.md" ]]; then
fi fi
stamp="$(basename "$run_dir")" stamp="$(basename "$run_dir")"
kept="$(find "$scratch_parent" -mindepth 1 -maxdepth 1 -type d -name 'wt-tracer-*' -print -quit)"
verdict_line="$(grep -m1 -E '^Aggregate label: .*verdict: \*\*(PASS|FAIL)\*\*' "$run_dir/trace1.md" || true)" verdict_line="$(grep -m1 -E '^Aggregate label: .*verdict: \*\*(PASS|FAIL)\*\*' "$run_dir/trace1.md" || true)"
verdict="$(sed -nE 's/.*verdict: \*\*(PASS|FAIL)\*\*.*/\1/p' <<<"$verdict_line")" verdict="$(sed -nE 's/.*verdict: \*\*(PASS|FAIL)\*\*.*/\1/p' <<<"$verdict_line")"
echo " run stamp $stamp" echo " run stamp $stamp"
echo " verdict ${verdict:-NONE}" echo " verdict ${verdict:-NONE}"
echo " line ${verdict_line:-<no Aggregate verdict line in trace1.md>}" echo " line ${verdict_line:-<no Aggregate verdict line in trace1.md>}"
echo " kept scratch ${kept:-<none found under $scratch_parent>} (memory/writer, memory/reader; kept 14 days)"
case "$verdict" in case "$verdict" in
PASS) echo "→ report to worldtree-dev today with run stamp $stamp (PASS builds the n)."; exit 0 ;; PASS) echo "→ report to worldtree-dev today with run stamp $stamp (PASS builds the n)."; exit 0 ;;
FAIL) echo "→ report to worldtree-dev TODAY with run stamp $stamp. Do not retry." >&2; exit 1 ;; FAIL) echo "→ report to worldtree-dev TODAY with run stamp $stamp. Do not retry." >&2; exit 1 ;;
+1 -1
View File
@@ -60,7 +60,7 @@ Media services on this box sit at `10.0.50.56` (Plex) and `10.0.50.57` (Jellyfin
rebooted this node without accounting for it. Before any planned reboot of either rebooted this node without accounting for it. Before any planned reboot of either
node, either accept the read-only window on the survivor or set node, either accept the read-only window on the survivor or set
`pvecm expected 1` on it for the duration. Discovered 2026-08-18. `pvecm expected 1` on it for the duration. Discovered 2026-08-18.
- ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.** - ⚠ **CT 103 `esh-nas` (10.0.50.50) runs on THIS host and serves `hard` NFS.** Since 2026-10-13 it has the `infra-ops` identity (uid 850, NOPASSWD sudo, Prime's ruling): `ssh infra-ops@10.0.50.50` works directly. Before that, the only way in was `pct exec 103` from this host.
Never reboot this host casually — quiesce the clients first. Measured from the Never reboot this host casually — quiesce the clients first. Measured from the
server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are server on 2026-08-18 (`ss -tn '( sport = :2049 )'` inside CT 103), there are
**five** clients, not the two long documented here: **five** clients, not the two long documented here:
+32
View File
@@ -229,3 +229,35 @@ Available to **every user/project** on this box — no per-home `playwright inst
/ `heid run` pollers) into Claude Code session orchestrators (heid commit / `heid run` pollers) into Claude Code session orchestrators (heid commit
`12aa5a9`); the `~/development/volva` dir + venvs are gone. `volva.service` had `12aa5a9`); the `~/development/volva` dir + venvs are gone. `volva.service` had
been crash-looping `203/EXEC`. Cleanup done by infra-ops at heid's request. been crash-looping `203/EXEC`. Cleanup done by infra-ops at heid's request.
## `/srv/nemi/inbox`: albok's drop-box (2026-10-13)
Nemi (lkraven's `albok-nemi.timer`) ingests markdown dropped here into albok's `personal/inbox` wing, once per file;
capture mode never tombstones, so a walked file may be moved away. Writers are group `nemi-drop` (lkraven,
infra-ops). The dir is `lkraven:nemi-drop 2770` with a default ACL (u:lkraven, g:nemi-drop rwX), and `/srv/nemi`
carries `u:lkraven:rx`.
⚠ lkraven's systemd user manager started before `nemi-drop` existed, so Nemi gets in through those ACL entries
for lkraven, NOT through the group. Do not "simplify" them away until lkraven's user manager has restarted.
`playbooks/nh3-dev-nemi-inbox.yaml` verifies both paths.
## `/srv/nemi/books`: the book drop (2026-10-16)
albok unit 8 (albok-dev, after Prime asked for a books drop folder with a rejection folder). Drop an epub into `fiction/` or
`nonfiction/`. Nemi validates it and moves it into `~lkraven/library/<shelf>/`, writing its yaml and MANIFEST line, then walks it
the same hour. If it refuses the file, it moves it into `rejected/` beside a reason file. Files are examined only after
2 minutes unchanged, and temporary names (dotfiles, `.part`, `.crdownload`, `.tmp`, `~`, `.syncthing.*`) are ignored.
Same shape as the inbox: all four dirs are `lkraven:nemi-drop 2770` with the default ACL. Same filesystem as `~/library`
(`/dev/sda1`), so Nemi's move is a rename. There is no network share: drops are local writes or `scp`/`sftp` as lkraven, as
for the inbox. **Prime ruled 2026-10-16: scp is the drop path; no SMB/Syncthing share** (`scp book.epub lkraven@nh3-dev:/srv/nemi/books/fiction/`). `playbooks/nh3-dev-nemi-books.yaml` provisions it and verifies an infra-ops drop is readable, movable
to `rejected/` and removable by lkraven.
## `/srv/nemi/vault`: Prime's notes for Nemi (2026-10-15)
A fresh clone of gitea `vh/VuongKB` (Prime's notes, which he syncs with git; his working copy is `/home/lkraven/kb`, never
touched by this). Nemi's `vault` source walks it into albok's `personal/notes` (unit 4.4 ruling 6). It is read-only by
convention: nothing writes into it but the pull. The lkraven user timer `nemi-vault-pull` (`configs/nemi-vault/`)
runs `git pull --ff-only` hourly at :50, ahead of Nemi's walk at ~:01. A force-push upstream fails the pull loudly
(`systemctl --user status nemi-vault-pull`) and the copy stays as it was. Re-clone it on purpose if that happens.
Because it is a CLONE, it holds only tracked files. The untracked `main/` (Muninn job dirs) and `fiction/` (Muninn
fiction concepts, 2,389 files) in Prime's working copy are deliberately not in it.
+62
View File
@@ -23,6 +23,68 @@ configuration, enrollment and acceptance.
| App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script | | App-consistent hook | svos-dev's `/etc/qemu/fsfreeze-hook.d/50-svos` stops the SVOS services on freeze and restarts the ones that were running on thaw. ⚠ **Debian's qemu-ga does NOT run fsfreeze hooks unless configured** (no `-F` in the unit, no `qemu-ga.conf` by default). Here `/etc/qemu/qemu-ga.conf` points it at **`/usr/local/sbin/fsfreeze-dispatch`** (`configs/qemu-ga/fsfreeze-dispatch`), which unlike the stock `/etc/qemu/fsfreeze-hook` propagates failure: if a freeze hook fails, it thaws that hook and the completed ones and exits 1, so qemu-ga refuses the freeze. **Live drill 2026-10-05 1952–1953 (services running):** quiesced backup `vm/111/2026-10-06T02:52:11Z` → kernel+interface down ~1–2 s, back by themselves, health ok. Refused freeze (temporary `60-drill-refuse` after 50-svos, since removed) `vm/111/2026-10-06T02:53:03Z` → the dispatcher unwound and restarted the services, and then **⚠ vzdump did NOT abort: it logged `ERROR: unable to freeze guest fs`, backed up UNFROZEN (crash-consistent, app live), and reported `Backup job finished successfully`, rc 0.** Both images mapped from PBS and mounted read-only (`ro,noload`) on nh3-pve-2, never booted: `svos.db` integrity ok (each image still carries its ~0.9 MB WAL: keep `-wal` with the db when restoring), interface `pins.sqlite3` integrity ok, config/key/config-generation hashes equal live, table counts equal live except `events`. **Monitoring:** `scripts/check-backup-freshness.sh` now reads the LOG of every OK vzdump task and reports ERROR lines; a freeze/thaw error is an APPLICATION-CONSISTENCY FAILURE, verdict **DEGRADED, exit 4** (alerted by the daily wrapper to infra-hermes). nh3-pve-2 was not in the check's node lists at all before 2026-10-05; it is now. Drill B is acknowledged by UPID in the script |
| Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 | | Monitoring | Beszel native agent 0.18.7 (`playbooks/beszel-native.yaml`, `configs/beszel-agent/hosts/nh3-svos.env`), hub system `t6y646ifa3x7xxv`, alerts CPU>95%/15m, Disk>85%/5m, Memory>90%/10m, offline/2m. Up 2026-10-05 |
## Release 57d90d6 / v0.0.15, the short spoken summons (2026-10-17 1100–1102; Prime's wording ruling via svos-dev, thread `01M5558E19Q3ZK4RRZXQG6A2Y9`)
- **Kit:** kernel 0.0.15 and app-voice 0.0.2 wheels from a clean archive of the tag. Everything else, the web build included, was carried byte-identical
from 3726d73 (the UI bar still reads 3726d73; its sources are unchanged). Kit sha256 `ffee056d…`. Playbook: `nh3-svos-57d90d6-v0.0.15.yaml`.
- **Spoken form:** a card summons now speaks "Pending card from <originator>: <action>. Created <HHMM>. Approve or countermand from your desk."
with no ids and no reference. The Matrix text is unchanged. The kernel's WAV cap is 12 MiB.
- **Config:** `voice.toml` `utterance_seconds` 30 → 60 (voice 0.0.2 admits 0.1..120). `[notifications.matrix] request_timeout_seconds`
60 (this morning's stopgap) → 30. Config gen `40f469b75b5d0782`. No migration (schema 14) and no manifest change, so no card.
Kernel READY in 0.76 s; all 7 apps up by 1101:21.
- **Backup / rollback:** `backup/pre-57d90d6/svos.db` (integrity ok, v14), plus `instance.toml.pre-57d90d6` and `apps/voice.toml.pre-57d90d6`.
To roll back: stop the apps and the kernel, restore both configs, then `current → releases/3726d73`, then start.
- ⚠ The front-door inventory header names the release dir, so regenerate it after every switch, even when the web build is identical.
## Release 3726d73 / v0.0.14, the task-seat success gate (2026-10-17 0908–0910; Prime's in-session go, svos-dev thread `01M5558E19Q3ZK4RRZXQG6A2Y9`)
- **Kit:** kernel 0.0.14, connector base 0.0.10, app-interface 0.0.6, app-terminal 0.0.5 wheels and web build UI 0.0.19, all from a clean archive of
the tag. Everything else was carried byte-identical from 33a7077 (no source change; uv.lock changed only versions). Kit sha256 `380e64f6…`.
Staged first (`nh3-svos-3726d73-v0.0.14-stage.yaml`), then three phases in svos-dev's order:
- **A** (`nh3-svos-3726d73-v0.0.14.yaml`): offline backup to `backup/pre-3726d73` (schema 13 `svos.db` plus the v1 `briefs.sqlite3`, both via the
sqlite backup API with integrity ok, plus the config tar and `instance.toml.pre-3726d73`). Kernel READY in 0.73 s, with migration 0014, now schema 14.
- **B** (`nh3-dev-svos-3726d73-v0.0.14.yaml`): nh3-dev `current → 3726d73`, plus `[seat_host].scan_patterns` pointing at hermes's
`tools/threat_patterns.py` (backup `terminal.toml.pre-3726d73`). Only svos-terminal was restarted.
- **C** (`…-interface.yaml`): interface start (briefs v1→v2), then the front-door inventory (33a7077 becomes previous).
- **Cards:** terminal revision 5 (`9822cb2f…`, raised 0908:53) and interface revision 4 (`898902b4…`, raised 0909:04). Prime approved both,
at 0909:20 and 0909:27. All 7 apps were up by 0909:33.
- **Hermes query-file route** (Prime, same go): `task_argv` is now `hermes chat --query-file {task} --oneshot -Q --ignore-rules -t {toolsets}
--max-turns {max_turns} --run-budget {deadline_seconds} -m {model}`, with `task_toolsets = ["terminal", "file"]` and `task_max_turns = 200`
(`nh3-svos-hermes-query-file-route.yaml`, backup `instance.toml.pre-queryfile`). Config gen is now `c0e9beefe9bde65b`.
- **Rollback:**
- nh3-svos: stop the apps and the kernel, then restore `svos.db` and `state/apps/interface/briefs.sqlite3` from `backup/pre-3726d73`, then
restore `instance.toml.pre-3726d73`, then `current → releases/33a7077`, then start.
- nh3-dev: `current → releases/33a7077`, then restore `terminal.toml.pre-3726d73`, then restart svos-terminal.
- Both apps' older revisions would raise cards again.
- ⚠ **Card audio has failed every time:** 12 of 12 `synthesis_failed` since 10-10, each at **exactly 15.0 s**. That is the kernel's
`[notifications.matrix] request_timeout_seconds` (default 15) running out while synthesis is still going. The kernel speaks the Matrix
summons text, which includes three card ids spelled aloud, so it runs far past what 15 s covers (miranda-is ≈ 28 chars of text per
second; 167 chars took 5.85 s median, n=3). **Stopgap 2026-10-17 0917 (svos-dev's word): `request_timeout_seconds = 60`** (range
15..60; config gen `d69e7ae87d14ed45`; revert = `instance.toml.pre-matrix-timeout`). Long cards still fail at the voice app's 30 s
audio cap (`utterance_seconds`, max 30, about 480 chars). The real fix is a shorter spoken summons, a kernel patch whose
wording is Prime's call. Restore the 15 s default once it ships.
## Release 452a592 / v0.0.11, U15 task seats slice 1 (2026-10-13 2131–2133; Prime's 2010 go via svos-dev, thread `01M4WAGAS2Z0P8J6GZRSCRPH2Q`)
- **Order:** tonight's 21:30 vzdump of VM 111 finished 21:31:30, then `playbooks/svos-releases/nh3-svos-452a592-v0.0.11.yaml`
(the kit was built in-session: kernel 0.0.11, connector base 0.0.8 and app-terminal 0.0.4 wheels from a clean archive of
the tag; everything else, including the web build of 7f95e54, carried byte-identical).
- **Backup before the migration:** `/var/lib/svos/backup/pre-452a592/` (the schema 12 `svos.db` via the sqlite backup API,
integrity ok, plus the config/keys/apps tar) and `config/instance.toml.pre-452a592`.
- **Config:** a `[tasks]` table with `scratch_root = "/var/lib/svos/apps/terminal/tasks"` (all ceilings at their defaults),
plus a `[[notifications.artifacts]]` for that root. Its **host is `nh3-svos`, not `nh3-dev`**: the terminal's
`artifact_host` stayed `nh3-svos` through U13, and its ResultReader names results by that host, so the mapping must
match it, as the injects mapping does. No runtime has `task_argv` yet, so a task-spawn is refused
`runtime_not_task_capable` until Prime names the command.
- **Result:** kernel 0.0.11 READY in 1.17 s, migration 0013 run, `/health` schema 13, config_gen `5ade279d2fb2dd1c`, apps up 7.
- **nh3-dev:** `/opt/svos/current → releases/452a592` (terminal 0.0.4, connector 0.0.8). `/var/lib/svos/apps/terminal/tasks`
(0700) and `terminal.toml [seat_host].tasks_root` (backup `.pre-452a592`). Only svos-terminal was restarted. Its manifest
revision 4 (0.0.4, `seat-host`) was **pinned directly, with no card**. The seats read unknown for 6 s, then live.
svos-miranda, svos-postoffice and svos-agent still run adc862e's code until their next restart.
- **Rollback:** stop the kernel, put `svos.db` back from `backup/pre-452a592` (schema 12), restore `instance.toml.pre-452a592`,
`current → releases/7f95e54`, then start. On nh3-dev: `current → releases/adc862e`, restore `terminal.toml.pre-452a592`,
then restart svos-terminal.
## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`) ## Release afab123 / v0.0.9 + U13 phase B (2026-10-10 1646–1648, Prime-approved 1639; svos-dev thread `01M4M2Z2H3BQH375M2G30GQHDM`)
- **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables; - **`/opt/svos/current` → `releases/afab123`: kernel 0.0.9, schema 11 → 12** (migration 0012, the page verb's tables;
+21
View File
@@ -0,0 +1,21 @@
# h2o decision front
The front of the h2o-lightning-4b decision seat on fv-ml1 (seat swap ordered by Prime 2026-10-12, through brokkr).
`front.py` sits in front of the vendor's **unmodified** `h2o_lightning_shim.py`, which sits in front of stock vLLM
0.30.0. It adds what the seat contract needs and the vendor stack lacks:
- **Admission** (infra-hermes's `tools/h2o-admission/admission_proxy.py` @ 30f36f2, logic unchanged): count every
decision request with h2o's own tokenizer over the shim's own renders, and refuse over 32,000 tokens with 422 + the
count. The shim would otherwise trim silently.
- **The seat's bearer token** on everything but `GET /health`, as intern-decision-serve has.
- **`/decide` and `/decide/shared` continuity** for albok-nemi's sensitivity judge: intern-decision-serve's own
mapping (one Jev `choice` question per decision, calls of at most 16, fields `q`/`q1..qN`), re-keyed into the semif
result shape (`results[i].native.probabilities[option id]`). `orderings` other than `none` → 422.
- **`/health`** with `model: {name, source, revision}`, which the judge stamps into its cache key.
`run.sh` runs the shim (loopback) and the front in one container (`python:3.12-slim`, host network). Stacks copy both
files into `conf/front/`: `stacks/h2o-eval` (TEMPORARY, GPU 3, :8034) today.
Verified 2026-10-12 1300 on the eval seat: 401 without the token; `/decide/shared` in the judge's shape; a planted
personal record → `personal` p_yes .911, a neutral note → .056 (n=1 each, smoke test only); 40,088 tokens → 422 with
the count; `/v1/systemone` passthrough with `_admission` counts; reachable from nh3-dev.
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
the three things the seat contract needs and the vendor stack does not have:
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
the judge stamps into its cache key.
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
"""
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
TOKEN = os.environ["FRONT_TOKEN"]
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
PORT = int(os.environ.get("FRONT_PORT", "8034"))
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
CAP = int(os.environ.get("FRONT_CAP", "32000"))
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
MAX_QUESTIONS_PER_CALL = 16
MIN_OPTIONS, MAX_OPTIONS = 2, 16
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
shim_mod = importlib.util.module_from_spec(spec)
sys.modules["h2o_shim"] = shim_mod
spec.loader.exec_module(shim_mod)
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
class Refused(Exception):
def __init__(self, code, obj):
super().__init__(obj)
self.code, self.obj = code, obj
def sent_tokens(body):
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
state = body.get("state") or ""
if isinstance(state, (dict, list)):
state = json.dumps(state)
state_txt = shim_mod.render_state(state, CFG.state_compact)
all_toks = []
for qid, q in body.get("questions", {}).items():
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
req = urllib.request.Request(VLLM + "/tokenize",
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=30) as r:
all_toks.append(json.loads(r.read())["tokens"])
mx = max(len(t) for t in all_toks)
shared = shim_mod.common_prefix_len(all_toks)
return mx, shared + sum(len(t) - shared for t in all_toks)
def admit(body):
try:
mx, total = sent_tokens(body)
except Refused:
raise
except Exception as e: # noqa: BLE001
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
if mx > CAP or total > CAP:
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
"shim_accounted_total": total, "cap": CAP})
return mx, total
def post_up(path, obj):
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
method="POST")
try:
with urllib.request.urlopen(req, timeout=300) as r:
return r.status, json.loads(r.read())
except urllib.error.HTTPError as e:
raw = e.read() or b"{}"
try:
return e.code, json.loads(raw)
except ValueError:
return e.code, {"error": raw[:300].decode(errors="replace")}
def systemone(body):
mx, total = admit(body)
code, obj = post_up("/v1/systemone", body)
if isinstance(obj, dict):
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
return code, obj
def check_decision(d, i):
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
opts = d.get("options")
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
for o in opts):
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
if len({o["id"] for o in opts}) != len(opts):
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
if d.get("orderings", "none") != "none":
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
"seat (h2o); send orderings 'none' and combine on the client"})
def decide(state, decisions):
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
for i, d in enumerate(decisions):
check_decision(d, i)
started, results, sizes, tokens = time.perf_counter(), [], [], []
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
questions = {f: {"type": "choice", "instructions": d["question"],
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
code, resp = systemone({"state": state, "questions": questions})
if code != 200:
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
sizes.append(len(chunk))
tokens.append((resp.get("usage") or {}).get("input_tokens"))
for f, d in zip(fields, chunk):
answer = (resp.get("answers") or {}).get(f)
ids = [o["id"] for o in d["options"]]
probs = (answer or {}).get("probabilities") or {}
if answer is None or any(i not in probs for i in ids):
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
"top": answer.get("choice"), "confidence": answer.get("confidence"),
"calibration": resp.get("calibration"), "native": answer,
"input_tokens": tokens[-1], "model": MODEL,
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
"questions_per_call": sizes, "input_tokens": tokens}}
class H(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args):
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
def _send(self, code, obj):
b = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(b)))
self.end_headers()
self.wfile.write(b)
def _authorized(self):
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
def do_GET(self):
path = self.path.split("?")[0]
if path == "/health":
try:
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
shim = json.loads(r.read())
except Exception as e: # noqa: BLE001
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
ok = bool(shim.get("ok"))
return self._send(200 if ok else 503, {
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
if path == "/v1/models":
try:
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
return self._send(r.status, json.loads(r.read()))
except urllib.error.HTTPError as e:
return self._send(e.code, {"error": "upstream"})
return self._send(404, {"error": "not found"})
def do_POST(self):
path = self.path.split("?")[0]
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
n = int(self.headers.get("Content-Length", 0) or 0)
if n > 4 * 1024 * 1024:
return self._send(413, {"error": "body over 4 MiB"})
try:
body = json.loads(self.rfile.read(n) or b"null")
except ValueError:
return self._send(422, {"error": "body is not JSON"})
if not isinstance(body, dict):
return self._send(422, {"error": "body must be a JSON object"})
try:
if path == "/v1/systemone":
return self._send(*systemone(body))
if path == "/api/alpha/decisions":
admit(body)
return self._send(*post_up(path, body))
if path == "/decide/shared":
return self._send(200, decide(body.get("state"), body.get("decisions")))
if path == "/decide":
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
out = decide(body.get("state"), [d])
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
except Refused as r:
return self._send(r.code, r.obj)
return self._send(404, {"error": "not found"})
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
+13
View File
@@ -0,0 +1,13 @@
#!/bin/bash
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
set -eu
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
--host 127.0.0.1 --port "$SHIM_PORT" &
SHIM=$!
python3 /front/front.py &
FRONT=$!
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
kill "$SHIM" "$FRONT" 2>/dev/null || true
exit 1
+55 -1
View File
@@ -8,7 +8,17 @@ The FastAPI process that owns Albok's buildings-and-wings store (`vh/albok`,
(`servers/nh3-albok/README.md` says why and what must never be added there). Before that: nh3-docker, 2026-10-02 to 2026-10-04. (`servers/nh3-albok/README.md` says why and what must never be added there). Before that: nh3-docker, 2026-10-02 to 2026-10-04.
- **URL:** `http://albok.nh3.internal:8392` (= `http://10.100.50.44:8392`). Container port 8390. - **URL:** `http://albok.nh3.internal:8392` (= `http://10.100.50.44:8392`). Container port 8390.
Host port **8392** is kept from nh3-docker, where 8390 was the althing post office, so no client changed. Host port **8392** is kept from nh3-docker, where 8390 was the althing post office, so no client changed.
- **Image:** `gitea.phasefinal.com/pfi/albok-service:0.1.14` @ `sha256:01b46f64…` since 2026-10-07 1804 (healthy 1804:35, - **Image (current):** `gitea.phasefinal.com/pfi/albok-service:1.0.0a1` @ `sha256:6067dfbb…` since 2026-10-16 1046
(healthy 1046:36, 8 wings ready). Version-only rebuild of 0.1.24 from vh/albok `0f099e4` (Prime cut the v1 alpha;
all four albok packages at 1.0.0a1). No config change. **Rollback:** `compose.yaml.pre-1.0.0a1` (0.1.24), free.
- **Previous image:** `gitea.phasefinal.com/pfi/albok-service:0.1.24` @ `sha256:75be71c1…` 2026-10-16 0917–1046
(healthy 0918:04, `/health` 8 wings ready). infra-ops deployed it at albok-dev's ask (althing thread `01M52KF3VM9S3K22A7TWV4DC4A`),
built from a clean archive of vh/albok `67eabb7` (albok-service 0.1.24, albok 0.1.12). This is unit 6: a locked encrypted wing's
journal entries are deferred rather than driven into the unmounted mountpoint; accept-head replays them. It shipped with the
`personal/sealed` stanza in `albok.yaml`. **Rollback:** `compose.yaml.pre-0.1.24` (0.1.23) on the same store, but put back
`/srv/albok/etc/albok.yaml.pre-0.1.24` (no sealed stanza) FIRST: 0.1.23 drives a locked wing's journal entry into the
immutable mountpoint and fails to start. Releases 0.1.15 to 0.1.23 are not recorded in this README; see git log.
- **Earlier image:** `gitea.phasefinal.com/pfi/albok-service:0.1.14` @ `sha256:01b46f64…` since 2026-10-07 1804 (healthy 1804:35,
after the 1803 walk). infra-ops deployed it at albok-dev's ask (althing 8524), from a clean archive of vh/albok `7ae0d2a6` after the 1803 walk). infra-ops deployed it at albok-dev's ask (althing 8524), from a clean archive of vh/albok `7ae0d2a6`
(tag albok-service/v0.1.14). This is unit 4.3, with three additions: (tag albok-service/v0.1.14). This is unit 4.3, with three additions:
- `GET .../sidecars`, a batch sidecar listing. Over 5,000 rows or 8 MiB it answers 413. - `GET .../sidecars`, a batch sidecar listing. Over 5,000 rows or 8 MiB it answers 413.
@@ -126,6 +136,19 @@ grants plus read+write on both library wings) and albok-dev `01M4HAP9HFYZCFV76WX
`read:library/*`), vaulted over `albok/token-nemi` / `albok/token-albok-dev`. albok-dev swaps them into nemi.yaml and `read:library/*`), vaulted over `albok/token-nemi` / `albok/token-albok-dev`. albok-dev swaps them into nemi.yaml and
albok-check (swapped 1459, proven by the 1503 walk) and revoked the previous nemi `01M40JWJ3NZ6VJ77KS8B06BCYZ` and albok-check (swapped 1459, proven by the 1503 walk) and revoked the previous nemi `01M40JWJ3NZ6VJ77KS8B06BCYZ` and
albok-dev `01M3ZMCXE8P66MRX16MZN1D44D` at 1555 10-09. Live set: bootstrap-admin, svos-knowledge-nh3-svos, the new nemi, the new albok-dev. albok-dev `01M3ZMCXE8P66MRX16MZN1D44D` at 1555 10-09. Live set: bootstrap-admin, svos-knowledge-nh3-svos, the new nemi, the new albok-dev.
**Wing `personal/inbox` (2026-10-13, unit 4.4 slice C's capture target, albok-dev's ask after Prime's 1530 intake ruling):**
like `personal/notes`, so `default_group: albok-personal` (1511) and not service-restricted. The hiding is the sensitivity
classes plus Nemi's `restricted_wings`. vh reads it through 1511, and no API token besides Nemi's holds it. The live
config backup is `albok.yaml.pre-inbox`; added by `playbooks/albok-personal-inbox.yaml`, with health reading 7 wings
ready. Nemi's token was re-minted for it: `01M4VX207PJ09ZSECASCSZPCFE`, its previous ten grants plus read+write on
`personal/inbox`, vaulted over `albok/token-nemi`. The previous nemi token `01M4HAP9G90J79HBRJB6VW6PQA` was
revoked at 1742 after albok-dev's swap; the capture source walks (generation 0, empty root).
**2026-10-15:** re-minted again for the vault source (Prime's notes, `/srv/nemi/vault` on nh3-dev → `personal/notes`):
`01M4ZBJG3DMRF9Q9XEVZ8Q20GH` = the twelve grants above plus read+write on `personal/notes`, vaulted over `albok/token-nemi`.
`01M4VX207PJ09ZSECASCSZPCFE` was revoked at 0148 after albok-dev's swap. The first vault walk created 232 objects (129 notes
+ 103 children of the 7 split ones); the judge classed 2 personal, 0 adult, 0 secret. The capture ROOT is on nh3-dev: `/srv/nemi/inbox`
(`playbooks/nh3-dev-nemi-inbox.yaml`).
**Nemi (albok's tagger, runs on nh3-dev):** `albok/token-nemi`, a service token minted 0255 with read+write on **Nemi (albok's tagger, runs on nh3-dev):** `albok/token-nemi`, a service token minted 0255 with read+write on
`fleet/memory`, `fleet/stash` and `personal/agent-feedback`; it sees exactly those three wings (`personal/notes` hidden). `fleet/memory`, `fleet/stash` and `personal/agent-feedback`; it sees exactly those three wings (`personal/notes` hidden).
The previous token (`01M40979…`, without agent-feedback) is still live until albok-dev revokes it. The previous token (`01M40979…`, without agent-feedback) is still live until albok-dev revokes it.
@@ -171,3 +194,34 @@ docker push gitea.phasefinal.com/pfi/albok-service:<ver> # then pin the new
Mount `/srv/albok/store` **`:ro`**, `group_add` the read group's **numeric** gid (1510 / 1511), set Mount `/srv/albok/store` **`:ro`**, `group_add` the read group's **numeric** gid (1510 / 1511), set
`GIT_OPTIONAL_LOCKS=0`, and add `safe.directory` for the path as the viewer sees it. Nothing else `GIT_OPTIONAL_LOCKS=0`, and add `safe.directory` for the path as the viewer sees it. Nothing else
may mount the store read-write. may mount the store read-write.
## Sealed (encrypted) wings: unit 6 (2026-10-16)
The first is `personal/sealed`: restricted, default-deny, gid **1513 `albok-sealed`**. It is empty, has no Nemi source,
and what goes in it is Prime's call. albok-dev provisioned it on Prime's goal "land all of u6". **It is in
`albok.yaml` since 0.1.24 (2026-10-16 0917)**; the first start with it configured and mounted created `store/` and
`private/` under the mount.
| | |
|---|---|
| container | `/srv/albok/luks/personal.sealed.img`: a 20 GiB file (root 0600, dir 0700), LUKS2 aes-xts, ext4 inside (label `personal.sealed`). The file grows later with `truncate` + `cryptsetup resize` + `resize2fs` |
| key | vault **`albok/wing-key-personal-sealed`** (64 random base64 chars). It is never written to nh3-albok's disk: it is piped in over ssh, and infra-ops' sudo there logs output, not input |
| mapping | `/dev/mapper/albok-personal-sealed` |
| mountpoint | **`/srv/albok/sealed/personal.sealed`**: EMPTY and `chattr +i` while unmounted, so nothing can write plaintext into a locked wing (root's `touch` there fails, tested). The mounted root is `1500:1513 0750`, holding `.albok-wing` (`1500:1513 0640`) and ext4's `lost+found`. The service creates `store/` and `private/` there on its first start with the wing configured |
| container view | the parent `/srv/albok/sealed` is bound with `propagation: rslave`, so a lock or unlock on the host shows inside the RUNNING container (tested both ways) |
```bash
# UNLOCK (after a VM reboot the wing comes up LOCKED, by design: no crypttab). No service restart is needed.
# The open takes ~25 s (argon2id keyslot: 215 MiB, time cost 55) and may print "keyslot operation could fail as it
# requires more than available memory": benign, cryptsetup compares against MemFree, not MemAvailable (2026-10-16 harness).
secret get albok/wing-key-personal-sealed | tr -d '\n' | ssh infra-ops@10.100.50.44 \
'sudo -n cryptsetup open --type luks2 --key-file=- /srv/albok/luks/personal.sealed.img albok-personal-sealed \
&& sudo -n mount /dev/mapper/albok-personal-sealed /srv/albok/sealed/personal.sealed'
# LOCK: the service holds the wing's SQLite/Chroma open, so umount refuses while it runs.
ssh infra-ops@10.100.50.44 'cd /opt/docker/compose/albok-service && sudo -n docker compose stop \
&& sudo -n umount /srv/albok/sealed/personal.sealed && sudo -n cryptsetup close albok-personal-sealed \
&& sudo -n docker compose start'
```
**Backups:** nh3-albok has NO file-level backup, so nothing ever copies the plaintext. Its whole-VM vzdump (nh3-pve-2,
21:30) holds the LUKS file, which is ciphertext; the key lives only in the vault. Never add a file-level job that walks
`/srv/albok/sealed`. ⚠ Losing the vault entry loses the wing.
+10 -1
View File
@@ -13,9 +13,10 @@
# ⚠ Nothing else may mount /srv/albok/store read-write. Viewers mount it :ro with group_add by gid # ⚠ Nothing else may mount /srv/albok/store read-write. Viewers mount it :ro with group_add by gid
# (1510 albok-read, 1511 albok-personal) and need GIT_OPTIONAL_LOCKS=0 + a safe.directory entry. # (1510 albok-read, 1511 albok-personal) and need GIT_OPTIONAL_LOCKS=0 + a safe.directory entry.
# 1512 albok-feedback (restricted wing personal/agent-feedback) is default-deny: no viewer gets it. # 1512 albok-feedback (restricted wing personal/agent-feedback) is default-deny: no viewer gets it.
# 1513 albok-sealed (encrypted wing personal/sealed, unit 6) likewise; its plaintext exists only while unlocked.
services: services:
albok-service: albok-service:
image: gitea.phasefinal.com/pfi/albok-service:0.1.19@sha256:6f8332a51abb27ae43071b36a4a0371c51a8c75a5c91236f812ce6ade5968c23 image: gitea.phasefinal.com/pfi/albok-service:1.0.0a1@sha256:6067dfbb034d5234794e9114da5a90c7905481de809aca100d69018edba26ae7
container_name: albok-service container_name: albok-service
restart: unless-stopped restart: unless-stopped
ports: ports:
@@ -25,12 +26,20 @@ services:
- "1510" - "1510"
- "1511" - "1511"
- "1512" # albok-feedback: restricted wing personal/agent-feedback - "1512" # albok-feedback: restricted wing personal/agent-feedback
- "1513" # albok-sealed: encrypted wing personal/sealed (unit 6, 2026-10-16)
volumes: volumes:
- /srv/albok/store:/srv/albok/store - /srv/albok/store:/srv/albok/store
- /srv/albok/private:/srv/albok/private - /srv/albok/private:/srv/albok/private
- /srv/albok/etc/albok.yaml:/etc/albok/albok.yaml:ro - /srv/albok/etc/albok.yaml:/etc/albok/albok.yaml:ro
# the read groups must RESOLVE BY NAME inside the container (layout.py uses grp.getgrnam) # the read groups must RESOLVE BY NAME inside the container (layout.py uses grp.getgrnam)
- /opt/docker/conf/albok-service/group:/etc/group:ro - /opt/docker/conf/albok-service/group:/etc/group:ro
# Unit 6 encrypted wings: the PARENT of the LUKS mountpoints, bound with slave propagation, so a wing unlocked
# (cryptsetup open + mount) after the container started is visible inside it without a restart (README § Sealed).
- type: bind
source: /srv/albok/sealed
target: /srv/albok/sealed
bind:
propagation: rslave
mem_limit: 8g # 4g on nh3-docker (7.7 GiB shared host); 12 GiB VM to itself now mem_limit: 8g # 4g on nh3-docker (7.7 GiB shared host); 12 GiB VM to itself now
healthcheck: healthcheck:
test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8390/health', timeout=5).status == 200 else 1)"] test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8390/health', timeout=5).status == 200 else 1)"]
@@ -35,7 +35,7 @@ listen:
port: 8390 # container port; published on the host as 8392 (8390 is the post office) port: 8390 # container port; published on the host as 8392 (8390 is the post office)
# First instance, per albok-dev's ask (2026-10-02): fleet/{memory,stash}, personal/notes. # First instance, per albok-dev's ask (2026-10-02): fleet/{memory,stash}, personal/notes.
# 2026-10-03: + personal/agent-feedback (restricted, gid 1512). The vault wing comes later (gid 1513+). # 2026-10-03: + personal/agent-feedback (restricted, gid 1512). 2026-10-16: + personal/sealed (unit 6, encrypted, gid 1513; albok-service 0.1.24+).
buildings: buildings:
- id: fleet - id: fleet
default_group: albok-read # host gid 1510 default_group: albok-read # host gid 1510
@@ -49,9 +49,14 @@ buildings:
default_group: albok-personal # host gid 1511 default_group: albok-personal # host gid 1511
wings: wings:
- id: notes - id: notes
- id: inbox # unit 4.4 capture wing (Nemi's drop-box /srv/nemi/inbox on nh3-dev); like notes (2026-10-13)
- id: agent-feedback # auto-memory (Nemi's third source); restricted, default-deny (2026-10-03) - id: agent-feedback # auto-memory (Nemi's third source); restricted, default-deny (2026-10-03)
restricted: true restricted: true
group: albok-feedback # host gid 1512 group: albok-feedback # host gid 1512
- id: sealed # unit 6: the first encrypted wing (LUKS2 /srv/albok/luks/personal.sealed.img; key vault albok/wing-key-personal-sealed)
restricted: true
group: albok-sealed # host gid 1513
encrypted_mount: /srv/albok/sealed/personal.sealed
- id: library # unit 4.5 (Nemi books), albok-dev ask 2026-10-09: epubs from nh3-dev ~/library - id: library # unit 4.5 (Nemi books), albok-dev ask 2026-10-09: epubs from nh3-dev ~/library
default_group: albok-read # host gid 1510, readable like fleet; neither wing restricted, no encrypted mount default_group: albok-read # host gid 1510, readable like fleet; neither wing restricted, no encrypted mount
wings: wings:
+1
View File
@@ -40,3 +40,4 @@ albok:x:1500:
albok-read:x:1510:albok albok-read:x:1510:albok
albok-personal:x:1511:albok albok-personal:x:1511:albok
albok-feedback:x:1512:albok albok-feedback:x:1512:albok
albok-sealed:x:1513:albok
+49 -2
View File
@@ -5,8 +5,7 @@ runner. Polls `gitea.phasefinal.com` for jobs from any repo that has a
`.gitea/workflows/` directory and runs them in ephemeral docker `.gitea/workflows/` directory and runs them in ephemeral docker
containers on this host. containers on this host.
**Server:** ana-docker (single central runner — see "Topology" below **Server:** ana-docker (the central runner), plus the opt-in fast runner on nh3-docker (see "Topology")
for when to add more)
**Image:** `gitea/act_runner:latest` **Image:** `gitea/act_runner:latest`
**Outbound only** — no host port published; the runner connects out to **Outbound only** — no host port published; the runner connects out to
gitea, gitea never connects in. gitea, gitea never connects in.
@@ -35,6 +34,54 @@ invocation — not a copy-pasted playbook.
Until one of those bites, one runner is enough. Until one of those bites, one runner is enough.
### The opt-in fast runner on nh3-docker (Prime's CI fix A, 2026-10-12)
`nh3-docker-runner` (gitea runner id 4, act_runner v0.6.1) carries **one label, `nh3-docker`**, and NOT `pfi-fleet`.
Nothing lands on it unless a workflow says `runs-on: nh3-docker`. It exists because ana-docker's disk is slow to
fsync: pfi-pve's `ospool` is HDD mirrors with no SLOG. Jobs that fsync a lot (a Postgres service, pytest temp dirs)
crawled into the 30 m timeout there.
Measured 2026-10-12 1725 in a throwaway repo (`vh/ci-probe-nh3`, deleted after). One run had a job on each runner
with the same steps (8 KiB `dd oflag=dsync` × 200, n = 3 each):
| | nh3-docker-runner | ana-docker-runner |
|---|---|---|
| 200 synced 8 KiB writes | 0.51 / 0.54 / 0.57 s (~2.7 ms each) | 6.36 / 6.82 / 7.21 s (~34 ms each) |
| docker socket in the job | **absent** | present (root over the host) |
| job memory / CPU limit | 4 GiB / 6 CPUs | none |
| `services:` container | Postgres reachable after 1 s | (not tested here) |
Its config is `conf/config.nh3-docker.yaml`, tighter than ana-docker's because nh3-docker also runs NH3's DNS
(adguardhome) and the althing post office on 8 vCPU / 7 GiB:
- `docker_host: "-"` keeps the docker socket out of jobs (the probe above proves it). `services:` still work.
- `capacity: 1`, and each job container is capped at `--memory=4g --memory-swap=4g --cpus=6`.
- `network: ""` gives every job its own network instead of traefik-net.
- The timeout stays at 30 m.
**First real use:** vastblue/platform's python job moved here at c02048c. Run 11042 passed in 5 m; pytest
(~1,234 tests) took 4m45s, against 14m17s on ana-docker in run 11033. That is one run each, at different times of
day, so read it as a ballpark. The Postgres service, apt and npm all worked without the socket.
**Second repo: vh/albok** (test job moved at 27d2bb5). Green as of run 11051 (2026-10-12 2312): 1,415 passed in
9m43s, a job of 11m29s. ⚠ **Correction to infra-ops's 10-12 diagnosis:** albok's CI was never a hanging test. The 169
service tests fsync their journal and stores on every test: they take 57 s on a laptop and took over 25 min on
ana-docker, and pytest-timeout never fired. The 4 GiB job cap here then exposed a real bug: a chromadb thread leak,
26,710 threads / 3.2 GB at peak, and EAGAIN at test 1,273. albok fixed it with a conftest. Note also that job
containers run as root, so tests relying on `chmod 0` must skip.
A job that needs docker inside it (buildx and the like) cannot run here; keep those on `pfi-fleet`. ⚠ nh3-docker's
vCPU has no AVX2 either, so CPU-bound torch tests are no faster here. This runner fixes the disk, not the CPU.
Deployed with:
```bash
scripts/elway infra-ops@10.100.50.40 --playbook playbooks/deploy-gitea-runner.yaml \
--var runner_name=nh3-docker-runner --var runner_labels=nh3-docker:docker://node:20-bookworm-slim \
--var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
```
Use `infra-ops@<ip>`, not the `nh3-docker` alias: the alias logs in as lkraven, and the first attempt left a
lkraven-owned data dir that infra-ops could not write. The registration token came from the admin API
(`POST /api/v1/admin/actions/runners/registration-token`) and was cleared from `.env` once the runner registered.
## What a job on this runner can actually do ## What a job on this runner can actually do
Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted). Measured 2026-09-02 on `pfi-fleet` (throwaway repo, three jobs, since deleted).
+1 -1
View File
@@ -28,7 +28,7 @@ services:
- tnet - tnet
labels: labels:
- homepage.group=Agents (no UI) - homepage.group=Agents (no UI)
- homepage.name=gitea-runner - homepage.name=${GITEA_RUNNER_NAME}
- homepage.icon=mdi-cog-play - homepage.icon=mdi-cog-play
- homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME}) - homepage.description=Gitea Actions self-hosted runner (${GITEA_RUNNER_NAME})
@@ -0,0 +1,43 @@
# act_runner config for the OPT-IN fast runner on nh3-docker (Prime's CI fix A, 2026-10-12).
# Mounted at /data/config.yaml. Deployed by playbooks/deploy-gitea-runner.yaml with
# --var runner_config=stacks/gitea-runner/conf/config.nh3-docker.yaml
#
# Why it exists: the central runner on ana-docker sits on pfi-pve's `ospool` (HDD mirrors, no SLOG); 8 KiB fsync
# writes measured ~145 ms there vs ~2.7 ms on nh3-docker. Jobs that fsync a lot (a Postgres service, pytest temp
# dirs) ran past the 30 m job timeout. Repos opt in with `runs-on: nh3-docker`. The runner carries ONLY that label,
# never `pfi-fleet`, so nothing lands here by accident.
#
# Why it is tighter than ana-docker's config: nh3-docker also runs NH3's DNS (adguardhome) and the althing post
# office, on 8 vCPU / 7 GiB. So:
# - docker_host "-": the job containers do NOT get the host docker socket (ana-docker's runner hands every job root
# over its host). `services:` still work, because the runner creates them through its own socket.
# - capacity 1, and each job container is capped at 4 GiB / 6 CPUs, leaving room for DNS and the bus.
# - network "": every job gets its own throwaway network, not traefik-net.
log:
level: info
runner:
file: /data/.runner
capacity: 1
timeout: 30m
shutdown_timeout: 1m
insecure: false
fetch_timeout: 5s
fetch_interval: 2s
cache:
enabled: true
dir: /data/cache
container:
network: ""
privileged: false
options: "--memory=4g --memory-swap=4g --cpus=6"
workdir_parent: /data/workspace
valid_volumes: []
docker_host: "-"
force_pull: false
host:
workdir_parent: /data/host-workspace
+35
View File
@@ -0,0 +1,35 @@
# h2o-eval (TEMPORARY) — fv-ml1 GPU 3, :8034
> **DOWN since 2026-10-12 1619 PT** (round 2 done: the floor was ruled out, with identical verdicts on and off, and albok-nemi 0.1.18
> re-fitted the judge's note question). The host keeps `/opt/docker/compose/h2o-eval` and the weights.
>
> **Round 2 (was UP 2026-10-12 1428–1619 PT)** (Prime 1420, via albok-dev): "I'd like to understand why h2o failed, so
> stand it up on gpu3 and refit it and let's get an understanding." albok-dev runs the investigation and the judge
> re-fit (thread `01M4S026GWS354WZGXZ5E34F31`). **No load tests or other traffic while their runs are in.** Round 1
> (1300–1319) was the pre-cutover validation. Take it down with `docker compose down` when albok-dev is done.
>
> | port | front | floor | judge stamp (`/health` model) |
> |---|---|---|---|
> | `:8034` | `h2o-eval-front` | ON, 0.801, as the live cutover ran | `h2o-lightning-4b@b43f759b…`, identical to the cutover's |
> | `:8035` | `h2o-eval-front-nofloor` | OFF (`SHIM_NOUL_FLOOR=0`) | `h2o-lightning-4b+noul-floor-0@b43f759b…` |
>
> Both fronts sit over ONE vLLM (14,420 MiB, KV 135,070 tokens, util-sized because GPU 3 is otherwise empty). Each
> `/health` also reports `backend.noul_floor {value, source}`.
>
> Checked at 1428:
> - `/decide/shared` 200 on both, with the cutover's smoke values reproduced (adult 0.052, personal 0.876).
> - An https image URL is refused on both: 422 "not accepted here".
> - **The vLLM backstop answered for the first time.** An https image URL posted straight to vLLM got 400 "The URL must
> be from one of the allowed domains: ['media.invalid']", with no fetch.
h2o-lightning-4b @ b43f759b behind `services/h2o-decision-front` (copied into `conf/front/`), for albok-dev's
re-validation of Nemi's sensitivity judge before the GPU 1 cutover (Prime 2026-10-12 via brokkr). **Remove right after
the cutover:** GPU 3 is for transient work only (Scriberr and MIA use it).
- Weights: `/tank/aimodels/h2o-lightning-4b-b43f759b` (copied from the bench kit; model.safetensors sha256
`2d51561f221e1549d91bd1c38b3512a1435dd32950d454a9f040f1d4d11497b6`, tokenizer.json `5f9e4d49…`, shim git oid
`fee7807e…`, serve_config `5e6fef1c…`: all equal to Hugging Face at b43f759b).
- vLLM exactly as benched: `--max-model-len 40960 --max-num-seqs 16 --gpu-memory-utilization 0.1506`, vision limits as
the vendor's serve.sh; healthy ~3 min after `up` (graph capture), ~14.4 GB on GPU 3.
- `.env` on the host: `FRONT_TOKEN` = intern-decision's token (= vault `albok/judge-token`).
- Remove: `cd /opt/docker/compose/h2o-eval && docker compose down` (the weights stay for the GPU 1 seat).
+90
View File
@@ -0,0 +1,90 @@
# TRANSIENT h2o-lightning-4b eval seat on fv-ml1 GPU 3 (GPU 3 is for transient work only).
# Round 1, 2026-10-12 1300-1319: albok-dev validated Nemi's judge on h2o before the GPU 1 cutover (Prime via brokkr).
# Round 2, from 2026-10-12 ~1430 (Prime 1420, via albok-dev): "I'd like to understand why h2o failed, so stand it up
# on gpu3 and refit it and let's get an understanding." albok-dev investigates the 154 texts h2o alone called adult
# and re-fits the judge. Two fronts over ONE vLLM:
# :8034 front the shim's yes/no floor ON (0.801), exactly as the live cutover ran; /health model stamp
# h2o-lightning-4b@b43f759b..., identical to the cutover's
# :8035 front-nofloor SHIM_NOUL_FLOOR=0 (floor OFF, brokkr's lever); stamp h2o-lightning-4b+noul-floor-0@b43f759b...
# Each /health also reports backend.noul_floor {value, source}. NO load tests or other traffic while albok-dev's
# runs are in (they name start/end on thread 01M4S026GWS354WZGXZ5E34F31). INC keeps :8033 on GPU 1, untouched.
# Serving exactly as benched (docs/pfi/decision-sweep-followon-bench-2026-10-12.md, Amendment 1): stock vLLM 0.30.0
# + the vendor's unmodified shim + the admission front (services/h2o-decision-front, copied into conf/front).
# Weights: /tank/aimodels/h2o-lightning-4b-b43f759b (hashes match HF at b43f759b; see the stack README).
# Inline images only (Prime 2026-10-12): LIGHTNING_IMAGE_URLS=off on both fronts + vLLM --allowed-media-domains.
name: h2o-eval
services:
vllm:
image: vllm/vllm-openai:v0.30.0
container_name: h2o-eval-vllm
restart: unless-stopped
ports:
- "127.0.0.1:18199:8000"
environment:
VLLM_USE_FLASHINFER_SAMPLER: "0"
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- h2o-eval-vllm-cache:/root/.cache/vllm
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
"--max-model-len", "40960", "--max-num-seqs", "16", "--gpu-memory-utilization", "0.1506",
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
"--allowed-media-domains", "media.invalid"]
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["3"]
capabilities: [gpu]
front:
image: python:3.12-slim
container_name: h2o-eval-front
restart: unless-stopped
network_mode: host
depends_on: [vllm]
env_file: .env
environment:
SHIM_VLLM: http://127.0.0.1:18199
SHIM_PORT: "18191"
SHIM_NOUL_FLOOR: "0.801"
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
FRONT_SHIM: http://127.0.0.1:18191
FRONT_VLLM: http://127.0.0.1:18199
FRONT_BIND: 0.0.0.0
FRONT_PORT: "8034"
FRONT_CAP: "32000"
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
FRONT_SERVE_CONFIG: /model/serve_config.json
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- /opt/docker/conf/h2o-eval/front:/front:ro
command: ["bash", "/front/run.sh"]
front-nofloor:
image: python:3.12-slim
container_name: h2o-eval-front-nofloor
restart: unless-stopped
network_mode: host
depends_on: [vllm]
env_file: .env
environment:
SHIM_VLLM: http://127.0.0.1:18199
SHIM_PORT: "18192"
SHIM_NOUL_FLOOR: "0" # the floor OFF (the shim: 0 turns it off)
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
FRONT_SHIM: http://127.0.0.1:18192
FRONT_VLLM: http://127.0.0.1:18199
FRONT_BIND: 0.0.0.0
FRONT_PORT: "8035"
FRONT_CAP: "32000"
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
FRONT_SERVE_CONFIG: /model/serve_config.json
FRONT_MODEL_NAME: h2o-lightning-4b+noul-floor-0 # a distinct judge stamp
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- /opt/docker/conf/h2o-eval/front:/front:ro
command: ["bash", "/front/run.sh"]
volumes:
h2o-eval-vllm-cache:
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
the three things the seat contract needs and the vendor stack does not have:
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
the judge stamps into its cache key.
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
"""
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
TOKEN = os.environ["FRONT_TOKEN"]
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
PORT = int(os.environ.get("FRONT_PORT", "8034"))
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
CAP = int(os.environ.get("FRONT_CAP", "32000"))
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
MAX_QUESTIONS_PER_CALL = 16
MIN_OPTIONS, MAX_OPTIONS = 2, 16
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
shim_mod = importlib.util.module_from_spec(spec)
sys.modules["h2o_shim"] = shim_mod
spec.loader.exec_module(shim_mod)
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
class Refused(Exception):
def __init__(self, code, obj):
super().__init__(obj)
self.code, self.obj = code, obj
def sent_tokens(body):
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
state = body.get("state") or ""
if isinstance(state, (dict, list)):
state = json.dumps(state)
state_txt = shim_mod.render_state(state, CFG.state_compact)
all_toks = []
for qid, q in body.get("questions", {}).items():
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
req = urllib.request.Request(VLLM + "/tokenize",
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=30) as r:
all_toks.append(json.loads(r.read())["tokens"])
mx = max(len(t) for t in all_toks)
shared = shim_mod.common_prefix_len(all_toks)
return mx, shared + sum(len(t) - shared for t in all_toks)
def admit(body):
try:
mx, total = sent_tokens(body)
except Refused:
raise
except Exception as e: # noqa: BLE001
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
if mx > CAP or total > CAP:
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
"shim_accounted_total": total, "cap": CAP})
return mx, total
def post_up(path, obj):
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
method="POST")
try:
with urllib.request.urlopen(req, timeout=300) as r:
return r.status, json.loads(r.read())
except urllib.error.HTTPError as e:
raw = e.read() or b"{}"
try:
return e.code, json.loads(raw)
except ValueError:
return e.code, {"error": raw[:300].decode(errors="replace")}
def systemone(body):
mx, total = admit(body)
code, obj = post_up("/v1/systemone", body)
if isinstance(obj, dict):
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
return code, obj
def check_decision(d, i):
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
opts = d.get("options")
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
for o in opts):
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
if len({o["id"] for o in opts}) != len(opts):
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
if d.get("orderings", "none") != "none":
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
"seat (h2o); send orderings 'none' and combine on the client"})
def decide(state, decisions):
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
for i, d in enumerate(decisions):
check_decision(d, i)
started, results, sizes, tokens = time.perf_counter(), [], [], []
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
questions = {f: {"type": "choice", "instructions": d["question"],
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
code, resp = systemone({"state": state, "questions": questions})
if code != 200:
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
sizes.append(len(chunk))
tokens.append((resp.get("usage") or {}).get("input_tokens"))
for f, d in zip(fields, chunk):
answer = (resp.get("answers") or {}).get(f)
ids = [o["id"] for o in d["options"]]
probs = (answer or {}).get("probabilities") or {}
if answer is None or any(i not in probs for i in ids):
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
"top": answer.get("choice"), "confidence": answer.get("confidence"),
"calibration": resp.get("calibration"), "native": answer,
"input_tokens": tokens[-1], "model": MODEL,
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
"questions_per_call": sizes, "input_tokens": tokens}}
class H(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args):
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
def _send(self, code, obj):
b = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(b)))
self.end_headers()
self.wfile.write(b)
def _authorized(self):
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
def do_GET(self):
path = self.path.split("?")[0]
if path == "/health":
try:
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
shim = json.loads(r.read())
except Exception as e: # noqa: BLE001
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
ok = bool(shim.get("ok"))
return self._send(200 if ok else 503, {
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
if path == "/v1/models":
try:
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
return self._send(r.status, json.loads(r.read()))
except urllib.error.HTTPError as e:
return self._send(e.code, {"error": "upstream"})
return self._send(404, {"error": "not found"})
def do_POST(self):
path = self.path.split("?")[0]
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
n = int(self.headers.get("Content-Length", 0) or 0)
if n > 4 * 1024 * 1024:
return self._send(413, {"error": "body over 4 MiB"})
try:
body = json.loads(self.rfile.read(n) or b"null")
except ValueError:
return self._send(422, {"error": "body is not JSON"})
if not isinstance(body, dict):
return self._send(422, {"error": "body must be a JSON object"})
try:
if path == "/v1/systemone":
return self._send(*systemone(body))
if path == "/api/alpha/decisions":
admit(body)
return self._send(*post_up(path, body))
if path == "/decide/shared":
return self._send(200, decide(body.get("state"), body.get("decisions")))
if path == "/decide":
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
out = decide(body.get("state"), [d])
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
except Refused as r:
return self._send(r.code, r.obj)
return self._send(404, {"error": "not found"})
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
+13
View File
@@ -0,0 +1,13 @@
#!/bin/bash
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
set -eu
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
--host 127.0.0.1 --port "$SHIM_PORT" &
SHIM=$!
python3 /front/front.py &
FRONT=$!
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
kill "$SHIM" "$FRONT" 2>/dev/null || true
exit 1
+129 -3
View File
@@ -1,6 +1,132 @@
# intern-decision # intern-decision — the decision seat (fv-ml1 GPU 1, :8033)
> **LIVE since 2026-09-30 0941 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168. > **LIVE: h2o-lightning-4b, since 2026-10-12 1807 PT: the production seat.** Prime, 1805, via albok-dev: "okay, go
> to h2o, and fix forward. do not fall back to inc." `compose.yaml` = `compose.h2o.yaml`. Nemi's judge runs the v2
> note question at concurrency 4.
>
> ⚠ **No fallback to INC.** `rollback-to-inc.sh` and `compose.inc.yaml` stay on the host as an artifact, but problems
> on this seat are fixed forward (judge wording, thresholds, `visibility` declarations), not by switching back. Using
> INC again would need Prime's explicit word.
>
> **What happened on 2026-10-12:**
> 1. **h2o 1311–1403 (judge v1), rolled back.** Adult flags went from 88 to 242. The v1 question asked two things at
> once.
> 2. **The GPU 3 investigation.** The floor was ruled out, and the v2 question was shipped (albok-nemi 0.1.18).
> 3. **h2o 1704–1747 (v2), rolled back.** Two margins moved:
> - nonfiction adult 0 → 79, from one textbook chapter at 0.531;
> - memory personal 6 → 18.
> 4. **h2o for good from 1807.** albok-dev fixed the chapter margin first, by declaring the two published nonfiction
> titles `visibility: public`, so the judge is no longer consulted for them. The personal-question reading is
> being re-fitted forward on the live seat.
### The h2o seat (`compose.h2o.yaml`; production since 2026-10-12 1807)
| | |
|---|---|
| **URL** | `http://10.251.50.54:8033` = `http://intern-decision.fv.internal:8033`. `/health` is open; everything else needs `Authorization: Bearer $(secret get intern-decision/api-token)` (the same value as `albok/judge-token`) |
| **Model** | [`h2oai/h2o-lightning-4b`](https://huggingface.co/h2oai/h2o-lightning-4b) @ `b43f759b6385d21182f87c3829931ec3d4dfa98a`, from `/tank/aimodels/h2o-lightning-4b-b43f759b` (read-only). `model.safetensors` sha256 `2d51561f…d4d11497b6`; tokenizer, shim, serve_config, config and chat template all equal Hugging Face at that revision |
| **Serving** | `vllm/vllm-openai:v0.30.0` (container `intern-decision-vllm`, loopback `:18299`), args exactly as benched: `--max-model-len 40960 --max-num-seqs 16`, the vendor's vision limits, `VLLM_USE_FLASHINFER_SAMPLER=0` |
| **KV** | **pinned in bytes**: `--kv-cache-memory 4810363371` (4.48 GiB = 135,558 tokens, 3.31× concurrency at 40,960). GPU 1 is shared with meromero-rp and gen-small, so a util-sized KV would drift on a co-boot. `--gpu-memory-utilization 0.1506` now only gates vLLM's boot check |
| **Front** | container `intern-decision` (python:3.12-slim, host network, `:8033`), running `conf/front/run.sh`: the vendor's **unmodified** `h2o_lightning_shim.py` on loopback `:18291`, behind `services/h2o-decision-front/front.py` |
| **State** | none beyond the `intern-decision_vllm-cache` volume (compile cache) |
| **Rollback** | `compose.inc.yaml` is the INC compose, same `.env`. See **Rollback to INC** below |
### What the front adds (`services/h2o-decision-front`)
- **Admission, refuse-not-trim, at 32,000 tokens.** It counts with h2o's own tokenizer through vLLM `/tokenize`, over
the shim's own render of each question, and checks both the largest prompt and the shim's own total (shared prefix
plus tails). Over the cap gets `422 "admission: N max-prompt / M total tokens against the 32000-token seat limit;
refused, not trimmed"`. The logic is infra-hermes's `tools/h2o-admission/admission_proxy.py` (30f36f2), unchanged.
- **The bearer token** on every path except `GET /health`.
- **`/decide` and `/decide/shared`**, translated onto `/v1/systemone`. This keeps albok-nemi's judge, the seat's only
production caller, working unchanged. One call holds at most 16 questions and is split greedily beyond that, as
INC did. `orderings` other than `none` → 422. Each result carries `probabilities`, `top`, `confidence`,
`calibration`, `native` (the shim's answer object, unchanged), `input_tokens`, `model` and `call`. INC's
`prompt_sha256`, `readout` and timing fields are gone.
- **`/health`** reports `model {name, source, revision}`, the cap, the endpoints and vLLM's `max_model_len`, and
returns 503 when the shim is not healthy.
- `/v1/systemone` and `/api/alpha/decisions` pass through after admission, and `/v1/models` passes through as is.
### Verification at the 1311 cutover (2026-10-12, 1311–1318 PT, on fv-ml1 loopback)
| check | result | expected |
|---|---|---|
| `/health` | `ok`, model `h2o-lightning-4b` @ `b43f759b…`, cap 32,000, `max_model_len` 40,960 | the pin |
| over the cap: one question over a ~40k-token state | **422**, "40078 max-prompt / 40078 total … refused, not trimmed" | 422 |
| 16 questions at the cap (31,528 max-prompt / 31,999 shim total), warm, N = 3 | **200**; 2.105 / 2.061 / 1.999 s | ~2 s (bench) |
| `/decide/shared` smoke: two yes/no decisions over a line holding a phone number and a street address | 200 in 0.52 s; p(yes) explicit 0.052, personal info 0.876 | the right way round |
| GPU 1, h2o's vLLM process, at rest / after the 16q@32k calls | 14,420 / 15,590 MiB | budget 16,847, bench peak 15,566 |
The 16q@32k and `/decide/shared` rows show the path works; the speed and memory numbers are infra-hermes's, below.
**infra-hermes, 1316–1318, same seat**: the bench's own harness (jevbench.cli v1.2.16 with the typesafe adapter, plus
the Amendment 1 gate probe driver). Bearer passed by `--key-env`. Artifacts are on fv-ml1 in `/tank/bench/cutover/jevbench/`
and `/tank/bench/cutover-probe.json`.
| check | result | reference |
|---|---|---|
| **JevBench public, one pass** | **205/231** | 205 ×3 in the Amendment 1 bench |
| depth cells: evidence at depth 1k / 8k / 16k / 30k | `supported` at every depth (p 0.89–0.94), admitted at 31,505 / 31,212 / 12,208 / 31,744 tokens | the 16k cell was under-filled by the probe's calibration, not by the seat. It passed at 29,508 in Amendment 1, and the 30k depth passes here |
| over the cap | 422 at 32,020 / 32,046 / 33,463 / 39,845 tokens, each naming the count | |
| `usage.input_tokens` against the admitted count | equal in every admitted cell | the admission honesty property |
| largest admissible, 1 question / 16 questions (31,417 / 31,907 tokens) | first 251 / 2,395 ms; warm median 226 / **2,159 ms** | bench front 2,033 ms; the bar is 2.93 s. The +126 ms is the front's translation and auth |
| **4 concurrent 16q × 32k calls** | all 4 correct, wall 7,178 ms. GPU 1 peak at 50 ms sampling: **95,178 MiB** = meromero 43,998 + gen-small 34,994 + **seat 16,164** | seat budget 16,847, so **683 MiB headroom**; the bench peak was 15,566 |
⚠ **683 MiB is the seat's margin under 4-way 32k load.** That is tighter than the bench's peak, because of the byte-pinned KV
size and concurrent growth. It passes, but it is the number to re-measure before anything else lands on GPU 1.
### Before any (re)create
GPU 1 holds meromero-rp (~44.0 GB) and gen-small (~35.0 GB). Read the free memory first, with
`scripts/oneoff/fv-ml1-gpucheck.sh 1 0.1506 intern-decision-vllm`, and stop if it fails. The vLLM container takes about
3 minutes to reach healthy (graph capture). The front waits on it, and the healthcheck has a 300 s start period.
### Images: inline only (Prime 2026-10-12, via brokkr)
The h2o seat must never fetch an image URL. `compose.h2o.yaml` sets `LIGHTNING_IMAGE_URLS=off` on the front, and the
vendor shim then refuses every image URL with 422 "image URLs are not accepted here". Inline `data:` URIs and base64
keep working. vLLM also runs with `--allowed-media-domains media.invalid` as a backstop. That is a reserved name that
never resolves, and vLLM 0.30.0 checks every http(s) media URL against it before fetching, while `data:` URLs bypass
the check.
The shim switch was verified live at 1402, on the h2o seat with the front recreated alone:
- the same `https://media.invalid/red.png` request went from "could not be fetched" (a fetch was attempted) to "not
accepted here" (refused before any fetch);
- inline red still answered red at 0.984;
- before the change, inline red/blue/green answered at 0.98+ and no-image came out near-uniform.
The vLLM backstop was first exercised on the GPU 3 eval seat at 1428 (same image and flag): an https image URL
posted straight to vLLM got 400 "The URL must be from one of the allowed domains: ['media.invalid']", with no fetch.
Repeat that check on this seat's loopback port (`:18299`) on the next switch. INC needs neither setting, because its served checkpoint has no vision tower.
### Switching between the two
The final switch: `switch-to-h2o.sh` ran from 180513, h2o healthy at 180740, with the same checks all passing. The 1704 switch: `switch-to-h2o.sh` ran from 170223 (rc 0) and h2o was healthy at 170454. The 1705 checks: `/health` at the
pin, with floor 0.801; `/decide/shared` 200; a 422 over the cap; inline red → red at 0.984; an https image URL refused
by the shim; the vLLM backstop refusing example.com (400); GPU 1 resting at 14,430 MiB.
Both run ON fv-ml1 and share one `.env`.
- **To h2o:** `sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh`. It stops INC and checks that GPU 1 has
16,900 MiB free (h2o's measured budget). If not, it restarts INC and exits 2. Otherwise it copies
`compose.h2o.yaml` over `compose.yaml` and waits up to 300 s for `/health`. Only do this once albok has re-fitted
the judge.
- **To INC:** `sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh`. It stops h2o, copies `compose.inc.yaml`
over `compose.yaml` and waits up to 180 s. Image `intern-decision-serve:0.1.3` and its
`intern-decision_triton-cache` volume survive the `down`. Used 2026-10-12, 140328 → healthy 140356, and again
174746 → healthy 174811.
After either switch, make `stacks/intern-decision/compose.yaml` match the host again (it is h2o since 1807).
⚠ **Tell albok-dev on either switch: Nemi's judge concurrency depends on the seat.** INC scores on one inference
thread (INV-2, ~16 calls/s at most), so the judge runs with concurrency 1 against it (albok's `nemi.yaml`, since
2026-10-12). With 4 in flight on INC, the 1533–1619 walk queued behind itself and lost 339 calls to 30 s timeouts. On
h2o, vLLM batches, so it goes back to 4.
---
# History: the Intern-Decision-4B seat (2026-09-30 → 10-12 1308, 1403–1702, 1748–1805; retired, no fallback per Prime)
> **LIVE from 2026-09-30 0941 PT to 2026-10-12 1308 PT, and again since 2026-10-12 1403 PT** on fv-ml1 GPU 1, with cap 9.0 GiB and `MAX_TOKENS` 7,168.
> Acceptance on the live service is below. `semif` is stopped and kept as the rollback. > Acceptance on the live service is below. `semif` is stopped and kept as the rollback.
**Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`, **Intern-Decision-4B typed decisions** on **fv-ml1 GPU 1**, the utility card beside `vllm-coder`,
@@ -293,7 +419,7 @@ Read `docker logs intern-decision`. Record the deploy with `scripts/ops-log`.
Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and Re-run the bench's sets through the service (`services/intern-decision-serve/acceptance/`), and
re-measure the VRAM before changing the cap. re-measure the VRAM before changing the cap.
## Rollback ## Rollback (INC → semif)
`semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word: `semif` is **stopped, not removed**, and is kept as the rollback. Only on Prime's word:
+75
View File
@@ -0,0 +1,75 @@
# STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh
# copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge
# flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8).
# h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
# conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
# /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
# Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
# GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
# util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
# Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's
# --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes.
name: intern-decision
services:
vllm:
image: vllm/vllm-openai:v0.30.0
container_name: intern-decision-vllm
restart: unless-stopped
ports:
- "127.0.0.1:18299:8000"
environment:
VLLM_USE_FLASHINFER_SAMPLER: "0"
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- vllm-cache:/root/.cache/vllm
command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
"--max-model-len", "40960", "--max-num-seqs", "16",
"--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
"--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
"--allowed-media-domains", "media.invalid"]
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["${GPU_ID:-1}"]
capabilities: [gpu]
front:
image: python:3.12-slim
container_name: intern-decision
restart: unless-stopped
network_mode: host
depends_on: [vllm]
environment:
FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
SHIM_VLLM: http://127.0.0.1:18299
SHIM_PORT: "18291"
SHIM_NOUL_FLOOR: "0.801"
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
FRONT_SHIM: http://127.0.0.1:18291
FRONT_VLLM: http://127.0.0.1:18299
FRONT_BIND: 0.0.0.0
FRONT_PORT: ${PORT:-8033}
FRONT_CAP: "32000"
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
FRONT_SERVE_CONFIG: /model/serve_config.json
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- /opt/docker/conf/intern-decision/front:/front:ro
command: ["bash", "/front/run.sh"]
healthcheck:
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 300s
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Decision seat (h2o-lightning-4b)
- homepage.icon=mdi-scale-balance
- homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
volumes:
vllm-cache:
+74
View File
@@ -0,0 +1,74 @@
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls).
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now").
#
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code +
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on
# fv-ml1 from that dir.
#
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
# gets 503 out_of_memory and the service stays up. See the README before changing it.
#
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
# (vault intern-decision/api-token, >= 32 chars).
name: intern-decision
services:
intern-decision:
image: ${IMAGE:?set IMAGE}
container_name: intern-decision
restart: unless-stopped
ports:
- "${PORT:-8033}:8000"
environment:
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
INTERN_DECISION_DEVICE: cuda
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
# POSTs in progress (queued + scoring) before new ones get 429 busy.
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
volumes:
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads.
- /tank/aimodels/huggingface:/hf:ro
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts.
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable.
- triton-cache:/tmp/triton-cache
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["${GPU_ID:-1}"]
capabilities: [gpu]
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
start_period: 300s
networks:
- tnet
labels:
- homepage.group=AI - Eval & Retrieval
- homepage.name=Intern-Decision — typed decisions
- homepage.icon=mdi-scale-balance
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1)
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
networks:
tnet:
name: traefik-net
external: true
volumes:
# -> intern-decision_triton-cache on the host
triton-cache:
+52 -51
View File
@@ -1,46 +1,33 @@
# intern-decision: Intern-Decision-4B (internlm, Apache-2.0) behind intern-decision-serve, on # STAGED, NOT LIVE (since the 2026-10-12 1403 rollback): the h2o-lightning-4b decision seat, which switch-to-h2o.sh
# fv-ml1 GPU 1 (the utility card, beside vllm-coder and the erp/meromero seats; scriberr moved to # copies over compose.yaml. It served 1311-1403 on 2026-10-12 (Prime via brokkr) and was rolled back when Nemi's judge
# GPU 3 on 2026-09-30 1322, Prime, to free this card's headroom for 32k-token calls). # flagged 242 texts adult against INC's 88 at fleet scale. Address intern-decision.fv.internal:8033 (brokkr rule 8).
# Replaces semif (Prime, 2026-09-30: "replace semif with intern-decision now"). # h2o-lightning-4b @ b43f759b. Stock vLLM 0.30.0 + the vendor's UNMODIFIED shim + services/h2o-decision-front (copied to
# # conf/front): 32,000-token REFUSE-not-trim admission, the seat's bearer token (the same INTERN_DECISION_API_TOKEN),
# One forward pass per call, scored by the checkpoint's OWN inference.py (sha256-pinned); the # /decide + /decide/shared translated onto /v1/systemone, /health model {name, revision}.
# service keeps semif-serve's HTTP surface (/decide, /decide/shared, /health). Service code + # Rollback to Intern-Decision-4B: compose.inc.yaml (the previous compose, same .env): see the stack README.
# contract: services/intern-decision-serve/ (intern-decision-serve.contract.md). Image built on # GPU 1 is shared (meromero-rp, gen-small): KV is pinned in BYTES (4.48 GiB = 135,070 tokens, the benched size);
# fv-ml1 from that dir. # util only gates the boot check. Gate on scripts/oneoff/fv-ml1-gpucheck.sh before any (re)create.
# # Images: INLINE ONLY (Prime 2026-10-12). LIGHTNING_IMAGE_URLS=off makes the shim refuse every image URL; vLLM's
# ⚠ VRAM_CAP_GIB is a HARD cap on torch's allocator (per-process memory fraction), set so the # --allowed-media-domains media.invalid (a reserved name that never resolves) is the backstop if the shim ever changes.
# container's WHOLE nvidia-smi footprint, CUDA context included, fits GPU 1's free memory beside
# the static vLLM seats (infra-ops budget, 2026-09-30); MAX_TOKENS keeps every accepted call under the cap. A request that needs more
# gets 503 out_of_memory and the service stays up. See the README before changing it.
#
# .env (tunables): IMAGE, PORT, GPU_ID, VRAM_CAP_GIB, MAX_TOKENS, HOST_IP, INTERN_DECISION_API_TOKEN
# (vault intern-decision/api-token, >= 32 chars).
name: intern-decision name: intern-decision
services: services:
intern-decision: vllm:
image: ${IMAGE:?set IMAGE} image: vllm/vllm-openai:v0.30.0
container_name: intern-decision container_name: intern-decision-vllm
restart: unless-stopped restart: unless-stopped
ports: ports:
- "${PORT:-8033}:8000" - "127.0.0.1:18299:8000"
environment: environment:
INTERN_DECISION_API_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN} VLLM_USE_FLASHINFER_SAMPLER: "0"
INTERN_DECISION_DEVICE: cuda
INTERN_DECISION_VRAM_CAP_GIB: ${VRAM_CAP_GIB:?set VRAM_CAP_GIB}
# Coupled to VRAM_CAP_GIB: the largest call measured to fit under the cap (README "VRAM").
INTERN_DECISION_MAX_TOKENS: ${MAX_TOKENS:?set MAX_TOKENS}
INTERN_DECISION_MAX_DECISIONS: ${MAX_DECISIONS:-64}
# POSTs in progress (queued + scoring) before new ones get 429 busy.
INTERN_DECISION_MAX_QUEUE: ${MAX_QUEUE:-32}
volumes: volumes:
# Pinned weights AND the checkpoint's inference.py, read offline. Never downloads. - /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- /tank/aimodels/huggingface:/hf:ro - vllm-cache:/root/.cache/vllm
# Triton/fla autotune cache (README "Cold-shape latency"): the ~6.5 s first-call-per-bucket command: ["--model", "/model", "--served-model-name", "h2oai/h2o-lightning-4b",
# autotune results survive RECREATES (deploys, image upgrades, .env edits), not just restarts. "--max-model-len", "40960", "--max-num-seqs", "16",
# The mount point exists in the image owned by 10001, so the fresh volume is intern-writable. "--gpu-memory-utilization", "0.1506", "--kv-cache-memory", "4810363371",
- triton-cache:/tmp/triton-cache "--limit-mm-per-prompt", "{\"image\": 4, \"video\": 0}",
"--mm-processor-kwargs", "{\"max_pixels\": 1605632}",
"--allowed-media-domains", "media.invalid"]
deploy: deploy:
resources: resources:
reservations: reservations:
@@ -48,27 +35,41 @@ services:
- driver: nvidia - driver: nvidia
device_ids: ["${GPU_ID:-1}"] device_ids: ["${GPU_ID:-1}"]
capabilities: [gpu] capabilities: [gpu]
front:
image: python:3.12-slim
container_name: intern-decision
restart: unless-stopped
network_mode: host
depends_on: [vllm]
environment:
FRONT_TOKEN: ${INTERN_DECISION_API_TOKEN:?set INTERN_DECISION_API_TOKEN}
SHIM_VLLM: http://127.0.0.1:18299
SHIM_PORT: "18291"
SHIM_NOUL_FLOOR: "0.801"
LIGHTNING_IMAGE_URLS: "off" # Prime 2026-10-12: inline images only, the seat never fetches an image URL
FRONT_SHIM: http://127.0.0.1:18291
FRONT_VLLM: http://127.0.0.1:18299
FRONT_BIND: 0.0.0.0
FRONT_PORT: ${PORT:-8033}
FRONT_CAP: "32000"
FRONT_SHIM_FILE: /model/h2o_lightning_shim.py
FRONT_SERVE_CONFIG: /model/serve_config.json
FRONT_MODEL_REVISION: b43f759b6385d21182f87c3829931ec3d4dfa98a
volumes:
- /tank/aimodels/h2o-lightning-4b-b43f759b:/model:ro
- /opt/docker/conf/intern-decision/front:/front:ro
command: ["bash", "/front/run.sh"]
healthcheck: healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=5)"] test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8033/health', timeout=5)"]
interval: 30s interval: 30s
timeout: 10s timeout: 10s
retries: 3 retries: 3
# Startup loads ~9 GB of weights and scores the warm-up three times before it serves.
start_period: 300s start_period: 300s
networks:
- tnet
labels: labels:
- homepage.group=AI - Eval & Retrieval - homepage.group=AI - Eval & Retrieval
- homepage.name=Intern-Decision — typed decisions - homepage.name=Decision seat (h2o-lightning-4b)
- homepage.icon=mdi-scale-balance - homepage.icon=mdi-scale-balance
- homepage.description=Typed decisions from one forward pass (Intern-Decision-4B, fv-ml1 GPU1) - homepage.description=Typed decisions, 32k refusing (h2o-lightning-4b on vLLM, fv-ml1 GPU1; /decide/shared kept)
- homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health - homepage.href=http://${HOST_IP:-10.251.50.54}:${PORT:-8033}/health
networks:
tnet:
name: traefik-net
external: true
volumes: volumes:
# -> intern-decision_triton-cache on the host vllm-cache:
triton-cache:
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
"""The front of the h2o-lightning-4b decision seat (fv-ml1). Stdlib only.
It sits in front of the UNMODIFIED shipped h2o_lightning_shim.py (which sits in front of stock vLLM 0.30.0) and adds
the three things the seat contract needs and the vendor stack does not have:
1. ADMISSION (from infra-hermes's tools/h2o-admission/admission_proxy.py @ 30f36f2, logic unchanged): the shim
silently head-and-tail-trims state above ~32,000 tokens, so every decision request is counted first with h2o's own
tokenizer (vLLM /tokenize over the shim's own per-question renders) and refused with 422 + the count when either the
largest prompt or the shim-accounted total (shared head + tails) exceeds FRONT_CAP. Refused, never trimmed.
2. THE SEAT'S AUTH: a bearer token on everything but GET /health, as intern-decision-serve has (OPEN_PATHS /health).
3. CONTINUITY for /decide and /decide/shared (semif's surface, albok-nemi's sensitivity judge): each decision becomes
one Jev `choice` question exactly as intern-decision-serve builds it ({type: choice, instructions: question,
criteria: {option id: description}}), asked through /v1/systemone in calls of at most 16 questions, and re-keyed
into the semif result shape the judge reads (results[i].native.probabilities[option id]). orderings other than
"none" are refused (422): the judge does its own forward/reverse. /health carries {model: {name, revision}}, which
the judge stamps into its cache key.
Env: FRONT_TOKEN (required), FRONT_BIND, FRONT_PORT, FRONT_SHIM (shim URL), FRONT_VLLM (vLLM URL), FRONT_CAP (32000),
FRONT_SHIM_FILE (vendor shim path, imported for its renderers), FRONT_SERVE_CONFIG, FRONT_MODEL_NAME,
FRONT_MODEL_SOURCE, FRONT_MODEL_REVISION, FRONT_MAX_DECISIONS (64).
"""
import hmac, importlib.util, json, os, sys, time, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
TOKEN = os.environ["FRONT_TOKEN"]
BIND = os.environ.get("FRONT_BIND", "127.0.0.1")
PORT = int(os.environ.get("FRONT_PORT", "8034"))
UP = os.environ.get("FRONT_SHIM", "http://127.0.0.1:18191")
VLLM = os.environ.get("FRONT_VLLM", "http://127.0.0.1:18199")
CAP = int(os.environ.get("FRONT_CAP", "32000"))
SHIM_FILE = os.environ.get("FRONT_SHIM_FILE", "/model/h2o_lightning_shim.py")
SERVE_CONFIG = os.environ.get("FRONT_SERVE_CONFIG", "/model/serve_config.json")
MODEL = {"name": os.environ.get("FRONT_MODEL_NAME", "h2o-lightning-4b"),
"source": os.environ.get("FRONT_MODEL_SOURCE", "h2oai/h2o-lightning-4b"),
"revision": os.environ.get("FRONT_MODEL_REVISION", "b43f759b6385d21182f87c3829931ec3d4dfa98a")}
MAX_DECISIONS = int(os.environ.get("FRONT_MAX_DECISIONS", "64"))
MAX_QUESTIONS_PER_CALL = 16
MIN_OPTIONS, MAX_OPTIONS = 2, 16
spec = importlib.util.spec_from_file_location("h2o_shim", SHIM_FILE)
shim_mod = importlib.util.module_from_spec(spec)
sys.modules["h2o_shim"] = shim_mod
spec.loader.exec_module(shim_mod)
CFG = shim_mod.Contract(json.load(open(SERVE_CONFIG)))
CFG.max_state_tokens = 10**9 # count the UNTRUNCATED render; the shim's own trim is what admission polices
class Refused(Exception):
def __init__(self, code, obj):
super().__init__(obj)
self.code, self.obj = code, obj
def sent_tokens(body):
"""(max tokens of one question's prompt, shim-accounted total = shared head + tails), counted by vLLM's own
tokenizer over the shim's own per-question renders (admission_proxy.py @ 30f36f2, unchanged)."""
state = body.get("state") or ""
if isinstance(state, (dict, list)):
state = json.dumps(state)
state_txt = shim_mod.render_state(state, CFG.state_compact)
all_toks = []
for qid, q in body.get("questions", {}).items():
prompt_text = CFG.question_prompt(state_txt, q, qid)[0]
req = urllib.request.Request(VLLM + "/tokenize",
data=json.dumps({"model": CFG.model, "prompt": prompt_text, "add_special_tokens": False}).encode(),
headers={"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=30) as r:
all_toks.append(json.loads(r.read())["tokens"])
mx = max(len(t) for t in all_toks)
shared = shim_mod.common_prefix_len(all_toks)
return mx, shared + sum(len(t) - shared for t in all_toks)
def admit(body):
try:
mx, total = sent_tokens(body)
except Refused:
raise
except Exception as e: # noqa: BLE001
raise Refused(502, {"error": f"admission: tokenizer count failed: {type(e).__name__}: {e}"})
if mx > CAP or total > CAP:
raise Refused(422, {"error": f"admission: {mx} max-prompt / {total} total tokens against the {CAP}-token seat "
f"limit; refused, not trimmed", "max_prompt_tokens": mx,
"shim_accounted_total": total, "cap": CAP})
return mx, total
def post_up(path, obj):
req = urllib.request.Request(UP + path, data=json.dumps(obj).encode(), headers={"Content-Type": "application/json"},
method="POST")
try:
with urllib.request.urlopen(req, timeout=300) as r:
return r.status, json.loads(r.read())
except urllib.error.HTTPError as e:
raw = e.read() or b"{}"
try:
return e.code, json.loads(raw)
except ValueError:
return e.code, {"error": raw[:300].decode(errors="replace")}
def systemone(body):
mx, total = admit(body)
code, obj = post_up("/v1/systemone", body)
if isinstance(obj, dict):
obj.setdefault("_admission", {}).update({"max_prompt_tokens": mx, "shim_accounted_total": total, "cap": CAP})
return code, obj
def check_decision(d, i):
if not isinstance(d, dict) or not isinstance(d.get("id"), str) or not isinstance(d.get("question"), str):
raise Refused(422, {"error": f"decision {i}: needs string id and question"})
opts = d.get("options")
if not isinstance(opts, list) or not MIN_OPTIONS <= len(opts) <= MAX_OPTIONS:
raise Refused(422, {"error": f"decision {d['id']!r}: options must contain {MIN_OPTIONS}-{MAX_OPTIONS} entries"})
if any(not isinstance(o, dict) or not isinstance(o.get("id"), str) or not isinstance(o.get("description"), str)
for o in opts):
raise Refused(422, {"error": f"decision {d['id']!r}: every option needs string id and description"})
if len({o["id"] for o in opts}) != len(opts):
raise Refused(422, {"error": f"decision {d['id']!r}: option ids must be unique"})
if d.get("orderings", "none") != "none":
raise Refused(422, {"error": f"decision {d['id']!r}: orderings {d.get('orderings')!r} is not served by this "
"seat (h2o); send orderings 'none' and combine on the client"})
def decide(state, decisions):
"""semif /decide/shared on top of /v1/systemone: intern-decision-serve's mapping (one Jev choice question per
decision, calls of at most 16, fields q / q1..qN), re-keyed into its result shape."""
if not isinstance(decisions, list) or not 1 <= len(decisions) <= MAX_DECISIONS:
raise Refused(422, {"error": f"decisions must be a list of 1..{MAX_DECISIONS}"})
for i, d in enumerate(decisions):
check_decision(d, i)
started, results, sizes, tokens = time.perf_counter(), [], [], []
for k in range(0, len(decisions), MAX_QUESTIONS_PER_CALL):
chunk = decisions[k:k + MAX_QUESTIONS_PER_CALL]
fields = ["q"] if len(chunk) == 1 else [f"q{j + 1}" for j in range(len(chunk))]
questions = {f: {"type": "choice", "instructions": d["question"],
"criteria": {o["id"]: o["description"] for o in d["options"]}} for f, d in zip(fields, chunk)}
code, resp = systemone({"state": state, "questions": questions})
if code != 200:
raise Refused(code, resp if isinstance(resp, dict) else {"error": str(resp)})
sizes.append(len(chunk))
tokens.append((resp.get("usage") or {}).get("input_tokens"))
for f, d in zip(fields, chunk):
answer = (resp.get("answers") or {}).get(f)
ids = [o["id"] for o in d["options"]]
probs = (answer or {}).get("probabilities") or {}
if answer is None or any(i not in probs for i in ids):
raise Refused(500, {"error": f"scoring_failed: the model's answer for field {f!r} lacks option ids"})
results.append({"id": d["id"], "option_ids": ids, "probabilities": [probs[i] for i in ids],
"top": answer.get("choice"), "confidence": answer.get("confidence"),
"calibration": resp.get("calibration"), "native": answer,
"input_tokens": tokens[-1], "model": MODEL,
"call": {"index": len(sizes) - 1, "field": f, "questions": len(chunk)}})
return {"results": results, "timing": {"total_seconds": time.perf_counter() - started, "calls": len(sizes),
"questions_per_call": sizes, "input_tokens": tokens}}
class H(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, fmt, *args):
sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args))
def _send(self, code, obj):
b = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(b)))
self.end_headers()
self.wfile.write(b)
def _authorized(self):
scheme, _, tok = (self.headers.get("Authorization") or "").partition(" ")
return scheme.lower() == "bearer" and hmac.compare_digest(tok.strip().encode(), TOKEN.encode())
def do_GET(self):
path = self.path.split("?")[0]
if path == "/health":
try:
with urllib.request.urlopen(UP + "/health", timeout=10) as r:
shim = json.loads(r.read())
except Exception as e: # noqa: BLE001
return self._send(503, {"status": "unavailable", "model": MODEL, "error": f"shim: {type(e).__name__}"})
ok = bool(shim.get("ok"))
return self._send(200 if ok else 503, {
"status": "ok" if ok else "starting", "model": MODEL, "cap_tokens": CAP, "admission": "refuse, never trim",
"endpoints": ["/decide", "/decide/shared", "/v1/systemone"], "max_decisions": MAX_DECISIONS,
"max_questions_per_call": MAX_QUESTIONS_PER_CALL, "orderings": ["none"], "backend": shim})
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
if path == "/v1/models":
try:
with urllib.request.urlopen(UP + "/v1/models", timeout=10) as r:
return self._send(r.status, json.loads(r.read()))
except urllib.error.HTTPError as e:
return self._send(e.code, {"error": "upstream"})
return self._send(404, {"error": "not found"})
def do_POST(self):
path = self.path.split("?")[0]
if not self._authorized():
return self._send(401, {"error": "unauthorized"})
n = int(self.headers.get("Content-Length", 0) or 0)
if n > 4 * 1024 * 1024:
return self._send(413, {"error": "body over 4 MiB"})
try:
body = json.loads(self.rfile.read(n) or b"null")
except ValueError:
return self._send(422, {"error": "body is not JSON"})
if not isinstance(body, dict):
return self._send(422, {"error": "body must be a JSON object"})
try:
if path == "/v1/systemone":
return self._send(*systemone(body))
if path == "/api/alpha/decisions":
admit(body)
return self._send(*post_up(path, body))
if path == "/decide/shared":
return self._send(200, decide(body.get("state"), body.get("decisions")))
if path == "/decide":
d = {k: body.get(k) for k in ("id", "question", "options", "orderings") if k in body}
out = decide(body.get("state"), [d])
return self._send(200, {**out["results"][0], "total_seconds": out["timing"]["total_seconds"]})
except Refused as r:
return self._send(r.code, r.obj)
return self._send(404, {"error": "not found"})
ThreadingHTTPServer((BIND, PORT), H).serve_forever()
+13
View File
@@ -0,0 +1,13 @@
#!/bin/bash
# One container: the unmodified vendor shim on loopback, then the front. If either dies, the container exits and
# Docker restarts it (restart: unless-stopped). Both inherit the container's whole environment, so the shim's own
# settings (SHIM_*, e.g. SHIM_NOUL_FLOOR, and LIGHTNING_*, e.g. LIGHTNING_IMAGE_URLS=off) set in compose reach it.
set -eu
python3 /model/h2o_lightning_shim.py --config /model/serve_config.json --vllm "$SHIM_VLLM" \
--host 127.0.0.1 --port "$SHIM_PORT" &
SHIM=$!
python3 /front/front.py &
FRONT=$!
wait -n "$SHIM" "$FRONT" 2>/dev/null || true
kill "$SHIM" "$FRONT" 2>/dev/null || true
exit 1
+12
View File
@@ -0,0 +1,12 @@
#!/bin/bash
# One-command rollback of the decision seat to Intern-Decision-4B (the pre-2026-10-12 seat), run ON fv-ml1:
# sudo bash /opt/docker/compose/intern-decision/rollback-to-inc.sh
# It stops the h2o stack, puts the INC compose back (same .env) and starts it (~35 s to healthy).
# Used 2026-10-12 1403. The h2o compose is kept as compose.h2o.yaml (canonical in eshpfi); switch-to-h2o.sh goes back.
set -euo pipefail
cd /opt/docker/compose/intern-decision
docker compose down
cp compose.inc.yaml compose.yaml
docker compose up -d
for i in $(seq 1 60); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "INC healthy"; exit 0; }; sleep 3; done
echo "INC not healthy after 180 s"; exit 1
+18
View File
@@ -0,0 +1,18 @@
#!/bin/bash
# Switch the decision seat to h2o-lightning-4b (the staged compose.h2o.yaml), run ON fv-ml1:
# sudo bash /opt/docker/compose/intern-decision/switch-to-h2o.sh
# Only once albok-dev has re-fitted Nemi's judge for h2o (2026-10-12 rollback; see the stack README).
# It stops INC, checks GPU 1 has room for h2o's measured peak, and either starts h2o or puts INC back.
# h2o takes ~3 min to healthy (graph capture). Undo with rollback-to-inc.sh.
set -euo pipefail
cd /opt/docker/compose/intern-decision
NEED_MIB=16900 # h2o seat budget 16,847 MiB (peak 16,164 under 4 x 16q x 32k on 2026-10-12)
docker compose down
FREE=$(nvidia-smi -i 1 --query-gpu=memory.free --format=csv,noheader,nounits | tr -d ' ')
if [ "$FREE" -lt "$NEED_MIB" ]; then
echo "GPU 1 free ${FREE} MiB < ${NEED_MIB}: not starting h2o; restarting INC"; docker compose up -d; exit 2
fi
cp compose.h2o.yaml compose.yaml
docker compose up -d
for i in $(seq 1 100); do curl -sf http://127.0.0.1:8033/health >/dev/null && { echo "h2o healthy"; exit 0; }; sleep 3; done
echo "h2o not healthy after 300 s: run rollback-to-inc.sh"; exit 1
+9
View File
@@ -1,5 +1,14 @@
# mimir-inbox # mimir-inbox
> **RETIRED 2026-10-13** (Prime 1530: albok subsumes library/note intake). `docker compose down` on corviduo-dev at 2123;
> the image, compose dir, config and the staging mount `/mnt/muninn-staging/mimir-inbox` are left in place for albok's unit 6
> (backup, then teardown, albok-dev's). Bring-back = `docker compose up -d` in the compose dir, nothing else changed.
> **TORN DOWN 2026-10-16 1031 (albok unit 6, on Prime's go).** The on-host compose dir, config, staging data and the personal
> instance's `state/ingestion` queue are DELETED; the images were already gone (cause unknown) and the build cache is pruned.
> Verified archives (sha256 + per-file restore manifests): `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016/`.
> Bring-back is now a rebuild from this directory plus a restore from that archive. This tree is kept as the record only.
UI front end for large-document / novel / book ingestion into the Muninn KB — UI front end for large-document / novel / book ingestion into the Muninn KB —
server-side FastAPI + HTMX over the **muninn-gate** API (#377). The submit path server-side FastAPI + HTMX over the **muninn-gate** API (#377). The submit path
is: **browser → mimir-inbox `/upload` (writes to staging) → path-addressed is: **browser → mimir-inbox `/upload` (writes to staging) → path-addressed
+9
View File
@@ -1,5 +1,14 @@
# muninn-gate # muninn-gate
> **RETIRED 2026-10-13** (Prime 1530: albok subsumes library/note intake). `docker compose down` on corviduo-dev at 2123;
> the image, compose dir, config and the staging mount `/mnt/muninn-staging/mimir-inbox` are left in place for albok's unit 6
> (backup, then teardown, albok-dev's). Bring-back = `docker compose up -d` in the compose dir, nothing else changed.
> **TORN DOWN 2026-10-16 1031 (albok unit 6, on Prime's go).** The on-host compose dir, config, staging data and the personal
> instance's `state/ingestion` queue are DELETED; the images were already gone (cause unknown) and the build cache is pruned.
> Verified archives (sha256 + per-file restore manifests): `nh3-nas:/volume1/Backup/retired/albok-u6-corviduo-dev-20261016/`.
> Bring-back is now a rebuild from this directory plus a restore from that archive. This tree is kept as the record only.
WG-internal HTTP front door for the Muninn ingestion queue (Worldtree #377). WG-internal HTTP front door for the Muninn ingestion queue (Worldtree #377).
Path-addressed submit, list/status, cancel/retry, watcher-liveness health. Path-addressed submit, list/status, cancel/retry, watcher-liveness health.
Upstream: `vh/muninn-gate` (no Dockerfile there — this stack owns containerization). Upstream: `vh/muninn-gate` (no Dockerfile there — this stack owns containerization).
+3 -2
View File
@@ -69,8 +69,9 @@ headroom.
## Rollback ## Rollback
The old seat was STOPPED, not removed: `docker stop parakeet-nemo && docker start parakeet` `docker stop parakeet-nemo && (cd /opt/docker/compose/parakeet && docker compose up -d)` restores the sherpa
restores the sherpa seat on :8300 exactly as before (container and image both kept). seat on :8300. Its container was REMOVED 2026-10-17, because its labels kept a stale "Parakeet ASR" on Homepage.
The image `local/parakeet:sherpa-onnx-v4` and the compose dir are kept, so `compose up` recreates it as it was.
## Deploy ## Deploy
+4
View File
@@ -0,0 +1,4 @@
PEWPEW_REV=26bba3c
PORT=8185
# Set both Worldtree vars or neither (WORLDTREE_BASE_URL is in compose.yaml):
WORLDTREE_API_KEY= # secret get nh3-dev/.config/worldtree/pewpewstudio-demo-key
+51
View File
@@ -0,0 +1,51 @@
# pewpewstudio
Laser job studio (FastAPI backend + React SPA, gitea `vh/pewpewstudio`). It composes Thunder Aurora marking jobs and
returns LightBurn `.lbrn` files as browser downloads, and never touches hardware. Deployed 2026-10-13 on Prime's directive
via pewpew-dev (thread `01M4WATRFQX3V7TSJ4XPGABWMN`).
| | |
|---|---|
| **URL** | `http://10.100.50.40:8185/` (= `http://nh3-docker.nh3.internal:8185/`). Homepage tile in *AI - Studios*. |
| **Host** | nh3-docker: fast local disk for SQLite WAL (fsync ~2.7 ms, unlike ana-docker's HDD pool), and it reaches arbo `100.64.0.6:8201` and Worldtree DEMO `10.250.50.152:8080`. |
| **Auth** | NONE in v1. LAN/mesh only: published on the host's LAN address, with no public ingress. |
| **Images** | `pewpewstudio:<rev>` (api) and `pewpewstudio-web:<rev>` (Caddy + `web/dist`), built on nh3-dev and loaded here; `PEWPEW_REV` in `.env` picks the pair. **Live: `3f73448`** (since 2026-10-15 2327; was `26bba3c`, whose images stay on the host for rollback). |
| **State** | volume `pewpewstudio_state` (`/state`: `shop.db` + `assets/`) and volume `pewpewstudio_config` (`/config`: machine profiles, lenses, denylist, sizing, fonts, which **the app edits from the browser**). Both are covered by nh3-pve's nightly 21:00 whole-VM vzdump of VM 100. |
| **Secrets** | `.env` (0600, infra-ops): `WORLDTREE_API_KEY` = vault `nh3-dev/.config/worldtree/pewpewstudio-demo-key`. |
## Topology
`web` (Caddy, published `10.100.50.40:8185 → :80`) serves the SPA at `/`. It sends `/api/*` to `api:8000` with the
prefix **stripped** (the backend has no `/api` routes and no CORS, so both must be one origin) and Host rewritten to
`localhost`, which the backend's TrustedHost accepts. Because of that rewrite, **Caddy refuses any Host other than
`10.100.50.40` / `nh3-docker.nh3.internal` with 400**, to keep the backend's DNS-rebinding defence. Add a name to
`conf/Caddyfile` before publishing one. Verified 2026-10-13: foreign Host → 400; IP and name → 200.
⚠ That check must stay inside `route { }`. Outside it, Caddy orders `respond` after `handle_path`, and the check
silently never runs: a foreign Host got 200 until it was wrapped.
⚠ The Caddy config is mounted as the DIRECTORY `/opt/docker/conf/pewpewstudio → /etc/caddy`. deploy-stack replaces the
file, and a single-file bind mount keeps showing the old inode.
## Build and deploy (from nh3-dev)
```bash
REV=<short sha>; C=$(mktemp -d)
git -C ~/development/pewpewstudio archive $REV | tar -x -C $C
cp stacks/pewpewstudio/build/Dockerfile $C/Dockerfile.eshpfi
# hamr is a git dependency over gitea SSH (10.250.50.70:222); BuildKit forwards lkraven's gitea key:
(cd $C && docker build -f Dockerfile.eshpfi --ssh default=$HOME/.ssh/id_rsa --target api -t pewpewstudio:$REV . \
&& docker build -f Dockerfile.eshpfi --target web -t pewpewstudio-web:$REV .)
docker save pewpewstudio:$REV pewpewstudio-web:$REV | ssh infra-ops@10.100.50.40 docker load
scripts/deploy-stack.sh nh3-docker pewpewstudio # compose + conf/Caddyfile; never .env
# on nh3-docker: set PEWPEW_REV=$REV in .env (in place, keep 0600), then
cd /opt/docker/compose/pewpewstudio && docker compose up -d
```
**First deploy only:** the config volume was seeded once from the repo's `config/` plus the gitignored
`config/machine.local.toml` from nh3-dev's checkout (machine parameters, no secrets). Without a `machine.*.toml` (the
registry skips `machine.example.toml`), startup fails "no machine.*.toml files". **Never reseed a populated volume**:
it holds the shop's own edits. The database started fresh; the dev `shop.db` was smoke-test data.
## Checks
- `GET /` 200 (the SPA); `GET /api/machines` 200 (one machine, `aurora-mopa-100w-f160`); `GET /api/generate/workflows` 200.
- The api healthcheck is `GET /machines` on loopback.
+33
View File
@@ -0,0 +1,33 @@
# syntax=docker/dockerfile:1.7
# pewpewstudio images, built on nh3-dev from a clean `git archive` of vh/pewpewstudio (see ../README.md).
# api: the FastAPI backend (core.api), venv built with uv; hamr is a git dependency over gitea SSH (BuildKit --ssh).
# web: Caddy serving the SPA (web/dist) and proxying /api/* to the api with the prefix stripped (Caddyfile mounted).
FROM python:3.12-slim-bookworm AS build
RUN apt-get update && apt-get install -y --no-install-recommends git openssh-client build-essential \
&& rm -rf /var/lib/apt/lists/* && pip install --no-cache-dir uv==0.13.0
# gitea's SSH is 10.250.50.70:222; the lockfile names ssh://git@gitea.phasefinal.com/vh/hamr.git, as nh3-dev's ssh config maps it
RUN install -d -m 700 /root/.ssh \
&& printf 'Host gitea.phasefinal.com\n HostName 10.250.50.70\n Port 222\n User git\n' > /root/.ssh/config \
&& ssh-keyscan -p 222 10.250.50.70 > /root/.ssh/known_hosts 2>/dev/null
ENV UV_LINK_MODE=copy UV_PYTHON_DOWNLOADS=never UV_PROJECT_ENVIRONMENT=/app/.venv
WORKDIR /app
COPY . /app
RUN --mount=type=ssh uv sync --frozen --no-dev && uv --version > /app/.uv-version
FROM python:3.12-slim-bookworm AS api
COPY --from=build /app /app
RUN install -d -o 1000 -g 1000 /state /state/assets /config
WORKDIR /app
ENV PATH=/app/.venv/bin:$PATH PYTHONUNBUFFERED=1
USER 1000:1000
EXPOSE 8000
CMD ["python", "-c", "import uvicorn; from core.api.app import create_app; from core.api.config import ApiConfig; uvicorn.run(create_app(ApiConfig.from_env()), host='0.0.0.0', port=8000)"]
FROM oven/bun:1.3 AS webbuild
WORKDIR /web
COPY web/ /web/
RUN bun install --frozen-lockfile && bun run build
FROM caddy:2-alpine AS web
COPY --from=webbuild /web/dist /srv
+51
View File
@@ -0,0 +1,51 @@
# pewpewstudio on nh3-docker: laser job studio (FastAPI + React SPA), composes Thunder Aurora marking jobs and exports
# LightBurn .lbrn. Prime 2026-10-13 via pewpew-dev. NO user auth in v1, so LAN/mesh only: published on the host's LAN
# address, never on a public ingress. Images are built on nh3-dev and loaded here (see README).
# State: the SQLite DB + assets on the named volume `state` (local disk, never NFS). The shop's own config (machine
# profiles, lenses, denylist, sizing) on the named volume `config`, because the app EDITS it from the browser (temp
# file + os.replace in the directory); a volume, not /opt/docker/conf, so deploy-stack's --delete can never touch it.
# Seeded once at first deploy (README). Both volumes ride nh3-docker's whole-VM backup.
name: pewpewstudio
services:
api:
image: pewpewstudio:${PEWPEW_REV:?set PEWPEW_REV}
container_name: pewpewstudio-api
restart: unless-stopped
env_file: .env # WORLDTREE_API_KEY (secret get nh3-dev/.config/worldtree/pewpewstudio-demo-key)
environment:
PEWPEW_DB_PATH: /state/shop.db
PEWPEW_ASSETS_ROOT: /state/assets
PEWPEW_PROFILE_DIR: /config
PEWPEW_DENYLIST: /config/materials.toml
PEWPEW_SIZING: /config/sizing.toml
PEWPEW_FONTS_DIR: /config/fonts
PEWPEW_BASELINE: /config/settings_baseline.toml
ARBO_BASE_URL: http://100.64.0.6:8201
WORLDTREE_BASE_URL: http://10.250.50.152:8080
volumes:
- state:/state
- config:/config
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/machines', timeout=5)"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
web:
image: pewpewstudio-web:${PEWPEW_REV:?set PEWPEW_REV}
container_name: pewpewstudio
restart: unless-stopped
depends_on: [api]
ports:
- "10.100.50.40:${PORT:-8185}:80"
volumes:
- /opt/docker/conf/pewpewstudio:/etc/caddy:ro # the DIRECTORY: deploy-stack replaces the file, and a single-file bind would keep the old inode
labels:
- homepage.group=AI - Studios
- homepage.name=pewpewstudio
- homepage.icon=mdi-laser-pointer
- "homepage.description=Laser job studio: composes Thunder Aurora marking jobs, exports LightBurn .lbrn"
- homepage.href=http://10.100.50.40:${PORT:-8185}/
volumes:
state:
config:
+21
View File
@@ -0,0 +1,21 @@
# pewpewstudio's front: the SPA at /, /api/* to the backend with the prefix STRIPPED (the backend has no /api routes and
# no CORS, so both must be one origin). The backend's TrustedHost accepts localhost, so Host is rewritten to it.
# No request-body cap here: asset uploads are capped at 32 MB by the app itself. LAN/mesh only, no auth in v1.
:80 {
encode zstd gzip
# `route` keeps this literal order; outside it Caddy sorts `respond` AFTER handle/handle_path, and the Host check
# would never run (measured: a foreign Host got 200 before this was a route).
route {
# Keep the backend's TrustedHost meaning (DNS-rebinding defence): the proxy rewrites Host to localhost below,
# so it must refuse any Host the app would refuse. Add a name here when one is published for it.
@foreign not host 10.100.50.40 nh3-docker.nh3.internal
respond @foreign 400
handle_path /api/* {
reverse_proxy api:8000 {
header_up Host localhost
}
}
root * /srv
file_server
}
}
+1 -2
View File
@@ -89,8 +89,7 @@ monitors:
url: http://10.250.50.70:4000/health/liveliness url: http://10.250.50.70:4000/health/liveliness
description: liveliness endpoint, so a model outage does not read as a gateway outage description: liveliness endpoint, so a model outage does not read as a gateway outage
- name: Asset Engine # Asset Engine: RETIRED 2026-10-17 (Prime), monitor deleted with `scripts/kuma delete`.
url: http://10.250.50.70:8200/api/v1/services
# ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ---- # ---- fleet embed/rerank: the one EXCEPTION to "seats are OUT" ----
# Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker` # Since 2026-09-25 these back the gateway's `qwen3-embedding` and `reranker`
+24
View File
@@ -0,0 +1,24 @@
# h2o admission proxy + gate probe set
Production-relevant because the seat swap to h2o-lightning-4b (ordered
2026-10-12) inherits a hard constraint: the shipped vendor shim SILENTLY
trims state at 32,000 tokens. The seat contract is a REFUSING 32,000-token
seat, and this proxy is what makes the refusal real. It must sit in front of
the shim on any deployment.
- `admission_proxy.py` — thin wrapper in front of the unmodified shipped
`h2o_lightning_shim.py`. Imports the vendor's own renderers, counts each
question's exact prompt with vLLM's `/tokenize`, and refuses over-cap with
422 + the named count (both currencies: per-question prompt max and the
shim's shared-head+tails total). Env: ADMIT_CAP (default 32000), ADMIT_UP
(shim URL), ADMIT_VLLM, ADMIT_PORT, ADMIT_SHIM (path to the vendor file).
Measured gate overhead: the shape numbers pass with it in-line.
- `gate_probe_set.py` — the Amendment 1 confirmation driver (calibrate to the
cap, depth-gate cells, cold/warm shapes, 4x concurrency, over-cap refusal).
Written for the bench layout on fv-ml1 (`/tank/bench`); paths are hardcoded,
adapt before re-running anywhere else.
Provenance: fv-ml1:/tank/bench/h2o_admission.py @ the confirmation run,
results in docs/pfi/decision-sweep-followon-bench-2026-10-12.md (Amendment 1
addendum). Measured boot for capacity planning: ~154 s launch-to-ready,
graph-capture bound (no warm-cache or eager-mode saving).

Some files were not shown because too many files have changed in this diff Show More