Persistent memory — eshpfi-management

Last updated: 2026-10-12 ~1410 PT (decision seat: h2o 1311–1403, rolled back to INC, h2o staged; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. Every v2 command was DELETED, not deprecated — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`	per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16)
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched.

2026-10-11: live now (as of 2026-10-11 ~1400 PT)

Nothing is mid-action. No claims held. eshpfi is pushed through df7161b; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with git commit -- <paths>. graphify-out/GRAPH_REPORT.md stays modified on purpose.
U13 cutover: executed 10-11 0721–0816, acceptance 1350. Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin svos, and the bridge app-postoffice on nh3-dev owning the miranda push route. The URGENT bus test (thread 01M4PBNX5PP0PGHDTY5BG7HG95) ingested, paged per the connector receipt, and a reply came back on the thread from svos. → persistent-memory.d/2026-10-11-u13-cutover-executed.md
- Acceptance closed 1354: Miranda's reply arrived on the thread through the bridge's own send. ⚠ CORRECTION (1412): card 05cf119e… was NOT approved (I had inferred it from the reply arriving); it was her own carded send, is moot, and Prime can deny it. Answered by svos-dev 1400: the thread text was Miranda answering bus mail with her own send (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in svos_miranda.turns) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, /opt/svos/current → releases/adc862e); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release 34476a7 (interface UI 0.0.9, tree strands follow speech) at 1712: /opt/svos/current → releases/34476a7 on nh3-svos, no restart, previous handle = adc862e; then ad255a2 (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then 627a9fb (UI 0.0.11: the top bar shows v0.0.<n> · <short commit>, from web/build-id.txt via git-archive export-subst, so ALWAYS build from a clean git archive; a plain copy reads dev) at 1722, previous = ad255a2; 61459d2 (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; 8fa229d (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. Static-only builds: scratchpad svos-static-release.sh <prev> <new> <ui> pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new static/), switch current without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from releases/adc862e (verified via /proc/<pid>/maps): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from releases/dc241f8 (svos-miranda from adc862e). The non-autonomous miranda → app:app-postoffice write grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s synthesis_failed). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos docs/u13-cutover-runbook.md).
- Rollback material, keep until the soak ends: nh3-svos releases/b588510, instance.toml.pre-u13-{A,B}, /etc/svos/kernel.env.pre-u13 (absent: there was none); nh3-dev ~/.hermes/config.yaml.pre-u13 and ~/.hermes/plugins/svos_miranda. After the soak: nh3-svos releases 6c6abbe, a737a06, b588510 and nh3-dev /opt/svos/releases/afab123 (unreferenced) can go.
fv-ml1: hard hang 10-11 ~0005, self-reset, no record (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit fv-ml1-sol-logger (journalctl --user -u fv-ml1-sol-logger). ⚠ HOLD zroot/ROOT/debian@pre-patch-20261008 (do not destroy) until the cause is known or a clean week passes after the last crash.
albok .browse/ share: the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in ~vh/albok-browse-test/ on nh3-albok, waits on Prime pasting the one-liner (scripts/oneoff/albok-browse-withdrawal/README.md); afterwards read results/, report to albok-dev, remove the replica.
Watching: jekyll on sec-reasoning (cyberprev, GPU 0) since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
Waiting on Prime: rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth ce095c1; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ irv-ml1: DO NOT PATCH (Prime); config changes are fine. ana-nas is out of scope.
Small clean-ups, ours, untracked: stale LiteLLM aliases erp-tune-v2 and gemma4-26b-a4b-it-base; ana-docker's Traefik (readTimeout 10m) and CrowdSec (use_wal, bouncer updateMaxFailure: 3) configs live only on the host, not in stacks/; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx .bak on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's patches/0003 lands only at the next scripts/scriberr-rebuild. The LiteLLM compose.yaml on ana-docker differs from stacks/litellm/compose.yaml (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
Open, Prime's calls (unchanged from 10-03): QNAP LCD visual check (services/qnap-lcd/); NH3 Mac mini (Mac-mini, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask openipmi.service on esh-nas-pve; run playbooks/pve-nag-patch.yaml on esh-pve-2 when it is plugged back in.
Older open items, one line each: esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (stacks/blender/README.md). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran 51168a2e on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.

fv-ml1 GPU layout (as of 2026-10-05 2249)

GPU	seat	max ctx	KV	KV tokens	concurrency
0	cyberprev (sec)	262,144	15 GiB	336,935	1.29×
0	erp-seat (char-rp-fast)	262,144	5.5 GiB	346,436	1.32×
0	voices	40,960	6 GiB (bf16)	43,680	1.07×
0	coder (coder-fast)	32,768	1.03 GiB	77,056	2.35×
0	parakeet-nemo	audio	rest 2,108 MiB, cap 3,840	—	—
1	gen-small (+summarizer/classifier)	262,144	5.5 GiB	461,215	1.76× (default output cap 32,768)
1	meromero-rp (char-rp)	262,144	22 GiB	346,436	1.32×
1	intern-decision = INC (h2o 1311–1403 on 10-12, rolled back; staged)	32,768/call	cap 14.4 GiB, peak 15,220 (rest 8,812)	—	—
2	flash-next (gen, gen-large, +6)	262,144	—	344,155	1.31×

Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (--kv-cache-memory), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (scripts/oneoff/fv-ml1-gpucheck.sh), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory feedback_idle_check_must_gate_the_restart).

GPU 3: clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot scripts/mia-run, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to irv-ml1's A6000, not back to GPU 1.

Scriberr (fv-ml1 GPU 3)

LIVE scriberr:local-blackwell-a353078-dropout2: upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, PARAKEET_MODEL_PATH), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). patches/0003 (pyannote env pins) is committed but in no image yet. scripts/scriberr-rebuild re-applies all. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
Diarization works (fixed 2026-10-06): model caches on /tank, HF_TOKEN a placeholder, saved profiles WhisperX large-v3 · GPU · speakers and Parakeet · GPU · speakers. A NEW gated model needs one download with vault fv-ml1/hf-token. → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
URL is https://scriberr.nh3.phasefinal.com since 2026-10-01 (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup .bak-20261001-scriberr). getUserMedia is refused on http://, so recording needs the HTTPS name, which is in ALLOWED_ORIGINS (playbooks/scriberr-https-origin.yaml). ⚠ /opt/docker/compose/scriberr is lkraven-owned, so deploy-stack.sh as infra-ops FAILS there; upload single files with elway --upload --sudo --owner lkraven:lkraven.

Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)

Incident 2026-10-01 04:21, fixed by nemo-0.1.1 (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: empty_cache around every window, MEM_CAP_MIB=3840 (a hard ceiling), CUDA_GRAPHS=0 (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory. Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
Rollback: docker stop parakeet-nemo && docker start parakeet (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with --http h11 (httptools 0.8.0 emits HTTP/1.1 200\x00OK, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

Worldtree U11 memory cutover (demo + personal)

Legacy plane OFF since 0115/0120 PT 2026-09-30 (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted agents/{forseti,lofn,mimir}/memory/<agent>.chroma + memory/context_promotion on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day JSONL_RETENTION_DAYS sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag. scripts/wt-memory-gate-batch now passes --legacy-mode only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. Still mine after b193 deploys: remove the residue memory/context_promotion/ledger.db a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM hermes-gateway (never read); reply to infra-hermes.
Legacy archive: DESTROY it whole by 2026-10-30, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
TODO: re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").

The decision seat `intern-decision` :8033 (Intern-Decision-4B; h2o staged)

LIVE: Intern-Decision-4B (intern-decision-serve:0.1.3), again since 2026-10-12 1403. The only production caller is albok-nemi's sensitivity judge (/decide/shared, threshold 0.5). Run scripts/intern-decision-warmup after an IMAGE change. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md
h2o-lightning-4b is STAGED (stacks/intern-decision/compose.h2o.yaml + switch-to-h2o.sh, with a GPU 1 free-memory gate; weights /tank/aimodels/h2o-lightning-4b-b43f759b; front services/h2o-decision-front). Inline images only (LIGHTNING_IMAGE_URLS=off and the vLLM --allowed-media-domains media.invalid backstop, both verified). It returns only after albok-dev re-fits the judge for it (a 4.4 contract revision on an eval seat). Round 2 (GPU 3 eval seat 1428–1619, Prime 1420) found the cause: the judge's v1 note question asked two things at once. INC answered it inconsistently and h2o answered it literally. The shim floor is ruled out (identical verdicts floor on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge.v2, 4.4 rev 1.15) re-fitted the question. Under v2 the two seats agree (work notes: INC 0/22, h2o 1/22 at 0.52) and fleet/memory adult went 86 → 0 on INC. The h2o cutover is now a cheap call for Prime, with the same sequence and the same rollback as this morning. Threads 01M4S026GWS354WZGXZ5E34F31, 01M4S6MD2GG1WS4FY5WJRP9AMD. ⚠ Its seat headroom under 4-way 32k load was 683 MiB.
Open: label ~50 real Wyrd/Cicada turns before trusting INC in production; its card makes no contamination claim.

nh3-pve + nh3-ml1 (2026-09-25/26)

nh3-pve: Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. AMT live: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as nh3-pve/amt-admin. → servers/nh3-pve/README.md
nh3-ml1 (CT 109 @ 10.100.50.80): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (qwen3-embedding esh-ml1 first, qwen3-embedding-bulk for albok writes nh3-ml1 first). brokkr's foundry seats: lfm-vl :8030 (gateway lfm25-vl-3b), lfm-vl-uncensored :8032 (direct), vibevoice-asr :8031 (audio.cpp, Q8_0). GPU ~11.3/16 GB. → servers/nh3-ml1/README.md

Recent decisions

[2026-10-12] Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did. albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads 01M4RSFC9J2R0J07EEBD4N04W3 (brokkr), 01M4RYSMP3DV9HZHJ0KNBH2S96 (albok).
[2026-10-12] U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403): the browser frames a pane at the registry base_url, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: base_url back to https://svos-terminal.phasefinal.com (config_gen acae45206bbc9fcb), plus POST /seats/{observe,check,launch,retire-launch} allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
[2026-10-12] Shared CI runner (ana-docker act_runner, capacity 2, timeout: 30m) is slow because pfi-pve's ospool is HDD mirrors with no SLOG: 8 KiB oflag=dsync writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. Prime's call, pending: A opt-in fast runner on nh3-docker (recommended), B SLOG for ospool on the BOSS (check endurance), C leave it. ana-docker vCPU is x86-64-v2-AES (no AVX2), a separate lean-against.
[2026-10-11] worldtree-dev is reading mail again (2219), answering four of our threads at once: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread 01M4BDETDGMNVDTFXN3T3Y63DP.
[2026-10-11] U13 cutover executed on Prime's go (0721) + his four rulings — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → persistent-memory.d/2026-10-11-u13-cutover-executed.md
[2026-10-11] fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD — and infra-ops' soft-off that landed mid-boot. → persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md
[2026-10-10] Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay matrix-relay. → persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md
[2026-10-10] U13 phase B built (B1–B6) — kernel 0.0.9/schema 12, svos-kernel.phasefinal.com (nh3-dev only, method-pinned paths), nh3-dev origins + apps env, enrollment secrets. → persistent-memory.d/2026-10-10-u13-phase-b-built.md
[2026-10-10] esh-nas-pve nvme/ROOT@pre-pve9 destroyed after a clean week (the two older snapshots followed on 10-11, Prime). → persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md
[2026-10-09] vh gets every albok read group + four Mac keys (Prime: "albok-everything") — no net new root path. → persistent-memory.d/2026-10-09-vh-albok-everything.md
[2026-10-09] nh3-dev: orphaned svos pytest killed 2242 (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.
[2026-10-09] albok .browse/ sshfs withdrawal test PASSED (Linux) + post-clear checks closed (fv-ml1 plateau, nh3-nas 71%, pre-pbs4 mislabel). → persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md
[2026-10-09] albok library building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev. → stacks/albok-service/README.md
[2026-10-09] Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".
[2026-10-09] SVOS app-traffic ingress for U13 approved (shape agreed with svos-dev) — EXECUTED 10-10 (phase B) and 10-11 (the edit). → persistent-memory.d/2026-10-09-u13-ingress-approved.md
[2026-10-09] nh3-svos → kernel 0.0.8 (b588510, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed. A 4-min browser soak held svos.db handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → servers/nh3-svos/README.md
[2026-10-08] nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the Type=notify unit as a pair); thaw race FIXED: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → servers/nh3-svos/README.md
[2026-10-08] SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door). /voice/* is routed BEFORE Basic auth, because the browser sends Authorization: Bearer <lease> there (web/src/lib/voice.ts) and Basic auth would reject it. All four app credentials are now vaulted as nh3-svos/svos-<app>-credential. → servers/nh3-svos/README.md
[2026-10-08] SVOS served on the LAN too (Prime): Caddy binds 10.100.50.43:443 as well as 100.64.0.9:443; the internal AdGuards answer svos/svos-terminal.phasefinal.com → 10.100.50.43 (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → servers/nh3-svos/README.md
[2026-10-08] fv-ml1 is OFF the mesh again, by design: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active.
[2026-10-08] heid's runaway sweep killed on Prime's order: two heidnotes/sweep.py processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66% — the sweep read its own output. heid fixed it at the source. → persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md
[2026-10-08] SVOS post office LIVE on nh3-svos — Prime declared svos; its ns:notifications write grant was approved on his word at 0731 (card card:4d3a8b26c17f472d828b06fd495192db); a test message was ingested and delivered; reach ok, unread 0. Reported to svos-dev.
[2026-10-07] No z.ai vision alias on the gateway (Prime). glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.
[2026-10-06] Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles. Patch 0003 waits for the next rebuild (tracked at stacks/scriberr/patches/README.md). → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
[2026-10-06] Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00. → persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md
[2026-10-08] Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md
[2026-10-07] nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps — acceptance 17/17 + Chromium; the thaw race. → persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md
[2026-10-07] Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime) — thinking: disabled is light thinking, not zero. → persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md
[2026-10-07] Hermes launchers repaired (Miranda relay 8462): svos-dev's scratch probe had rewritten ~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp} to exec a /tmp scratch Python. Repointed to ~/.hermes/tools/python-3.14.7…; the gateway was not restarted. ⚠ Don't run git in ~/.hermes/hermes-agent: a status there looped on auto-gc.
[2026-10-07] albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed — the .browse/ view exists on the box only; the share test is owed. → persistent-memory.d/2026-10-07-albok-service-0-1-14.md
[2026-10-07] albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed — rollback to 0.1.12 needs the strip snippet first. → persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md
[2026-10-06] albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199, built from clean archives, pinned by digest, single-file upload (deploy-stack's --delete would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → stacks/albok-service/README.md
[2026-10-06] The LiteLLM gateway keeps no request/response content, and store_prompts_in_spend_logs: false is now a marked consumer guarantee (comfy-dev, arbo incognito). Proved with a canary plus positive controls; per-request no-log does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes input/incognito-* (dry-run proved with a canary and a control). 847424c, 0f6c7a0.
[2026-10-06] jekyll moved from gen-large to sec-reasoning (Prime) after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md
[2026-10-06] vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention. Content verified (main sha, 20 branches, 29 tags).
[2026-10-05] fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime). → persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md
[2026-10-05] Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115): its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.
[2026-10-05] nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only https://svos.phasefinal.com with HTTP Basic auth, login Vuong (Prime, direct) — WhoIs gate retired (all mesh nodes are login pfi; Prime declined a mesh restructure). Password vaulted nh3-svos/basic-auth-vuong. qemu-ga freeze hooks were never run on Debian; now configs/qemu-ga/fsfreeze-dispatch (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun svos-front-inventory after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → servers/nh3-svos/README.md.
[2026-10-05] ESH Homepage also on https://10.0.50.45 (Prime) — a Traefik file-provider router (homepage-https.yml, tls: {}) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (2c4c7ca)
[2026-10-05] bench-jev servers killed and removed, their HF cache cleaned (Prime); pfi-pteradactyl fixed + patched + engine upgraded (3429b5c; manual dockerd + wings restart).
[2026-10-05] fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE) — run 1 aborted on its own guard (live rsync walked into an overlay merged view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (-x, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → servers/fv-ml1/README.md
[2026-10-05] ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime) — with live-restore: true the engine upgrade kept containers up, but 6 that bind-mount docker.sock (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → servers/ana-docker/README.md
[2026-10-05] RMM wildcard cert now auto-renews (Prime) — *.phasefinal.com on pfi-tacticalrmm was certbot manual (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault pfi-tacticalrmm/cloudflare-certbot-dns01-token); reissued to 2027-01-03, dry-run OK. → servers/pfi-tacticalrmm/README.md
[2026-10-05] CrowdSec SQLite → WAL on ana-docker (0418) — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (use_wal: false, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → servers/ana-docker/README.md
[2026-10-05] Fleet guest patching (park 100) DONE 0355 — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → docs/runbooks/fleet-guest-patch-2026-10-05.md § Result — [2026-10-05 later] reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.
[2026-10-05] irv-ml1 storage cleanup + text encoders on worktank (Prime) — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); /storetank/arbo/models/text_encoders is now ZFS worktank/arbo-text-encoders mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → servers/irv-ml1/README.md
[2026-10-05] ana-docker Traefik readTimeout 60s → 10m on both entrypoints — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in stacks/. → servers/ana-docker/README.md
[2026-10-05] Embedder routing split (Prime 0030, via albok-dev, confirmed directly): qwen3-embedding = esh-ml1 primary / nh3-ml1 failover; new qwen3-embedding-bulk = nh3-ml1 primary / esh-ml1 failover, for albok writes — LiteLLM order (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → stacks/litellm/conf/config.yaml
[2026-10-04] albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime) — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns nh3-pve-2; copy verified by digest (dir st_size differs on ext4, not data). → persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md
[2026-10-04] The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md
[2026-10-04] pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + infra-ops-pages tokens — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md
[2026-10-04] sudo session recordings: archived weekly, bulk data never kept (Prime) — BULK_DATA !log_output on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → services/sudo-io-archive/README.md
[2026-10-04] VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use) — off, onboot 0, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
[2026-10-04] PARKED (Prime): move fv-ml1's Docker data-root to tank — needs a model-seat outage window; tracked at park move-fv-ml1-s-docker-storage-off-the-root-ssd (id 99). — [2026-10-05] DONE (entry above).
[2026-10-04] PARKED (Prime): guest OS patch campaign — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park guest-os-patch-campaign-across-the-fleet-debian (id 100). — [2026-10-05] DONE (entry above).
[2026-10-04] PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway — tracked at park give-svos-and-miranda-a-model-path-that-doesn-t (id 98).
[2026-10-03] PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host"): an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → services/pve-nag-patch/README.md
[2026-10-03] nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 nic3, AMT on nic0 with auto kept up, 1 TB wiped into vmstore (913 GiB), stale boot entry gone → persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md
[2026-10-03] CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232) → persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md
[2026-10-03] Worldtree memory-gate duty REFRAMED: regression check, not a gate → persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md
[2026-10-03] albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1 → persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md
[2026-10-02] esh-pve-2 (MS-03 at ESH, future esh-dev host) onboarded: infra-ops (Prime), 1 TB → LVM-thin vmstore, AMT phoning home as esh-pve-2-amt → persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md
[2026-10-02] Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684) → persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md
[2026-10-02] nh3-pve's AMT is in TacticalRMM's MeshCentral (group PFI-AMT, device nh3-pve-amt) → persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md
[2026-10-02] Prime: dev-backup gets dailies + weeklies — DONE → persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md
[2026-10-02] ESH tank (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629 → persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md
[2026-10-02] albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): http://albok.nh3.internal:8392 → persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md
[2026-10-02] nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03 → persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md
[2026-10-02] Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as nh3-pve-2 at NH3 (Prime, ~0910) → persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md
[2026-10-02] nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot) → persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md
[2026-10-01] MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev) → persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md
[2026-10-01] Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed → persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md
[2026-10-01] Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
[2026-10-01] ⚠ Gitea's [webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8 silently REJECTS headscale mesh IPs (100.64.0.0/10) → persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md
[2026-10-01] vh/arbo push webhook repointed from the retired wg0 lifeline 10.100.79.3:9009 to irv-ml1's mesh address 100.64.0.6:9009 → persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md
[2026-10-01] Prime: delete the bench leftovers, no upstream for Scriberr, push → persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md
[2026-10-01] irv-ml1 /storetank reclaim done: Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread 01M3TCSYRSFNAPA9BPQTMFQ6KJ).
[2026-09-30] Parakeet speech seat → parakeet-unified-en-0.6b under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md
[2026-09-30] Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30. → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
[2026-09-30] SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev /v1/systemone at 32k tokens on GPU 1, with a Triton warm-up cache volume. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md
[2026-09-30] Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
[2026-09-29] Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches). → persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md
[2026-09-28] Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors. → persistent-memory.d/2026-09-28-worldtree-u10-backfill.md
[2026-09-28] Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired. → persistent-memory.d/2026-09-28-bonsai-ternary-spike.md
[2026-09-28] blender-run gained --cpu (no GPU attached) and a fixed hostname fv-ml1-blender (draupnir). The design stage never renders, so it stays off GPU 3.
[2026-09-28] Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (--extensions). SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → stacks/blender/README.md § Extensions
[2026-09-27] hermes-gateway restarted 0401 for highseat-dev → persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md
[2026-09-27] SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime): wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md
[2026-09-27] Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port. → persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md (canonical: stacks/blender/README.md)
[2026-09-27] Worldtree reward path: d8726a13 + providers.yaml reward_models.scalar-judge live on demo; personal waited on a newer image. → persistent-memory.d/2026-09-27-worldtree-reward-path-live.md
[2026-09-27] esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August). → servers/esh-docker-vm/README.md
[2026-09-27] Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only. Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md (canonical: stacks/zigbee2mqtt/README.md)
[2026-09-27] semif-serve 0.1.4: object states ending in ), ; or } no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart. Prime ruled; heid bug hunt folded. → stacks/semif/README.md
[2026-09-27] SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win. Build nothing (Prime). Henge 88 carries it. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md
[2026-09-27] SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21. Recommendations await Prime. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md
[2026-09-27] SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels. Tracked in the in-flight SemIf section. → persistent-memory.d/2026-09-27-semif-order-averaging.md — DONE: 0.1.3 live, with fast kernels adopted (77b8cb4).
[2026-09-27] restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime). Rotation stays Prime's. → persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md
[2026-09-27] infra-ops bootstrapped on vm-esh-nas by Prime (the first account at the fleet-pinned uid/gid 850; bootstrap-infra-ops-user.yaml now pins 850 when free, d775a01).
[2026-09-27] augaman: second instance on fv-ml1 benched, then REMOVED (Prime). esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (ad484c3, bench docs/pfi/augaman-speed-bench/).
[2026-09-27] Created empty private repo corviduo/svoperatingsystem (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. svos-dev → the OS session; The High Seat's session is highseat-dev again.
[2026-09-26] augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy. → persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md — [2026-09-27] DONE: deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (d8f59a1).
[2026-09-26] zellij-fleet@.service installed, NOT enabled (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling @Claude at boot is Prime's call when seat_up ships. Tracked: 0ad7799, services/zellij-fleet/README.md.
[2026-09-26] btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority). Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: servers/nh3-pve/README.md.
[2026-09-25] AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326). — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md
[2026-09-26] esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only — , for ha-dev (operator-approved, relayed). → persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md
[2026-09-26] Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime). — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md
[2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): — LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and… → persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md
[2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).
[2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.
[2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md
[2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md
[2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md
[2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). → persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md
[2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction → persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md

240 older entries archived to archival-memory.md.

Tried and abandoned

[2026-10-08] NVIDIA's network debian12 repo on trixie via cuda-keyring — trixie's sqv rejects the key's SHA1 binding signature, so apt update errors until the list is gone (dpkg -r keeps the conffile; it took dpkg -P). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → servers/fv-ml1/README.md
[2026-10-08] secret get <name> | head "to see its shape" — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with secret list or wc -c, never by printing.
[2026-10-07] An idle gate counting with grep -c … || echo ERR — grep -c exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use || true and test the gate against a known-idle state first.
[2026-10-07] pgrep -f <pattern> to check that a process died — it matched the checking shell's own command line and reported a survivor; check with ps -eo … | grep '[x]…', as the same self-match bit pkill -f before.
[2026-10-06] Restarting a user-facing service after an idle check that only PRINTS — nvidia-smi … | wc -l said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (&& / exit). auto-memory feedback_idle_check_must_gate_the_restart.
[2026-10-06] Aging PBS snapshots by directory mtime — verify jobs and notes edits rewrite index.json.blob, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory reference_pbs_snapshot_mtime_moves_on_verify.
[2026-10-06] deploy-stack.sh on a stack whose host dir holds rollback copies (albok-service compose.yaml.pre-*) — its rsync --delete removes them. Upload the single changed file with elway instead.
[2026-10-05] Sizing a vLLM seat's KV from --gpu-memory-utilization on a shared card — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin --kv-cache-memory. auto-memory reference_vllm_util_kv_coboot_race.
[2026-10-05] Reading GPU headroom just after a restart — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory feedback_vllm_headroom_at_plateau_not_boot.
[2026-10-04] apt-get run in-band through qm guest exec — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a systemd-run unit. (auto-memory feedback_no_inband_apt_through_guest_agent)
[2026-10-04] Pulling files with ssh host 'sudo cat …' — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and log_output re-recorded the whole file as a new session. Read without sudo (group-readable file).
[2026-10-01] Size + mtime scans to pick reclaimable "staging" on a model store → persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md
[2026-09-30] Shorter Parakeet slices as Scriberr's memory fix → persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md
[2026-09-30] Whole-file local-attention Parakeet in Scriberr (context 255/255). OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
[2026-09-30] Start-time midpoint stitching of overlapped Parakeet chunks. It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
[2026-09-30] int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime. The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
[2026-09-30] GPU budgets computed as total − used. nvidia-smi Free is ~640 MiB lower per card (driver reserve). Budget from Free.
[2026-09-27] The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel → persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md
[2026-09-27] A TCP connect as the "is Blender ready" probe → persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md
[2026-09-27] log.exception() in a GPU failure path — the record keeps exc_info, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log traceback.format_exc() text instead (semif-serve engine._guard).
[2026-09-27] fla/triton in a slim image without gcc — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
[2026-09-27] uv export → requirements.txt → uv pip install for a torch-from-cu128 project → persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md
[2026-09-27] Translating a CUDA OOM by raising inside except (or from exc) — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, gc.collect() before empty_cache().
[2026-09-27] deploy-stack.sh --yes / dns-sync.py fed a blind y — the harness refuses a blind apply. Review with echo n | first, then apply with echo y |.
[2026-09-27] Relying on the build cache surviving on esh-ml1 → persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md

128 older entries archived to archival-memory.md.

64 KiB Raw Blame History Unescape Escape