Persistent memory — eshpfi-management

Last updated: 2026-10-09 ~1955 PT (nh3-svos: SVOS served on the LAN too, voice live, refreshed to ac42ace then kernel 0.0.8 — thaw race and the SQLite-connection fd outage both fixed; SVOS app-traffic ingress for U13 APPROVED, build deferred; fv-ml1 off the mesh by design; albok library building + token swap; Worldtree gate 10-09 filing FAIL under watch; Homepage descriptions.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. Every v2 command was DELETED, not deprecated — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`	per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16)
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched.

2026-10-09: live now (as of 2026-10-09 ~1955 PT)

Nothing is mid-action. No claims held (scripts/ops-log claims). eshpfi is pushed through this snapshot's commit. ⚠ The working tree AND index are shared with infra-hermes and subagents: commit with git commit -- <paths> (auto-memory feedback_shared_git_index_commit_pathspecs). graphify-out/GRAPH_REPORT.md stays modified and uncommitted on purpose (auto-regenerated).
U13 phase B (infra) DONE 2026-10-10 1646–1650 on Prime's go: nh3-svos on v0.0.9 (afab123, schema 12) with app-terminal + app-postoffice HELD on b588510 by drop-ins (remove at the edit; keep releases/b588510 until then); kernel app origin svos-kernel.phasefinal.com (LAN, nh3-dev only); svos-seathost + svos-highseat + svos-miranda origins on nh3-dev (nh3-svos only); apps env /opt/svos/current/apps-env (dc241f8: bridge, terminal, miranda 0.0.3, agent) on nh3-dev; three enrollment secrets vaulted, app-high-seat declared (B5). Next is the atomic edit (highseat-dev's runbook svos/docs/u13-cutover-runbook.md, Prime's go; E2 is S→I). → servers/nh3-svos/README.md, servers/nh3-dev/README.md
Owed by me (no input needed):
- ⚠ HOLD the destroy of zroot/ROOT/debian@pre-patch-20261008 on fv-ml1 (was ~10-15): fv-ml1 hard-hung 2026-10-11 ~0005, the first crash since that patch (NVIDIA 580.178.04, kernel 6.12.111). Keep the rollback point until the cause is known or a clean week passes after the last crash.
- nh3-svos housekeeping: releases 6c6abbe and a737a06 are no longer referenced by the front door and can be deleted (ac42ace holds the same static as the current b588510).
Worldtree memory gate watch CLOSED 10-10: batch 20261010T143011Z PASS (user median 0.83 over runs 0.83/0.75/0.92), so the 10-09 filing fail did not repeat and flash-next is not investigated. ⚠ worldtree-dev still is not reading mail (Prime's call, below).
Watching: jekyll on sec-reasoning (cyberprev, GPU 0) since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens. jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it, as gen-large did (preemptions; decode down to 2–16 tok/s). jekyll's context grows ~1k/call toward the 262k ceiling. It runs on the shared all-agents-local key; a key of its own would make its traffic attributable.
Waiting on Prime:
- albok .browse/ share: vh holds every albok read group since 10-09 2244 (Prime: "vh gets albok-everything"; four Mac keys authorized for vh). The macOS run of the withdrawal test is staged on nh3-albok and waits on Prime running the one-liner in scripts/oneoff/albok-browse-withdrawal/README.md; then read ~vh/albok-browse-test/results/, report to albok-dev, remove the replica.
- Who sees Worldtree gate failures while worldtree-dev is not reading mail (Prime, Miranda, or keep waiting).
- Rotate the fv-ml1 BMC password? It was printed into a local session transcript on 10-08 (never left nh3-dev); infra-ops leans no.
- albok's 200-object load check, 2.01× against a 2× bar (albok-dev: the ruling is the operator's).
- The cleanup bundle offered 10-05: close stale parks 49/46/40; refresh the fv-ml1 README stack tables plus the serial-keyed NVMe table and zpool replace line (the 2084-media-error drive is now /dev/nvme6, not nvme7); wire ZED pool alerts into althing-alert-bridge; prune the exited one-shot containers on fv-ml1 (keep the parakeet rollback).
- Push Booth commit ce095c1. A trusted HTTPS name for ESH Homepage. nh3-dev reboot (kernel 6.1.0-53 installed).
- ⚠ irv-ml1: DO NOT PATCH (Prime); config changes are fine. ana-nas is out of scope.
Scheduled / unchecked: 10-11 0540: first weekly sudo-io archiver run (check it). 10-11 0900: infra-hermes deletes the two one-off PBS VM vzdumps. Never zpool upgrade any pool yet. Next scheduled tank scrub on fv-ml1: Sun 10-11 0024 (the September slot was missed); check it ran.
Small clean-ups, ours, untracked: stale LiteLLM aliases erp-tune-v2 and gemma4-26b-a4b-it-base; ana-docker's Traefik (readTimeout 10m) and CrowdSec (use_wal, bouncer updateMaxFailure: 3) configs live only on the host, not in stacks/; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx .bak on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's patches/0003 lands only at the next scripts/scriberr-rebuild. The LiteLLM compose.yaml on ana-docker differs from stacks/litellm/compose.yaml (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
Open, Prime's calls (unchanged from 10-03): QNAP LCD visual check (services/qnap-lcd/); NH3 Mac mini (Mac-mini, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask openipmi.service on esh-nas-pve; run playbooks/pve-nag-patch.yaml on esh-pve-2 when it is plugged back in.
Older open items, one line each: esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (stacks/blender/README.md). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran 51168a2e on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.

fv-ml1 GPU layout (as of 2026-10-05 2249)

GPU	seat	max ctx	KV	KV tokens	concurrency
0	cyberprev (sec)	262,144	15 GiB	336,935	1.29×
0	erp-seat (char-rp-fast)	262,144	5.5 GiB	346,436	1.32×
0	voices	40,960	6 GiB (bf16)	43,680	1.07×
0	coder (coder-fast)	32,768	1.03 GiB	77,056	2.35×
0	parakeet-nemo	audio	rest 2,108 MiB, cap 3,840	—	—
1	gen-small (+summarizer/classifier)	262,144	5.5 GiB	461,215	1.76× (default output cap 32,768)
1	meromero-rp (char-rp)	262,144	22 GiB	346,436	1.32×
1	intern-decision	32,768/call	cap 14.4 GiB, peak 15,220	—	—
2	flash-next (gen, gen-large, +6)	262,144	—	344,155	1.31×

Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (--kv-cache-memory), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (scripts/oneoff/fv-ml1-gpucheck.sh), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory feedback_idle_check_must_gate_the_restart).

GPU 3: clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot scripts/mia-run, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to irv-ml1's A6000, not back to GPU 1.

Scriberr (fv-ml1 GPU 3)

LIVE scriberr:local-blackwell-a353078-dropout2: upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, PARAKEET_MODEL_PATH), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). patches/0003 (pyannote env pins) is committed but in no image yet. scripts/scriberr-rebuild re-applies all. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
Diarization works (fixed 2026-10-06): model caches on /tank, HF_TOKEN a placeholder, saved profiles WhisperX large-v3 · GPU · speakers and Parakeet · GPU · speakers. A NEW gated model needs one download with vault fv-ml1/hf-token. → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
URL is https://scriberr.nh3.phasefinal.com since 2026-10-01 (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup .bak-20261001-scriberr). getUserMedia is refused on http://, so recording needs the HTTPS name, which is in ALLOWED_ORIGINS (playbooks/scriberr-https-origin.yaml). ⚠ /opt/docker/compose/scriberr is lkraven-owned, so deploy-stack.sh as infra-ops FAILS there; upload single files with elway --upload --sudo --owner lkraven:lkraven.

Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)

Incident 2026-10-01 04:21, fixed by nemo-0.1.1 (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: empty_cache around every window, MEM_CAP_MIB=3840 (a hard ceiling), CUDA_GRAPHS=0 (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory. Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
Rollback: docker stop parakeet-nemo && docker start parakeet (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with --http h11 (httptools 0.8.0 emits HTTP/1.1 200\x00OK, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

Worldtree U11 memory cutover (demo + personal)

Legacy plane OFF since 0115/0120 PT 2026-09-30 (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted agents/{forseti,lofn,mimir}/memory/<agent>.chroma + memory/context_promotion on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day JSONL_RETENTION_DAYS sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag. scripts/wt-memory-gate-batch now passes --legacy-mode only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. Still mine after b193 deploys: remove the residue memory/context_promotion/ledger.db a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM hermes-gateway (never read); reply to infra-hermes.
Legacy archive: DESTROY it whole by 2026-10-30, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
TODO: re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").

intern-decision (replaced SemIf on 2026-09-30)

LIVE 0.1.3 at intern-decision.fv.internal:8033: semif-compatible /decide plus Jev /v1/systemone, 32k tokens, a Triton cache volume. Run scripts/intern-decision-warmup after an IMAGE change. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md
Open: label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim.

nh3-pve + nh3-ml1 (2026-09-25/26)

nh3-pve: Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. AMT live: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as nh3-pve/amt-admin. → servers/nh3-pve/README.md
nh3-ml1 (CT 109 @ 10.100.50.80): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (qwen3-embedding esh-ml1 first, qwen3-embedding-bulk for albok writes nh3-ml1 first). brokkr's foundry seats: lfm-vl :8030 (gateway lfm25-vl-3b), lfm-vl-uncensored :8032 (direct), vibevoice-asr :8031 (audio.cpp, Q8_0). GPU ~11.3/16 GB. → servers/nh3-ml1/README.md

Recent decisions

[2026-10-11] fv-ml1 hard hang ~0005, self-reset ~0008, back by 0015 (all seats). Journal ends mid-normal-operation (no shutdown), pstore empty, SOL shows nothing (no serial kernel console), BMC restart cause unknown, SEL only shows the 00:08 boot events (⚠ the BMC clock is 7 h behind, so ipmitool prints them as 17:08 the day before). Idle at the time (Beszel 0004: CPU 0.15%, no Scriberr job). The sp5100 TCO watchdog exists (60 s). infra-ops sent an ACPI soft-off at 00:12:14 believing it still hung; it landed in early boot before logind watched the button, so no effect. Open (Prime): capture evidence next time (serial console on the kernel cmdline for SOL, and/or kdump); the pre-patch snapshot is HELD.
[2026-10-10] Prime's U13 rulings (~1705): seat roles dev + ops; artifact host label KEPT nh3-svos (old inbox links keep working; infra-ops copies the WHOLE /var/lib/svos/apps/terminal (E2 step 2) and /var/lib/svos/apps/postoffice (step 6) to nh3-dev at the same paths, owner lkraven, modes kept, each after its nh3-svos unit stops and before the nh3-dev one starts: the artifacts.sqlite index and the bridge's journal must travel with the files); Matrix relay persona matrix-relay. A two-label hybrid is impossible: svos_connector.artifacts.relative() refuses any host but the app's one artifact_host. Relay secret nh3-svos/matrix-relay-enrollment-secret minted. svos-dev's payload: svoperatingsystem docs/operations/u13-edit-payload.md @ 7d7ed1f (enrollment sha256 placeholders: fill at E2 from secret get <item> | sha256 — never commit them).
[2026-10-10] U13 phase B built (Prime 1639: "implement as asked"): B1 kernel 0.0.9 + schema 12 (vzdump vm/111/2026-10-10T23:45:41Z + offline pre-afab123 copy first), B2 svos-kernel.phasefinal.com → --app-listen 127.0.0.1:8410 (allowlist nh3-dev; /auth/kernel-key ADDED to svos-dev's path list because every connector client fetches it), B3 seat-host origin + bridge env on nh3-dev, B4 app-high-seat origin; then svos-dev's dc241f8 amendment: full method-pinned path list, Miranda's app + agent env and svos-miranda origin on nh3-dev, three secrets minted to the vault, app-high-seat declared. Found + fixed: nh3-dev's fleet Caddy shared admin :2019 with ttyd-caddy (SO_REUSEPORT), now :2021. Kernel answers 401 no principal through the origin; other sources 403.
[2026-10-10] esh-nas-pve nvme/ROOT@pre-pve9 destroyed 0750 (607M) after a week clean on PVE 9.2.21 / 7.0.14-20 (up 6 d, no err-level journal in 7 d, pools healthy, all guests up; only openipmi.service failed, the known item). Left: nvme/ROOT/pve-1@pre-upgrade-20260818T141652Z (now 1.43G, a pre-PVE-9 rollback point that is useless as one) and nvme/subvol-105-disk-0@pre-vaapi-20260911.
[2026-10-09] vh gets every albok read group (Prime 2244: "vh gets albok-everything"): albok-read, albok-personal and albok-feedback (the restricted wing's default-deny ends for vh only), plus his four Macs' keys on vh (playbooks/nh3-albok-vh-viewer.yaml). No net new root path: those Macs already reach root there via lkraven@nh3-dev and the infra-ops key. Later restricted groups (1513+) are NOT included.
[2026-10-09] nh3-dev: orphaned svos pytest killed 2242 (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.
[2026-10-09] Post-clear checks closed (2015–2025 PT): albok .browse/ sshfs pathname-withdrawal test PASSED (replica on nh3-albok, Linux sshfs 3.7.3 client in a container; fresh open ENOENT 5/5 default and 5/5 hour-long caches; null + positive controls ok; failed rename = path served; reported to albok-dev) → scripts/oneoff/albok-browse-withdrawal/README.md. fv-ml1 plateau (Beszel 120m, 42 h after the 0034 10-08 restart): GPU 0 flat at 5,606 MiB free (~3.9 GB at parakeet's cap), GPU 1 median 9,459 free (~3.3 GB at intern-decision's 32k peak), no creep; GPU 2 643 MiB free = flash-next util 0.96, single tenant, by design. nh3-nas /volume1 71% (79% on 10-07). The "sfsrv-ana pre-pbs4" item was a mislabel: the pre-pbs4 snapshots were pfi-pve VM 100 and nh3-pve VM 105, dropped by infra-hermes 10-05 and read gone 10-09.
[2026-10-09] albok library building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev. → stacks/albok-service/README.md
[2026-10-09] Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".
[2026-10-09] APPROVED by Prime (10-09), BUILD DEFERRED until after his snapshot/clear: SVOS app-traffic ingress for U13 (apps on nh3-dev reach the nh3-svos kernel). Shape agreed with svos-dev: (1) kernel --app-listen 127.0.0.1:8410, which never trusts loopback (svos-dev builds it in U9a); (2) a Caddy origin svos-kernel.phasefinal.com on 10.100.50.43, no Basic auth, bearer passed, app paths only, allowlisted to nh3-dev; (3) kernel→bridge over plain LAN HTTP; kernel→seat host (terminal) via nh3-dev's wildcard Caddy svos-seathost.nh3.phasefinal.com (TLS, because the leg carries secrets). nh3-dev is NOT on the mesh; the contracts now say "fleet network". --app-listen is ON MAIN (68c9933, 10-09). It ships in kernel 0.0.9 with schema 11 → 12 (migration 0012 rebuilds dispatches + summons, so the 'schema 11' acceptance line changes and a pre-refresh backup is mandatory). At the cutover the bridge gets three owner-approved runtime grants (app-postoffice → app:app-miranda write, app:app-postoffice write, ns:pages write, all autonomous) and installs as svos-postoffice[push] with althing-core from ~/development/althing in apps-env. Release is cut: svoperatingsystem afab123 (tag v0.0.9): kernel 0.0.9, svos-postoffice 0.0.2, svos-terminal 0.0.3, svos-connector 0.0.7, svos-agent 0.0.3, svos-miranda 0.0.2 (U9a + U5b). The SVOS side of the cutover is written up in docs/operations/u13-svos-edit.md (app listener on nh3-svos, terminal app moved to nh3-dev with [seat_host], bridge on nh3-dev with the converse mapping). Gotchas: the terminal app serves WITHOUT its seat host if the socket or launch dir fails (one stderr line); serve refuses --app-listen on the primary port. Build only when the edit is scheduled (highseat-dev owns the schedule) and NOT before the snapshot; the runbook refresh is ours at deploy. Tracking: svos-dev thread 01M4GPXSBKQHCQ30FVPH4R3N7G.
[2026-10-09] nh3-svos → kernel 0.0.8 (b588510, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed. A 4-min browser soak held svos.db handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → servers/nh3-svos/README.md
[2026-10-08] nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the Type=notify unit as a pair); thaw race FIXED: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → servers/nh3-svos/README.md
[2026-10-08] SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door). /voice/* is routed BEFORE Basic auth, because the browser sends Authorization: Bearer <lease> there (web/src/lib/voice.ts) and Basic auth would reject it. All four app credentials are now vaulted as nh3-svos/svos-<app>-credential. → servers/nh3-svos/README.md
[2026-10-08] SVOS served on the LAN too (Prime): Caddy binds 10.100.50.43:443 as well as 100.64.0.9:443; the internal AdGuards answer svos/svos-terminal.phasefinal.com → 10.100.50.43 (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → servers/nh3-svos/README.md
[2026-10-08] fv-ml1 is OFF the mesh again, by design: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active.
[2026-10-08] heid's runaway sweep killed on Prime's order: two heidnotes/sweep.py processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66% — the sweep read its own output. heid fixed it at the source. → persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md
[2026-10-08] SVOS post office LIVE on nh3-svos — Prime declared svos; its ns:notifications write grant was approved on his word at 0731 (card card:4d3a8b26c17f472d828b06fd495192db); a test message was ingested and delivered; reach ok, unread 0. Reported to svos-dev.
[2026-10-07] No z.ai vision alias on the gateway (Prime). glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.
[2026-10-06] Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles. Patch 0003 waits for the next rebuild (tracked at stacks/scriberr/patches/README.md). → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
[2026-10-06] Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00. → persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md
[2026-10-08] Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md
[2026-10-07] nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps — acceptance 17/17 + Chromium; the thaw race. → persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md
[2026-10-07] Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime) — thinking: disabled is light thinking, not zero. → persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md
[2026-10-07] Hermes launchers repaired (Miranda relay 8462): svos-dev's scratch probe had rewritten ~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp} to exec a /tmp scratch Python. Repointed to ~/.hermes/tools/python-3.14.7…; the gateway was not restarted. ⚠ Don't run git in ~/.hermes/hermes-agent: a status there looped on auto-gc.
[2026-10-07] albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed — the .browse/ view exists on the box only; the share test is owed. → persistent-memory.d/2026-10-07-albok-service-0-1-14.md
[2026-10-07] albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed — rollback to 0.1.12 needs the strip snippet first. → persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md
[2026-10-06] albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199, built from clean archives, pinned by digest, single-file upload (deploy-stack's --delete would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → stacks/albok-service/README.md
[2026-10-06] The LiteLLM gateway keeps no request/response content, and store_prompts_in_spend_logs: false is now a marked consumer guarantee (comfy-dev, arbo incognito). Proved with a canary plus positive controls; per-request no-log does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes input/incognito-* (dry-run proved with a canary and a control). 847424c, 0f6c7a0.
[2026-10-06] jekyll moved from gen-large to sec-reasoning (Prime) after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md
[2026-10-06] vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention. Content verified (main sha, 20 branches, 29 tags).
[2026-10-05] fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime). → persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md
[2026-10-05] Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115): its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.
[2026-10-05] nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only https://svos.phasefinal.com with HTTP Basic auth, login Vuong (Prime, direct) — WhoIs gate retired (all mesh nodes are login pfi; Prime declined a mesh restructure). Password vaulted nh3-svos/basic-auth-vuong. qemu-ga freeze hooks were never run on Debian; now configs/qemu-ga/fsfreeze-dispatch (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun svos-front-inventory after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → servers/nh3-svos/README.md.
[2026-10-05] ESH Homepage also on https://10.0.50.45 (Prime) — a Traefik file-provider router (homepage-https.yml, tls: {}) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (2c4c7ca)
[2026-10-05] bench-jev servers killed and removed, their HF cache cleaned (Prime); pfi-pteradactyl fixed + patched + engine upgraded (3429b5c; manual dockerd + wings restart).
[2026-10-05] fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE) — run 1 aborted on its own guard (live rsync walked into an overlay merged view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (-x, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → servers/fv-ml1/README.md
[2026-10-05] ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime) — with live-restore: true the engine upgrade kept containers up, but 6 that bind-mount docker.sock (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → servers/ana-docker/README.md
[2026-10-05] RMM wildcard cert now auto-renews (Prime) — *.phasefinal.com on pfi-tacticalrmm was certbot manual (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault pfi-tacticalrmm/cloudflare-certbot-dns01-token); reissued to 2027-01-03, dry-run OK. → servers/pfi-tacticalrmm/README.md
[2026-10-05] CrowdSec SQLite → WAL on ana-docker (0418) — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (use_wal: false, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → servers/ana-docker/README.md
[2026-10-05] Fleet guest patching (park 100) DONE 0355 — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → docs/runbooks/fleet-guest-patch-2026-10-05.md § Result — [2026-10-05 later] reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.
[2026-10-05] irv-ml1 storage cleanup + text encoders on worktank (Prime) — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); /storetank/arbo/models/text_encoders is now ZFS worktank/arbo-text-encoders mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → servers/irv-ml1/README.md
[2026-10-05] ana-docker Traefik readTimeout 60s → 10m on both entrypoints — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in stacks/. → servers/ana-docker/README.md
[2026-10-05] Embedder routing split (Prime 0030, via albok-dev, confirmed directly): qwen3-embedding = esh-ml1 primary / nh3-ml1 failover; new qwen3-embedding-bulk = nh3-ml1 primary / esh-ml1 failover, for albok writes — LiteLLM order (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → stacks/litellm/conf/config.yaml
[2026-10-04] albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime) — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns nh3-pve-2; copy verified by digest (dir st_size differs on ext4, not data). → persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md
[2026-10-04] The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md
[2026-10-04] pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + infra-ops-pages tokens — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md
[2026-10-04] sudo session recordings: archived weekly, bulk data never kept (Prime) — BULK_DATA !log_output on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → services/sudo-io-archive/README.md
[2026-10-04] VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use) — off, onboot 0, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
[2026-10-04] PARKED (Prime): move fv-ml1's Docker data-root to tank — needs a model-seat outage window; tracked at park move-fv-ml1-s-docker-storage-off-the-root-ssd (id 99). — [2026-10-05] DONE (entry above).
[2026-10-04] PARKED (Prime): guest OS patch campaign — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park guest-os-patch-campaign-across-the-fleet-debian (id 100). — [2026-10-05] DONE (entry above).
[2026-10-04] PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway — tracked at park give-svos-and-miranda-a-model-path-that-doesn-t (id 98).
[2026-10-03] PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host"): an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → services/pve-nag-patch/README.md
[2026-10-03] nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 nic3, AMT on nic0 with auto kept up, 1 TB wiped into vmstore (913 GiB), stale boot entry gone → persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md
[2026-10-03] CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232) → persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md
[2026-10-03] Worldtree memory-gate duty REFRAMED: regression check, not a gate → persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md
[2026-10-03] albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1 → persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md
[2026-10-02] esh-pve-2 (MS-03 at ESH, future esh-dev host) onboarded: infra-ops (Prime), 1 TB → LVM-thin vmstore, AMT phoning home as esh-pve-2-amt → persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md
[2026-10-02] Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684) → persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md
[2026-10-02] nh3-pve's AMT is in TacticalRMM's MeshCentral (group PFI-AMT, device nh3-pve-amt) → persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md
[2026-10-02] Prime: dev-backup gets dailies + weeklies — DONE → persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md
[2026-10-02] ESH tank (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629 → persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md
[2026-10-02] albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): http://albok.nh3.internal:8392 → persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md
[2026-10-02] nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03 → persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md
[2026-10-02] Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as nh3-pve-2 at NH3 (Prime, ~0910) → persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md
[2026-10-02] nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot) → persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md
[2026-10-01] MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev) → persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md
[2026-10-01] Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed → persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md
[2026-10-01] Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
[2026-10-01] ⚠ Gitea's [webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8 silently REJECTS headscale mesh IPs (100.64.0.0/10) → persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md
[2026-10-01] vh/arbo push webhook repointed from the retired wg0 lifeline 10.100.79.3:9009 to irv-ml1's mesh address 100.64.0.6:9009 → persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md
[2026-10-01] Prime: delete the bench leftovers, no upstream for Scriberr, push → persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md
[2026-10-01] irv-ml1 /storetank reclaim done: Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread 01M3TCSYRSFNAPA9BPQTMFQ6KJ).
[2026-09-30] Parakeet speech seat → parakeet-unified-en-0.6b under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md
[2026-09-30] Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30. → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
[2026-09-30] SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev /v1/systemone at 32k tokens on GPU 1, with a Triton warm-up cache volume. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md
[2026-09-30] Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
[2026-09-29] Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches). → persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md
[2026-09-28] Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors. → persistent-memory.d/2026-09-28-worldtree-u10-backfill.md
[2026-09-28] Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired. → persistent-memory.d/2026-09-28-bonsai-ternary-spike.md
[2026-09-28] blender-run gained --cpu (no GPU attached) and a fixed hostname fv-ml1-blender (draupnir). The design stage never renders, so it stays off GPU 3.
[2026-09-28] Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (--extensions). SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → stacks/blender/README.md § Extensions
[2026-09-27] hermes-gateway restarted 0401 for highseat-dev → persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md
[2026-09-27] SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime): wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md
[2026-09-27] Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port. → persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md (canonical: stacks/blender/README.md)
[2026-09-27] Worldtree reward path: d8726a13 + providers.yaml reward_models.scalar-judge live on demo; personal waited on a newer image. → persistent-memory.d/2026-09-27-worldtree-reward-path-live.md
[2026-09-27] esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August). → servers/esh-docker-vm/README.md
[2026-09-27] Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only. Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md (canonical: stacks/zigbee2mqtt/README.md)
[2026-09-27] semif-serve 0.1.4: object states ending in ), ; or } no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart. Prime ruled; heid bug hunt folded. → stacks/semif/README.md
[2026-09-27] SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win. Build nothing (Prime). Henge 88 carries it. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md
[2026-09-27] SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21. Recommendations await Prime. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md
[2026-09-27] SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels. Tracked in the in-flight SemIf section. → persistent-memory.d/2026-09-27-semif-order-averaging.md — DONE: 0.1.3 live, with fast kernels adopted (77b8cb4).
[2026-09-27] restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime). Rotation stays Prime's. → persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md
[2026-09-27] infra-ops bootstrapped on vm-esh-nas by Prime (the first account at the fleet-pinned uid/gid 850; bootstrap-infra-ops-user.yaml now pins 850 when free, d775a01).
[2026-09-27] augaman: second instance on fv-ml1 benched, then REMOVED (Prime). esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (ad484c3, bench docs/pfi/augaman-speed-bench/).
[2026-09-27] Created empty private repo corviduo/svoperatingsystem (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. svos-dev → the OS session; The High Seat's session is highseat-dev again.
[2026-09-26] augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy. → persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md — [2026-09-27] DONE: deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (d8f59a1).
[2026-09-26] zellij-fleet@.service installed, NOT enabled (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling @Claude at boot is Prime's call when seat_up ships. Tracked: 0ad7799, services/zellij-fleet/README.md.
[2026-09-26] btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority). Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: servers/nh3-pve/README.md.
[2026-09-26] GPU-LXC Temperature alerts watch the GPU, not the host CPU (SENSORS=-coretemp_*,acpitz); hypervisor CPU alerts at >95 °C on nh3-pve/esh-pve. The first nh3-ml1 alert was nh3-pve's CPU during vzdump. 6d901c0.
[2026-09-26] Abliterated LFM2.5-VL-3B parallel seat (:8032) passed brokkr's 6-image eval and stays (direct only; stock seat kept for SFW A/B). 9cc3824, 944bb36.
[2026-09-25] AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326). — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md
[2026-09-26] VibeVoice ASR → Q8_0 (Prime). WER on the 4 bundled LibriSpeech clips 3/69 → 2/69 (only the I'm/I am artifact left), ×3 identical; RTF 0.09–0.17; +1.1 GB VRAM (nh3-ml1 ~11.3/16 GB). Q4_K file removed.
[2026-09-26] esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only — , for ha-dev (operator-approved, relayed). → persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md
[2026-09-26] Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime). — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md
[2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): — LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and… → persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md
[2026-09-26] Coder seat STAYS on fv-ml1 (Prime). — The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5×… → persistent-memory.d/2026-09-26-coder-seat-stays-on-fv-ml1-prime.md
[2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).
[2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.
[2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md
[2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md
[2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md
[2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md
[2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). → persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md
[2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction → persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md

236 older entries archived to archival-memory.md.

Tried and abandoned

[2026-10-08] NVIDIA's network debian12 repo on trixie via cuda-keyring — trixie's sqv rejects the key's SHA1 binding signature, so apt update errors until the list is gone (dpkg -r keeps the conffile; it took dpkg -P). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → servers/fv-ml1/README.md
[2026-10-08] secret get <name> | head "to see its shape" — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with secret list or wc -c, never by printing.
[2026-10-07] An idle gate counting with grep -c … || echo ERR — grep -c exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use || true and test the gate against a known-idle state first.
[2026-10-07] pgrep -f <pattern> to check that a process died — it matched the checking shell's own command line and reported a survivor; check with ps -eo … | grep '[x]…', as the same self-match bit pkill -f before.
[2026-10-06] Restarting a user-facing service after an idle check that only PRINTS — nvidia-smi … | wc -l said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (&& / exit). auto-memory feedback_idle_check_must_gate_the_restart.
[2026-10-06] Aging PBS snapshots by directory mtime — verify jobs and notes edits rewrite index.json.blob, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory reference_pbs_snapshot_mtime_moves_on_verify.
[2026-10-06] deploy-stack.sh on a stack whose host dir holds rollback copies (albok-service compose.yaml.pre-*) — its rsync --delete removes them. Upload the single changed file with elway instead.
[2026-10-05] Sizing a vLLM seat's KV from --gpu-memory-utilization on a shared card — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin --kv-cache-memory. auto-memory reference_vllm_util_kv_coboot_race.
[2026-10-05] Reading GPU headroom just after a restart — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory feedback_vllm_headroom_at_plateau_not_boot.
[2026-10-04] apt-get run in-band through qm guest exec — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a systemd-run unit. (auto-memory feedback_no_inband_apt_through_guest_agent)
[2026-10-04] Pulling files with ssh host 'sudo cat …' — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and log_output re-recorded the whole file as a new session. Read without sudo (group-readable file).
[2026-10-01] Size + mtime scans to pick reclaimable "staging" on a model store → persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md
[2026-09-30] Shorter Parakeet slices as Scriberr's memory fix → persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md
[2026-09-30] Whole-file local-attention Parakeet in Scriberr (context 255/255). OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
[2026-09-30] Start-time midpoint stitching of overlapped Parakeet chunks. It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
[2026-09-30] int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime. The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
[2026-09-30] GPU budgets computed as total − used. nvidia-smi Free is ~640 MiB lower per card (driver reserve). Budget from Free.
[2026-09-27] The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel → persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md
[2026-09-27] A TCP connect as the "is Blender ready" probe → persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md
[2026-09-27] log.exception() in a GPU failure path — the record keeps exc_info, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log traceback.format_exc() text instead (semif-serve engine._guard).
[2026-09-27] fla/triton in a slim image without gcc — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
[2026-09-27] uv export → requirements.txt → uv pip install for a torch-from-cu128 project → persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md
[2026-09-27] Translating a CUDA OOM by raising inside except (or from exc) — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, gc.collect() before empty_cache().
[2026-09-27] deploy-stack.sh --yes / dns-sync.py fed a blind y — the harness refuses a blind apply. Review with echo n | first, then apply with echo y |.
[2026-09-27] Relying on the build cache surviving on esh-ml1 → persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md
[2026-09-26] Greedy exact-match as parity for a small generative seat (coder) — same-host repeats matched only 30–38% (run B hit the prefix cache, so a different numeric path; near-tie… → persistent-memory.d/2026-09-26-greedy-exact-match-as-parity-for-a-small-generative-seat.md
[2026-09-26] zsh traps in ad-hoc test loops: set -- $a does NOT word-split, so alert POSTs went out with empty fields; and local path=… inside a function CLOBBERS $PATH (zsh's tied array), so the failover test ran nothing while TEI sat stopped. Use other names and ${=var}.

126 older entries archived to archival-memory.md.

64 KiB Raw Blame History Unescape Escape