vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ad79071d20346373f2217ecaec18d7a2c4144c05
esh-pfi-infrastructure/persistent-memory.md
T

58 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-10-11 ~1400 PT (U13 cutover executed + acceptance; fv-ml1 hard hang and evidence capture staged; esh-nas-pve stale snapshots destroyed; vh albok-everything; eshpfi pushed.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (e6da607) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at http://10.100.50.40:8390 is the only stateful component; althing-po-herald one per box; althing-listen one per session; postbox is the client. Every v2 command was DELETED, not deprecated — althing-cli→postbox, althing-wake-listener→althing-listen, althing-light-monitor/althing-receiver gone. Sessions need BOTH ALTHING_POST_OFFICE and ALTHING_HANDLE; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → persistent-memory.d/2026-08-28-althing-v3-cutover.md per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at /opt/uv-tools, needs its own wheel install (playbooks/nh3-extdev-althing-v3.yaml)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
vh/zonos-gateway OpenAI-compatible TTS gateway over stock ZONOS2 (:8890 irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM ext-tts alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (resolve_preset(name,voice); angry/happy/startled_happy per-voice). 8 voices incl. 4 clones pushed to gitea (main 8f1885b/v0.2.1); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec docs/EMOTION-DIALS-SPEC.md; host-managed voices bind-mount (./voices:/app/voices, drop wav + restart, no rebuild)
vh/soong-lab Noonien Soong character-design studio (SPA + /api + WT /bifrost/tool-call); containerized 2026-07-18, LIVE on corviduo-dev :8443 (image vh/soong-lab:latest). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as deploy, compose pull && up -d from /opt/soong-lab, health-gated on /api/version). Manual redeploy sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'. → archival-memory.md (archived 2026-08-16)
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched.

2026-10-11: live now (as of 2026-10-11 ~1400 PT)

  • Nothing is mid-action. No claims held. eshpfi is pushed through df7161b; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with git commit -- <paths>. graphify-out/GRAPH_REPORT.md stays modified on purpose.
  • U13 cutover: executed 10-11 0721–0816, acceptance 1350. Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin svos, and the bridge app-postoffice on nh3-dev owning the miranda push route. The URGENT bus test (thread 01M4PBNX5PP0PGHDTY5BG7HG95) ingested, paged per the connector receipt, and a reply came back on the thread from svos. → persistent-memory.d/2026-10-11-u13-cutover-executed.md
    • Acceptance closed 1354: Miranda's reply arrived on the thread through the bridge's own send. ⚠ CORRECTION (1412): card 05cf119e… was NOT approved (I had inferred it from the reply arriving); it was her own carded send, is moot, and Prime can deny it. Answered by svos-dev 1400: the thread text was Miranda answering bus mail with her own send (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in svos_miranda.turns) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, /opt/svos/current → releases/adc862e); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release 34476a7 (interface UI 0.0.9, tree strands follow speech) at 1712: /opt/svos/current → releases/34476a7 on nh3-svos, no restart, previous handle = adc862e; then ad255a2 (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7. Static-only releases: build the full release dir (carried artifacts + new static/), switch current without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from releases/adc862e (verified via /proc/<pid>/maps): do not delete it before they restart. The non-autonomous miranda → app:app-postoffice write grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s synthesis_failed). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos docs/u13-cutover-runbook.md).
    • Rollback material, keep until the soak ends: nh3-svos releases/b588510, instance.toml.pre-u13-{A,B}, /etc/svos/kernel.env.pre-u13 (absent: there was none); nh3-dev ~/.hermes/config.yaml.pre-u13 and ~/.hermes/plugins/svos_miranda. After the soak: nh3-svos releases 6c6abbe, a737a06, b588510 and nh3-dev /opt/svos/releases/afab123 (unreferenced) can go.
  • fv-ml1: hard hang 10-11 ~0005, self-reset, no record (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit fv-ml1-sol-logger (journalctl --user -u fv-ml1-sol-logger). ⚠ HOLD zroot/ROOT/debian@pre-patch-20261008 (do not destroy) until the cause is known or a clean week passes after the last crash.
  • albok .browse/ share: the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in ~vh/albok-browse-test/ on nh3-albok, waits on Prime pasting the one-liner (scripts/oneoff/albok-browse-withdrawal/README.md); afterwards read results/, report to albok-dev, remove the replica.
  • Watching: jekyll on sec-reasoning (cyberprev, GPU 0) since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
  • Waiting on Prime: who sees Worldtree gate failures while worldtree-dev is not reading mail; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth ce095c1; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ irv-ml1: DO NOT PATCH (Prime); config changes are fine. ana-nas is out of scope.
  • Small clean-ups, ours, untracked: stale LiteLLM aliases erp-tune-v2 and gemma4-26b-a4b-it-base; ana-docker's Traefik (readTimeout 10m) and CrowdSec (use_wal, bouncer updateMaxFailure: 3) configs live only on the host, not in stacks/; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx .bak on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's patches/0003 lands only at the next scripts/scriberr-rebuild. The LiteLLM compose.yaml on ana-docker differs from stacks/litellm/compose.yaml (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
  • Open, Prime's calls (unchanged from 10-03): QNAP LCD visual check (services/qnap-lcd/); NH3 Mac mini (Mac-mini, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask openipmi.service on esh-nas-pve; run playbooks/pve-nag-patch.yaml on esh-pve-2 when it is plugged back in.
  • Older open items, one line each: esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (stacks/blender/README.md). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran 51168a2e on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.

fv-ml1 GPU layout (as of 2026-10-05 2249)

GPU seat max ctx KV KV tokens concurrency
0 cyberprev (sec) 262,144 15 GiB 336,935 1.29×
0 erp-seat (char-rp-fast) 262,144 5.5 GiB 346,436 1.32×
0 voices 40,960 6 GiB (bf16) 43,680 1.07×
0 coder (coder-fast) 32,768 1.03 GiB 77,056 2.35×
0 parakeet-nemo audio rest 2,108 MiB, cap 3,840 — —
1 gen-small (+summarizer/classifier) 262,144 5.5 GiB 461,215 1.76× (default output cap 32,768)
1 meromero-rp (char-rp) 262,144 22 GiB 346,436 1.32×
1 intern-decision 32,768/call cap 14.4 GiB, peak 15,220 — —
2 flash-next (gen, gen-large, +6) 262,144 — 344,155 1.31×

Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (--kv-cache-memory), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (scripts/oneoff/fv-ml1-gpucheck.sh), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory feedback_idle_check_must_gate_the_restart).

  • GPU 3: clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot scripts/mia-run, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to irv-ml1's A6000, not back to GPU 1.

Scriberr (fv-ml1 GPU 3)

  • LIVE scriberr:local-blackwell-a353078-dropout2: upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, PARAKEET_MODEL_PATH), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). patches/0003 (pyannote env pins) is committed but in no image yet. scripts/scriberr-rebuild re-applies all. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
  • Diarization works (fixed 2026-10-06): model caches on /tank, HF_TOKEN a placeholder, saved profiles WhisperX large-v3 · GPU · speakers and Parakeet · GPU · speakers. A NEW gated model needs one download with vault fv-ml1/hf-token. → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
  • URL is https://scriberr.nh3.phasefinal.com since 2026-10-01 (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup .bak-20261001-scriberr). getUserMedia is refused on http://, so recording needs the HTTPS name, which is in ALLOWED_ORIGINS (playbooks/scriberr-https-origin.yaml). ⚠ /opt/docker/compose/scriberr is lkraven-owned, so deploy-stack.sh as infra-ops FAILS there; upload single files with elway --upload --sudo --owner lkraven:lkraven.

Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)

  • Incident 2026-10-01 04:21, fixed by nemo-0.1.1 (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: empty_cache around every window, MEM_CAP_MIB=3840 (a hard ceiling), CUDA_GRAPHS=0 (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory. Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
  • Rollback: docker stop parakeet-nemo && docker start parakeet (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with --http h11 (httptools 0.8.0 emits HTTP/1.1 200\x00OK, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

Worldtree U11 memory cutover (demo + personal)

  • Legacy plane OFF since 0115/0120 PT 2026-09-30 (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
  • U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted agents/{forseti,lofn,mimir}/memory/<agent>.chroma + memory/context_promotion on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day JSONL_RETENTION_DAYS sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
  • b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag. scripts/wt-memory-gate-batch now passes --legacy-mode only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. Still mine after b193 deploys: remove the residue memory/context_promotion/ledger.db a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM hermes-gateway (never read); reply to infra-hermes.
  • Legacy archive: DESTROY it whole by 2026-10-30, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
  • TODO: re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").

intern-decision (replaced SemIf on 2026-09-30)

  • LIVE 0.1.3 at intern-decision.fv.internal:8033: semif-compatible /decide plus Jev /v1/systemone, 32k tokens, a Triton cache volume. Run scripts/intern-decision-warmup after an IMAGE change. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md
  • Open: label ~50 real Wyrd/Cicada turns before trusting it in production; its card makes no contamination claim.

nh3-pve + nh3-ml1 (2026-09-25/26)

  • nh3-pve: Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. AMT live: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as nh3-pve/amt-admin. → servers/nh3-pve/README.md
  • nh3-ml1 (CT 109 @ 10.100.50.80): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (qwen3-embedding esh-ml1 first, qwen3-embedding-bulk for albok writes nh3-ml1 first). brokkr's foundry seats: lfm-vl :8030 (gateway lfm25-vl-3b), lfm-vl-uncensored :8032 (direct), vibevoice-asr :8031 (audio.cpp, Q8_0). GPU ~11.3/16 GB. → servers/nh3-ml1/README.md

Recent decisions

  • [2026-10-11] U13 cutover executed on Prime's go (0721) + his four rulings — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → persistent-memory.d/2026-10-11-u13-cutover-executed.md

  • [2026-10-11] fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD — and infra-ops' soft-off that landed mid-boot. → persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md

  • [2026-10-10] Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay matrix-relay. → persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md

  • [2026-10-10] U13 phase B built (B1–B6) — kernel 0.0.9/schema 12, svos-kernel.phasefinal.com (nh3-dev only, method-pinned paths), nh3-dev origins + apps env, enrollment secrets. → persistent-memory.d/2026-10-10-u13-phase-b-built.md

  • [2026-10-10] esh-nas-pve nvme/ROOT@pre-pve9 destroyed after a clean week (the two older snapshots followed on 10-11, Prime). → persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md

  • [2026-10-09] vh gets every albok read group + four Mac keys (Prime: "albok-everything") — no net new root path. → persistent-memory.d/2026-10-09-vh-albok-everything.md

  • [2026-10-09] nh3-dev: orphaned svos pytest killed 2242 (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.

  • [2026-10-09] albok .browse/ sshfs withdrawal test PASSED (Linux) + post-clear checks closed (fv-ml1 plateau, nh3-nas 71%, pre-pbs4 mislabel). → persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md

  • [2026-10-09] albok library building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev. → stacks/albok-service/README.md

  • [2026-10-09] Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".

  • [2026-10-09] SVOS app-traffic ingress for U13 approved (shape agreed with svos-dev) — EXECUTED 10-10 (phase B) and 10-11 (the edit). → persistent-memory.d/2026-10-09-u13-ingress-approved.md

  • [2026-10-09] nh3-svos → kernel 0.0.8 (b588510, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed. A 4-min browser soak held svos.db handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → servers/nh3-svos/README.md

  • [2026-10-08] nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the Type=notify unit as a pair); thaw race FIXED: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → servers/nh3-svos/README.md

  • [2026-10-08] SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door). /voice/* is routed BEFORE Basic auth, because the browser sends Authorization: Bearer <lease> there (web/src/lib/voice.ts) and Basic auth would reject it. All four app credentials are now vaulted as nh3-svos/svos-<app>-credential. → servers/nh3-svos/README.md

  • [2026-10-08] SVOS served on the LAN too (Prime): Caddy binds 10.100.50.43:443 as well as 100.64.0.9:443; the internal AdGuards answer svos/svos-terminal.phasefinal.com → 10.100.50.43 (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → servers/nh3-svos/README.md

  • [2026-10-08] fv-ml1 is OFF the mesh again, by design: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active.

  • [2026-10-08] heid's runaway sweep killed on Prime's order: two heidnotes/sweep.py processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66% — the sweep read its own output. heid fixed it at the source. → persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md

  • [2026-10-08] SVOS post office LIVE on nh3-svos — Prime declared svos; its ns:notifications write grant was approved on his word at 0731 (card card:4d3a8b26c17f472d828b06fd495192db); a test message was ingested and delivered; reach ok, unread 0. Reported to svos-dev.

  • [2026-10-07] No z.ai vision alias on the gateway (Prime). glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.

  • [2026-10-06] Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles. Patch 0003 waits for the next rebuild (tracked at stacks/scriberr/patches/README.md). → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md

  • [2026-10-06] Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00. → persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md

  • [2026-10-08] Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md

  • [2026-10-07] nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps — acceptance 17/17 + Chromium; the thaw race. → persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md

  • [2026-10-07] Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime) — thinking: disabled is light thinking, not zero. → persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md

  • [2026-10-07] Hermes launchers repaired (Miranda relay 8462): svos-dev's scratch probe had rewritten ~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp} to exec a /tmp scratch Python. Repointed to ~/.hermes/tools/python-3.14.7…; the gateway was not restarted. ⚠ Don't run git in ~/.hermes/hermes-agent: a status there looped on auto-gc.

  • [2026-10-07] albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed — the .browse/ view exists on the box only; the share test is owed. → persistent-memory.d/2026-10-07-albok-service-0-1-14.md

  • [2026-10-07] albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed — rollback to 0.1.12 needs the strip snippet first. → persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md

  • [2026-10-06] albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199, built from clean archives, pinned by digest, single-file upload (deploy-stack's --delete would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → stacks/albok-service/README.md

  • [2026-10-06] The LiteLLM gateway keeps no request/response content, and store_prompts_in_spend_logs: false is now a marked consumer guarantee (comfy-dev, arbo incognito). Proved with a canary plus positive controls; per-request no-log does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes input/incognito-* (dry-run proved with a canary and a control). 847424c, 0f6c7a0.

  • [2026-10-06] jekyll moved from gen-large to sec-reasoning (Prime) after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md

  • [2026-10-06] vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention. Content verified (main sha, 20 branches, 29 tags).

  • [2026-10-05] fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime). → persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md

  • [2026-10-05] Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115): its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.

  • [2026-10-05] nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only https://svos.phasefinal.com with HTTP Basic auth, login Vuong (Prime, direct) — WhoIs gate retired (all mesh nodes are login pfi; Prime declined a mesh restructure). Password vaulted nh3-svos/basic-auth-vuong. qemu-ga freeze hooks were never run on Debian; now configs/qemu-ga/fsfreeze-dispatch (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun svos-front-inventory after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → servers/nh3-svos/README.md.

  • [2026-10-05] ESH Homepage also on https://10.0.50.45 (Prime) — a Traefik file-provider router (homepage-https.yml, tls: {}) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (2c4c7ca)

  • [2026-10-05] bench-jev servers killed and removed, their HF cache cleaned (Prime); pfi-pteradactyl fixed + patched + engine upgraded (3429b5c; manual dockerd + wings restart).

  • [2026-10-05] fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE) — run 1 aborted on its own guard (live rsync walked into an overlay merged view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (-x, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → servers/fv-ml1/README.md

  • [2026-10-05] ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime) — with live-restore: true the engine upgrade kept containers up, but 6 that bind-mount docker.sock (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → servers/ana-docker/README.md

  • [2026-10-05] RMM wildcard cert now auto-renews (Prime) — *.phasefinal.com on pfi-tacticalrmm was certbot manual (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault pfi-tacticalrmm/cloudflare-certbot-dns01-token); reissued to 2027-01-03, dry-run OK. → servers/pfi-tacticalrmm/README.md

  • [2026-10-05] CrowdSec SQLite → WAL on ana-docker (0418) — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (use_wal: false, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → servers/ana-docker/README.md

  • [2026-10-05] Fleet guest patching (park 100) DONE 0355 — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → docs/runbooks/fleet-guest-patch-2026-10-05.md § Result — [2026-10-05 later] reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.

  • [2026-10-05] irv-ml1 storage cleanup + text encoders on worktank (Prime) — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); /storetank/arbo/models/text_encoders is now ZFS worktank/arbo-text-encoders mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → servers/irv-ml1/README.md

  • [2026-10-05] ana-docker Traefik readTimeout 60s → 10m on both entrypoints — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in stacks/. → servers/ana-docker/README.md

  • [2026-10-05] Embedder routing split (Prime 0030, via albok-dev, confirmed directly): qwen3-embedding = esh-ml1 primary / nh3-ml1 failover; new qwen3-embedding-bulk = nh3-ml1 primary / esh-ml1 failover, for albok writes — LiteLLM order (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → stacks/litellm/conf/config.yaml

  • [2026-10-04] albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime) — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns nh3-pve-2; copy verified by digest (dir st_size differs on ext4, not data). → persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md

  • [2026-10-04] The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md

  • [2026-10-04] pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + infra-ops-pages tokens — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md

  • [2026-10-04] sudo session recordings: archived weekly, bulk data never kept (Prime) — BULK_DATA !log_output on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → services/sudo-io-archive/README.md

  • [2026-10-04] VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use) — off, onboot 0, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.

  • [2026-10-04] PARKED (Prime): move fv-ml1's Docker data-root to tank — needs a model-seat outage window; tracked at park move-fv-ml1-s-docker-storage-off-the-root-ssd (id 99). — [2026-10-05] DONE (entry above).

  • [2026-10-04] PARKED (Prime): guest OS patch campaign — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park guest-os-patch-campaign-across-the-fleet-debian (id 100). — [2026-10-05] DONE (entry above).

  • [2026-10-04] PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway — tracked at park give-svos-and-miranda-a-model-path-that-doesn-t (id 98).

  • [2026-10-03] PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host"): an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → services/pve-nag-patch/README.md

  • [2026-10-03] nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 nic3, AMT on nic0 with auto kept up, 1 TB wiped into vmstore (913 GiB), stale boot entry gone → persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md

  • [2026-10-03] CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232) → persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md

  • [2026-10-03] Worldtree memory-gate duty REFRAMED: regression check, not a gate → persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md

  • [2026-10-03] albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1 → persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md

  • [2026-10-02] esh-pve-2 (MS-03 at ESH, future esh-dev host) onboarded: infra-ops (Prime), 1 TB → LVM-thin vmstore, AMT phoning home as esh-pve-2-amt → persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md

  • [2026-10-02] Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684) → persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md

  • [2026-10-02] nh3-pve's AMT is in TacticalRMM's MeshCentral (group PFI-AMT, device nh3-pve-amt) → persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md

  • [2026-10-02] Prime: dev-backup gets dailies + weeklies — DONE → persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md

  • [2026-10-02] ESH tank (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629 → persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md

  • [2026-10-02] albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): http://albok.nh3.internal:8392 → persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md

  • [2026-10-02] nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03 → persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md

  • [2026-10-02] Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as nh3-pve-2 at NH3 (Prime, ~0910) → persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md

  • [2026-10-02] nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot) → persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md

  • [2026-10-01] MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev) → persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md

  • [2026-10-01] Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed → persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md

  • [2026-10-01] Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.

  • [2026-10-01] ⚠ Gitea's [webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8 silently REJECTS headscale mesh IPs (100.64.0.0/10) → persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md

  • [2026-10-01] vh/arbo push webhook repointed from the retired wg0 lifeline 10.100.79.3:9009 to irv-ml1's mesh address 100.64.0.6:9009 → persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md

  • [2026-10-01] Prime: delete the bench leftovers, no upstream for Scriberr, push → persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md

  • [2026-10-01] irv-ml1 /storetank reclaim done: Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread 01M3TCSYRSFNAPA9BPQTMFQ6KJ).

  • [2026-09-30] Parakeet speech seat → parakeet-unified-en-0.6b under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

  • [2026-09-30] Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30. → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md

  • [2026-09-30] SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev /v1/systemone at 32k tokens on GPU 1, with a Triton warm-up cache volume. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md

  • [2026-09-30] Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md

  • [2026-09-29] Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches). → persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md

  • [2026-09-28] Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors. → persistent-memory.d/2026-09-28-worldtree-u10-backfill.md

  • [2026-09-28] Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired. → persistent-memory.d/2026-09-28-bonsai-ternary-spike.md

  • [2026-09-28] blender-run gained --cpu (no GPU attached) and a fixed hostname fv-ml1-blender (draupnir). The design stage never renders, so it stays off GPU 3.

  • [2026-09-28] Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (--extensions). SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → stacks/blender/README.md § Extensions

  • [2026-09-27] hermes-gateway restarted 0401 for highseat-dev → persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md

  • [2026-09-27] SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime): wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md

  • [2026-09-27] Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port. → persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md (canonical: stacks/blender/README.md)

  • [2026-09-27] Worldtree reward path: d8726a13 + providers.yaml reward_models.scalar-judge live on demo; personal waited on a newer image. → persistent-memory.d/2026-09-27-worldtree-reward-path-live.md

  • [2026-09-27] esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August). → servers/esh-docker-vm/README.md

  • [2026-09-27] Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only. Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md (canonical: stacks/zigbee2mqtt/README.md)

  • [2026-09-27] semif-serve 0.1.4: object states ending in ), ; or } no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart. Prime ruled; heid bug hunt folded. → stacks/semif/README.md

  • [2026-09-27] SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win. Build nothing (Prime). Henge 88 carries it. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md

  • [2026-09-27] SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21. Recommendations await Prime. → persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md

  • [2026-09-27] SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels. Tracked in the in-flight SemIf section. → persistent-memory.d/2026-09-27-semif-order-averaging.md — DONE: 0.1.3 live, with fast kernels adopted (77b8cb4).

  • [2026-09-27] restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime). Rotation stays Prime's. → persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md

  • [2026-09-27] infra-ops bootstrapped on vm-esh-nas by Prime (the first account at the fleet-pinned uid/gid 850; bootstrap-infra-ops-user.yaml now pins 850 when free, d775a01).

  • [2026-09-27] augaman: second instance on fv-ml1 benched, then REMOVED (Prime). esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (ad484c3, bench docs/pfi/augaman-speed-bench/).

  • [2026-09-27] Created empty private repo corviduo/svoperatingsystem (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. svos-dev → the OS session; The High Seat's session is highseat-dev again.

  • [2026-09-26] augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy. → persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md — [2026-09-27] DONE: deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (d8f59a1).

  • [2026-09-26] zellij-fleet@.service installed, NOT enabled (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling @Claude at boot is Prime's call when seat_up ships. Tracked: 0ad7799, services/zellij-fleet/README.md.

  • [2026-09-26] btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority). Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: servers/nh3-pve/README.md.

  • [2026-09-25] AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326). — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md

  • [2026-09-26] esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only — , for ha-dev (operator-approved, relayed). → persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md

  • [2026-09-26] Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime). — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md

  • [2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): — LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and… → persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md

  • [2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).

  • [2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.

  • [2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.

  • [2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md

  • [2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md

  • [2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md

  • [2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md

  • [2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). → persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md

  • [2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction → persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md

240 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-10-08] NVIDIA's network debian12 repo on trixie via cuda-keyring — trixie's sqv rejects the key's SHA1 binding signature, so apt update errors until the list is gone (dpkg -r keeps the conffile; it took dpkg -P). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → servers/fv-ml1/README.md
  • [2026-10-08] secret get <name> | head "to see its shape" — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with secret list or wc -c, never by printing.
  • [2026-10-07] An idle gate counting with grep -c … || echo ERR — grep -c exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use || true and test the gate against a known-idle state first.
  • [2026-10-07] pgrep -f <pattern> to check that a process died — it matched the checking shell's own command line and reported a survivor; check with ps -eo … | grep '[x]…', as the same self-match bit pkill -f before.
  • [2026-10-06] Restarting a user-facing service after an idle check that only PRINTS — nvidia-smi … | wc -l said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (&& / exit). auto-memory feedback_idle_check_must_gate_the_restart.
  • [2026-10-06] Aging PBS snapshots by directory mtime — verify jobs and notes edits rewrite index.json.blob, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory reference_pbs_snapshot_mtime_moves_on_verify.
  • [2026-10-06] deploy-stack.sh on a stack whose host dir holds rollback copies (albok-service compose.yaml.pre-*) — its rsync --delete removes them. Upload the single changed file with elway instead.
  • [2026-10-05] Sizing a vLLM seat's KV from --gpu-memory-utilization on a shared card — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin --kv-cache-memory. auto-memory reference_vllm_util_kv_coboot_race.
  • [2026-10-05] Reading GPU headroom just after a restart — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory feedback_vllm_headroom_at_plateau_not_boot.
  • [2026-10-04] apt-get run in-band through qm guest exec — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a systemd-run unit. (auto-memory feedback_no_inband_apt_through_guest_agent)
  • [2026-10-04] Pulling files with ssh host 'sudo cat …' — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and log_output re-recorded the whole file as a new session. Read without sudo (group-readable file).
  • [2026-10-01] Size + mtime scans to pick reclaimable "staging" on a model store → persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md
  • [2026-09-30] Shorter Parakeet slices as Scriberr's memory fix → persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md
  • [2026-09-30] Whole-file local-attention Parakeet in Scriberr (context 255/255). OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
  • [2026-09-30] Start-time midpoint stitching of overlapped Parakeet chunks. It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
  • [2026-09-30] int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime. The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
  • [2026-09-30] GPU budgets computed as total − used. nvidia-smi Free is ~640 MiB lower per card (driver reserve). Budget from Free.
  • [2026-09-27] The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel → persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md
  • [2026-09-27] A TCP connect as the "is Blender ready" probe → persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md
  • [2026-09-27] log.exception() in a GPU failure path — the record keeps exc_info, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log traceback.format_exc() text instead (semif-serve engine._guard).
  • [2026-09-27] fla/triton in a slim image without gcc — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
  • [2026-09-27] uv export → requirements.txt → uv pip install for a torch-from-cu128 project → persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md
  • [2026-09-27] Translating a CUDA OOM by raising inside except (or from exc) — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, gc.collect() before empty_cache().
  • [2026-09-27] deploy-stack.sh --yes / dns-sync.py fed a blind y — the harness refuses a blind apply. Review with echo n | first, then apply with echo y |.
  • [2026-09-27] Relying on the build cache surviving on esh-ml1 → persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md

128 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink