vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9d599caa53f9fdf51a273e0f132d9b29363595ee
esh-pfi-infrastructure/persistent-memory.md
T

56 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-10-14 ~0005 PT (snapshot for clear: SVOS v0.0.11 live, pewpewstudio deployed, albok inbox live, decision seat on h2o; earlier: the h2o swap saga, U13.)

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under 8 hours old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than 8 hours: stale — delete it unread.

(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the handoff unread across any overnight gap, which is the exact case it exists for. 8 h also matches the global CLAUDE.md and the /snapshot skill default.)

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) — [2026-09-24] MOTHBALLED by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (e6da607) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback): ONE container on nh3-dev at http://10.100.50.40:8390 is the only stateful component; althing-po-herald one per box; althing-listen one per session; postbox is the client. Every v2 command was DELETED, not deprecated — althing-cli→postbox, althing-wake-listener→althing-listen, althing-light-monitor/althing-receiver gone. Sessions need BOTH ALTHING_POST_OFFICE and ALTHING_HANDLE; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → persistent-memory.d/2026-08-28-althing-v3-cutover.md per-box install (NOT CI-deploy); nh3-dev = container host + repo; nh3-extdev = system WHEEL at /opt/uv-tools, needs its own wheel install (playbooks/nh3-extdev-althing-v3.yaml)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
vh/zonos-gateway OpenAI-compatible TTS gateway over stock ZONOS2 (:8890 irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM ext-tts alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (resolve_preset(name,voice); angry/happy/startled_happy per-voice). 8 voices incl. 4 clones pushed to gitea (main 8f1885b/v0.2.1); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec docs/EMOTION-DIALS-SPEC.md; host-managed voices bind-mount (./voices:/app/voices, drop wav + restart, no rebuild)
vh/soong-lab Noonien Soong character-design studio (SPA + /api + WT /bifrost/tool-call); containerized 2026-07-18, LIVE on corviduo-dev :8443 (image vh/soong-lab:latest). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as deploy, compose pull && up -d from /opt/soong-lab, health-gated on /api/version). Manual redeploy sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'. → archival-memory.md (archived 2026-08-16)
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path) — [2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev (fleet ownership audit; also CLAUDE.md 2026-09-05). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-10-14 ~0005 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched.

2026-10-14: live now (as of 2026-10-14 ~0005 PT)

  • Nothing is mid-action. No ops-log claims held. eshpfi main is pushed through d1192ca (infra-ops has STANDING push on main since 2026-10-13, CLAUDE.md § Working rules); this snapshot's commit follows and is pushed. ⚠ The working tree and index are shared with infra-hermes, albok-dev and subagents: commit with git commit -- <paths> and read git log origin/main..main before pushing. graphify-out/GRAPH_REPORT.md stays modified on purpose.
  • SVOS: v0.0.11 (452a592, U15 task seats slice 1) is live on nh3-svos (schema 13, config_gen 5ade279d2fb2dd1c, apps up 7) and nh3-dev (/opt/svos/current → releases/452a592; svos-terminal 0.0.4 runs from it). svos-dev accepted it. Task seats stay OFF until Prime names a runtime task_argv (e.g. the claude command), and GET /tasks as Prime is his acceptance check. Static-only interface releases since: UI 0.0.14 89dad7a, 0.0.15 7f95e54 (the web build carried into 452a592). ⚠ Process→release map: nh3-svos kernel + local apps run 452a592; on nh3-dev svos-miranda runs adc862e, svos-postoffice and svos-agent still run dc241f8 (check /proc/<pid>/maps before deleting any release dir). Rollback material: nh3-svos /var/lib/svos/backup/pre-452a592/ (schema-12 DB + config tar), instance.toml.pre-452a592, instance.toml.pre-a171; nh3-dev terminal.toml.pre-452a592. U13 history and the static-release chain: → persistent-memory.d/2026-10-11-u13-cutover-executed.md. Phase R (24 h soak, High Seat removal) is highseat-dev's. After the soak: nh3-svos releases 6c6abbe, a737a06, b588510, afab123, adc862e and nh3-dev /opt/svos/releases/afab123 can go once no process maps them.
  • Decision seat = h2o-lightning-4b (production, Prime 1805 10-13: fix forward, no INC fallback). The one open margin is the judge's personal-question reading (h2o 18 vs INC 6 on fleet/memory); albok-dev re-fits it forward in small batches on the live seat.
  • albok note intake (Prime 1530 10-13): gate 2 DONE (wing personal/inbox, /srv/nemi/inbox on nh3-dev, nemi token 01M4VX207PJ09ZSECASCSZPCFE). Gate 1 (ruling 6: Prime's Obsidian vault synced onto nh3-dev) is APPROVED by Prime (~2120 10-13: "you may copy the notes") but BLOCKED on where the vault lives: nothing under nh3-dev or esh-nas's shares; asked Prime. Plan: Syncthing receive-only onto nh3-dev, root path to albok-dev (thread 01M4VWV5FEF64EP71W60N6Q1N0).
  • Worldtree: demo was deploying beba08a4 (albok 4.7 the rest; Worldtree is a pure albok-service HTTP customer) at 2120 10-13. Ours: strip the retired config blocks (providers library: + graph_store:, defaults kb_audit:; NOT defaults library:) from each instance config ONLY after that instance runs beba08a4+, demo first (vh/worldtree-instance-configs, deploy-wt-config; thread 01M4WA58P2M001K1420NN72WFM). Daily memory gate: 10-13 re-run 20261013T160414Z PASS; worldtree-dev owns the tagger-latch fix. muninn-gate + mimir-inbox are STOPPED (their data at rest kept for albok unit 6).
  • satellite-1: its own key is authorized for infra-ops on esh-nas only (playbooks/files/infra-ops-satellite-keys). Waiting on Prime's login test from the satellite and its machine name (the key's comment came through as bare infra-ops@).
  • fv-ml1: hard hang 10-11 ~0005, self-reset, no record (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit fv-ml1-sol-logger (journalctl --user -u fv-ml1-sol-logger). ⚠ HOLD zroot/ROOT/debian@pre-patch-20261008 (do not destroy) until the cause is known or a clean week passes after the last crash.
  • albok .browse/ share: the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in ~vh/albok-browse-test/ on nh3-albok, waits on Prime pasting the one-liner (scripts/oneoff/albok-browse-withdrawal/README.md); afterwards read results/, report to albok-dev, remove the replica.
  • Watching: jekyll on sec-reasoning (cyberprev, GPU 0) since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
  • Waiting on Prime: where his Obsidian vault lives (gate 1); satellite-1's login test + name; a task_argv for SVOS task seats; rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth ce095c1; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ irv-ml1: DO NOT PATCH (Prime); config changes are fine. ana-nas is out of scope.
  • Small clean-ups, ours, untracked: stale LiteLLM aliases erp-tune-v2 and gemma4-26b-a4b-it-base; ana-docker's Traefik (readTimeout 10m) and CrowdSec (use_wal, bouncer updateMaxFailure: 3) configs live only on the host, not in stacks/; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx .bak on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's patches/0003 lands only at the next scripts/scriberr-rebuild. The LiteLLM compose.yaml on ana-docker differs from stacks/litellm/compose.yaml (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%). The ana-docker gitea-runner's Homepage name changes to its runner name at its next recreate (stacks/gitea-runner/compose.yaml). nh3-dev's /opt/svos/releases/ and the h2o eval seat on fv-ml1 GPU 3 (stacks/h2o-eval, down) are kept as artifacts.
  • Open, Prime's calls (unchanged from 10-03): QNAP LCD visual check (services/qnap-lcd/); NH3 Mac mini (Mac-mini, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask openipmi.service on esh-nas-pve; run playbooks/pve-nag-patch.yaml on esh-pve-2 when it is plugged back in.
  • Older open items, one line each: esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (stacks/blender/README.md). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran 51168a2e on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.

fv-ml1 GPU layout (as of 2026-10-05 2249)

GPU seat max ctx KV KV tokens concurrency
0 cyberprev (sec) 262,144 15 GiB 336,935 1.29×
0 erp-seat (char-rp-fast) 262,144 5.5 GiB 346,436 1.32×
0 voices 40,960 6 GiB (bf16) 43,680 1.07×
0 coder (coder-fast) 32,768 1.03 GiB 77,056 2.35×
0 parakeet-nemo audio rest 2,108 MiB, cap 3,840 — —
1 gen-small (+summarizer/classifier) 262,144 5.5 GiB 461,215 1.76× (default output cap 32,768)
1 meromero-rp (char-rp) 262,144 22 GiB 346,436 1.32×
1 intern-decision = h2o-lightning-4b since 10-12 1807 (production; no INC fallback per Prime) 40,960 (front admits 32,000/call) 4.48 GiB (4,810,363,371 B) 135,558 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847)
2 flash-next (gen, gen-large, +6) 262,144 — 344,155 1.31×

Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (--kv-cache-memory), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (scripts/oneoff/fv-ml1-gpucheck.sh), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory feedback_idle_check_must_gate_the_restart).

  • GPU 3: clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot scripts/mia-run, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to irv-ml1's A6000, not back to GPU 1.

Scriberr (fv-ml1 GPU 3)

  • LIVE scriberr:local-blackwell-a353078-dropout2: upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, PARAKEET_MODEL_PATH), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). patches/0003 (pyannote env pins) is committed but in no image yet. scripts/scriberr-rebuild re-applies all. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md
  • Diarization works (fixed 2026-10-06): model caches on /tank, HF_TOKEN a placeholder, saved profiles WhisperX large-v3 · GPU · speakers and Parakeet · GPU · speakers. A NEW gated model needs one download with vault fv-ml1/hf-token. → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md
  • URL is https://scriberr.nh3.phasefinal.com since 2026-10-01 (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup .bak-20261001-scriberr). getUserMedia is refused on http://, so recording needs the HTTPS name, which is in ALLOWED_ORIGINS (playbooks/scriberr-https-origin.yaml). ⚠ /opt/docker/compose/scriberr is lkraven-owned, so deploy-stack.sh as infra-ops FAILS there; upload single files with elway --upload --sudo --owner lkraven:lkraven.

Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)

  • Incident 2026-10-01 04:21, fixed by nemo-0.1.1 (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: empty_cache around every window, MEM_CAP_MIB=3840 (a hard ceiling), CUDA_GRAPHS=0 (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory. Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
  • Rollback: docker stop parakeet-nemo && docker start parakeet (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with --http h11 (httptools 0.8.0 emits HTTP/1.1 200\x00OK, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

Worldtree U11 memory cutover (demo + personal)

  • Legacy plane OFF since 0115/0120 PT 2026-09-30 (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md
  • U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted agents/{forseti,lofn,mimir}/memory/<agent>.chroma + memory/context_promotion on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day JSONL_RETENTION_DAYS sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
  • b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag. scripts/wt-memory-gate-batch now passes --legacy-mode only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. Still mine after b193 deploys: remove the residue memory/context_promotion/ledger.db a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM hermes-gateway (never read); reply to infra-hermes.
  • Legacy archive: DESTROY it whole by 2026-10-30, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
  • TODO: re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").

The decision seat intern-decision :8033 (h2o-lightning-4b, production since 2026-10-12 1807)

  • LIVE: h2o-lightning-4b @ b43f759b (compose.yaml = compose.h2o.yaml) behind services/h2o-decision-front: 32k refuse-not-trim admission, bearer, /decide/shared translator. Inline images only (LIGHTNING_IMAGE_URLS=off + vLLM --allowed-media-domains media.invalid, both verified). Nemi's judge runs v2 (albok-nemi 0.1.18) at concurrency 4.
  • ⚠ Prime 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc." rollback-to-inc.sh + compose.inc.yaml remain on the host as artifacts only. Seat problems are fixed forward in the judge (wording, thresholds, visibility declarations); INC again needs Prime's explicit word.
  • ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1.
  • INC history: → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md.

nh3-pve + nh3-ml1 (2026-09-25/26)

  • nh3-pve: Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. AMT live: static 10.100.250.61 on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as nh3-pve/amt-admin. → servers/nh3-pve/README.md
  • nh3-ml1 (CT 109 @ 10.100.50.80): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (qwen3-embedding esh-ml1 first, qwen3-embedding-bulk for albok writes nh3-ml1 first). brokkr's foundry seats: lfm-vl :8030 (gateway lfm25-vl-3b), lfm-vl-uncensored :8032 (direct), vibevoice-asr :8031 (audio.cpp, Q8_0). GPU ~11.3/16 GB. → servers/nh3-ml1/README.md

Recent decisions

  • [2026-10-13] pewpewstudio LIVE on nh3-docker :8185 (Prime via pewpew-dev): api + Caddy front; the Host allowlist must sit in a route (foreign Host 400); DB, assets and shop config on named volumes. → stacks/pewpewstudio/README.md

  • [2026-10-13] infra-ops on esh-nas (CT 103) + per-host satellite keys (Prime) — bootstrapped via pct from esh-pve-nas; satellite-1 authorized on esh-nas only (playbooks/files/infra-ops-satellite-keys); its login test + name pending.

  • [2026-10-13] Worldtree memory gate: 0730 harness error (the tagger's MalformedClassification latched the writer), re-run 20261013T160414Z PASS (median 0.83) — the latch fix is worldtree-dev's (thread 01M4V035SF73XKVG3NRVX37XQ6).

  • [2026-10-13] SVOS v0.0.11 (452a592, U15 task seats slice 1) deployed — schema 13, terminal 0.0.4 pinned without a card, tasks mapping host nh3-svos. → persistent-memory.d/2026-10-13-svos-v0-0-11-deployed.md

  • [2026-10-13] muninn-gate + mimir-inbox STOPPED (retired) — data at rest kept for albok unit 6; Worldtree retired-config strip order. → persistent-memory.d/2026-10-13-muninn-gate-mimir-inbox-stopped.md

  • [2026-10-13] Prime ~2120: notes copy approved; fleet infra-ops key vaulted — restore tested; vault location still unknown. → persistent-memory.d/2026-10-13-notes-copy-approved-fleet-key-vaulted.md

  • [2026-10-13] Albok note intake: gate 2 done (personal/inbox, /srv/nemi/inbox), gate 1 waits on Prime — wing, token, ACL trap. → persistent-memory.d/2026-10-13-albok-note-intake-gate-2.md

  • [2026-10-13] A171: Miranda's bus sends no longer card — one grant made autonomous; config_gen and revert. → persistent-memory.d/2026-10-13-a171-miranda-postoffice-autonomous.md

  • [2026-10-13] app-terminal degraded since U13: its unit's PATH lacked ~/.cargo/bin — zellij unreachable; dump/inject broken too. → persistent-memory.d/2026-10-13-app-terminal-path-fix.md

  • [2026-10-12] Prime's rulings ~1715: CI fix A (fast runner on nh3-docker); push eshpfi main (done, 3aa715c..6a01da7); deny old card 05cf119e (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); no rotation of albok/judge-token (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.

  • [2026-10-12] Decision seat: h2o tried, rolled back twice, then production on Prime's fix-forward — the judge's v1 question, the margins, the lessons. → persistent-memory.d/2026-10-12-decision-seat-h2o-rollbacks-and-cutover.md

  • [2026-10-12] U13 regression fixed (app panes 403) — base_url back to the terminal origin, seat routes allowed there. → persistent-memory.d/2026-10-12-u13-regression-fixed.md

  • [2026-10-12] Shared CI runner slow = ospool fsync; fix A (nh3-docker fast runner) done — measurements, the corrected albok diagnosis. → persistent-memory.d/2026-10-12-shared-ci-runner-slow-fsync.md

  • [2026-10-11] worldtree-dev answering again; #416 residual reopened — four threads answered at once. → persistent-memory.d/2026-10-11-worldtree-dev-reading-mail.md

  • [2026-10-11] U13 cutover executed on Prime's go (0721) + his four rulings — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → persistent-memory.d/2026-10-11-u13-cutover-executed.md

  • [2026-10-11] fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD — and infra-ops' soft-off that landed mid-boot. → persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md

  • [2026-10-10] Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay matrix-relay. → persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md

  • [2026-10-10] U13 phase B built (B1–B6) — kernel 0.0.9/schema 12, svos-kernel.phasefinal.com (nh3-dev only, method-pinned paths), nh3-dev origins + apps env, enrollment secrets. → persistent-memory.d/2026-10-10-u13-phase-b-built.md

  • [2026-10-10] esh-nas-pve nvme/ROOT@pre-pve9 destroyed after a clean week (the two older snapshots followed on 10-11, Prime). → persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md

  • [2026-10-09] vh gets every albok read group + four Mac keys (Prime: "albok-everything") — no net new root path. → persistent-memory.d/2026-10-09-vh-albok-everything.md

  • [2026-10-09] nh3-dev: orphaned svos pytest killed 2242 (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.

  • [2026-10-09] albok .browse/ sshfs withdrawal test PASSED (Linux) + post-clear checks closed (fv-ml1 plateau, nh3-nas 71%, pre-pbs4 mislabel). → persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md

  • [2026-10-09] albok library building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev. → stacks/albok-service/README.md

  • [2026-10-09] Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".

  • [2026-10-09] SVOS app-traffic ingress for U13 approved (shape agreed with svos-dev) — EXECUTED 10-10 (phase B) and 10-11 (the edit). → persistent-memory.d/2026-10-09-u13-ingress-approved.md

  • [2026-10-09] nh3-svos → kernel 0.0.8 (b588510, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed. A 4-min browser soak held svos.db handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → servers/nh3-svos/README.md

  • [2026-10-08] nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the Type=notify unit as a pair); thaw race FIXED: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → servers/nh3-svos/README.md

  • [2026-10-08] SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door). /voice/* is routed BEFORE Basic auth, because the browser sends Authorization: Bearer <lease> there (web/src/lib/voice.ts) and Basic auth would reject it. All four app credentials are now vaulted as nh3-svos/svos-<app>-credential. → servers/nh3-svos/README.md

  • [2026-10-08] SVOS served on the LAN too (Prime): Caddy binds 10.100.50.43:443 as well as 100.64.0.9:443; the internal AdGuards answer svos/svos-terminal.phasefinal.com → 10.100.50.43 (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → servers/nh3-svos/README.md

  • [2026-10-08] fv-ml1 is OFF the mesh again, by design: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active.

  • [2026-10-08] heid's runaway sweep killed on Prime's order: two heidnotes/sweep.py processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66% — the sweep read its own output. heid fixed it at the source. → persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md

  • [2026-10-08] SVOS post office LIVE on nh3-svos — Prime declared svos; its ns:notifications write grant was approved on his word at 0731 (card card:4d3a8b26c17f472d828b06fd495192db); a test message was ingested and delivered; reach ok, unread 0. Reported to svos-dev.

  • [2026-10-07] No z.ai vision alias on the gateway (Prime). glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.

  • [2026-10-06] Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles. Patch 0003 waits for the next rebuild (tracked at stacks/scriberr/patches/README.md). → persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md

  • [2026-10-06] Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00. → persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md

  • [2026-10-08] Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md

  • [2026-10-07] nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps — acceptance 17/17 + Chromium; the thaw race. → persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md

  • [2026-10-07] Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime) — thinking: disabled is light thinking, not zero. → persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md

  • [2026-10-07] Hermes launchers repaired (Miranda relay 8462): svos-dev's scratch probe had rewritten ~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp} to exec a /tmp scratch Python. Repointed to ~/.hermes/tools/python-3.14.7…; the gateway was not restarted. ⚠ Don't run git in ~/.hermes/hermes-agent: a status there looped on auto-gc.

  • [2026-10-07] albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed — the .browse/ view exists on the box only; the share test is owed. → persistent-memory.d/2026-10-07-albok-service-0-1-14.md

  • [2026-10-07] albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed — rollback to 0.1.12 needs the strip snippet first. → persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md

  • [2026-10-06] albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199, built from clean archives, pinned by digest, single-file upload (deploy-stack's --delete would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → stacks/albok-service/README.md

  • [2026-10-06] The LiteLLM gateway keeps no request/response content, and store_prompts_in_spend_logs: false is now a marked consumer guarantee (comfy-dev, arbo incognito). Proved with a canary plus positive controls; per-request no-log does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes input/incognito-* (dry-run proved with a canary and a control). 847424c, 0f6c7a0.

  • [2026-10-06] jekyll moved from gen-large to sec-reasoning (Prime) after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md

  • [2026-10-06] vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention. Content verified (main sha, 20 branches, 29 tags).

  • [2026-10-05] fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime). → persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md

  • [2026-10-05] Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115): its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.

  • [2026-10-05] nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only https://svos.phasefinal.com with HTTP Basic auth, login Vuong (Prime, direct) — WhoIs gate retired (all mesh nodes are login pfi; Prime declined a mesh restructure). Password vaulted nh3-svos/basic-auth-vuong. qemu-ga freeze hooks were never run on Debian; now configs/qemu-ga/fsfreeze-dispatch (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun svos-front-inventory after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → servers/nh3-svos/README.md.

  • [2026-10-05] ESH Homepage also on https://10.0.50.45 (Prime) — a Traefik file-provider router (homepage-https.yml, tls: {}) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (2c4c7ca)

  • [2026-10-05] bench-jev servers killed and removed, their HF cache cleaned (Prime); pfi-pteradactyl fixed + patched + engine upgraded (3429b5c; manual dockerd + wings restart).

  • [2026-10-05] fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE) — run 1 aborted on its own guard (live rsync walked into an overlay merged view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (-x, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → servers/fv-ml1/README.md

  • [2026-10-05] ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime) — with live-restore: true the engine upgrade kept containers up, but 6 that bind-mount docker.sock (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → servers/ana-docker/README.md

  • [2026-10-05] RMM wildcard cert now auto-renews (Prime) — *.phasefinal.com on pfi-tacticalrmm was certbot manual (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault pfi-tacticalrmm/cloudflare-certbot-dns01-token); reissued to 2027-01-03, dry-run OK. → servers/pfi-tacticalrmm/README.md

  • [2026-10-05] CrowdSec SQLite → WAL on ana-docker (0418) — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (use_wal: false, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → servers/ana-docker/README.md

  • [2026-10-05] Fleet guest patching (park 100) DONE 0355 — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → docs/runbooks/fleet-guest-patch-2026-10-05.md § Result — [2026-10-05 later] reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.

  • [2026-10-05] irv-ml1 storage cleanup + text encoders on worktank (Prime) — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); /storetank/arbo/models/text_encoders is now ZFS worktank/arbo-text-encoders mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → servers/irv-ml1/README.md

  • [2026-10-05] ana-docker Traefik readTimeout 60s → 10m on both entrypoints — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in stacks/. → servers/ana-docker/README.md

  • [2026-10-05] Embedder routing split (Prime 0030, via albok-dev, confirmed directly): qwen3-embedding = esh-ml1 primary / nh3-ml1 failover; new qwen3-embedding-bulk = nh3-ml1 primary / esh-ml1 failover, for albok writes — LiteLLM order (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → stacks/litellm/conf/config.yaml

  • [2026-10-04] albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime) — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns nh3-pve-2; copy verified by digest (dir st_size differs on ext4, not data). → persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md

  • [2026-10-04] The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md

  • [2026-10-04] pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + infra-ops-pages tokens — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md

  • [2026-10-04] sudo session recordings: archived weekly, bulk data never kept (Prime) — BULK_DATA !log_output on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → services/sudo-io-archive/README.md

  • [2026-10-04] VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use) — off, onboot 0, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.

  • [2026-10-04] PARKED (Prime): move fv-ml1's Docker data-root to tank — needs a model-seat outage window; tracked at park move-fv-ml1-s-docker-storage-off-the-root-ssd (id 99). — [2026-10-05] DONE (entry above).

  • [2026-10-04] PARKED (Prime): guest OS patch campaign — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park guest-os-patch-campaign-across-the-fleet-debian (id 100). — [2026-10-05] DONE (entry above).

  • [2026-10-04] PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway — tracked at park give-svos-and-miranda-a-model-path-that-doesn-t (id 98).

  • [2026-10-03] PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host"): an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → services/pve-nag-patch/README.md

  • [2026-10-03] nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 nic3, AMT on nic0 with auto kept up, 1 TB wiped into vmstore (913 GiB), stale boot entry gone → persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md

  • [2026-10-03] CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232) → persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md

  • [2026-10-03] Worldtree memory-gate duty REFRAMED: regression check, not a gate → persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md

  • [2026-10-03] albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1 → persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md

  • [2026-10-02] esh-pve-2 (MS-03 at ESH, future esh-dev host) onboarded: infra-ops (Prime), 1 TB → LVM-thin vmstore, AMT phoning home as esh-pve-2-amt → persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md

  • [2026-10-02] Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684) → persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md

  • [2026-10-02] nh3-pve's AMT is in TacticalRMM's MeshCentral (group PFI-AMT, device nh3-pve-amt) → persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md

  • [2026-10-02] Prime: dev-backup gets dailies + weeklies — DONE → persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md

  • [2026-10-02] ESH tank (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629 → persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md

  • [2026-10-02] albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): http://albok.nh3.internal:8392 → persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md

  • [2026-10-02] nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03 → persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md

  • [2026-10-02] Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as nh3-pve-2 at NH3 (Prime, ~0910) → persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md

  • [2026-10-02] nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot) → persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md

  • [2026-10-01] MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev) → persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md

  • [2026-10-01] Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed → persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md

  • [2026-10-01] Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.

  • [2026-10-01] ⚠ Gitea's [webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8 silently REJECTS headscale mesh IPs (100.64.0.0/10) → persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md

  • [2026-10-01] vh/arbo push webhook repointed from the retired wg0 lifeline 10.100.79.3:9009 to irv-ml1's mesh address 100.64.0.6:9009 → persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md

  • [2026-10-01] Prime: delete the bench leftovers, no upstream for Scriberr, push → persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md

  • [2026-10-01] irv-ml1 /storetank reclaim done: Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread 01M3TCSYRSFNAPA9BPQTMFQ6KJ).

  • [2026-09-30] Parakeet speech seat → parakeet-unified-en-0.6b under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137. → persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md

  • [2026-09-30] Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30. → persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md

  • [2026-09-30] SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev /v1/systemone at 32k tokens on GPU 1, with a Triton warm-up cache volume. → persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md

  • [2026-09-30] Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept. → persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md

  • [2026-09-27] Worldtree reward path: d8726a13 + providers.yaml reward_models.scalar-judge live on demo; personal waited on a newer image. → persistent-memory.d/2026-09-27-worldtree-reward-path-live.md

  • [2026-09-26] augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy. → persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md — [2026-09-27] DONE: deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (d8f59a1).

  • [2026-09-26] zellij-fleet@.service installed, NOT enabled (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling @Claude at boot is Prime's call when seat_up ships. Tracked: 0ad7799, services/zellij-fleet/README.md.

  • [2026-09-26] btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority). Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: servers/nh3-pve/README.md.

  • [2026-09-25] AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326). — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md

  • [2026-09-26] esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only — , for ha-dev (operator-approved, relayed). → persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md

  • [2026-09-26] Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime). — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md

  • [2026-09-26] Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed): — LFM2.5-VL-3B on llama.cpp :8030 (gateway lfm25-vl-3b, LiteLLM restarted 36 s at 0039) and… → persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md

  • [2026-09-25] nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333). Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: servers/nh3-pve/README.md (2b49be4, cdd7605).

  • [2026-09-25] Direct ESH→esh-ml1 consumer path — PARKED (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md.

  • [2026-09-24] Worldtree admin.memory.forget stays OFF on demo/personal until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.

  • [2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md

  • [2026-09-17] The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev — dev_launch.py has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md

  • [2026-09-17] PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the… → persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md

  • [2026-09-13] STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint. → persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md

  • [2026-08-25] Fused MoE kernel path — DEFERRED, tracked at park fused-moe-kernel-path-for-gemma-4-moe-training (id 47). → persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md

  • [2026-08-24] nconnect=8 on /mnt/smithy — approved but DEFERRED at operator instruction → persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md

258 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-10-08] NVIDIA's network debian12 repo on trixie via cuda-keyring — trixie's sqv rejects the key's SHA1 binding signature, so apt update errors until the list is gone (dpkg -r keeps the conffile; it took dpkg -P). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → servers/fv-ml1/README.md
  • [2026-10-08] secret get <name> | head "to see its shape" — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with secret list or wc -c, never by printing.
  • [2026-10-07] An idle gate counting with grep -c … || echo ERR — grep -c exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use || true and test the gate against a known-idle state first.
  • [2026-10-07] pgrep -f <pattern> to check that a process died — it matched the checking shell's own command line and reported a survivor; check with ps -eo … | grep '[x]…', as the same self-match bit pkill -f before.
  • [2026-10-06] Restarting a user-facing service after an idle check that only PRINTS — nvidia-smi … | wc -l said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (&& / exit). auto-memory feedback_idle_check_must_gate_the_restart.
  • [2026-10-06] Aging PBS snapshots by directory mtime — verify jobs and notes edits rewrite index.json.blob, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory reference_pbs_snapshot_mtime_moves_on_verify.
  • [2026-10-06] deploy-stack.sh on a stack whose host dir holds rollback copies (albok-service compose.yaml.pre-*) — its rsync --delete removes them. Upload the single changed file with elway instead.
  • [2026-10-05] Sizing a vLLM seat's KV from --gpu-memory-utilization on a shared card — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin --kv-cache-memory. auto-memory reference_vllm_util_kv_coboot_race.
  • [2026-10-05] Reading GPU headroom just after a restart — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory feedback_vllm_headroom_at_plateau_not_boot.
  • [2026-10-04] apt-get run in-band through qm guest exec — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a systemd-run unit. (auto-memory feedback_no_inband_apt_through_guest_agent)
  • [2026-10-04] Pulling files with ssh host 'sudo cat …' — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and log_output re-recorded the whole file as a new session. Read without sudo (group-readable file).
  • [2026-10-01] Size + mtime scans to pick reclaimable "staging" on a model store → persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md

141 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink