340 lines
66 KiB
Markdown
340 lines
66 KiB
Markdown
# Persistent memory — eshpfi-management
|
||
|
||
_Last updated: 2026-10-12 ~1810 PT (decision seat: h2o production since 1807 on Prime's fix-forward ruling, after two rollbacks; earlier: U13 cutover + acceptance, fv-ml1 evidence capture, vh albok-everything.)_
|
||
|
||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||
> handoff from the previous session), then delete it. Older than 8 hours:
|
||
> stale — delete it unread.
|
||
>
|
||
> _(Raised from 1 h to 8 h by operator 2026-09-13 — a one-hour window deleted the
|
||
> handoff unread across any overnight gap, which is the exact case it exists for.
|
||
> 8 h also matches the global CLAUDE.md and the `/snapshot` skill default.)_
|
||
|
||
## Repo purpose
|
||
|
||
Reference workspace for PFI infrastructure: server inventory, canonical
|
||
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
||
copies of compose files live on the servers under
|
||
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
|
||
control, editing, planning, and CI-driven deploys. **It was originally
|
||
spun up to handle the fleet backups** — keep that lens when triaging
|
||
backup/storage issues.
|
||
|
||
## Tools and conventions
|
||
|
||
Sister repos (separate gitea repos, deployed by playbooks here):
|
||
|
||
| Repo | Role | CI status |
|
||
|---|---|---|
|
||
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) — **[2026-09-24] MOTHBALLED** by Prime (superseded by the High Seat + ledger); container removed on ana-docker, data/image/compose kept (`e6da607`) | push-to-main → CI deploys (2026-04-29) |
|
||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
||
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
|
||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
||
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
|
||
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
|
||
| `vh/zonos-gateway` | OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion **dials-first** + voice mapping; reached via LiteLLM `ext-tts` alias. **v0.2.1 (2026-07-18): voice-resolved emotion presets** (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones | pushed to gitea (main `8f1885b`/`v0.2.1`); **deployed irv-ml1 tree still NON-git** (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild) |
|
||
| `vh/soong-lab` | Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); **containerized 2026-07-18**, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host | CI = Gitea Actions build+push+**DEPLOY** on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; **auto-redeploy LIVE 2026-07-18** — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from **/opt/soong-lab**, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `archival-memory.md` (archived 2026-08-16) |
|
||
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
|
||
|
||
(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
|
||
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
|
||
no longer deployed sidecars here. See Recent decisions.)
|
||
|
||
- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
|
||
fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
|
||
VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
|
||
cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
|
||
for the blast-radius matrix. **⚠️ The restic file+DB layer routes
|
||
through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
|
||
ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
|
||
nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
|
||
export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)
|
||
|
||
- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
|
||
model/dataset onto ana-ml2's shared cache at
|
||
`/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.
|
||
|
||
- **Worldtree admin auth — per-instance.** Each Worldtree deployment
|
||
(demo :8080, personal :8081, pinned :8082) has its own Heimdall
|
||
registry and its own bootstrap admin key. Infra-ops's stored
|
||
long-lived admin key (`key_id 61419c92`) at
|
||
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
|
||
auths against **demo only**. Personal-instance admin (the
|
||
`~/.config/worldtree/personal-admin-token`, mode 600) POSTs
|
||
`/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
|
||
scope param** — scopes are tier-derived). **On-instance mint recipe
|
||
(cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
|
||
`/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
|
||
once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.
|
||
|
||
- **Per-project user keys against personal Worldtree** (issued
|
||
2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
|
||
`althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
|
||
value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
|
||
shreds (DO NOT cat to chat transcript).
|
||
|
||
- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
|
||
pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
|
||
`playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
|
||
SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.
|
||
|
||
- **gitea internal route for fleet hosts.** gitea is a container on
|
||
**ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
|
||
hosts must use this internal route, NOT public `gitea.phasefinal.com`
|
||
(`38.120.12.44`) — the public path fail2bans the host egress IP. Full
|
||
gotcha in `docs/orientation.md` → Git/gitea.
|
||
|
||
- **docker-as-root pattern** (for ops with no admin API, or to edit
|
||
deploy-owned/root-owned files without sudo): `docker run --rm -v
|
||
<target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
|
||
effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
|
||
resolve against the sandbox CWD but the daemon interprets them against the
|
||
HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**
|
||
|
||
- **`scripts/elway` sudo handling** — elway prompts for the sudo password
|
||
ONCE via `getpass` before the first `sudo: true` step → can't run
|
||
unattended from a non-TTY tool if any step needs sudo. Sudo-free
|
||
playbooks run fully non-interactive over key SSH.
|
||
|
||
- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
|
||
on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
|
||
ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
|
||
passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
|
||
For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
|
||
infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
|
||
is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path) — **[2026-09-23] measured: infra-ops HAS NOPASSWD sudo on nh3-extdev** (fleet ownership audit; also CLAUDE.md 2026-09-05). **irv-ml1:
|
||
`ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
|
||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||
|
||
## Current state / in-flight
|
||
|
||
_As of 2026-10-09 ~1955 PT. The newest subsection is first; older subsections carry their own dates. What was DONE lives in Recent decisions; this section keeps only what is open, scheduled or being watched._
|
||
|
||
### 2026-10-11: live now (as of 2026-10-11 ~1400 PT)
|
||
|
||
- **Nothing is mid-action.** No claims held. eshpfi is pushed through `df7161b`; this snapshot's commit follows. ⚠ The working tree and index are shared with infra-hermes and subagents: commit with `git commit -- <paths>`. `graphify-out/GRAPH_REPORT.md` stays modified on purpose.
|
||
- **U13 cutover: executed 10-11 0721–0816, acceptance 1350.** Miranda now runs in SVOS: app-miranda + svos-agent on nh3-dev, the Hermes gateway on plugin `svos`, and the bridge app-postoffice on nh3-dev owning the `miranda` push route. The URGENT bus test (thread `01M4PBNX5PP0PGHDTY5BG7HG95`) ingested, paged per the connector receipt, and a reply came back on the thread from `svos`. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
|
||
- **Acceptance closed 1354:** Miranda's reply arrived on the thread through the bridge's own send. ⚠ RE-CORRECTED 2026-10-12: card `card:05cf119e604649838e2a4c642c161e26` (Miranda's carded `send`) WAS approved by `prime` at 2026-10-11 20:54:25Z (1354:25 PT), per the kernel DB (`/var/lib/svos/state/svos.db`, read-only). The 1412 "NOT approved" correction was itself wrong, because it came from inference, not a read. Nothing to deny; 0 cards pending. **Answered by svos-dev 1400:** the thread text was Miranda answering bus mail with her own `send` (carded) instead of letting the bridge carry her turn's reply; the fix (a bus-tag clause in `svos_miranda.turns`) SHIPPED in v0.0.10 (adc862e), deployed 1410–1411 (nh3-svos kernel 0.0.10 + UI 0.0.8; nh3-dev miranda 0.0.4, `/opt/svos/current → releases/adc862e`); bus check passed 1412 with no card. Its manifest still says 0.0.3 (hardcoded), so it did not re-register. Then a static-only release `34476a7` (interface UI 0.0.9, tree strands follow speech) at 1712: `/opt/svos/current → releases/34476a7` on nh3-svos, no restart, previous handle = adc862e; then `ad255a2` (UI 0.0.10, stronger strands, Prime found 0.0.9 too subtle) at 1719, previous = 34476a7; then `627a9fb` (UI 0.0.11: the top bar shows `v0.0.<n> · <short commit>`, from `web/build-id.txt` via git-archive export-subst, so ALWAYS build from a clean `git archive`; a plain copy reads `dev`) at 1722, previous = ad255a2; `61459d2` (UI 0.0.12: voice chorus/flanger toggle, lighter header) at 1739, previous = 627a9fb; `8fa229d` (UI 0.0.13: strands span the tree, deeper voice character) at 1800, previous = 61459d2. `89dad7a` (UI 0.0.14: the pane lease race, a frame loading before the attach reply now gets its lease) on 10-13 at 0857, previous = 8fa229d. `7f95e54` (UI 0.0.15: the terminal leaves the app list's Open; a bare embedded terminal can only ever wait for a lease, so it is attached per seat) at 0906:37, previous = 89dad7a. Static-only builds: `scratchpad svos-static-release.sh <prev> <new> <ui>` pattern (rebuild it from the 61459d2 kit if the scratchpad is gone). Static-only releases: build the full release dir (carried artifacts + new `static/`), switch `current` without restarting, regenerate the inventory. ⚠ The kernel + apps still RUN from `releases/adc862e` (verified via `/proc/<pid>/maps`): do not delete it before they restart. Same on nh3-dev: svos-terminal, svos-postoffice and svos-agent still run from `releases/dc241f8` (svos-miranda from `adc862e`). The non-autonomous `miranda → app:app-postoffice write` grant is intended (her own sends); making it autonomous is Prime's call, surfaced by svos-dev. Speech leg unconfirmed (svos-dev asked Prime; n=1 vs the earlier 15 s `synthesis_failed`). Phase R (24 h soak, doctrine + broadcast texts, High Seat removal) is highseat-dev's (svos `docs/u13-cutover-runbook.md`).
|
||
- **Rollback material, keep until the soak ends:** nh3-svos `releases/b588510`, `instance.toml.pre-u13-{A,B}`, `/etc/svos/kernel.env.pre-u13` (absent: there was none); nh3-dev `~/.hermes/config.yaml.pre-u13` and `~/.hermes/plugins/svos_miranda`. After the soak: nh3-svos releases `6c6abbe`, `a737a06`, `b588510` and nh3-dev `/opt/svos/releases/afab123` (unreferenced) can go.
|
||
- **fv-ml1: hard hang 10-11 ~0005, self-reset, no record** (pstore/ERST registered and empty, so not a kernel panic). Evidence capture staged: kernel console on ttyS1 from its next boot + nh3-dev user unit `fv-ml1-sol-logger` (`journalctl --user -u fv-ml1-sol-logger`). ⚠ **HOLD** `zroot/ROOT/debian@pre-patch-20261008` (do not destroy) until the cause is known or a clean week passes after the last crash.
|
||
- **albok `.browse/` share:** the Linux-client withdrawal test passed (10-09); vh holds every albok read group; Prime's Mac keys on vh stay full access (Prime 10-11). Owed: the macOS run, staged as vh in `~vh/albok-browse-test/` on nh3-albok, waits on Prime pasting the one-liner (`scripts/oneoff/albok-browse-withdrawal/README.md`); afterwards read `results/`, report to albok-dev, remove the replica.
|
||
- **Watching: jekyll on `sec-reasoning` (cyberprev, GPU 0)** since ~1435 10-06 (Prime moved it off gen-large). sec holds 337k KV tokens; jekyll's ~180k context plus a long sec prompt (up to 192k) can overflow it. It runs on the shared all-agents-local key.
|
||
- **Waiting on Prime:** rotate the fv-ml1 BMC password (printed into a local transcript 10-08; infra-ops leans no); albok's 2.01× load check against a 2× bar; the 10-05 cleanup bundle (stale parks 49/46/40, fv-ml1 README stack + NVMe tables, ZED alerts into althing-alert-bridge, prune exited one-shot containers on fv-ml1); push Booth `ce095c1`; ESH Homepage HTTPS name; nh3-dev reboot (kernel 6.1.0-53). ⚠ **irv-ml1: DO NOT PATCH (Prime)**; config changes are fine. ana-nas is out of scope.
|
||
- **Small clean-ups, ours, untracked:** stale LiteLLM aliases `erp-tune-v2` and `gemma4-26b-a4b-it-base`; ana-docker's Traefik (`readTimeout` 10m) and CrowdSec (`use_wal`, bouncer `updateMaxFailure: 3`) configs live only on the host, not in `stacks/`; TEI weights revision unpinned on esh-ml1 + nh3-ml1; a stray nginx `.bak` on pfi-tacticalrmm; meromero's Homepage label still says "Dormant / rollback"; Scriberr's `patches/0003` lands only at the next `scripts/scriberr-rebuild`. The LiteLLM compose.yaml on ana-docker differs from `stacks/litellm/compose.yaml` (found 10-07; only the conf side was pushed). nh3-dev holds ~33 GB of unused TAGGED Docker images (rollbacks; not pressing at 66%).
|
||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||
- **Older open items, one line each:** esh-matter: Prime shares the Aqara W200 into HA via Matter multi-admin (if commissioning misbehaves, the UniFi mDNS reflector on esh-iot is the first knob). Restic: the rest-server password rotation stays Prime's (the units no longer leak it). esh-ml1: the nh3-docker Dozzle agent has been stopped by hand since ~2026-04 (revive or drop), and the Beszel superuser password rotation was offered to Prime. Blender: agent-drawn CAD Sketcher geometry and MeasureIt overlays in MCP screenshots are still open (`stacks/blender/README.md`). Worldtree reward path: confirm personal's image now includes d8726a13 (it ran `51168a2e` on 09-27). nh3-dev's root disk is at ~66% since heid's runaway sweep was killed 10-08 (it flapped on the Beszel 85% line before). ESH has a single outside route (esh-scale on esh-pve), noted and untracked.
|
||
|
||
### fv-ml1 GPU layout (as of 2026-10-05 2249)
|
||
|
||
| GPU | seat | max ctx | KV | KV tokens | concurrency |
|
||
|---|---|---|---|---|---|
|
||
| 0 | cyberprev (sec) | 262,144 | 15 GiB | 336,935 | 1.29× |
|
||
| 0 | erp-seat (char-rp-fast) | 262,144 | 5.5 GiB | 346,436 | 1.32× |
|
||
| 0 | voices | 40,960 | 6 GiB (bf16) | 43,680 | 1.07× |
|
||
| 0 | coder (coder-fast) | 32,768 | 1.03 GiB | 77,056 | 2.35× |
|
||
| 0 | parakeet-nemo | audio | rest 2,108 MiB, cap 3,840 | — | — |
|
||
| 1 | gen-small (+summarizer/classifier) | 262,144 | 5.5 GiB | 461,215 | 1.76× (default output cap 32,768) |
|
||
| 1 | meromero-rp (char-rp) | 262,144 | 22 GiB | 346,436 | 1.32× |
|
||
| 1 | intern-decision = **h2o-lightning-4b** since 10-12 1807 (production; no INC fallback per Prime) | 40,960 (front admits 32,000/call) | 4.48 GiB (4,810,363,371 B) | 135,558 | 3.31× (seat peak 16,164 under 4×16q×32k, budget 16,847) |
|
||
| 2 | flash-next (gen, gen-large, +6) | 262,144 | — | 344,155 | 1.31× |
|
||
|
||
Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache-memory`), so util only gates the boot check (Free at start ≥ util × total). flash-next is alone on GPU 2 and stays util-sized (0.96). ⚠ Before restarting any seat on a shared card, gate on the arithmetic (`scripts/oneoff/fv-ml1-gpucheck.sh`), never trial-boot. Before restarting a user-facing service, the idle check must STOP the restart, not just print (auto-memory `feedback_idle_check_must_gate_the_restart`).
|
||
|
||
- **GPU 3:** clear for ad-hoc work (Prime, 2026-10-05); no always-resident tenant goes here. On-demand tenants: Blender, Scriberr (0 idle, ~5.5 GB per job), and MIA auto-rig (one-shot `scripts/mia-run`, peak 3.4 GB, added 2026-10-01 for dread-dev). When a full-size seat claims GPU 3, Scriberr steps aside to **irv-ml1's A6000**, not back to GPU 1.
|
||
|
||
### Scriberr (fv-ml1 GPU 3)
|
||
|
||
- **LIVE `scriberr:local-blackwell-a353078-dropout2`:** upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, `PARAKEET_MODEL_PATH`), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). `patches/0003` (pyannote env pins) is committed but in no image yet. `scripts/scriberr-rebuild` re-applies all. → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||
- **Diarization works (fixed 2026-10-06):** model caches on /tank, `HF_TOKEN` a placeholder, saved profiles `WhisperX large-v3 · GPU · speakers` and `Parakeet · GPU · speakers`. A NEW gated model needs one download with vault `fv-ml1/hf-token`. → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
|
||
- **URL is https://scriberr.nh3.phasefinal.com since 2026-10-01** (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup `.bak-20261001-scriberr`). `getUserMedia` is refused on http://, so recording needs the HTTPS name, which is in `ALLOWED_ORIGINS` (`playbooks/scriberr-https-origin.yaml`). ⚠ `/opt/docker/compose/scriberr` is lkraven-owned, so `deploy-stack.sh` as infra-ops FAILS there; upload single files with `elway --upload --sudo --owner lkraven:lkraven`.
|
||
|
||
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
||
|
||
- **Incident 2026-10-01 04:21, fixed by nemo-0.1.1** (infra-hermes 963f9ed). gen-small OOM'd on a lazily allocated workspace while the seat held its window cache. The fix: `empty_cache` around every window, `MEM_CAP_MIB=3840` (a hard ceiling), `CUDA_GRAPHS=0` (empty_cache poisons the graph pool). The seat rests at 2,108 MiB and peaks at 3,028 on a 12-min file. **Lesson: a shared-card tenant must carry a HARD cap, whether or not it returns memory.** Prime kept graphs OFF. His 10-01 "no insurance trim" on gen-small was superseded on 10-05 (KV 8 → 5.5 GiB) once the GPU 0 plateau turned out to be ~60 MiB free, not the ~1.05 GB read just after a restart.
|
||
- **Rollback:** `docker stop parakeet-nemo && docker start parakeet` (old container and image kept). Seat invariants (in its README): cast to bf16 AFTER change_attention_model; uvicorn pinned with `--http h11` (httptools 0.8.0 emits `HTTP/1.1 200\x00OK`, which LiteLLM/httpx rejects). NVIDIA Open Model License accepted for internal use. → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
|
||
|
||
### Worldtree U11 memory cutover (demo + personal)
|
||
|
||
- **Legacy plane OFF since 0115/0120 PT 2026-09-30** (config repo 63cf268; personal /metrics b6fdd81; the repo is pushed). → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||
- **U11b STEP 5 DONE 2026-10-02 14:54:03Z on Prime's direct go** (in this channel, ~0750 PT). H2 re-check exit 0 on both (demo rows 4/12/5, personal 0/6/2200, agent_self 0); deleted `agents/{forseti,lofn,mimir}/memory/<agent>.chroma` + `memory/context_promotion` on demo and personal with literal paths; apis healthy, no restart; stamp sent to worldtree-dev (b193 is theirs to cut). Demo had already lost 22 context_promotion JSONL (2026-07-03) to ledger.py's 90-day `JSONL_RETENTION_DAYS` sweep on 10-01 07:10:30 — TTL, not an erasure; flagged to worldtree-dev as retirement owner.
|
||
- **b193 is cut and pushed (c2d87263, worldtree-dev 2026-10-02); demo deploys via CI, personal on Prime's staging tag.** `scripts/wt-memory-gate-batch` now passes `--legacy-mode` only when the DEPLOYED sha's harness still has it (git grep at that sha; tested b192 → passed, b193 → omitted), so no timed edit is needed. ⚠ It will REFUSE (exit 3) while demo's deployed sha lags a tree HEAD whose pyproject/uv.lock/packages changed (b193, then the albok commit): by design. **Still mine after b193 deploys:** remove the residue `memory/context_promotion/ledger.db` a b192 restart may have re-created; then the parity config retirements (detail file). Verdict ccs come FROM `hermes-gateway` (never read); reply to `infra-hermes`.
|
||
- **Legacy archive:** DESTROY it whole by **2026-10-30**, or at retirement-done, or on a subject-erasure request, whichever comes first. The runbook is in the detail file.
|
||
- **TODO:** re-sweep both api logs after real traffic. After the b193 push, remove the retired config keys (list: detail file, "Post-b193 config retirements").
|
||
|
||
### The decision seat `intern-decision` :8033 (h2o-lightning-4b, production since 2026-10-12 1807)
|
||
|
||
- **LIVE: h2o-lightning-4b @ b43f759b** (`compose.yaml` = `compose.h2o.yaml`) behind `services/h2o-decision-front`: 32k refuse-not-trim admission, bearer, `/decide/shared` translator. Inline images only (`LIGHTNING_IMAGE_URLS=off` + vLLM `--allowed-media-domains media.invalid`, both verified). Nemi's judge runs v2 (albok-nemi 0.1.18) at **concurrency 4**.
|
||
- ⚠ **Prime 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** `rollback-to-inc.sh` + `compose.inc.yaml` remain on the host as artifacts only. Seat problems are fixed forward in the judge (wording, thresholds, `visibility` declarations); INC again needs Prime's explicit word.
|
||
- ⚠ Seat headroom under 4-way 32k load is 683 MiB (peak 16,164 against 16,847). Re-measure before anything else lands on GPU 1.
|
||
- INC history: → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`.
|
||
|
||
### nh3-pve + nh3-ml1 (2026-09-25/26)
|
||
|
||
- **nh3-pve:** Secure Boot OFF; IGFX restored; NVIDIA 580.178.04 DKMS. **AMT live**: static `10.100.250.61` on nh3-mgmt (UDM port 6), KVM on, Opt-in None, password in the vault as `nh3-pve/amt-admin`. → `servers/nh3-pve/README.md`
|
||
- **nh3-ml1** (CT 109 @ `10.100.50.80`): TEI rerank load-shared with esh-ml1 through the gateway; embeddings primary/failover since 2026-10-05 (`qwen3-embedding` esh-ml1 first, `qwen3-embedding-bulk` for albok writes nh3-ml1 first). brokkr's foundry seats: `lfm-vl` `:8030` (gateway `lfm25-vl-3b`), `lfm-vl-uncensored` `:8032` (direct), `vibevoice-asr` `:8031` (audio.cpp, Q8_0). GPU ~11.3/16 GB. → `servers/nh3-ml1/README.md`
|
||
|
||
## Recent decisions
|
||
|
||
- `[2026-10-13]` **app-terminal was degraded from the U13 cutover until 0856 on 10-13: its user unit's PATH lacked `~/.cargo/bin`, so every `zellij` call failed to launch** (rc 127). That broke /health AND every dump/inject. Fixed with `Environment=PATH=` in `configs/svos-apps/nh3-dev/svos-terminal.service` (3c92a12). The kernel logged app-terminal up at 15:56:40Z, the manifest was unchanged, and no card was raised; kernel health reads up 7 / degraded 0. The restart moved its process from release dc241f8 to adc862e (same svos_terminal 0.0.3). ⚠ Any new user unit that shells out to a cargo/uv/bun tool needs PATH set explicitly.
|
||
- `[2026-10-12]` **Prime's rulings ~1715:** CI fix A (fast runner on nh3-docker); push eshpfi main (done, `3aa715c..6a01da7`); deny old card `05cf119e` (moot: the kernel DB shows Prime approved it 10-11 1354:25, so there is nothing to deny); **no rotation of `albok/judge-token`** (it was printed only into a local transcript). The ana-docker vCPU-type change is dropped.
|
||
- `[2026-10-12]` **Decision seat: h2o-lightning-4b served 1311–1403, then ROLLED BACK to INC** on albok-dev's word, with brokkr concurring (Prime's plan made the swap conditional on the judge's sign-off). Every eval-set check had passed (JevBench 205/231, 422 refusal, 16q@32k ~2.1 s, seat peak 16,164/16,847), but albok's full walk over 10,841 same texts (both verdicts, same prompt and policy) found h2o flags 242 adult at 0.5 against INC's 88, with 154 extra at 0.80–0.90 that no threshold separates; for personal, 5 vs 11 with only 2 in common. **Lesson: a passing benchmark plus a small borderline set (9 entries, 2 → 5) could not resolve a 2× shift in a production classifier; only the same-text fleet comparison did.** albok's re-stamp walk on INC (1406–1408) put the hidden set back exactly to its pre-cutover counts (fleet/memory 86, agent-feedback 21, nonfiction 0). Prime's inline-images-only order (via brokkr) is carried in the staged h2o compose. h2o returns only after a judge re-fit. Threads `01M4RSFC9J2R0J07EEBD4N04W3` (brokkr), `01M4RYSMP3DV9HZHJ0KNBH2S96` (albok). **Then:** the round 2 eval seat on GPU 3 (1428–1619, Prime 1420) found the cause. The judge's v1 note question asked two things at once: INC answered it inconsistently, h2o literally. The shim floor is ruled out (identical verdicts on and off). Prime ruled "text itself", and albok-nemi 0.1.18 (judge v2) re-fitted the question. Both seats agree under v2 (work notes INC 0/22, h2o 1/22 at 0.52). **Prime ruled the cutover at 1640 ("do it", via albok-dev): h2o LIVE again since 1704** (`switch-to-h2o.sh` 170223, healthy 170454, probes clean). **The v2 walk on h2o (1705–1745) moved two margins past albok-dev's named line**: nonfiction adult 0 → 79 (the Schell chapter at 0.531 on the book profile) and memory personal 6 → 18. Notes held at 0 adult. **Rolled back to INC at 1748 on albok-dev's word.** The 1749–1751 re-stamp walk restored the counts (memory 0 adult / 6 personal, feedback 0 / 2, nonfiction 0 / 0), and the timer was back at 1751. **Then Prime, 1805 (via albok-dev): "go to h2o, and fix forward. do not fall back to inc."** albok-dev declared the two published nonfiction titles `visibility: public` (the chapter margin cannot recur). **h2o has been production since 1807** (`switch-to-h2o.sh` 180513, healthy 180740, all checks clean). The cutover walk (1809–1813, 0 degraded) left memory 0 adult / 18 personal, feedback 2 / 0, nonfiction 0 / 0 and fiction 0 / 0, and the timer is back. **The one open margin is the personal reading (18 vs INC's 6).** albok-dev is re-fitting it forward on the live seat in small batches. Threads `01M4S026GWS354WZGXZ5E34F31`, `01M4S91WPPPN0B17JBNX98K5CY`.
|
||
- `[2026-10-12]` **U13 regression fixed 0943 (svos-dev's URGENT, Prime saw every app pane 403):** the browser frames a pane at the registry `base_url`, and U13 had set app-terminal's to the seat-host origin, which admits only nh3-svos. Fix: `base_url` back to `https://svos-terminal.phasefinal.com` (config_gen `acae45206bbc9fcb`), plus POST `/seats/{observe,check,launch,retire-launch}` allowed from the box on that origin. svos-dev's one-line fix alone would have 404'd the kernel's seat-host calls; the origin never carried them before. svos-dev owes a separate browser-facing origin field; then the seat-host goes back to being kernel-facing.
|
||
- `[2026-10-12]` **Shared CI runner (ana-docker act_runner, capacity 2, `timeout: 30m`) is slow because pfi-pve's `ospool` is HDD mirrors with no SLOG:** 8 KiB `oflag=dsync` writes measured ~145 ms on ana-docker vs ~2.7 ms on nh3-docker (n=3 each; pfi-pve host on ospool 18–32 ms; corviduo-dev ~75 ms). Jobs past 30 min are killed with a truncated log and no footer (albok every run since 10-04; vastblue 10935). Worse since the 10-04 PVE 9 reboot, cause not pinned (drive write cache was the same before and after: sda WCE on, sdb–sdd off). vastblue fixed its side with PostgreSQL on tmpfs (run 11033 green, pytest 14m17s vs 7–8 min per 72-test batch); albok is adding pytest-timeout + durations in 0.1.19. **Prime ruled A (2026-10-12 ~1715): an opt-in fast runner on nh3-docker.** **Done 1725** (`nh3-docker-runner`, label `nh3-docker`, no docker socket in jobs, a 4 GiB cap). vastblue's pytest went 14m17s → 4m45s; albok is green (run 11051). ⚠ albok's failures were the slow fsync, never a hang: my 10-12 "hangs" diagnosis was wrong. The ana-docker vCPU-type question (x86-64-v2-AES, no AVX2) is DROPPED: the measured bottleneck was disk fsync plus a hanging network test, not CPU, so nothing rides on it.
|
||
- `[2026-10-11]` **worldtree-dev is reading mail again (2219), answering four of our threads at once**: #416 residual REOPENED for a second writer pass (a filer-side deterministic expiry guard, U4 rev 1.17, after #422/#423; acceptance = 0 leaks over 10 daily batches with the filing/user median in band). Keep the daily report's "known #416 residual" classification until then. The writer/gate never call embed/rerank and the gate fails CLOSED, so the 10-04 leak was the writer's judgment. No confirmatory batch for 10-09 (10-10/10-11 = 0.83). Nothing of theirs sets a small max_tokens on gen-frontier. This makes Prime's "who sees gate failures" question moot. Thread `01M4BDETDGMNVDTFXN3T3Y63DP`.
|
||
- `[2026-10-11]` **U13 cutover executed on Prime's go (0721) + his four rulings** — E1 0733, E2 steps 1–6 0734–0816, URGENT acceptance test 1350; fv-ml1 evidence capture, stale snapshots, Mac keys, no key rotation. → `persistent-memory.d/2026-10-11-u13-cutover-executed.md`
|
||
- `[2026-10-11]` **fv-ml1 hard hang ~0005, self-reset, no panic record; pre-patch snapshot HELD** — and infra-ops' soft-off that landed mid-boot. → `persistent-memory.d/2026-10-11-fv-ml1-hard-hang.md`
|
||
- `[2026-10-10]` **Prime's U13 rulings: roles dev+ops, artifact label KEPT nh3-svos (whole app dirs copied at E2), relay `matrix-relay`.** → `persistent-memory.d/2026-10-10-u13-rulings-p1-p3.md`
|
||
- `[2026-10-10]` **U13 phase B built (B1–B6)** — kernel 0.0.9/schema 12, `svos-kernel.phasefinal.com` (nh3-dev only, method-pinned paths), nh3-dev origins + apps env, enrollment secrets. → `persistent-memory.d/2026-10-10-u13-phase-b-built.md`
|
||
- `[2026-10-10]` **esh-nas-pve `nvme/ROOT@pre-pve9` destroyed after a clean week** (the two older snapshots followed on 10-11, Prime). → `persistent-memory.d/2026-10-10-esh-nas-pve-pre-pve9-destroyed.md`
|
||
- `[2026-10-09]` **vh gets every albok read group + four Mac keys (Prime: "albok-everything")** — no net new root path. → `persistent-memory.d/2026-10-09-vh-albok-everything.md`
|
||
- `[2026-10-09]` **nh3-dev: orphaned svos pytest killed 2242** (14.4 GB RSS, +2.3 GB/h, swap 100%, Beszel 90% alert; SIGTERM ignored, SIGKILL; used 25.4 → 11.3 GB). highseat-dev's retry-loop test with an instant fake sleep; they added pytest-timeout 300 s and a 50-attempt bound.
|
||
- `[2026-10-09]` **albok `.browse/` sshfs withdrawal test PASSED (Linux) + post-clear checks closed** (fv-ml1 plateau, nh3-nas 71%, pre-pbs4 mislabel). → `persistent-memory.d/2026-10-09-post-clear-checks-albok-withdrawal-pass.md`
|
||
- `[2026-10-09]` **albok `library` building (fiction, nonfiction) live for unit 4.5; Nemi + albok-dev re-minted with library grants (token grants are immutable), old tokens revoked 1555 by albok-dev.** → `stacks/albok-service/README.md`
|
||
- `[2026-10-09]` **Homepage: The High Seat's description no longer says "SVOS" (Prime: author it yourself); SVOS's now says "Internal network + mesh".**
|
||
- `[2026-10-09]` **SVOS app-traffic ingress for U13 approved (shape agreed with svos-dev)** — EXECUTED 10-10 (phase B) and 10-11 (the edit). → `persistent-memory.d/2026-10-09-u13-ingress-approved.md`
|
||
- `[2026-10-09]` **nh3-svos → kernel 0.0.8 (`b588510`, a bounded SQLite reader pool): the 10-08 22:55 EMFILE outage is fixed at the source and the LimitNOFILE stopgap is removed.** A 4-min browser soak held `svos.db` handles flat at 9–12 (0.0.7 grew +22 in 2 min). Root cause: one cached connection per short-lived anyio worker thread. → `servers/nh3-svos/README.md`
|
||
- `[2026-10-08]` **nh3-svos refreshed to ac42ace (interface UI 0.0.7 polish + kernel 0.0.7 with the `Type=notify` unit as a pair); thaw race FIXED**: the pre-refresh drill (old kernel) made two apps fail at thaw, the post-refresh drill made 0. Previous release is now a737a06; 6c6abbe is deletable. → `servers/nh3-svos/README.md`
|
||
- `[2026-10-08]` **SVOS voice ON (Prime, via svos-dev): app-voice on 127.0.0.1:8402, apps up 4 + terminal degraded; Connect → "Voice connected", Enable lists the gateway's 30 voices (Chromium over the LAN door).** `/voice/*` is routed BEFORE Basic auth, because the browser sends `Authorization: Bearer <lease>` there (`web/src/lib/voice.ts`) and Basic auth would reject it. All four app credentials are now vaulted as `nh3-svos/svos-<app>-credential`. → `servers/nh3-svos/README.md`
|
||
- `[2026-10-08]` **SVOS served on the LAN too (Prime):** Caddy binds `10.100.50.43:443` as well as `100.64.0.9:443`; the internal AdGuards answer `svos`/`svos-terminal.phasefinal.com` → `10.100.50.43` (MagicDNS still answers 100.64.0.9). Verified from LAN hosts at all three sites. → `servers/nh3-svos/README.md`
|
||
- `[2026-10-08]` **fv-ml1 is OFF the mesh again, by design**: its tailscaled is disabled, and the break-glass watchdog joins only on fleet loss (09-15 design, archived). The 0030 reboot closed a door that had been left open, so mesh-side tests need another peer, not fv-ml1. The watchdog timer is active.
|
||
- `[2026-10-08]` **heid's runaway sweep killed on Prime's order: two `heidnotes/sweep.py` processes had written ~76 GB into a deleted-but-open file; nh3-dev root 86% → 66%** — the sweep read its own output. heid fixed it at the source. → `persistent-memory.d/2026-10-08-heid-runaway-sweep-killed.md`
|
||
- `[2026-10-08]` **SVOS post office LIVE on nh3-svos** — Prime declared `svos`; its `ns:notifications` write grant was approved on his word at 0731 (card `card:4d3a8b26c17f472d828b06fd495192db`); a test message was ingested and delivered; `reach` ok, unread 0. Reported to svos-dev.
|
||
- `[2026-10-07]` **No z.ai vision alias on the gateway (Prime).** glm-5.x is text-only at z.ai; glm-4.6v and glm-5.3-flash read images on our plan but stay off the gateway.
|
||
- `[2026-10-06]` **Scriberr diarization fixed: gated pyannote, a torchcodec/torch mismatch, and WhisperX's CPU defaults; model caches on /tank, HF_TOKEN a placeholder (Prime), GPU+speaker profiles.** Patch 0003 waits for the next rebuild (tracked at `stacks/scriberr/patches/README.md`). → `persistent-memory.d/2026-10-06-scriberr-diarization-fixed.md`
|
||
- `[2026-10-06]` **Backup checker: destroyed guests reported as residue, an ssh stdin swallow fixed, snapshot age from the name; residue pruned (Prime); pbs-nh3 mirror keeps pbs-ana's history (Prime "match history"), first prune 10-07 00:00.** → `persistent-memory.d/2026-10-06-backup-checker-fixes-and-mirror-retention.md`
|
||
- `[2026-10-08]` **Patch night (Prime's go): fv-ml1 kernel/NVIDIA/ZFS/toolkit + Docker 29.8.2 on esh-docker-vm and nh3-docker, all verified** — ZBM boot, the hand-verified NVIDIA driver path, holds re-applied. → `persistent-memory.d/2026-10-08-patch-night-fv-ml1-kernel-nvidia-zfs-docker-hosts.md`
|
||
- `[2026-10-07]` **nh3-svos refreshed to SVOS a737a06 (schema 11): knowledge, terminal (own origin) and post office apps** — acceptance 17/17 + Chromium; the thaw race. → `persistent-memory.d/2026-10-07-nh3-svos-refresh-a737a06-apps.md`
|
||
- `[2026-10-07]` **Gateway: glm-5.3 added, gen-frontier pair moved to 5.3 (effort low/high), 1M/128K advertised on GLM 5.2/5.3 (Prime)** — `thinking: disabled` is light thinking, not zero. → `persistent-memory.d/2026-10-07-gateway-glm-5-3-gen-frontier.md`
|
||
- `[2026-10-07]` **Hermes launchers repaired (Miranda relay 8462):** svos-dev's scratch probe had rewritten `~/.hermes/hermes-agent/.hermes/bin/{hermes,hermes-acp}` to exec a /tmp scratch Python. Repointed to `~/.hermes/tools/python-3.14.7…`; the gateway was not restarted. ⚠ Don't run `git` in `~/.hermes/hermes-agent`: a `status` there looped on auto-gc.
|
||
- `[2026-10-07]` **albok-service 0.1.14 (unit 4.3) deployed + all four wings reindexed** — the `.browse/` view exists on the box only; the share test is owed. → `persistent-memory.d/2026-10-07-albok-service-0-1-14.md`
|
||
- `[2026-10-07]` **albok-service 0.1.13 (unit 4.2, the sensitivity gate) deployed** — rollback to 0.1.12 needs the strip snippet first. → `persistent-memory.d/2026-10-07-albok-service-0-1-13-sensitivity-gate.md`
|
||
- `[2026-10-06]` **albok-service 0.1.11 (0202) then 0.1.12 (1254) deployed on nh3-albok at albok-dev's asks 7991/8199**, built from clean archives, pinned by digest, single-file upload (deploy-stack's `--delete` would wipe the on-host rollback copies). Every wing is now migrated to generation collections, so a 0.1.10 rollback needs a full reindex. albok's 2.01×-vs-2× load check awaits Prime. → `stacks/albok-service/README.md`
|
||
- `[2026-10-06]` **The LiteLLM gateway keeps no request/response content, and `store_prompts_in_spend_logs: false` is now a marked consumer guarantee (comfy-dev, arbo incognito).** Proved with a canary plus positive controls; per-request `no-log` does NOT cover the proxy's own spend-log callback (v1.97 source). irv-ml1 restic also excludes `input/incognito-*` (dry-run proved with a canary and a control). `847424c`, `0f6c7a0`.
|
||
- `[2026-10-06]` **jekyll moved from gen-large to sec-reasoning (Prime)** after its ~180k-token agent loop and Prime's ~140k session overflowed flash-next's KV. → `persistent-memory.d/2026-10-06-gen-large-contention-jekyll.md`
|
||
- `[2026-10-06]` **vh/rea created on gitea as a private pull mirror of github.com/morluto/rea (8h, wiki + LFS), matching the vh/ mirror convention.** Content verified (main sha, 20 branches, 29 tags).
|
||
- `[2026-10-05]` **fv-ml1: vLLM KV pinned in bytes on every shared-card seat, GPUs 0/1 rebalanced by measured usage, contexts at published (native) maximums, gen-small default output cap 32,768; YaRN stays OFF (Prime).** → `persistent-memory.d/2026-10-05-fv-ml1-kv-pins-and-rebalance.md`
|
||
- `[2026-10-05]` **Park 75 (TTS stack → fv-ml1) DROPPED (Prime, 2115):** its trigger fired 09-25, but intern-decision used the room, and the cross-site-hop objection still stands. Henge comment 63.
|
||
- `[2026-10-05]` **nh3-svos (VM 111, nh3-pve-2) built for the new SVOS (svos-dev relaying Prime); front door = mesh-only `https://svos.phasefinal.com` with HTTP Basic auth, login `Vuong` (Prime, direct)** — WhoIs gate retired (all mesh nodes are login `pfi`; Prime declined a mesh restructure). Password vaulted `nh3-svos/basic-auth-vuong`. qemu-ga freeze hooks were never run on Debian; now `configs/qemu-ga/fsfreeze-dispatch` (fails the freeze on a hook failure). Drill 2026-10-05: quiesced backup ok (~1–2 s app outage); ⚠ a REFUSED freeze does not abort vzdump (ERROR line, unfrozen backup, job "successful"); the backup check now flags that as DEGRADED (exit 4) and covers nh3-pve-2, which it had never read. Offline shell (+ previous release's assets) served before auth; rerun `svos-front-inventory` after every SVOS release. Hosting bounds 1 MiB / 15 s body / 5 s headers. → `servers/nh3-svos/README.md`.
|
||
- `[2026-10-05]` **ESH Homepage also on https://10.0.50.45 (Prime)** — a Traefik file-provider router (`homepage-https.yml`, `tls: {}`) on esh-docker-vm serves Traefik's default cert; the ESH ACME resolver (Route53) is dead, so a trusted name is Prime's open call. (`2c4c7ca`)
|
||
- `[2026-10-05]` **bench-jev servers killed and removed, their HF cache cleaned (Prime)**; pfi-pteradactyl fixed + patched + engine upgraded (`3429b5c`; manual dockerd + wings restart).
|
||
- `[2026-10-05]` **fv-ml1 Docker → tank zvol + host patched (Prime, park 99 DONE)** — run 1 aborted on its own guard (live rsync walked into an overlay `merged` view; seats down 0218–0228, rolled back); run 2 ok 0231–0240 (`-x`, merged excluded, path-list verify, auto-restore trap). Kernel 6.12.111 broke NVIDIA DKMS → purged + kernel metas removed; ZFS stack + nvidia-container-toolkit held. Root 49%→8%. → `servers/fv-ml1/README.md`
|
||
- `[2026-10-05]` **ana-docker: Docker CE repo bullseye → bookworm, 29.8.2 (Prime)** — with `live-restore: true` the engine upgrade kept containers up, but 6 that bind-mount `docker.sock` (crowdsec, traefik, dozzle, beszel-agent, dockge, gitea-runner) went stale and needed restarts. → `servers/ana-docker/README.md`
|
||
- `[2026-10-05]` **RMM wildcard cert now auto-renews (Prime)** — `*.phasefinal.com` on pfi-tacticalrmm was certbot `manual` (could never renew; was 16 days from expiry). Now dns-cloudflare with a phasefinal.com-only token (vault `pfi-tacticalrmm/cloudflare-certbot-dns01-token`); reissued to 2027-01-03, dry-run OK. → `servers/pfi-tacticalrmm/README.md`
|
||
- `[2026-10-05]` **CrowdSec SQLite → WAL on ana-docker (0418)** — three public 403 blips (0310/0342/0415) were the Traefik bouncer failing closed while LAPI took 26–58 s per stream request (`use_wal: false`, blocklist-mirror polling). WAL brought it to 0.04–2.4 s. → `servers/ana-docker/README.md`
|
||
- `[2026-10-05]` **Fleet guest patching (park 100) DONE 0355** — 15 hosts current, no reboots; parked: pfi-pteradactyl (pre-broken), pfi-tacticalrmm (certbot failing), pfi-gx10 (dgx-release + bootloader + kernel), ana-nas, irv-ml1 (Prime: off-limits). Deferred: kernel reboots (6 hosts), Docker engine bumps (8 hosts), fv-ml1 ZFS/toolkit/kernel. → `docs/runbooks/fleet-guest-patch-2026-10-05.md` § Result — `[2026-10-05 later]` reboots all done except nh3-dev; engines done except nh3-docker + esh-docker-vm (held); pfi-pteradactyl, pfi-tacticalrmm, pfi-gx10 done; see in-flight.
|
||
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
|
||
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
|
||
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
||
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
||
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
||
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99). — `[2026-10-05]` DONE (entry above).
|
||
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100). — `[2026-10-05]` DONE (entry above).
|
||
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
|
||
|
||
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
||
- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone** → `persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md`
|
||
- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** → `persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md`
|
||
- `[2026-10-03]` **Worldtree memory-gate duty REFRAMED: regression check, not a gate** → `persistent-memory.d/2026-10-03-worldtree-memory-gate-duty-reframed-regression-check-not-a-gate.md`
|
||
- `[2026-10-03]` **albok-service 0.1.2 LIVE on nh3-docker (albok-dev ask 6955, 0251), fixing a leak that had broken 0.1.1** → `persistent-memory.d/2026-10-03-albok-service-0-1-2-live-on-nh3-docker-albok-dev-ask.md`
|
||
- `[2026-10-02]` **esh-pve-2 (MS-03 at ESH, future `esh-dev` host) onboarded: infra-ops (Prime), 1 TB → LVM-thin `vmstore`, AMT phoning home as `esh-pve-2-amt`** → `persistent-memory.d/2026-10-02-esh-pve-2-ms-03-at-esh-future-esh-dev-host-onboarded-infra-ops.md`
|
||
- `[2026-10-02]` **Demo outage ~13 min, ROLLED BACK (worldtree-dev URGENT 6684)** → `persistent-memory.d/2026-10-02-demo-outage-13-min-rolled-back-worldtree-dev-urgent-6684.md`
|
||
- `[2026-10-02]` **nh3-pve's AMT is in TacticalRMM's MeshCentral (group `PFI-AMT`, device `nh3-pve-amt`)** → `persistent-memory.d/2026-10-02-nh3-pve-s-amt-is-in-tacticalrmm-s-meshcentral-group.md`
|
||
- `[2026-10-02]` **Prime: dev-backup gets dailies + weeklies — DONE** → `persistent-memory.d/2026-10-02-prime-dev-backup-gets-dailies-weeklies-done.md`
|
||
- `[2026-10-02]` **ESH `tank` (esh-nas-pve, 175 T raidz2×2): was DEGRADED since ~2026-08-20 (found while planning PVE 9); HEALTHY since 2026-10-03 0629** → `persistent-memory.d/2026-10-02-esh-tank-esh-nas-pve-175-t-raidz2-2-was-degraded.md`
|
||
- `[2026-10-02]` **albok-service 0.1.0 LIVE on nh3-docker (albok-dev ask, operator-approved): `http://albok.nh3.internal:8392`** → `persistent-memory.d/2026-10-02-albok-service-0-1-0-live-on-nh3-docker-albok-dev-ask.md`
|
||
- `[2026-10-02]` **nh3-pve-2 status at end of day: AMT done, Proxmox on the WRONG NVMe; Prime reinstalls ON SITE 2026-10-03** → `persistent-memory.d/2026-10-02-nh3-pve-2-status-at-end-of-day-amt-done-proxmox.md`
|
||
- `[2026-10-02]` **Prime: MS-03s get Proxmox; dummy plugs in hand. One MS-03 is being deployed as `nh3-pve-2` at NH3 (Prime, ~0910)** → `persistent-memory.d/2026-10-02-prime-ms-03s-get-proxmox-dummy-plugs-in-hand-one.md`
|
||
- `[2026-10-02]` **nh3-dev root grown 250 → 378 GB (Prime resized scsi0; I grew the guest online, no reboot)** → `persistent-memory.d/2026-10-02-nh3-dev-root-grown-250-378-gb-prime-resized-scsi0.md`
|
||
- `[2026-10-01]` **MIA (Make-It-Animatable v2) auto-rigger LIVE on fv-ml1 GPU 3, one-shot (Prime via dread-dev)** → `persistent-memory.d/2026-10-01-mia-make-it-animatable-v2-auto-rigger-live-on-fv-ml1-gpu-3.md`
|
||
- `[2026-10-01]` **Prime: "clean up the 1700+ backups" — dev-backup retention FIXED, 1,740 husks removed** → `persistent-memory.d/2026-10-01-prime-clean-up-the-1700-backups-dev-backup-retention-fixed.md`
|
||
- `[2026-10-01]` **Prime: gen-small keeps its 8 GiB KV pin, and parakeet-nemo keeps CUDA graphs OFF** (~2–8 ms at short clips, accepted). GPU 0's ~1 GB of spare memory is enough with the seat's hard 3,840 MiB cap.
|
||
- `[2026-10-01]` **⚠ Gitea's `[webhook] ALLOWED_HOST_LIST = external, 10.0.0.0/8` silently REJECTS headscale mesh IPs (100.64.0.0/10)** → `persistent-memory.d/2026-10-01-gitea-s-webhook-allowed-host-list-external-10-0.md`
|
||
- `[2026-10-01]` **vh/arbo push webhook repointed from the retired wg0 lifeline `10.100.79.3:9009` to irv-ml1's mesh address `100.64.0.6:9009`** → `persistent-memory.d/2026-10-01-vh-arbo-push-webhook-repointed-from-the-retired-wg0.md`
|
||
- `[2026-10-01]` **Prime: delete the bench leftovers, no upstream for Scriberr, push** → `persistent-memory.d/2026-10-01-prime-delete-the-bench-leftovers-no-upstream-for-scriberr.md`
|
||
- `[2026-10-01]` **irv-ml1 /storetank reclaim done:** Prime ruled through comfy-dev, which deleted 84 files of its own (260 → 299 GB free). Tiers B/C/D got no ruling (infra-hermes thread `01M3TCSYRSFNAPA9BPQTMFQ6KJ`).
|
||
- `[2026-09-30]` **Parakeet speech seat → `parakeet-unified-en-0.6b` under NeMo (bf16) APPROVED by Prime, NVIDIA Open Model License accepted. DONE 2026-10-01 0126 by infra-hermes; infra-ops audit passed 0137.** → `persistent-memory.d/2026-09-30-parakeet-seat-switch-approved.md`
|
||
- `[2026-09-30]` **Worldtree U11a: legacy memory plane OFF on demo and personal. The U11b data deletion is gated on 3 consecutive PASS and step 5 is mine; the legacy archive must be destroyed by 2026-10-30.** → `persistent-memory.d/2026-09-30-worldtree-u11a-off-u11b-gate.md`
|
||
- `[2026-09-30]` **SemIf replaced by intern-decision (Intern-Decision-4B, the Jev bench pick): semif-compatible plus Jev `/v1/systemone` at 32k tokens on GPU 1, with a Triton warm-up cache volume.** → `persistent-memory.d/2026-09-30-semif-replaced-by-intern-decision.md`
|
||
- `[2026-09-30]` **Scriberr moved to GPU 3 (on demand); our build carries the overlap slicer (0001) and the Parakeet gap retry (0002); v3 kept.** → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md`
|
||
- `[2026-09-29]` **Worldtree U11a prepped, not flipped: a staged demo config plus the agreed U8 window plan (infra-hermes runs the batches).** → `persistent-memory.d/2026-09-29-worldtree-u11a-prepped.md`
|
||
- `[2026-09-28]` **Worldtree U10 backfill done on demo (5) and personal (797). model_roles drift needed a memory_tagger sync first; mimir had missing vectors.** → `persistent-memory.d/2026-09-28-worldtree-u10-backfill.md`
|
||
- `[2026-09-28]` **Bonsai ternary vs Q4_K_XL at concurrency on the 275 W card (1.93x at N=1 falls to 1.06x at N=8, 1.21x with the MMVQ fix); the weights are acquired.** → `persistent-memory.d/2026-09-28-bonsai-ternary-spike.md`
|
||
- `[2026-09-28]` **blender-run gained `--cpu` (no GPU attached) and a fixed hostname `fv-ml1-blender` (draupnir).** The design stage never renders, so it stays off GPU 3.
|
||
- `[2026-09-28]` **Blender extensions live in a read-only System repo built from a sha256 lock, enabled by a hook, opt-in for blender-run (`--extensions`).** SurfacePsycho's eval() is patched to literal_eval (a proven safe-mode escape). → `stacks/blender/README.md` § Extensions
|
||
- `[2026-09-27]` **hermes-gateway restarted 0401 for highseat-dev** → `persistent-memory.d/2026-09-27-hermes-gateway-restarted-0401-for-highseat-dev.md`
|
||
- `[2026-09-27]` **SemIf LIVE on fv-ml1 GPU 1 (semif-serve 0.1.2, Prime):** wrapper + contract + 39 tests, 142/144 upstream parity, two card-only memory defects fixed. → `persistent-memory.d/2026-09-27-semif-live-on-fv-ml1-gpu1.md`
|
||
- `[2026-09-27]` **Blender 5.2 on fv-ml1 GPU 3 (on demand), agent-driven via mcp-for-blender running in-container over ssh stdio; safe mode on, no published port.** → `persistent-memory.d/2026-09-27-blender-on-fv-ml1-gpu3.md` (canonical: `stacks/blender/README.md`)
|
||
- `[2026-09-27]` **Worldtree reward path: d8726a13 + `providers.yaml` `reward_models.scalar-judge` live on demo; personal waited on a newer image.** → `persistent-memory.d/2026-09-27-worldtree-reward-path-live.md`
|
||
- `[2026-09-27]` **esh-docker-vm host→HA traffic needs a /32 over macvlan-shim (if-up.d hook); without it HA silently loses MQTT (3× since August).** → `servers/esh-docker-vm/README.md`
|
||
- `[2026-09-27]` **Zigbee2MQTT live on esh-docker-vm :8099 (PAN 0xCFF4, ch 25) replacing ZHA; network key vaulted, data host-only.** Prime go-ahead in this session; a peer-relayed approval was blocked by the permission gate. → `persistent-memory.d/2026-09-27-zigbee2mqtt-on-esh-docker-vm.md` (canonical: `stacks/zigbee2mqtt/README.md`)
|
||
- `[2026-09-27]` **semif-serve 0.1.4: object states ending in `)`, `;` or `}` no longer 422 (INV-7, a prefix wrapper proven at startup); numerics are deterministic within a process but a bf16 near-tie can flip across a restart.** Prime ruled; heid bug hunt folded. → `stacks/semif/README.md`
|
||
- `[2026-09-27]` **SemIf as Cicada's mood source: slower (+32 ms async, +94 ms sequential) and worse (67% vs 92% apt; carry 7/15 vs 14/15); only the gesture restraint is a win.** Build nothing (Prime). Henge 88 carries it. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||
- `[2026-09-27]` **SemIf consumer-fit spikes (Prime): Cicada affect gate 30/31 with descriptive wording and 19/31 terse; Wyrd "left this place?" 21/21 on the second wording, exit choice 18/21.** Recommendations await Prime. → `persistent-memory.d/2026-09-27-semif-consumer-fit-spikes.md`
|
||
- `[2026-09-27]` **SemIf order-averaging spiked (+9.1 pts accuracy, agreement = strong confidence signal); Prime ruled: build it in as 0.1.3 and trial the fast kernels.** Tracked in the in-flight SemIf section. → `persistent-memory.d/2026-09-27-semif-order-averaging.md` — **DONE:** 0.1.3 live, with fast kernels adopted (`77b8cb4`).
|
||
- `[2026-09-27]` **restic repository URL moved out of world-readable systemd units on all 8 hosts, and vaulted (Prime).** Rotation stays Prime's. → `persistent-memory.d/2026-09-27-restic-repository-file-fleetwide.md`
|
||
- `[2026-09-27]` **infra-ops bootstrapped on vm-esh-nas by Prime** (the first account at the fleet-pinned uid/gid 850; `bootstrap-infra-ops-user.yaml` now pins 850 when free, `d775a01`).
|
||
- `[2026-09-27]` **augaman: second instance on fv-ml1 benched, then REMOVED (Prime).** esh-ml1 on v0.1.3 does 48 ms/face in the house with the verified backup (`ad484c3`, bench `docs/pfi/augaman-speed-bench/`).
|
||
- `[2026-09-27]` Created empty private repo `corviduo/svoperatingsystem` (SVOS, the operating system) as claude-bot, at brokkr-smithy-dev's relay of Prime's ruling. `svos-dev` → the OS session; The High Seat's session is `highseat-dev` again.
|
||
- `[2026-09-26]` **augaman: infra-ops owes the gallery backup + first deploy — DEFERRED until augaman-dev reaches deploy.** → `persistent-memory.d/2026-09-26-augaman-infra-ops-owes-the-gallery-backup-first-deploy.md` — `[2026-09-27]` **DONE:** deployed (v0.1.3 on esh-ml1), backup wired, identity-level restore verified (`d8f59a1`).
|
||
- `[2026-09-26]` **`zellij-fleet@.service` installed, NOT enabled** (for svos-dev's seat_up; owns the fleet zellij server in its own cgroup). Enabling `@Claude` at boot is Prime's call when seat_up ships. Tracked: `0ad7799`, `services/zellij-fleet/README.md`.
|
||
- `[2026-09-26]` **btusb blacklist on nh3-pve + esh-pve — proposed, awaiting Prime (low priority).** Every 6.8.12-4x boot oopses in btmtk (benign). Also noted for the next NH3 visit: nh3-pve runs ~20 °C warmer than esh-pve (workload-confounded; check airflow). Tracked: `servers/nh3-pve/README.md`.
|
||
- `[2026-09-25]` **AMT follow-ups PARKED until the two MS-03s arrive and esh-pve's MS-01 is on AMT (Prime 2326).** — Parked: MeshCommander test; dummy HDMI plug, then NanoKVM → gx10; an OOB path not through nh3-scale… → `persistent-memory.d/2026-09-25-amt-follow-ups-parked-until-the-two-ms-03s-arrive-and-esh.md`
|
||
- `[2026-09-26]` **esh-matter LIVE: a Matter server (matter.js 1.4.0) on CT 111 @ 10.0.90.20, VLAN 90 only** — , for ha-dev (operator-approved, relayed). → `persistent-memory.d/2026-09-26-esh-matter-live-a-matter-server-matter-js-1-4-0-on-ct-111.md`
|
||
- `[2026-09-26]` **Embed/rerank LOAD-SHARED across esh-ml1 + nh3-ml1 (Prime).** — Second deployments were added for qwen3-embedding and reranker (config) and for reranker-a3-bge-v2-m3 (DB… → `persistent-memory.d/2026-09-26-embed-rerank-load-shared-across-esh-ml1-nh3-ml1-prime.md`
|
||
- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** — LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and… → `persistent-memory.d/2026-09-26-two-dataset-foundry-utility-seats-live-on-nh3-ml1-brokkr.md`
|
||
- `[2026-09-25]` **nh3-pve OOB: HOLD blind until the next NH3 visit (Prime via Miranda, 1333).** Target: NanoKVM → pfi-gx10, and the MS-01 on its own vPro. The IGFX BIOS fix rides the same visit, since AMT KVM captures only the iGPU. Tracked: `servers/nh3-pve/README.md` (`2b49be4`, `cdd7605`).
|
||
- `[2026-09-25]` **Direct ESH→esh-ml1 consumer path — PARKED** (no ESH-side callers in 7 days; consumers go through the gateway). Tracked at `persistent-memory.d/2026-09-24-esh-ml1-embed-rerank.md`.
|
||
- `[2026-09-24]` **Worldtree `admin.memory.forget` stays OFF on demo/personal** until an instance needs it — a destructive erase; enabling it is a per-change operator yes via deploy-wt-config.
|
||
|
||
|
||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
||
|
||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md`
|
||
|
||
- `[2026-09-13]` **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** → `persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md`
|
||
|
||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** → `persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md`
|
||
|
||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||
|
||
_240 older entries archived to archival-memory.md._
|
||
|
||
## Tried and abandoned
|
||
|
||
- `[2026-10-08]` **NVIDIA's network debian12 repo on trixie via `cuda-keyring`** — trixie's sqv rejects the key's SHA1 binding signature, so `apt update` errors until the list is gone (`dpkg -r` keeps the conffile; it took `dpkg -P`). Fetch the debs and verify InRelease (gpgv) → Packages.gz → sha256 instead. → `servers/fv-ml1/README.md`
|
||
- `[2026-10-08]` **`secret get <name> | head` "to see its shape"** — a single-value secret IS its shape; it printed the fv-ml1 BMC password into the local transcript. Check with `secret list` or `wc -c`, never by printing.
|
||
- `[2026-10-07]` **An idle gate counting with `grep -c … || echo ERR`** — `grep -c` exits 1 on a ZERO count, so the value was "0\nERR" and never equalled "0"; two 75-min LiteLLM restart gates were blind to any quiet window. Use `|| true` and test the gate against a known-idle state first.
|
||
- `[2026-10-07]` **`pgrep -f <pattern>` to check that a process died** — it matched the checking shell's own command line and reported a survivor; check with `ps -eo … | grep '[x]…'`, as the same self-match bit `pkill -f` before.
|
||
- `[2026-10-06]` **Restarting a user-facing service after an idle check that only PRINTS** — `nvidia-smi … | wc -l` said 1 and the same command restarted Scriberr anyway, killing Prime's transcript run. Make the check gate (`&&` / exit). auto-memory `feedback_idle_check_must_gate_the_restart`.
|
||
- `[2026-10-06]` **Aging PBS snapshots by directory mtime** — verify jobs and notes edits rewrite `index.json.blob`, so a 10-04 snapshot read as 10-06. Age by the snapshot NAME. auto-memory `reference_pbs_snapshot_mtime_moves_on_verify`.
|
||
- `[2026-10-06]` **`deploy-stack.sh` on a stack whose host dir holds rollback copies** (albok-service `compose.yaml.pre-*`) — its rsync `--delete` removes them. Upload the single changed file with elway instead.
|
||
- `[2026-10-05]` **Sizing a vLLM seat's KV from `--gpu-memory-utilization` on a shared card** — co-booting neighbours corrupt the profile (non-torch −0.56 GiB), so the KV is a lottery per boot. Pin `--kv-cache-memory`. auto-memory `reference_vllm_util_kv_coboot_race`.
|
||
- `[2026-10-05]` **Reading GPU headroom just after a restart** — 10-01's "Free ~1.05 GB" was really a ~60 MiB plateau 2 days later. Size against Beszel history. auto-memory `feedback_vllm_headroom_at_plateau_not_boot`.
|
||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
|
||
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
|
||
- `[2026-09-30]` **Start-time midpoint stitching of overlapped Parakeet chunks.** It duplicated a word at 26 of 108 stitches, because Parakeet timestamps a post-pause word anywhere inside the pause. Hand over at a word both chunks agree on instead.
|
||
- `[2026-09-30]` **int8 ONNX (sherpa-onnx) as the low-latency Parakeet runtime.** The int8 graph runs on ONE CPU thread with the GPU at 2–9%. Unified-en int8 was slower than the seat; fp32 ONNX was 4–12× faster, and NeMo was fastest.
|
||
- `[2026-09-30]` **GPU budgets computed as total − used.** nvidia-smi `Free` is ~640 MiB lower per card (driver reserve). Budget from `Free`.
|
||
- `[2026-09-27]` **The Blender MCP server on nh3-dev, reaching the add-on socket over an SSH tunnel** → `persistent-memory.d/2026-09-27-the-blender-mcp-server-on-nh3-dev-reaching-the-add-on.md`
|
||
- `[2026-09-27]` **A TCP connect as the "is Blender ready" probe** → `persistent-memory.d/2026-09-27-a-tcp-connect-as-the-is-blender-ready-probe.md`
|
||
- `[2026-09-27]` **`log.exception()` in a GPU failure path** — the record keeps `exc_info`, so any retaining handler (pytest's capture does) pins the traceback's frames and tensors. Log `traceback.format_exc()` text instead (semif-serve `engine._guard`).
|
||
- `[2026-09-27]` **fla/triton in a slim image without gcc** — triton compiles its CUDA driver shim at runtime ("Failed to find C compiler"). The warm-up died and startup failed closed. The semif image now installs gcc + libc6-dev with the fast extra.
|
||
- `[2026-09-27]` **`uv export` → requirements.txt → `uv pip install` for a torch-from-cu128 project** → `persistent-memory.d/2026-09-27-uv-export-requirements-txt-uv-pip-install-for-a.md`
|
||
- `[2026-09-27]` **Translating a CUDA OOM by raising inside `except` (or `from exc`)** — the chained exception's traceback pins the failed call's frames, and their GPU tensors (11.9 GiB after the 503). Raise after the block, unchained, `gc.collect()` before `empty_cache()`.
|
||
- `[2026-09-27]` **`deploy-stack.sh --yes` / `dns-sync.py` fed a blind `y`** — the harness refuses a blind apply. Review with `echo n |` first, then apply with `echo y |`.
|
||
- `[2026-09-27]` **Relying on the build cache surviving on esh-ml1** → `persistent-memory.d/2026-09-27-relying-on-the-build-cache-surviving-on-esh-ml1.md`
|
||
|
||
_128 older entries archived to archival-memory.md._
|