diff --git a/archival-memory.md b/archival-memory.md index 0b8c84d..d5efecc 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -345,6 +345,24 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re service. _Archived 2026-06-13._ +- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** Persistent box services there need SYSTEM systemd units (see Tried). + _Archived 2026-06-14._ + +- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI). + _Archived 2026-06-14._ + +- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (`d8d776c`, auto-memory `reference_worldtree_deploys_cicd`) + _Archived 2026-06-14._ + +- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (`8c32a05`, auto-memory `reference_infra_ops_sudo_identity`) + _Archived 2026-06-14._ + +- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's** (Ollama's omits the system `<|end|>`). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; "headgun" lesson in Tried.) + _Archived 2026-06-14._ + +- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision] + _Archived 2026-06-14._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with @@ -736,3 +754,18 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re restart gitea (act_runner job containers survive a restart). gitea runs as a container on ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`). _Archived 2026-06-13._ + +- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state can come from dead `siteMonitor` widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`) + _Archived 2026-06-14._ + +- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and **R/soundgen `-lgfortran` fails** on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install `libgfortran-11-dev`. + _Archived 2026-06-14._ + +- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with enable-linger). Use `/etc/systemd/system/`. + _Archived 2026-06-14._ + +- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. + _Archived 2026-06-14._ + +- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`) + _Archived 2026-06-14._ diff --git a/persistent-memory.md b/persistent-memory.md index d3186a0..5a65072 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-13_ +_Last updated: 2026-06-14_ ## Repo purpose @@ -25,6 +25,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | | `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` | +| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (catalog-only restart; infra side = `stacks/arbo/`) | (`vh/volva` + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev `.service` units were removed — @@ -68,6 +69,13 @@ no longer deployed sidecars here. See Recent decisions.) needs `docker login gitea.phasefinal.com` once (read:package PAT) — not currently in the workflow. +- **gitea internal route for fleet hosts.** gitea is a container on + **ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo + hosts must use this internal route, NOT public `gitea.phasefinal.com` + (`38.120.12.44`, ana-srv1) — the public path fail2bans the host egress + IP and wedges webhook deploys. `:22` on `10.250.50.70` is ana-docker's + HOST sshd, not gitea. Full gotcha in `docs/orientation.md` → Git/gitea. + - **docker-as-root pattern** (for ops that have no admin API, e.g. `SqliteUserStore.set_bifrost_credentials`): on hosts where the SSH user is in the `docker` group but lacks passwordless sudo, run @@ -91,289 +99,148 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-06-13:_ +_As of 2026-06-14:_ -- **ana-ml2 went Ada → Blackwell** (dual NVIDIA RTX PRO 6000 Blackwell Max-Q, 96 GB each, cc 12.0 / - sm_120 — was dual RTX 6000 Ada 48 GB / cc 8.9, confirmed live via `nvidia-smi`). GPU layout now: - **GPU 0** held free for large-model hot-loads (llama-swap pinned there, `edf0f91`); **GPU 1** is the - steady-tenant card — granite (131k ctx) + qwen3.5-vl (65k) + the embed/rerank/reward trio, ~3.5 GB - free after the rebalance. CLAUDE.md's server table + the granite/qwen "Ada (cc 8.9)" compose comments - are STALE — doc-fix offered, **pending operator go-ahead** (/snapshot doesn't auto-edit CLAUDE.md). +- **Arbo prod is current + the deploy loop is closed.** Backend **v0.11.6** + / frontend **v0.11.8** (frontend rides the git mount). Auth is OFF + (WireGuard boundary). Pipeline is ban-immune (internal gitea route) + + restart-correct (catalog-only) + version-controlled (`stacks/arbo/`). + OPEN: the gitea registry has **never** held an arbo image — every deploy + is local-image-only on irv-ml1, a rollback SPOF. Operator may mint a vh + `package:write` PAT to backfill; the internal-route fix made it + non-blocking, so NOT done. The public-IP fail2ban ban on irv-ml1's egress + (`38.120.94.3`) on the gitea host — operator was handling the unban, but + the internal-route repoint made it MOOT; ban-cleanup is optional hygiene + (status unverified). -- **R16 "vmoan" TTS LoRA POC — awaiting the operator's ear** (Brokkr's designated gate). Two auditions - on NFS: `/mnt/smithy/scratch/r16-vmoan-pilot/vmoan_audition_v1.wav` (42 s, run-on) vs `_v2.wav` (10 s, - tight). v2 fixed the run-on; Brokkr holds the pilot verdict on the operator's listen. Grab: - `scp irv-ml1:/mnt/smithy/scratch/r16-vmoan-pilot/vmoan_audition_v{1,2}.wav .`. LOCAL-ONLY corpus - (soundgasm-derived) — distribution barred; never echo transcripts to the bus. +- **R16 vmoan pilot is CLOSED** — v1 (default decode) is the final + Chatterbox-tag inline artifact (see Recent decisions). The only forward + thread: a splice-pivot **de-risk yield-probe** brokkr surfaced 2026-06-14 + (45 standalone-NVV clips + speaker-embedding/acoustic metrics) — + **surfaced to operator, awaiting go**; it needs Resemblyzer/ECAPA + + librosa/parselmouth installs in the harness venv. -- **R17 v2 corpus characterization still on the local irv-ml1 branch, push HELD** (`r17-v2-character - ization`, 18 MULTI / 101 ELIGIBLE + 19936 candidates + `speech.json` ASR). No-push rule AND ASR of - the intimate-audio batch (content-exposure call). Awaiting brokkr-collect or explicit push approval. +- **ana-ml2 on Blackwell** (dual RTX PRO 6000 Blackwell Max-Q, 96 GB ea, + cc 12.0). GPU 0 held free for large-model hot-loads (llama-swap pinned); + GPU 1 steady tenants — granite 131k + qwen3.5-vl 65k + embed/rerank/reward + trio, ~3.5 GB free. CLAUDE.md GPU-spec doc-fix LANDED (`355a240`). -- **Mac Pro migration is still the big open project** — `migration-plan.md` (repo root): WORKSTATION- - ONLY move of nh3-dev's Hat-1 dev env (~40 repos, all of `~/.claude`, dotfiles, toolchain) to an M2 - Ultra Mac Pro Rack on the 10.100 subnet. Hat-2 fleet sidecars (egress SOCKS5, mead-hall, ttyd seats, - `althing-forseti`) STAY on the Linux VM. **Phase 0** (push pushables + confirm the ~6 local-only repos) - is the only time-sensitive step. Cutover = `rsync` working trees, NOT re-clone (re-clone loses unsaved - work in ~15 repos). macOS `/home/lkraven`→`/Users/lkraven` repath. Everything else waits on hardware. +- **R17 v2 corpus characterization still on the local irv-ml1 branch, push + HELD** (`r17-v2-characterization`). No-push rule AND ASR-content-exposure + call on the intimate-audio batch. Awaiting brokkr-collect or explicit push + approval. (LOCAL-ONLY soundgasm-derived corpus — distribution barred, + never echo transcripts to the bus.) -- **sglang-vs-vLLM bench stack staged but parked** (`5f049cb`) — the originating question (SGLang - RadixAttention caching) resolved without it: vLLM v1 already defaults prefix-caching ON, now pinned - explicit on granite+qwen. The NVFP4 chase is abandoned. Bench stack stays staged if ever revisited; - the MAX_JOBS-on-shared-prod foot-gun (see Tried) caps any future from-source build on ana-ml2. +- **Mac Pro migration is still the big open project** — `migration-plan.md` + (repo root): WORKSTATION-ONLY move of nh3-dev's Hat-1 dev env to an M2 + Ultra Mac Pro on the 10.100 subnet. Hat-2 fleet sidecars STAY on the Linux + VM. **Phase 0** (push pushables + confirm the ~6 local-only repos) is the + only time-sensitive step. Cutover = `rsync` working trees, NOT re-clone. + Everything else waits on hardware. -- **pi + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against `glm-5.1` (thinking-off) - via the LiteLLM gateway; `glm-5.1-reasoning` for opt-in thinking. +- **sglang-vs-vLLM bench stack staged but parked** (`5f049cb`) — the + originating SGLang-RadixAttention question resolved (vLLM v1 defaults + prefix-caching ON, now pinned explicit on granite+qwen). NVFP4 chase + abandoned. MAX_JOBS-on-shared-prod foot-gun caps any from-source build on + ana-ml2. -- **Worldtree config-propagation lane is mature + humming** — pre-merge delta-ping → sync-on-merge to - the demo+personal bind-mounts; pinned stays pre-cutover (full migration at its next re-image). +- **pi + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against + `glm-5.1` (thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for + opt-in thinking. -- **Disclosed-keys hygiene queue** (rotate at convenience): HF token, the wt-personal keys, Gitea - runner reg token, MINIFLUX_PASSWORD, ZAI_API_KEY. (The shared all-agents LiteLLM key is intentional, - not hygiene-debt — see decisions; rotatable via infra-ops only if it leaks.) +- **Worldtree config-propagation lane is mature + humming** — pre-merge + delta-ping → sync-on-merge to demo+personal bind-mounts; pinned stays + pre-cutover. -- **Still open from prior:** clean legacy `news-digest` on ana-docker; the `docker push 60s ceiling` - mystery uninstrumented. (llama-swap GPU-0 pin — DONE this session, `edf0f91`.) +- **Disclosed-keys hygiene queue** (rotate at convenience): HF token, the + wt-personal keys, Gitea runner reg token, MINIFLUX_PASSWORD, ZAI_API_KEY. + (The shared all-agents LiteLLM key is intentional, not hygiene-debt.) + +- **Still open from prior:** clean legacy `news-digest` on ana-docker; the + `docker push 60s ceiling` mystery uninstrumented. ## Recent decisions -- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / - sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 - tensor cores) and doubles VRAM headroom. CLAUDE.md's server table + the granite/qwen compose - "Ada (cc 8.9)" comments are now stale; doc-fix **offered, pending operator go-ahead** (snapshot doesn't - auto-edit CLAUDE.md). Tracking surface: this snapshot + commits `19a07b9`/`1e2a3a1` ("Blackwell 96GB"). +- `[2026-06-14]` **R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact.** Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + `tokenizer.json.v3bak` preserved on `irv-ml1:~/r16-vmoan-harness`. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread `01KV010WGSSMPWRNCPAGSPK15Y`. -- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** - W4A4 (4-bit weights + activations) collapses at 30k context, proven **producer-independent** (modelopt - AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both - axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup), so - the tempting ~40% uplift doesn't materialize without quality loss. **30B retired** (not enough quality - for the VRAM/perf hit for our use case). (auto-memory `reference_nvfp4_w4a4_granite_infeasible`) +- `[2026-06-14]` **Arbo deploy pipeline fixed, hardened, and version-controlled.** Prod rebuilt v0.11.1 → **v0.11.6** backend; the webhook machinery (`arbo-deploy.sh` + `arbo-webhook.py`, :9009 HMAC listener) is now repo-tracked at `stacks/arbo/` (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (`10.250.50.70:222`) and restarts the engine ONLY on `catalog/` changes (graphs/frontend per-request; warn on `src/`|`Dockerfile` only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in **eshpfi** (not migrated to comfy-dev's repo). Secret + `.env` stay host-only. Tracking: `6d66bc2`, `6e58e57`, `stacks/arbo/README` Q5. -- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway - alias `qwen3.5-9b-fp8`. Clean FP8 (no NVFP4 for vision). **Pinned nightly digest, not `:latest`**: the - stable release quantizes the Qwen3.5-VL *vision tower* under `--quantization fp8` → garbage vision (LM - fine, "sees" noise); the nightly correctly excludes the vision tower. Re-pin to `:latest` + drop the - pin once that exclusion lands stable. (`2e3dcc2`) +- `[2026-06-13]` **Arbo prod bearer auth turned OFF — WireGuard is the access boundary** (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be **ABSENT, not empty** (empty-string still gates) — removed from BOTH the host `.env` AND the compose `environment:` injection line. Original token backed up at `irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak`; comfy-dev updated their ADR-0001. Tracking: `db97899` + `playbooks/arbo-disable-engine-token.yaml`. -- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). - `arbo` is the consuming app (ComfyUI-backed prompt/image pipeline); models live under its name on the - bigger pool. Overlay bind-mount via `COMFYUI_MODELS_DIR` in the comfyui compose; inventory at - `docs/arbo-comfyui-model-catalog.md` (retain-large-portion-for-future-use decision). (`38186be`) +- `[2026-06-13]` **Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G).** Curated `/storetank/image-models/comfy` (was 919 G, the native `/opt/ComfyUI/models` symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into `/storetank/arbo/models` (same-fs move, skip-existing protects prod). Tracking: `docs/storetank-image-models-archive.md` + `docs/arbo-comfyui-model-catalog.md` (`1902425`→`5007ec1`). -- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads - (llama-swap pinned there, `edf0f91`); GPU 1 is the steady-tenant card — granite 131k ctx (was 51k), - qwen 65k, embed/rerank/reward trio, ~3.5 GB free after rebalance (`1e2a3a1`, `19a07b9`; trio re-floored - for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs are chunked BEFORE embedding, - so a longer embed ctx buys nothing. PagedAttention note: max-model-len is a ceiling, not a reservation, - so small requests aren't blocked by the big ceiling; concurrency = KV-pool-tokens / actual-request-size. +- `[2026-06-13]` **GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env** at `~/.arbo_granite_key` (0600) for the hero gen+judge script — verbatim copy of the prod `arbo-prompt-enhance` vkey (now extended to reach BOTH `granite-4.1-8b` AND `qwen3.5-9b-fp8`); nothing minted. The vkey README's "granite-only" wording was stale → corrected (`f32c6dd`). -- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT (45 ms - vs 292 ms) on a shared ~4.5k-token summarizer template; soft/evictable KV, neutral when prefixes don't - repeat. vLLM v1 `:latest` defaults it ON (granite) but the qwen nightly defaults it OFF — pin both so a - version flip can't silently disable it. (`a9a2be7`) +- `[2026-06-13]` **ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q** (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via `nvidia-smi`). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix **LANDED `355a240`** (operator). Tracking: `19a07b9`/`1e2a3a1` ("Blackwell 96GB"). -- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier endpoint + a - shared all-agents key minted** (operator-directed). Added to the global `~/.claude/CLAUDE.md` Global- - tools section; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen- - vision + embed/rerank, NOT the paid GLM), internal-gateway-only, rotatable. The standing "reach for - this before spending premium tokens on low-caliber high-volume work" lever. (auto-memory - `reference_litellm_gateway`) +- `[2026-06-13]` **NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format.** W4A4 collapses at 30k context, proven **producer-independent** (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). **30B retired**. (auto-memory `reference_nvfp4_w4a4_granite_infeasible`) -- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference - engine (`ee57e69`), python-based healthcheck (the slim image ships no curl/wget, `bdb3312`), frontend - ro-mounted from the v0.11.2 checkout (`922e8ad`, ADR-0001 D2). arbo = the ComfyUI-consuming app whose - models now live at `/storetank/arbo`. +- `[2026-06-13]` **Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1** — `qwen35-vl` stack, :8007, gateway alias `qwen3.5-9b-fp8`. **Pinned nightly digest, not `:latest`**: the stable release quantizes the VL *vision tower* under `--quantization fp8` → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (`2e3dcc2`) -- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults - thinking-OFF; new `glm-5.1-reasoning` alias = same z.ai upstream with thinking ON (opt-in). - Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — LiteLLM `drop_params` STRIPS a - top-level `thinking`/`reasoning_effort`, but forwards `extra_body` verbatim to z.ai (the only channel - that works; verified reasoning_tokens 0 vs >0). Shared-gateway change — affects ALL glm-5.1 callers - (brokkr's all-models key included). (`95b2701`, auto-memory `reference_litellm_gateway`) +- `[2026-06-13]` **comfyui 325 G model tree migrated worktank → `/storetank/arbo`** (worktank 97% → 26%). `arbo` is the consuming app; overlay bind-mount via `COMFYUI_MODELS_DIR`. (`38186be`) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.) -- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/ - pi-coding-agent` via **bun** (user-level; npm's global prefix is `/usr` → needs sudo, bun avoids it). - Config `~/.pi/agent/models.json` (litellm provider → gateway), launcher `~/.local/bin/glm` sources - the gateway key + selects the model. pi is OpenAI-compatible; proxy-safe compat flags for the GLM path. +- `[2026-06-13]` **GPU layout settled on the Blackwell box.** GPU 0 held free for large-model hot-loads (llama-swap pinned, `edf0f91`); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (`1e2a3a1`, `19a07b9`; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation. -- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** Direct - `api.z.ai/api/paas/v4/web_search` → 429/1113 "insufficient balance" (coding-plan keys bill tools on a - separate quota path); `open.bigmodel.cn` is the China platform (404, different account). WORKS: MCP - streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer - $ZAI_API_KEY` (the **MCP** key — distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's - Leif agent (`tools/web/zai_client.py` + `core/clients/mcp.py`). +- `[2026-06-13]` **Prefix caching pinned explicit on granite + qwen** — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (`a9a2be7`) -- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); - sidecars stay Linux. `migration-plan.md`. (See in-flight.) +- `[2026-06-13]` **granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted** (operator-directed). Global `~/.claude/CLAUDE.md` Global-tools entry; key alias `all-agents-local`, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory `reference_litellm_gateway`) -- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). - worldtree-dev pings the config delta on every config-touching commit (pre-merge); infra-ops syncs - `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal - (Heimdall hot-reloads policies; defaults are code-defaulted). The **v0.33.8 9-HOUR demo outage** — a - `model_roles.yaml` hard-startup-dep that shipped in canonical 3 releases earlier but never reached the - VM — is the failure mode this lane prevents. providers.yaml stays hand-tuned (artemis graft on - personal). corviduo-dev emergency-ops = `ssh vh@10.250.50.152` (alias doesn't resolve, lkraven denied, - infra-ops key excluded), docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, - `reference_corviduo_dev_emergency_ops`) +- `[2026-06-13]` **arbo engine + frontend stack stood up** (ADR-0001) — irv-ml1 co-located inference engine (`ee57e69`), python-based healthcheck (slim image, no curl/wget, `bdb3312`), frontend ro-mounted from the checkout (`922e8ad`, ADR-0001 D2). -- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr- - smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev, granite-only). Pattern: mint via - `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. - (auto-memory `reference_litellm_gateway`) +- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults thinking-OFF; `glm-5.1-reasoning` = same z.ai upstream, thinking ON. Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — `drop_params` strips a top-level `thinking`/`reasoning_effort` but forwards `extra_body` verbatim to z.ai. Shared-gateway change. (`95b2701`, auto-memory `reference_litellm_gateway`) -- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; - Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid commit - `12aa5a9`); binaries+dirs gone, volva.service was crash-looping 203/EXEC. Cleanup at heid's request - (the bus-content-driven sudo got the harness guardrail; operator green-lit). (`6e2f80e`) +- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/pi-coding-agent` via **bun** (npm's global prefix is `/usr` → needs sudo, bun avoids it). Config `~/.pi/agent/models.json`, launcher `~/.local/bin/glm`. -- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer** (supersedes - the 2026-06-04 phi4 decision below). Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not - Q4/AWQ** — primary workload (agent memory + summarization) is high-concurrency, where FP8 scales - ~linearly (profiled 2010 tok/s @ C=32; single-stream 67.5 is batch-1 GEMV physics, not a config bug). - vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8, CUDA graphs. (Then on Ada cc 8.9; - the box has since gone Blackwell — see 2026-06-13.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`) +- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** WORKS: MCP streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer $ZAI_API_KEY` (the **MCP** key, distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's Leif agent. -- `[2026-06-05]` **Langfuse v3 stood up on ana-docker (:3001) as the gateway trace UI**; LiteLLM - `success_callback:[langfuse]` live (project `gateway`). Pretty prompt/completion/reasoning traces + - an `outputTokensPerSecond` tok/s dashboard. NOT a prerequisite — spend_logs already capture - tokens+latency. (`9171e6a`, auto-memory `reference_litellm_gateway`) +- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. `migration-plan.md`. (See in-flight.) -- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any - found; serve via llama-swap or vLLM. Torn down irv-ml1 :11434 (freed 19 GB). (auto-memory - `feedback_avoid_ollama`) +- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs `config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a `model_roles.yaml` startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = `ssh vh@10.250.50.152`, docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`, `reference_corviduo_dev_emergency_ops`) -- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). - FLUX.2-klein (fp8 + q8 GGUF, stock + uncensored encoders) installed on the irv-ml1 Docker ComfyUI; - eshpfi keeps the `comfyui` stack compose, comfy-dev owns the model/workflow knowledge. (auto-memory - `reference_irv_ml1_ampere_quant`) +- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev — granite, later extended to qwen-vision). Mint via `/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory `reference_litellm_gateway`) -- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is - the structured-output profile, ON HOLD, no live consumer; the conversation summarizer defaults to - claude-haiku — the "phi4 erroring" premise was wrong). No instance changes now; worldtree-dev hands - the exact providers.yaml + consumer config when #254 un-holds, infra-ops applies to the bind mounts. - **CORRECTION to the 2026-06-04 "deploys ALL CICD" line:** the bind-mount CONFIGS (providers.yaml, - vh-owned on corviduo `/opt/worldtree*/config`) ARE infra-ops's to apply directly — only the - app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`) +- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`) -- `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer/dreaming agent; - granite-4-small retired** from llama-swap (config-only; GGUFs on disk). 50K ctx (dropped from - Phi-4's 128K max to fit GPU 1's ~10 GB free) + FP8 KV. (`40a374b`) +- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`) -- `[2026-06-04]` **phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's.** - Ollama's bundled template omits the system `<|end|>` — that flattered brokkr's R15 eval but is - the DIVERGENT scaffold (Dvalin: the system `<|end|>` is Microsoft's intended format). Applied an - Ollama-matching override then reverted — ship correct, not the benchmark quirk. (`90e08f0`→`27eb537`; - "headgun" lesson in Tried.) +- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`) -- `[2026-06-04]` **infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes** (+esh-docker-vm - by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. - Dedicated key, sudo log_output, key-gated. (`8c32a05`) +- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`) -- `[2026-06-04]` **Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops** — a "deploy - vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. The granite→phi4 - repoint: worldtree-dev self-served via their CI/CD (v0.30.10). (`d8d776c`) +- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`) -- `[2026-06-04]` **ollama upgraded 0.9.0→0.30.4 on irv-ml1** (Ministral-3 is a Dec-2025 model the - old engine refused); A6000 pinned by **UUID** not index (native fastest-first ≠ nvidia-smi PCI). +- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`) -- `[2026-06-04]` **`brokkr` user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr.** - Persistent box services there need SYSTEM systemd units (see Tried). - -_44 older entries archived to archival-memory.md._ +_50 older entries archived to archival-memory.md._ ## Tried and abandoned -- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128` sglang fork build) on the shared PROD GPU - box PINS it** — load hit 187, prod vLLM services restarted, killed an in-flight quant. ana-ml2 hosts - live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or - cgroup-constrain. (Operator ran the kill; infra-ops NOPASSWD-sudo confirmed working on ana-ml2 — retry - infra-ops on an ssh-255 before concluding "no access.") +- `[2026-06-14]` **Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP.** `gitea.phasefinal.com` = public `38.120.12.44` (ana-srv1); gitea is a container on ana-docker, git-SSH `10.250.50.70:222` + HTTP `:3000`. A fleet host egressing to public `:22` gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (`git fetch` times out under `set -euo pipefail`, aborts before reset). Bit irv-ml1's arbo deploy. `:22` on `10.250.50.70` is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented `docs/orientation.md` (`6e58e57`). -- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** - (Qwen3.5-VL on the stable vLLM: gray-grid output; the LM answers text fine, so it "looks" healthy - until you actually feed it an image). The nightly excludes the vision tower. Lesson: validate the - VISION path on a quantized VLM, not just text — and pin the engine digest that has the exclusion. +- `[2026-06-14]` **Chatterbox-Turbo decode-knob foot-guns** (R16 v1-polish + emergent probes): the turbo length cap is `max_gen_len` (default 1000) on `t3.inference_turbo`, NOT `max_new_tokens` — and `tts_turbo.generate` does NOT forward it (wrap inference_turbo to cap). `rep_pen 2.0 / temp 0.5` BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words. -- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup - (`free >= util*total`), not total** — on a shared card, growing one service before trimming a - co-tenant OOMs ("free 48.56 < desired 80.72" at util 0.85 on a half-occupied 96 GB card). Start-order - matters: trim the shrinking service FIRST, then grow the other. Size to the FREE budget, not the total. +- `[2026-06-13]` **Loading an old LoRA adapter after a vocab bump fails on embedding size.** The harness config + `tokenizer.json` are now at `new_vocab_size=50279` (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set `cfg.new_vocab_size=50277` before `load_finetuned_engine_lora` (else PeftModel state_dict size mismatch). `tokenizer.json.v3bak` is the 50277 tokenizer for a clean restore. -- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — the compose - `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve` (or `--model X`) - yields "unrecognized arguments". Same-class gotchas this session: `tee` masks the real exit code (use - a `>` redirect to keep rc); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`). +- `[2026-06-13]` **Heavy from-source compile (`MAX_JOBS=128`) on the shared PROD GPU box PINS it** — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap `MAX_JOBS≤32`, build off-box, or cgroup-constrain. -- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — - `merge_and_save_turbo_tokenizer()` pulls gpt2-medium + a grapheme merge file (len mismatch) instead of - the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override with the correct - tokenizer + delete the grapheme `tokenizer.json`; `[vmoan]` token → new_vocab_size 50277 (1-row resize), - lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Also: a unique-stem corpus collision (53 - rows, 32 wavs) needs `{index}_{stem}` IDs. (irv-ml1 `~/r16-vmoan-harness`) +- `[2026-06-13]` **`--quantization fp8` on a VL model can quantize the VISION TOWER → garbage vision** (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion. -- `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell - argv** — the poll's command line contains the script name, so its own `pgrep -f` always finds itself - → the loop never exits, the poll never fires. Use a match pattern ABSENT from the poll command (pgrep - the python stage, or a sentinel file), not the driver's own name. (Caught only because the operator - asked "check status"; the job had already finished cleanly.) +- `[2026-06-13]` **vLLM's `--gpu-memory-utilization` is checked against FREE VRAM at startup, not total** — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget. -- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` - requires torchcodec → dies AFTER separating (0 stems written, rc=1).** Same class as the torch-2.12 - torchcodec foot-gun. Fix = pin `torch==torchaudio==2.4.1` (pre-torchcodec save backend) + - `UV_LINK_MODE=copy` for the EPERM-hardlink quirk. Lesson restated: validate the SAVE path, not just - import + GPU inference, on a bleeding-edge torch. +- `[2026-06-13]` **The `vllm/vllm-openai` entrypoint is already `["vllm","serve"]`** — compose `command:` supplies the model as the first POSITIONAL arg + flags; a second `serve`/`--model X` → "unrecognized arguments". Same-class: `tee` masks the real exit code (use `>`); HF `datasets` rejects bare `wikitext` (needs `Salesforce/wikitext`). -- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — it fills the KV cache to the - `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs - AFTER weights+KV load (model/KV log looks healthy, then crash-loops; saw 11 restarts at util 0.36 - with 237 MB free). Fix: free co-tenant room (right-size the other vLLM services) OR `--enforce-eager` - (no graphs, ~15-25% slower decode). FP8 single-stream is batch-1 GEMV (memory-bound, FP8 tensor cores - need batch>1) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) +- `[2026-06-13]` **Chatterbox-Turbo LoRA finetune: the repo's `setup.py` loads the WRONG tokenizer** — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme `tokenizer.json`; `[vmoan]` → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs `{index}_{stem}` IDs. (`irv-ml1:~/r16-vmoan-harness`) -- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres - rows (`dashboards`/`dashboard_widgets`); build by cloning a default-dashboard row + swapping the - measure. tok/s is NOT a per-generation field (null on the observation) — it's the - `outputTokensPerSecond` MEASURE, computed at metrics-API/dashboard query time; no native per-call - tok/s display exists (streaming doesn't change that). langfuse-web needs `HOSTNAME=0.0.0.0` (Next.js - standalone binds one net-IP otherwise, unreachable via the published port once also on tnet). Host - 3000 is gitea's → langfuse on 3001. +- `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name. -- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + - corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read - world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are - vh-owned (editable), the `.env` is deploy-owned 600 (vh can't edit it, no sudo). +- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. -- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL` on - the containers; it defaults to claude-haiku in code, opt-in via config not `.env`. Don't trust an - ".env-flip" recipe — inspect the live container env + the vh-owned config files first. (Inspection - corrected a wrong "summarizer erroring on phi4" premise → saved churning 3 live instances.) +- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) -- `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — - the "headgun" lesson.** Ollama's phi4 template drops the system `<|end|>`; serving vLLM with the - model's HF tokenizer template (canonical, has it) regressed brokkr's Ollama-measured R15 baseline - -33pp type-F1 while valid_format held 1.0. An Ollama-matching `--chat-template` "fixed" it but was - the WRONG fix (the bundled scaffold is the divergent one). PRINCIPLE: serve each model's canonical - `tokenizer.apply_chat_template`, not the bundled template — bundled ones corrupt baselines. Verify - the applied prompt via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`) +- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001. -- `[2026-06-04]` **GPU pin by INDEX is ambiguous on irv-ml1** — native CUDA orders fastest-first - (A6000=0) but nvidia-smi/docker use PCI order (A6000=1), so an index pin can land on the wrong - card. Pin by **UUID** (`CUDA_VISIBLE_DEVICES=GPU-…`); verify via nvidia-smi compute-apps. Check - loaded-model VRAM with `ollama ps` (Ministral-3 @ its 256K default ctx = ~30 GB; cap num_ctx). +- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600. -- `[2026-06-04]` **Persistent services on irv-ml1 need SYSTEM systemd units** — the box reaps - user-session processes on ssh disconnect, and `--user` systemd isn't reachable over non-login - ssh, so nohup/setsid/`screen -dmS`/`systemd-run --user` all die (even with `enable-linger`). Use - `/etc/systemd/system/`. +- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first. -- `[2026-06-04]` **pyworld needs `setuptools<81`** (imports the removed `pkg_resources`); and - **R/soundgen `-lgfortran` fails** on irv-ml1 because the default `gcc` is gcc-11 but only - gfortran-12 is present (libgfortran.so lives only in the gcc-12 dir) → install `libgfortran-11-dev`. - -- `[2026-06-04]` **homepage "crash" ≠ always NFS** — a wedged container in unkillable D-state - ("tried to kill container, but did not receive an exit event") can come from dead `siteMonitor` - widget targets (retired ESH firewall IPs) hanging the node event loop into `exit_mmap`, needing a - host reboot. Check homepage's siteMonitors against retired hosts. (`incident_esh_docker_nfs_boot_race`) - -_48 older entries archived to archival-memory.md._ +_53 older entries archived to archival-memory.md._