diff --git a/archival-memory.md b/archival-memory.md index d5efecc..78f868e 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -363,6 +363,24 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re - `[2026-06-04]` **phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired** from llama-swap. 50K ctx + FP8 KV. (`40a374b`) [superseded by the 2026-06-05 granite-4.1-8b decision] _Archived 2026-06-14._ +- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid `12aa5a9`); volva.service was crash-looping 203/EXEC. (`6e2f80e`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer.** Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not Q4/AWQ** — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM `vllm-granite` :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (`34a43a0`, auto-memory `reference_ana_ml2_vllm_granite`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **Langfuse v3 on ana-docker (:3001) as the gateway trace UI**; LiteLLM `success_callback:[langfuse]` live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (`9171e6a`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **Ollama BANNED fleet-wide** (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory `feedback_avoid_ollama`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **ComfyUI / FLUX.2 work split to `~/development/comfy-dev`** (dedicated repo + agent). eshpfi keeps the `comfyui`/`arbo` stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory `reference_irv_ml1_ampere_quant`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **Worldtree summarizer config refresh DEFERRED to Worldtree #254** (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the `.env` is deploy-owned. (auto-memory `reference_worldtree_deploys_cicd`) + _Archived 2026-06-16._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with @@ -769,3 +787,18 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re - `[2026-06-04]` **Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson.** Serve each model's canonical `tokenizer.apply_chat_template`, not the bundled template (bundled ones corrupt baselines). Verify via vLLM `/tokenize`→`/detokenize`. (`90e08f0`/`27eb537`) _Archived 2026-06-14._ + +- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. + _Archived 2026-06-16._ + +- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) + _Archived 2026-06-16._ + +- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001. + _Archived 2026-06-16._ + +- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600. + _Archived 2026-06-16._ + +- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first. + _Archived 2026-06-16._ diff --git a/persistent-memory.md b/persistent-memory.md index 2bffc79..f2e0c1a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-15_ +_Last updated: 2026-06-16_ ## Repo purpose @@ -99,24 +99,67 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-06-15:_ +_As of 2026-06-16:_ + +- **ana-ml2 GPU layout RESHAPED again (2026-06-15/16) — both cards now full + with NVFP4 tenants.** **GPU 0 = Mistral Small 4** (`mistral-small-4` stack, + `mistralai/Mistral-Small-4-119B-2603-NVFP4`, 119B/6.5B-active MoE, :8010, + gateway `mistral-small-4` + `mistral-small-4-reasoning`@effort=high). Pinned + **vLLM v0.22.0** — the LAST release with working Mistral *vision* (#44911 + `fetch_images` regression breaks it on 0.22.1+/0.23.0). Serves the full native + **256K context** (max-model-len 262144, max-num-seqs 32 — fits the tight card, + ~5 GB free). Text + vision both work; reasoning via `reasoning_effort` (BINARY: + none|high). Dedicated single-tenant; the operator's creative-writing model + (abliteration planned → it succeeds llama-swap). **GPU 1 = qwen36 swapped + FP8→NVFP4** (`nvidia/Qwen3.6-35B-A3B-NVFP4`, fp16 KV, util 0.34, :8007, gateway + name `qwen3.6-35b-a3b` UNCHANGED + `-thinking` variant) — the ModelOpt NVFP4 + MoE LOADS on 0.23.0 now (the 2026-06-14 "blocked" finding is RESOLVED). + + **granite restored** (0.34/131072) + **Selene FP8 judge added** (`selene-1-mini-8b`, + AtlaAI Selene-1-Mini-Llama-3.1-8B dynamic fp8, util 0.17, :8011, ctx 32768) + + embed/rerank/reward. ~5.6 GB free. Prefix-caching ON on all 4 generative. + **llama-swap is DOWN** (decommissioned from GPU 0 for Mistral; its qwen GGUF + consumers migrated to the gateway). (auto-memory `reference_nvfp4_moe_loads_on_vllm_023`) + +- **arbo engine builds handed to comfy-dev; Gitea Actions runner LIVE on irv-ml1.** + Operator approved comfy-dev owning arbo engine deploys (`deploy-engine.sh`, + build-local). Runner = `irv-ml1-arbo` (act_runner v0.2.13, **host-executor as + lkraven**, repo-scoped to vh/arbo, labels `self-hosted`+`irv-ml1`, systemd unit + `act_runner.service`, polls the INTERNAL gitea route). comfy-dev to commit + `.gitea/workflows/deploy.yml` — Phase 1 = engine-only auto-deploy (catalog stays + on the :9009 webhook), Phase 2 catalog cutover later. infra-ops = box-level + escalation only (runner, the `package:write` PAT once corviduo-org exists, webhook, + the arbo-deploy.sh HEAD-baseline restart-detection bug still open). + +- **claude-bot Gitea service account STOOD UP** (credential-migration foundation) — + user id=3, non-admin/scoped, token `arbo-ci`=write:package+read:repository, creds + at `~/.config/claude-bot/{gitea-password,gitea-token}` (mode 600). Registry-push + DEFERRED: vh is a USER not an org, so Gitea won't grant a scoped service account + package-write to `vh/*`. (auto-memory `project_migrate_infra_access_to_claude_credentials`) + +- **Worldtree demo + personal MIGRATED to v0.35.10 capability-named profiles** — + persona substrate moved qwen→Mistral (profiles `dialogue`/`classification`→ + mistral-small-4, `deep-reasoning`→mistral-small-4-reasoning), Domari→selene; qwen + heretic/heavy retired. Both green. The win: future model swaps (the abliterated + Mistral) are now bind-mount-only `providers.yaml` edits — no image/agent/coupling. - **R18 clip+caption auto-assist models STAGED on ana-ml2** (brokkr/Eitri) — WhisperX + PANNs CNN14_DecisionLevelAtt + Qwen3-Omni-30B-A3B-Captioner at Eitri's `/srv/r18-clip-caption` path contract (symlink → /tank; dedicated `r18clip` user, `workspaces/` 0700). Stub-level offline smoke PASSED (lo-only netns); Eitri closed the scaffold deliverable. AWAITING: Eitri/brokkr's concrete model runners (replace the deterministic stubs) → re-smoke the real path; AND the **hard real-voice gate** — no real victim voice through the shared-box captioner until ephemeral-(b) controls (in-VRAM-only, no shared-disk persistence, access-logged + auditd stood up) are built AND explicitly confirmed to brokkr. (auto-memory `project-r18-clip-caption-staging`) - **MastMed prototype LIVE + public** — gated static-SPA container on ana-docker:8089 (caddy `basic_auth`, user `mastmed`) fronted by a cloudflared **quick tunnel** (ephemeral `*.trycloudflare.com` — dies on tunnel-container restart → new URL). Swap to a named tunnel (operator's scoped CF token + hostname) for a stable URL anytime; container + auth unchanged. -- **comfyui (irv-ml1) crash-loop = a VAE-decode SEGFAULT, NOT OOM** (diagnosed 2026-06-15 → comfy-dev, thread `01KV6FHCJ9…`). Per cycle: startup OK → `got prompt` → sampling 8/8 OK → `Requested to load AutoencodingEngine` → `Fatal Python error: Segmentation fault` → restart → queued prompt re-runs → loop. SERVING now (:8188 200, loop stopped — no VAE-decode gen since 20:08:51Z); recurs on the next such gen. comfy-dev owns the fix (suspects: torch/cu130 mismatch + `comfy_kitchen` cuda backend disabled, the injected `aimdo` cuda-funchooks, or the VAE). Infra-side offered to chase what injects `aimdo`. +- **comfyui (irv-ml1) VAE-decode SEGFAULT — RESOLVED 2026-06-15.** Root cause was + `comfy_aimdo`'s cudaMalloc hooks conflicting with ComfyUI's load-bearing + `--disable-cuda-malloc`: with the optimized backend disabled (cu129`. + (auto-memory `feedback_compose_up_recreates_whole_stack`) + - `[2026-06-15]` **A comfyui restart-loop with `exit 0` + `OOMKilled=false` is NOT OOM — suspect a native SEGFAULT.** `docker inspect` shows only the LAST exit code; a GPU OOM is a catchable `RuntimeError: CUDA out of memory`, NEVER `Fatal Python error: Segmentation fault`. Diagnostic: `RestartCount>0` = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full `docker logs` for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading `AutoencodingEngine`) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected `aimdo` cuda-funchooks lib, the VAE. - `[2026-06-15]` **NEVER blanket-kill `althing-light-monitor` by PID** — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A `kill $(pgrep -f light-monitor)` downs four peers' monitors (near-miss caught by checking `/proc//environ` `ALTHING_HANDLE` + cwd first). Recover your OWN via `althing-cli stop-monitor` (v0.14.1+); a raw `&`-backgrounded monitor is untracked (no harness fire-notification) — always arm via `run_in_background`. (extends the pkill-self-match lesson below) @@ -289,14 +392,4 @@ _50 older entries archived to archival-memory.md._ - `[2026-06-11]` **A completion-poll `while pgrep -f ` SELF-MATCHES its own remote shell argv** — its own `pgrep -f` always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name. -- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()` requires torchcodec → dies AFTER separating** (0 stems, rc=1). Fix = pin `torch==torchaudio==2.4.1` + `UV_LINK_MODE=copy`. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. - -- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — fills KV to the `--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR `--enforce-eager`. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (`reference_ana_ml2_vllm_granite`) - -- `[2026-06-05]` **Langfuse has NO public dashboard-creation API** — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the `outputTokensPerSecond` MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs `HOSTNAME=0.0.0.0`. Host 3000 is gitea's → langfuse on 3001. - -- `[2026-06-05]` **`sudo` over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD** (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = `vh@10.250.50.152`; bind-mount configs are vh-owned, the `.env` is deploy-owned 600. - -- `[2026-06-05]` **Worldtree summarizer-model is NOT an env var** — no `WORLDTREE_SUMMARIZER_MODEL`; defaults to claude-haiku in code, opt-in via config not `.env`. Inspect the live container env + vh-owned config files first. - -_53 older entries archived to archival-memory.md._ +_58 older entries archived to archival-memory.md._