diff --git a/archival-memory.md b/archival-memory.md index af69406..65bb0fe 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -549,6 +549,42 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread `01KV6X0M…`) _Archived 2026-06-20._ +- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3) + _Archived 2026-06-21._ + +- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae) + _Archived 2026-06-21._ + +- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice) + _Archived 2026-06-21._ + +- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514) + _Archived 2026-06-21._ + +- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice) + _Archived 2026-06-21._ + +- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04) + _Archived 2026-06-21._ + +- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`) + _Archived 2026-06-21._ + +- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad) + _Archived 2026-06-21._ + +- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1. + _Archived 2026-06-21._ + +- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff) + _Archived 2026-06-21._ + +- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35) + _Archived 2026-06-21._ + +- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab) + _Archived 2026-06-21._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with @@ -1074,3 +1110,30 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re _76 older entries archived to archival-memory.md._ _Archived 2026-06-20._ + +- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce. + _Archived 2026-06-21._ + +- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.** + _Archived 2026-06-21._ + +- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util <= ~0.95 on a tight MoE). + _Archived 2026-06-21._ + +- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output. + _Archived 2026-06-21._ + +- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first. + _Archived 2026-06-21._ + +- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`) + _Archived 2026-06-21._ + +- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`. + _Archived 2026-06-21._ + +- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS). + _Archived 2026-06-21._ + +- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. + _Archived 2026-06-21._ diff --git a/persistent-memory.md b/persistent-memory.md index 256de49..86ee878 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-20_ +_Last updated: 2026-06-21_ ## Repo purpose @@ -109,185 +109,44 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-06-20:_ +_As of 2026-06-21:_ -- **Worldtree capability-gateway migration — LANDED + DEPLOY TRIGGERED.** - The role→capability indirection (worldtree-dev's #257/ADR-0012 direction; - operator: NO `wt-` prefix, reuse the existing capability-alias convention) - is live on the LiteLLM gateway: `gen`/`gen-reasoning` (→qwen3.5-122-a10b), - `classifier`/`summarizer` (→granite), `chat-judge` (→selene-1-mini-8b), - `reranker` (→qwen3-reranker), `scalar-judge` (passthrough → Skywork-Reward-V2 - @ ana-ml2:8003 — LiteLLM has no reward MODE), `gen-frontier`/`-reasoning` - (→GLM 5.2, PAID). **`qwen3-embedding` kept model-named ON PURPOSE** — the - swap-transparency rule: embedding vectors aren't swap-transparent, so a - generic `embedding` alias is a footgun. The `*` wildcard → dead llama-swap - was REMOVED (unknown models now 404 cleanly). WT's full re-point (granite - SPLIT into granite-classifier/granite-summarizer + role rebind + edges→gen + - selene→chat-judge) landed as worldtree-dev's commit **3fbf338 / v0.37.5**; - I staged its `providers.yaml`+`model_roles.yaml` byte-identical into - demo+personal bind-mounts and triggered build-and-deploy via claude-bot. - Pinned rides CI. (d372103, d0eb09c, 7819f96) +- **Backups — RECOVERED + hardened (2026-06-20).** rest-server-ana is back (the + ~6.5-wk silent ana-side restic outage is closed); ana-ml2 + ana-docker + re-snapshotted; daily freshness alert live (nh3-dev systemd timer → + `scripts/check-backup-freshness.sh` → althing on any stale/down layer); both + ana-docker NFS mounts moved to self-healing automount. Topology + 2-min check: + `docs/runbooks/backups.md`. STILL OPEN: rotate the 5 disclosed rest-server creds + (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps + (esh-pve-nas has NO PBS namespace → it is restic-only). Detail in Recent decisions. -- **Worldtree demo → v0.37.7 GREEN (2026-06-20) — two-half fix.** worldtree-dev's - v0.37.5 config (3fbf338) carried a dangling `granite-4.1-8b` catalog ref → - demo crash-blocked at startup validation. Fix needed BOTH halves: (a) gitea - was silently not creating Actions runs so no deploy could fire (the earlier - disk-full wedged its notifier — `docker restart gitea` cleared it; see - Tried-and-abandoned); (b) the v0.37.7 IMAGE alone wouldn't fix demo because - demo+personal read BIND-MOUNTED config — I re-staged `config/providers.yaml` - + `config/model_roles.yaml` from gitea at **439bebf** byte-identical into BOTH - `/opt/worldtree{,-personal}/config` (sha256-verified across source + both - instances; `granite-4.1-8b`→`summarizer`, `granite-classifier/summarizer`→ - `classifier/summarizer`; old configs saved `.bak-pre-439bebf`; NO container - recreate — avoided the stale-`:latest` footgun). worldtree-dev self-served the - CI re-run via the claude-bot Actions token → **run 1290 GREEN, demo on - v0.37.7**, capability routing live. claude-bot token migration DONE for - worldtree-dev (off vh's personal PAT). **RATATOSKR → READONLY (2026-06-20, final):** - operator first said admin (worldtree-dev minted interim `ratatoskr-admin` 90db1fbd), - then flipped to least-privilege → that key REVOKED (DELETE 200). I re-staged 439bebf's - `config/policies.yaml` to `/opt/worldtree-personal/config` (byte-identical, sha256 - f3ca3e6…, `.bak-pre-439bebf`) and reloaded Heimdall via `docker restart - worldtree-personal-worldtree-api-1` (same d2e9f05 SHA — the SAFE reload, no recreate/ - no `:latest` flip); personal came back healthy + the `readonly-admin` tier (7 read - scopes incl. admin.events.read) is now in the loaded policy. **FOLLOW-UP cb2a79a - (v0.37.9):** the 439bebf tier-only stage was NOT sufficient — WT's PDP is RULE-based, - so a tier with scopes but NO allow RULES is default-denied; cb2a79a adds the 4 - readonly-admin allow rules. Re-staged cb2a79a `config/policies.yaml` to BOTH demo + - personal (sha256 d853e51…, `.bak-pre-cb2a79a`) + restarted BOTH API containers - (`worldtree-worldtree-api-1` + `…-personal-…`); both healthy + serving. (Still on - d2e9f05 image — CI cb2a79a deploy pending, but the bind-mount policy is the - load-bearing fix.) worldtree-dev mints the ratatoskr readonly key via their admin key. - **Lessons: (1) reload bind-mounted WT config via `docker restart `, NEVER - `compose up` (the `:latest` footgun); (2) a WT tier needs allow RULES, not just - scopes — the PDP is rule-based, scopes alone default-deny.** +- **Worldtree — demo + personal GREEN (v0.37.7/.9).** Capability-gateway migration + live; ratatoskr readonly-admin tier staged (cb2a79a). worldtree-dev fully + self-serve: admin keys (demo d113207c / personal f4f75adb) + one consolidated + claude-bot Gitea token (Actions+deploys+issues). Nothing pending. Detail in + Recent decisions. -- **🟠 BACKUP DIAGNOSIS (2026-06-20, full probe) — REVISED from "all ana - backups down": PBS + nh3-restic are HEALTHY & CURRENT; only the ANA-side - restic file+DB layer is dead, and the one host with NO other cover is - ana-ml2.** Three layers: - - ✅ **PBS-ANA (VM images)** — healthy, snapshots TODAY. All PFI guests - backed up under PBS namespaces `ana-pve` / `esh-pve` / `nh3-pve` (+ tenant - `sfsrv-pve`); PBS-ANA's own NFS datastore mounts ana-nas fine (1.2T/20T). - Gaps to note: `ana-pve/vm/100` has no snapshots; **there is NO `esh-pve-nas` - PBS namespace** → vm-esh-nas + esh-vm-db may be PBS-UNCOVERED (VERIFY). - - ✅ **restic file+DB nh3-side** (`rest-server-nh3` @ nh3-nas:8000, UP→401) — - irv-ml1 + nh3-docker, last snapshot TODAY 01:00. Repos at - `nh3-nas:/volume1/Backup/restic/{irv-ml1,nh3-docker}`. - - 🔴 **restic file+DB ANA-side** (`rest-server-ana` @ ana-docker:8000) — DOWN, - crash-looping. Last successful snapshot **2026-05-06 (~6.5 weeks)** across - ALL 5 clients (ana-docker, ana-ml2, esh-docker-vm, esh-vm-db, vm-esh-nas). - - **Scheduler:** NOT Backrest (it has **0 plans** — it's just a repo viewer). - Backups run via **`resticprofile` systemd timers** (`resticprofile-backup@ - profile-default.timer`, daily 01:00; `…-check@…` Sun 05:00). The ana timer - fires daily and **FAILS SILENTLY** (no alerting → why the gap went unnoticed - 6.5 wks). - **Root cause:** ana-nas NFS fstab bare-`defaults` (no `_netdev`/retry) → - `mnt-backup.mount` stuck `failed` → rest-server-ana's bind source - `/mnt/backup/restic/repo/ana` resolves to an empty LOCAL ghost dir (no - `.htpasswd`) → crash-loop (`cannot load /data/.htpasswd: permission denied`). - The REAL repos + `.htpasswd` are intact on ana-nas at - `/mnt/backup/restic/repo/ana/{/,.htpasswd}`. ana-nas itself is FINE. - **Real exposure, ranked:** (1) 🔴 **ana-ml2** — BARE METAL, no PBS coverage; - ana-side restic is its ONLY layer → ~6.5 wks fully unprotected (bespoke - at-risk: in-house NVFP4 quants / configs / scripts; bulk `/tank` models are - re-downloadable). (2) 🟠 **esh-vm-db** (a DB VM: postgres+mongo) + vm-esh-nas - IF PBS-uncovered (verify the esh-pve-nas gap). (3) 🟡 **ana-docker / - esh-docker-vm** — VMs, PBS image-CURRENT today, only the granular restic - layer is stale (full-VM restore available; no recent file/DB-granular point). - **RECOVERY (needs root on ana-docker; the session there = `lkraven`, NO - NOPASSWD sudo → operator `!`-sudo or bootstrap infra-ops there):** `sudo - mount -a` → `sudo docker restart rest-server` (recreate if the bind doesn't - re-resolve onto the NFS; if the ghost persists: stop container → umount → rm - the ghost `repo/ana` → remount → start) → verify → next 01:00 succeeds. - **PREVENTION (the systemic fixes):** fstab `_netdev`+retry so it self-heals - on reboot; a **backup-freshness alert** (the real miss — 6.5 wks unnoticed). - Recovery steps mirror `docs/runbooks/disaster-recovery.md` Tier-0. - **⚠️ Disclosed creds:** diagnosing dumped the Backrest config, exposing the 5 - per-repo rest-server basic-auth passwords into the session transcript → - rotate after recovery (add to the disclosed-keys rotation queue). - **✅ RECOVERED (2026-06-20):** via `ssh infra-ops@ana-docker` (NOT the default - lkraven session) — `sudo mount -a` + `reset-failed mnt-backup.mount` re-attached - the NFS, then `docker compose up -d --force-recreate` rebound rest-server onto - the real `/data` (now `Loaded htpasswd file`, 401, healthy). Verified end-to-end: - triggered backups on the two no-PBS hosts' priority — **ana-ml2 snapshot landed - 22:53Z** (total 19→20); ana-docker's larger catch-up was still running at hand-off - (will land). esh-docker-vm/esh-vm-db/vm-esh-nas auto-catch-up at their next 01:00 - timers (or trigger manually). **Full topology + 2-min freshness-check now documented - in `docs/runbooks/backups.md`.** **PREVENTION DONE (2026-06-20):** (1) freshness - alert — `scripts/check-backup-freshness.sh` + daily systemd user timer on nh3-dev - (08:00) → althing alert to infra-ops on any stale/down layer (proven end-to-end); - (2) fstab hardened — ana-docker `/mnt/backup` → `noauto,x-systemd.automount,…` - (autofs self-heals, active next boot; `.bak-pre-harden` saved); (3) esh-pve-nas - VERIFIED **not** PBS-covered → esh-vm-db (a DB!) + vm-esh-nas are restic-only. - `/mnt/compose` ALSO hardened + automount activated live (2026-06-20, binds no - container — proved the autofs pattern works on ana-docker). STILL OPEN: rotate the - 5 rest-server creds (operator, offline); the 3 esh + ana-docker restic clients - clear at tonight's 01:00. +- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model + hooks) CANCELLED (Worldtree model-agnostic → no deploy path). R22 whole footprint + = gateway calls to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at + `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete). + Detail in Recent decisions. -- **ana-docker disk incident RESOLVED — root cause was a 94 GB unrotated - `langfuse-clickhouse` log, NOT (just) image bloat.** ClickHouse logs - verbosely to stdout + there was NO log rotation anywhere (no per-stack - `logging:`, no daemon default) → its json.log hit ~94 GB and filled `/` - (100%, 28/48 containers unhealthy on `runc /tmp ENOSPC`). Fixed: truncated - the log, pruned ~101 GB of images (the gitea-runner accumulated 18 - worldtree build versions), retired Langfuse. Disk → ~51%. Installed a - daily image+buildx prune cron (`lkraven` crontab on ana-docker). **DEFERRED - to clean context:** a **docker-daemon default log cap** (`daemon.json` - log-opts + docker restart, needs root → infra-ops) — the real systemic fix - so no container can ever spew unbounded again. (Note: named buildx-builder - state volumes aren't cleared by `docker image/builder prune` — orphaned - ones need `docker volume rm`.) +- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the + gateway (lists in /v1/models but 400s — backend displaced); docker-daemon default + log-cap on ana-docker (root → infra-ops; systemic fix behind the 94 GB disk + incident). -- **Langfuse RETIRED (2026-06-20, operator).** Redundant with LiteLLM's - native logging (`store_prompts_in_spend_logs: true` → full prompts/ - responses/tokens/cost at `:4000/ui`); nothing used its trace-grouping/eval - features; and it's a 6-container crash-prone surface (the 94 GB spew). - Dropped the `success_callback`/`failure_callback: [langfuse]` from the - litellm config (gateway observability stays native) and `compose down -v` - the stack. (f8eda1c) +- **Standing / parked (from prior):** Mac Pro migration (`migration-plan.md`, + hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys + rotation queue; clean legacy `news-digest` on ana-docker. -- **OmniVoice gained a streaming `/tts` + language-safe sanitizer** (irv-ml1 - :8199, `stacks/omnivoice/`), driven by a VENDORED copy of chatterbox-fast's - buffer-ratchet scheduler. KEY FINDING: OmniVoice is DIFFUSION → a ~fixed - per-call overhead sets a TTFA FLOOR (~0.7 s @16 steps), and measured RTF is - ~2.8×/5.6× on the 3090 (NOT the upstream-claimed 40×). Tuned: streaming - num_step=16/batch 32, rtf_prior=20. **chatterbox-fast (autoregressive, - ~0.5 s TTFA) stays the lower-latency front-end** — OmniVoice is the - multilingual/voice-design complement, NOT a replacement. (288d085, cd92b85; - auto-memory `reference_omnivoice_streaming_tts`) - -- **arbo v0.14.9 deployed** (muse prompt-enhance → `summarizer-large`, was - granite which mode-collapsed at temp 1.0) via the canonical health-gated - `scripts/deploy-engine.sh` on irv-ml1. arbo's gitea Actions CI is now LIVE - (runner-label mismatch fixed by comfy-dev) → future src pushes auto-deploy; - manual deploy only when CI is down. (auto-memory `reference_arbo_engine_deploy`) - -- **ratatoskr `:8392` (composite Bifrost provider) added** to the personal - WT `BIFROST_CLIENT_ALLOWED_HOSTS` (footgun-safe recreate: inline sha pin + - `--pull never`, the `.env` still pins stale `:latest`). And ratatoskr's - **admin.events.read key (ratatoskr#11) BROKERED** with worldtree-dev — the - scope is tier-derived (not mint-settable), so worldtree-dev defines the - isolated admin-tier user/scope, then I mint + secure-deliver. NON-URGENT. - -- **claude-bot granted ADMIN on vh/Worldtree** (2026-06-20, via operator - vh-admin → migration-enabler) so I can self-serve WT deploy re-runs + - provision worldtree-dev's own Actions token (pending their gitea username). - -- **arbo + brokkr litellm keys = `all-proxy-models`** (verified — full gateway - access incl. paid GLM, no key change needed). `all-agents-local` shared key - stays fenced to free local models only. claude-bot Gitea service account is - the credential-migration foundation (admin on vh/arbo + now vh/Worldtree). - -- **Worldtree config-propagation:** demo+personal bind-mount their - `model_roles.yaml`/`providers.yaml` (`/opt/worldtree{,-personal}/config`, - infra-ops-deployable); **pinned is IN-IMAGE** (worldtree-dev CI bakes it). - worldtree-dev authors the canonical config; infra-ops deploys to the - bind-mounted instances byte-identical. - -- **Still open from prior:** Mac Pro migration (`migration-plan.md`, - workstation-only, hardware-gated); R17 v2 corpus push HELD (local-only - soundgasm-derived, distribution barred); disclosed-keys hygiene rotation - queue; clean legacy `news-digest` on ana-docker. +- **Worldtree config-propagation (reference):** demo+personal bind-mount + `providers.yaml`/`model_roles.yaml`/`policies.yaml` from + `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from + worldtree-dev canonical sha); pinned is IN-IMAGE. Reload via `docker restart + `, NEVER `compose up` (stale-`:latest` footgun); WT PDP is rule-based + (a tier needs allow RULES, not just scopes). ## Recent decisions @@ -380,33 +239,9 @@ _As of 2026-06-20:_ unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context. -- `[2026-06-19]` **litellm task-aliases: `classifier`→granite-4.1-8b, `summarizer-large`→gen/qwopus** (`summarizer`→granite already existed). Also added then REMOVED `gen-nt`/`gen-reasoning-nt` (redundant with the global `strip_empty_tools` hook). (ef45f6d, 2e5ab72, 75bd4c3) - -- `[2026-06-19]` **gateway-chat: model-smoking web chat → PERSISTENT container.** `/v1/models` auto-discovery + image-upload; `stacks/gateway-chat/` (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae) - -- `[2026-06-19]` **pi on nh3-dev wired to `gen`** — `~/.pi/agent/models.json` + a `gen` launcher mirroring `glm`. Local box config, not a repo artifact. (untracked by operator choice) - -- `[2026-06-19]` **`gen` model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8.** ana-ml2 GPU 0, served-name `qwen3.5-122-a10b`. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514) - -- `[2026-06-19]` **TP=2 across the two ana-ml2 Blackwells REJECTED** — `PIX` (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice) - -- `[2026-06-19]` **qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1**, aliased `image-judge`. (bfae924, 5dfce04) - -- `[2026-06-18]` **heretic abliterated Mistral Small 4 NVFP4 built + LIVE as `mistral-small-4`** (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, `tools/mistral-small4-nvfp4/`) - -- `[2026-06-18]` **irv-ml1 VRAM consolidation + comfy-dev torch-pin** — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; `DISABLE_UPGRADES=true` @ torch 2.12.1. (a8550ad) - -- `[2026-06-18]` **ComfyUI acceleration set (9 nodes) installed for comfy-dev** on irv-ml1. - -- `[2026-06-17]` **ana-ml2 `vm.overcommit_memory=1` made durable** (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff) - -- `[2026-06-17]` **GLM 5.2 wired into litellm** (`glm-5.2` + `-reasoning`, z.ai passthrough, `extra_body.thinking.type` toggle). (fe77a35) - -- `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab) - - `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`) -_101 older entries archived to archival-memory.md._ +_113 older entries archived to archival-memory.md._ ## Tried and abandoned @@ -476,22 +311,4 @@ _101 older entries archived to archival-memory.md._ reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm config, most stack compose/conf) take plain `cp` under either identity. -- `[2026-06-19]` **litellm CONFIG-loaded models can't be hot-removed — `/model/delete` is DB-only.** Only a process restart drops a config model. The INVERSE works: `/model/new` live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce. - -- `[2026-06-19]` **the litellm `*` wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading `Connection error`, not `model not found`.** Bit a brokkr call to the renamed `qwen-image-judge`. **→ RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly.** - -- `[2026-06-19]` **A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor**, allocated OUTSIDE `gpu-memory-utilization`. FIX: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` + leave ~3.2 GB free (util ≤ ~0.95 on a tight MoE). - -- `[2026-06-19]` **vLLM decode tok/s: ALWAYS discard the first generation (cold warmup).** Cold ≈ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3–4×. Measure run 2+ over a ≥256-token output. - -- `[2026-06-19]` **For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE.** Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first. - -- `[2026-06-18]` **mmartial `comfyui-nvidia-docker`: root pip installs CRASH-LOOP the container.** `docker exec -u 0 pip install` leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: `chown -R 1000:1000 /comfy/mnt/venv`; pin with `DISABLE_UPGRADES=true`. (auto-memory `reference_irv_ml1_comfyui_mmartial`) - -- `[2026-06-18]` **ComfyUI custom nodes break on version-assumption drift** — box runs 0.24.1; TeaCache imports `precompute_freqs_cis` at MODULE level (guard the LTX-only import). `pip install tensorrt` defaults to cu13 → use `tensorrt-cu12`. - -- `[2026-06-17]` **Mistral HF→NVFP4 quant: the placement knob is the whole game.** `device_map="cpu"` (CPU-resident, sequential onload) works; `"auto"` OOMs, `max_memory` offloads to meta. Read shards with plain `read()` + `safetensors.torch.load(bytes)`, NOT `safe_open` (mmaps the 50 GB shard → ENOMEM on `/tank` ZFS). - -- `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. - -_84 older entries archived to archival-memory.md._ +_93 older entries archived to archival-memory.md._