diff --git a/archival-memory.md b/archival-memory.md index 65bb0fe..ac2bc35 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -585,6 +585,21 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re - `[2026-06-17]` **nh3-extdev stood up as a sudo-LESS infra-ops manager box** (successor to nh3-ansible); hosts `/opt/externs` pi-on-GLM-5.2 client workspaces. (a841eab) _Archived 2026-06-21._ +- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` prefix (operator) — reuse the existing capability convention.** gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: embedding stays model-named (`qwen3-embedding`) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96) + _Archived 2026-06-25._ + +- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c) + _Archived 2026-06-25._ + +- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c) + _Archived 2026-06-25._ + +- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's scheduler, not its whole package** (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85) + _Archived 2026-06-25._ + +- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing automation behind the recurring image bloat) — `lkraven` crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context. + _Archived 2026-06-25._ + ## Tried and abandoned (archived) - `[2026-04-30]` task-board workflow with @@ -1137,3 +1152,18 @@ _76 older entries archived to archival-memory.md._ - `[2026-06-17]` **HF-format Mistral Small 4 is UNSERVEABLE on vLLM** — no HF `Mistral4` backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. _Archived 2026-06-21._ + +- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). + _Archived 2026-06-25._ + +- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending operator OK to free. + _Archived 2026-06-25._ + +- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** No per-stack `logging:` limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir `du`, not `docker system df`. The systemic fix is a daemon-default `log-opts` (`max-size`/`max-file`) — NOT just per-service patching. + _Archived 2026-06-25._ + +- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end. + _Archived 2026-06-25._ + +- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes MISS.** `docker image prune -a` + `docker builder prune` don't touch `buildx_buildkit_-builder0_state` volumes (per-builder buildkit state). Prune them with `docker buildx prune --builder -af`, or `docker volume rm` orphaned ones (builder already removed). + _Archived 2026-06-25._ diff --git a/persistent-memory.md b/persistent-memory.md index 86ee878..1e83132 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-21_ +_Last updated: 2026-06-25_ ## Repo purpose @@ -22,7 +22,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | -| `vh/althing` | Inter-agent message bus (chamber UI port 7881, forseti + agent-runner daemons, valkey IPC) | push-to-main → CI deploys (2026-05-14) | +| `vh/althing` | Lean trusted inter-agent message bus — **v0.17 multi-machine** (re-architected 2026-06): each box runs a local-SQLite bus + a courier/receiver for P2P over the 10.x net. The v0.15 lean-bus cut RIPPED moderation / chamber (UI 7881) / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-extdev** added as a mesh peer 2026-06-25 (model B: althing-svc svc account + shared `/srv/althing`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker** (the image-bloat source); claude-bot now ADMIN collaborator (2026-06-20) | push-to-main → CI build-and-deploy (runner on ana-docker) | @@ -41,8 +41,7 @@ no longer deployed sidecars here. See Recent decisions.) through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS - export of `/mnt/backup`. **As of 2026-06-20 `rest-server-ana` is DOWN - on a failed ana-nas NFS mount — see Current state.** + export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.) - **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at @@ -105,51 +104,86 @@ no longer deployed sidecars here. See Recent decisions.) 2026-06-20). **→ For any sudo op on ana-docker (mount, root-owned files, service control), use `ssh infra-ops@ana-docker`, NOT the default session.** lkraven-owned files (litellm config, most stack compose/conf) still take - plain `cp`/edit under either identity. + plain `cp`/edit under either identity. **NEW (2026-06-25):** `ssh infra-ops@10.100.10.50` + (nh3-dev) ALSO has NOPASSWD sudo; on **nh3-extdev** infra-ops is sudo-LESS by + design and **`ssh lkraven@10.100.50.42` is the NOPASSWD path** there. ## Current state / in-flight -_As of 2026-06-21:_ +_As of 2026-06-25:_ -- **Backups — RECOVERED + hardened (2026-06-20).** rest-server-ana is back (the - ~6.5-wk silent ana-side restic outage is closed); ana-ml2 + ana-docker - re-snapshotted; daily freshness alert live (nh3-dev systemd timer → - `scripts/check-backup-freshness.sh` → althing on any stale/down layer); both - ana-docker NFS mounts moved to self-healing automount. Topology + 2-min check: - `docs/runbooks/backups.md`. STILL OPEN: rotate the 5 disclosed rest-server creds - (operator, offline); confirm esh-vm-db's resticprofile includes DB dumps - (esh-pve-nas has NO PBS namespace → it is restic-only). Detail in Recent decisions. +- **Worldtree persona-render config arc COMPLETE (demo + personal).** Pre-synced + + deployed three bind-mount config deltas on corviduo-dev demo+personal: #314 + relational-stance (relational_extractor role + relational_stance block + stance/ dir), + #322 affect/mood gate (affect-render-baseline-allow policy rule + mood_tier_deployment_cap), + #317 valence teardown (REMOVED relational_valence — a **boot-blocking removal**). All + deployed green; the full two-layer persona render (relational + mood) is live on + demo+personal; Worldtree at v0.37.38. Recipe + the additions-vs-removals lesson in + auto-memory `reference_corviduo_dev_emergency_ops`. -- **Worldtree — demo + personal GREEN (v0.37.7/.9).** Capability-gateway migration - live; ratatoskr readonly-admin tier staged (cb2a79a). worldtree-dev fully - self-serve: admin keys (demo d113207c / personal f4f75adb) + one consolidated - claude-bot Gitea token (Actions+deploys+issues). Nothing pending. Detail in +- **nh3-extdev (10.100.50.42) stood up as an althing v0.17.1 mesh PEER — MODEL B.** Full + provision in one session: system zellij + althing (`/usr/local/bin`, both users), + CLAUDE.md, and the mesh (handles `mailman`+`ldp-dev`, ↔nh3-dev). MODEL B (operator's + multi-user choice): receiver runs as the **`althing-svc`** service account + a + group-shared **`/srv/althing`** root (group `althing`; infra-ops+lkraven members; + `ALTHING_ROOT=/srv/althing` via `/etc/profile.d/althing.sh`). Live + verified (two OS + users sharing ONE config/DB). Layout + access facts in auto-memory + `reference_nh3_extdev_althing_mesh`. Open: forseti's multi-user ADR. + +- **zellij native web client piloted on nh3-dev** (`zellij-web.service` :8443, native TLS) + ALONGSIDE the ttyd seats (ttyd KEPT as the no-auth fallback). Verdict: wins on + clipboard/auth/TLS/simplicity, loses device-independent fonts; bind left `0.0.0.0` per + operator. auto-memory `reference_zellij_web_seat`. + +- **Backups — recovered + hardened (2026-06-20), STILL OPEN:** rotate the 5 disclosed + rest-server creds (operator, offline); confirm esh-vm-db's resticprofile includes DB + dumps (esh-pve-nas restic-only). Topology + 2-min check: `docs/runbooks/backups.md`. + +- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model hooks) + CANCELLED (Worldtree model-agnostic → no deploy path). Whole footprint = gateway calls + to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at + `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete). Detail in Recent decisions. -- **R22 (brokkr/dwarves) — CONCLUDED, gateway-only.** Phase B (internal-model - hooks) CANCELLED (Worldtree model-agnostic → no deploy path). R22 whole footprint - = gateway calls to free `qwen3.5-122-a10b` via a PERSISTENT full-access key at - `/home/lkraven/.r22-gateway-key` (mode 600, carries paid GLM, do NOT delete). - Detail in Recent decisions. - -- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the - gateway (lists in /v1/models but 400s — backend displaced); docker-daemon default - log-cap on ana-docker (root → infra-ops; systemic fix behind the 94 GB disk - incident). +- **Open follow-ups (low-priority):** clean phantom `qwen3.6-35b-a3b` off the gateway + (lists in /v1/models but 400s — backend displaced); docker-daemon default log-cap on + ana-docker (systemic fix behind the 94 GB disk incident). - **Standing / parked (from prior):** Mac Pro migration (`migration-plan.md`, - hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys - rotation queue; clean legacy `news-digest` on ana-docker. + hardware-gated); R17 v2 corpus push HELD (distribution barred); disclosed-keys rotation + queue; clean legacy `news-digest` on ana-docker. -- **Worldtree config-propagation (reference):** demo+personal bind-mount - `providers.yaml`/`model_roles.yaml`/`policies.yaml` from - `/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from - worldtree-dev canonical sha); pinned is IN-IMAGE. Reload via `docker restart - `, NEVER `compose up` (stale-`:latest` footgun); WT PDP is rule-based - (a tier needs allow RULES, not just scopes). +- **Worldtree config-propagation (reference):** demo+personal bind-mount config + (`providers.yaml`/`model_roles.yaml`/`policies.yaml`/`defaults.yaml`, now incl. the + #314/#322/#317 additions) from `/opt/worldtree{,-personal}/config` + (infra-ops-deployable, byte-identical from worldtree-dev canonical sha); pinned is + IN-IMAGE. Reload via `docker restart `, NEVER `compose up` (stale-`:latest` + footgun); WT PDP is rule-based (a tier needs allow RULES, not just scopes). **#317 + lesson: config REMOVALS are NOT backward-compatible with the still-running image — push + promptly, don't bounce the instance in the window.** ## Recent decisions +- `[2026-06-25]` **althing re-architected to the v0.17 lean multi-machine bus; nh3-extdev + stood up as a mesh peer (MODEL B).** v0.15.0's lean-bus cut RIPPED + moderation/chamber/forseti-daemon/agent-runner/redis-valkey; **v0.17 = per-box + local-SQLite bus + a courier/receiver for P2P over the 10.x net** (installed per-box via + `uv tool install`, NOT CI). Migrated nh3-dev's client v0.15→v0.17. nh3-extdev + provisioned as a full mesh peer with the operator's **MODEL B** (dedicated `althing-svc` + service account + group-shared `/srv/althing` root, so multiple OS users share one + config/DB; `ALTHING_ROOT` via /etc/profile.d). Verified multi-user concurrent rw. + Supersedes the old althing Tools-row (was chamber/daemons/valkey). auto-memory + `reference_nh3_extdev_althing_mesh`. + +- `[2026-06-23]` **zellij native web client piloted on nh3-dev** (`zellij-web.service` + :8443, native TLS), system-installed for all users, ALONGSIDE ttyd (kept as the no-auth + fallback). Bind left `0.0.0.0` per operator. auto-memory `reference_zellij_web_seat`. + +- `[2026-06-22]` **Worldtree persona-render config arc (#314 relational-stance / #322 + affect-mood gate / #317 valence teardown) pre-synced + deployed green on demo+personal** + — #317 a boot-blocking config REMOVAL (relational_valence). Config-sync recipe + + additions-vs-removals lesson in auto-memory `reference_corviduo_dev_emergency_ops`. + - `[2026-06-20]` **R22 (brokkr/eitri/dwarves) stood down to gateway; full-access R22 key minted; Phase B parked.** Operator caught the raw-weights/GPU ask as premature — Phase A (P00 + NL-rendering) is PROMPT-LEVEL on the LiteLLM gateway, not raw weights @@ -173,15 +207,10 @@ _As of 2026-06-21:_ approach has no deploy path (the pragmatic-outcome gate firing). R22's ENTIRE compute is now the gateway (free qwen3.5-122-a10b MUT); NO raw-weights/GPU ever; the A6000 slice reservation is released (was never provisioned). The full-scope R22 key stays - (pinned to the free qwen-122, GLM untouched). **FOLLOW-UP: clean phantom - `qwen3.6-35b-a3b` off the gateway** (shows in /v1/models but 400s "invalid model" — - backend displaced 2026-06-19; the no-false-alias footgun). **OPERATOR STEER - (2026-06-20): R22 is research for a PRAGMATIC/deployable outcome, NOT advancing-the-art - ("another day's work").** Before the dwarves invest in Phase B, establish a deploy path - (distilled steering-vector / fine-tune applied to a model we already serve, e.g. - Worldtree's) — don't research an avenue we won't implement. Phase A/P00 (the cheap - prompt-level gate) runs regardless (it just tells us if the affect signal exists). - Operator is taking the dwarf discussion directly. + (pinned to the free qwen-122, GLM untouched). **OPERATOR STEER (2026-06-20): R22 is + research for a PRAGMATIC/deployable outcome, NOT advancing-the-art.** Before the dwarves + invest in Phase B, establish a deploy path (distilled steering-vector / fine-tune + applied to a model we already serve). Operator is taking the dwarf discussion directly. - `[2026-06-20]` **claude-bot issue-scope token minted for worldtree-dev self-serve** (closes their last `tea`-as-vh fallback — issues; they already self-serve Actions + @@ -189,105 +218,38 @@ _As of 2026-06-21:_ `worldtree-dev-actions-issues-20260621` (id 16) with `write:repository`+`write:issue` via basic-auth (`claude-bot:gitea-password`, POST `/users/claude-bot/tokens`); dropped mode-600 `~/.claude-bot-token-issues` → worldtree-dev swaps. **Gitea COLLAPSES - `read:issue` into `write:issue`** (write implies read) so the granted set shows - write-only but read IS satisfied (GET issues → 200). Old token (id 15, - worldtree-dev-actions-20260620) REVOKED (DELETE 204) after worldtree-dev verified - read+write end-to-end via `core.issues` (show/list/comment on #311) → claude-bot now - carries ONE consolidated worldtree-dev token (id 16) + arbo-ci (id 14), no stale dups. - Advances the credential-migration directive (worldtree-dev fully self-serve: Actions + - deploys + issues on one token). ALSO this session: re-staged cb2a79a `policies.yaml` (readonly-admin allow - rules) to demo+personal (see Current-state ratatoskr entry). + `read:issue` into `write:issue`** (write implies read). Old token (id 15) REVOKED after + verification → claude-bot carries ONE consolidated worldtree-dev token (id 16) + arbo-ci + (id 14). Advances the credential-migration directive. - `[2026-06-20]` **rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned** (operator-directed). rest-server-ana fixed (mount-a + force-recreate via `infra-ops@ana-docker`); freshness alert + fstab hardening landed - (see Current-state BACKUP entry + `docs/runbooks/backups.md`). Minted worldtree-dev - **admin-tier Heimdall keys** on demo (key_id d113207c) + personal (f4f75adb) via - `POST /admin/keys {tier:admin}` + `POST /admin/users/worldtree-dev/tier`, verified - read+write, dropped mode-600 to `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev - now self-serves key minting (cuts infra-ops out of the per-key loop, like the - claude-bot gitea token). Cred rotation (5 rest-server pw) BELAYED per operator. - -- `[2026-06-20]` **Worldtree capability-gateway aliases stood up; NO `wt-` - prefix (operator) — reuse the existing capability convention.** gen/ - gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge - (passthrough)/gen-frontier/-reasoning. The **swap-transparency rule**: - embedding stays model-named (`qwen3-embedding`) because vectors aren't - swap-transparent; everything else can swap behind a stable alias. (d372103, - 7819f96) - -- `[2026-06-20]` **Removed the litellm `*` wildcard → llama-swap** (operator; - llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now - 404 cleanly instead of a misleading "Connection error" — closes the - long-standing footgun. (d0eb09c) - -- `[2026-06-20]` **Langfuse retired** (operator) — redundant with LiteLLM - native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse - spew). Callbacks dropped, stack torn down. (f8eda1c) - -- `[2026-06-20]` **OmniVoice streaming `/tts` — vendor chatterbox-fast's - scheduler, not its whole package** (operator-approved vendor-copy). Diffusion - TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox- - fast remains the lowest-latency front-end. (288d085, cd92b85) + (see `docs/runbooks/backups.md`). Minted worldtree-dev **admin-tier Heimdall keys** on + demo (key_id d113207c) + personal (f4f75adb), dropped mode-600 to + `~/.wt-admin-{demo,personal}` on nh3-dev → worldtree-dev now self-serves key minting. + Cred rotation (5 rest-server pw) BELAYED per operator. - `[2026-06-20]` **claude-bot → ADMIN on vh/Worldtree** (operator-authorized; one-time use of operator vh-admin to enable the migration) — claude-bot self-serves WT deploys/tokens henceforth, reducing personal-cred fallbacks. -- `[2026-06-20]` **Daily image+buildx prune cron on ana-docker** (the missing - automation behind the recurring image bloat) — `lkraven` crontab, prune - unused images >48 h + per-builder buildx cache. The real systemic fix - (daemon log cap) is deferred to clean context. - - `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.** Stand up service accounts (claude-bot Gitea user + scoped tokens, distinct litellm admin key); re-mint consumer creds under them; flag personal-cred fallbacks until done. (auto-memory `project_migrate_infra_access_to_claude_credentials`) -_113 older entries archived to archival-memory.md._ +_118 older entries archived to archival-memory.md._ ## Tried and abandoned -- `[2026-06-20]` **A full disk wedges gitea's Actions notifier SILENTLY — and it - STAYS wedged after disk recovers; only a `docker restart gitea` clears it.** After - the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push - (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while - the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea - errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree - (worldtree-dev's deploys) → it's global, not repo-specific. Fix: `docker restart gitea` - on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively - fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred - docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). - -- `[2026-06-20]` **Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — - ABANDONED, too slow (operator confirmed).** Stood up as a temporary speed-check - displacing mistral-small-4; measured base **~19.2 tok/s** decode on one Blackwell - (vs `gen`/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" - suspicion. KEEP these serving findings: (1) vLLM **nightly** loads the HF-format - NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same `mistral3` - arch) — via the FlashInfer Cutlass NVFP4 kernel + `turboquant_4bit_nc` KV; TP=1 - fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) **EAGLE was a dead end**: the - official native FP8 head (`mistralai/Mistral-Medium-3.5-128B-EAGLE`) crashes in - CUDA-graph replay with graphs on, and nets NO speedup with `--enforce-eager` - (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 - (non-official-FP8) base. Untracked workspace files (`stacks/mistral-medium-3.5/` + - `playbooks/serve-mistral-medium-3.5.yaml`) removed. ⚠️ Weights still staged on - ana-ml2 `/tank/aimodels/mistral-medium-3.5-{nvfp4,eagle}` (~70 GB+) — pending - operator OK to free. - -- `[2026-06-20]` **A 94 GB unrotated `langfuse-clickhouse` json.log was the - REAL disk-filler (not image bloat) — there was NO log rotation anywhere.** - No per-stack `logging:` limit and no docker-daemon default → ClickHouse's - verbose stdout grew unbounded to ~94 GB and filled ana-docker's `/` - (healthchecks then failed fleet-wide on `runc /tmp: no space left`). Lesson: - on a full disk, the symptom (all-unhealthy) is the FORK-can't-write - healthcheck, not the containers; find the runaway via per-container-dir `du`, - not `docker system df`. The systemic fix is a daemon-default `log-opts` - (`max-size`/`max-file`) — NOT just per-service patching. - -- `[2026-06-20]` **OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — - diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk - streaming is a net loss.** Measured ~2.8× (32-step) / 5.6× (16-step) on the - 3090; every chunk costs ~the same regardless of length, so naive - per-sentence streaming triples total gen time + starves. Use few/aggressive - chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end. +- `[2026-06-25]` **althing "unreachable: — retry later" can MASK an app-level + 500.** A multi-hour "intermittent :8087 / firewall flap" hunt was a red herring: raw + network was always clean (curl POST to the receiver `:8087` worked; a connect probe = 0 + fails). Root cause = the receiver's DB agents-table wasn't synced with the config roster, + so delivery 500'd `"unknown to: "`, which the courier MAPPED to "unreachable" + (looks like network/DNS). **Diagnose: raw curl to :8087 + a connect-probe pass ⇒ NOT + network — look up-stack.** Fixed in althing **v0.17.1** (receiver auto-ensures the + recipient on delivery). Stop-gap on older builds: `ALTHING_HANDLE= althing-cli inbox` + as the receiver user syncs config→DB; re-run after any roster change. auto-memory + `reference_nh3_extdev_althing_mesh`. - `[2026-06-20]` **rest-server `.htpasswd: permission denied` = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission.** @@ -297,12 +259,6 @@ _113 older entries archived to archival-memory.md._ a crash-looping legacy container until you've checked fstab + the docs — it was the live ana-side restic target. -- `[2026-06-20]` **Named buildx builders accumulate cache the standard prunes - MISS.** `docker image prune -a` + `docker builder prune` don't touch - `buildx_buildkit_-builder0_state` volumes (per-builder buildkit - state). Prune them with `docker buildx prune --builder -af`, or - `docker volume rm` orphaned ones (builder already removed). - - `[2026-06-20]` **The DEFAULT `ssh ana-docker` is `lkraven` (no NOPASSWD) — but `ssh infra-ops@ana-docker` HAS NOPASSWD root** (corrected later same day). Early on a `sudo cp` as lkraven silently failed (password prompt) → one @@ -311,4 +267,4 @@ _113 older entries archived to archival-memory.md._ reach for `infra-ops@ana-docker` for sudo ops; lkraven-owned files (litellm config, most stack compose/conf) take plain `cp` under either identity. -_93 older entries archived to archival-memory.md._ +_98 older entries archived to archival-memory.md._