esh-pfi-infrastructure/archival-memory.md

Archival memory — eshpfi-management

Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns.

Recent decisions (archived)

• [2026-05-12] corviduo-dev (Worldtree-team dev VM, 10.250.50.152, CT 106 on pfi-pve) added to servers/ inventory. Treat like SF client hosts: PFI hosts + provides emergency-ops backstop; Worldtree team owns OS config + deploys + backup decisions. Archived 2026-05-27.

• [2026-05-12] Worldtree :latest tag drift bug — fixed by health-gated :latest advance in vh/worldtree's deploy workflow (architect commit 8ef3801): only tag :latest AFTER the new container's /health probe passes. Build-on-host stacks here don't have this problem because the playbook always builds the SHA-tagged image from a git reset --hard <ref> checkout. Archived 2026-05-27.

• [2026-05-12] asset-engine stack scaffolded LAN-direct at http://10.250.50.70:8200. Initially included Traefik labels for public hostname; user pulled them out (internal tool, no public TLS surface needed). Pattern: internal tools default LAN-direct; Traefik wiring only when external/TLS required. Archived 2026-05-27.

• [2026-05-12] asset-engine catalog gains lifecycle: { stack, vram_gb, gpu_device_id } per irv-ml1 service for the orchestrator feature. SSH keypair scaffolded at ana-docker:/opt/docker/conf/asset-engine/ssh/ for asset-engine container → irv-ml1 orchestration via dedicated ed25519 key. Archived 2026-05-27.

• [2026-05-13] pull-hf-repo.yaml is the canonical HF-fetch playbook on ana-ml2. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download calls. Archived 2026-05-27.

• [2026-05-13] Selene-1-Mini-Llama-3.1-8B added to llama-swap as judge model. mradermacher i1-Q6_K imatrix quant (~6.5GB). AtlaAI reward/eval model — temp 0.01, ctx 32K, q8_0 KV cache. New JUDGE / EVAL MODELS section in stacks/llama-swap/conf/config.yaml. Archived 2026-05-27.

• [2026-05-13] vllm-qwen3 → vllm stack rename. Added vllm-reward service (Skywork-Reward-V2-Llama-3.1-8B-AWQ classifier). Three vLLM services share GPU 1 (embed 0.20, rerank 0.20, reward 0.30 utilization; 30% headroom). All use --runner pooling; classification drives via model's architectures: [LlamaForSequenceClassification] in config.json, NOT --task classify (deprecated in vLLM 0.19.1). Archived 2026-05-27.

• [2026-05-13] /tend-docs first pass deletions: stacks/infinity/ removed (retired by vllm). Archived docs/asset-engine/design-brief.md → docs/archive/asset-engine/ with archival header. Fixed pfi-pve VM list to full qm list enumeration. Dropped stale weak-password section from pfi-postgres (rotation done 2026-04-23). Archived 2026-05-27.

• [2026-05-14] althing-chamber stack scaffolded: chamber + forseti. Internal LAN-only at port 7881 (chamber default 7878 collides with task-board). Two-service compose, shared SQLite bind-mount, build-on-host pattern via vh/althing's gitea-workflow. Forseti is the canonical dev for this stack (galdrabok is on a different project). Archived 2026-05-31.

• [2026-05-16] althing-chamber Phase 2: added althing-agent-runner as third compose service (worldtree-driver agent dispatcher). All three althing services use the same image; command: selects entrypoint. Safe to enable preemptively (sleeps when no driver=worldtree handles declared). Archived 2026-05-31.

• [2026-05-17] Phase 3.1 cross-process streaming uses Valkey 8 alpine as a sibling compose service in stacks/althing-chamber/, redis-protocol pub/sub for high-volume msg_delta / msg_thinking / msg_start / msg_complete event kinds. DB bridge keeps msg_curated + floor_grant (structured / canonical). Two-channel architecture, no overlap. chamber + agent-runner depends_on: valkey: service_healthy. Archived 2026-05-31.

• [2026-05-17] Worldtree admin workflow shift (per vh): infra-ops gets its own permanent admin-tier key (61419c92, stored at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin). Future admin ops route through this key, not the bootstrap admin via docker-as-root. Archived 2026-05-31.

• [2026-05-17] Worldtree env-var addition checklist: anytime introducing os.environ.get("FOO") in worldtree code, update BOTH .env.example AND compose.yaml's &worldtree-env anchor in the same PR. Same Z_AI_API_KEY-shape footgun bit BIFROST_CLIENT_ALLOWED_HOSTS (#170) until worldtree-dev added the passthrough line in 08f02b2. Archived 2026-05-31.

• [2026-05-18] Volva systemd install complete after three-stage debug. Final unit at /etc/systemd/system/volva.service runs as User=lkraven with ProtectHome=read-only + ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex carve-outs for state writes. VOLVA_ALTHING_CLI=/home/lkraven/ .local/bin/althing-cli + ALTHING_HANDLE=volva both pinned in env.sh. Archived 2026-05-31.

• [2026-05-19] Worldtree CD disk-hygiene strategy: watermark gate (env-tunable threshold + window, fail-loud on still-low post-prune)

  • eager post-deploy prune (only after :latest advance succeeds, uses docker image prune -a --filter "until=24h" which respects in-use semantic — protects pinned + personal images automatically). Combined: demo VM holds ~24h of deploy history instead of unbounded accumulation. Shipped in vh/Worldtree PR #184 (306cd61 + 613dac2 + bd91df5). Archived 2026-05-31.

• [2026-05-19] Skaldsong CD shape: shape (1) of three operator options — container + Gitea registry + pull-restart, matching Worldtree's pattern. Target host ana-docker (NOT nh3-dev where skaldsong-dev runs for iteration). SHA-pin only for now; health-gated :latest advance is a follow-up once /health exercises Worldtree

  • Kokoro reachability. Archived 2026-05-31.

• [2026-05-19] Skaldsong prod (ana-docker) switched from demo Worldtree (:8080) to personal (:8081). Same user_id=skaldsong as the nh3-dev hand-launch key — shared Heimdall agent slot (skaldsong:wizard-v2), different key_ids for independent rotation. Demo Worldtree stays for isolation; personal becomes the multi-consumer dev iteration instance. Archived 2026-05-31.

• [2026-05-19] mead-hall Bifrost v0.3 end-to-end smoke green. Closed task #32 (althing thread 01KRV1M2KW6N6HBEXGTH72QXCA). Wire layer (handshake + binding + dispatch) + data-flow (per-dispatch JWT claims → ctx.session_id populated → real session-scoped data) + agent-loop (LLM reads + quotes back) all proven. Resolves the "stalled mid-Worldtree" state from the 2026-05-17 snapshot. Archived 2026-05-31.

• [2026-05-25] v0.25.3 lofn tuning: temperature 0.6 → 1.0 + repetition_penalty 1.0 → 1.15 on default+fast profiles. Heretic-abliterated qwen3.6 was locking into degenerate attractors at the model's thinking-mode floor (0.6). Pattern: abliterated/uncensored Qwen variants need higher temp + non-trivial rep-penalty than base, NOT the model-card's documented floors. Archived 2026-05-31.

• [2026-05-25] Worldtree #205 v0.25.2 ships /app/config/ as bind-mount + root-then-drop entrypoint shim (gosu). Operators get persistent per-instance config without container-rebuild. Same bind-mount pattern hit twice subsequently in v0.27.0 (selene) and v0.29.9 (echo) — bind-mount shadows image-baked defaults, so every new required key surfaces as a crash-loop on existing deployments. The v0.29.12 canonical example files close this loop. Archived 2026-05-31.

• [2026-05-26] Worldtree v0.27.0/v0.27.1 fixes Tier 3 GET visibility. available_agents() helper was over-applied to GET /agents/<id>, masking ALL Tier 3 agents regardless of row state. Bug only visible as "agent not found" via GET; storage was fine (silent-2xx PATCHes had persisted correctly). v0.27.1 added fail-fast hardening for the startup pre-resolve fragility class. Archived 2026-05-31.

• [2026-05-26] Skaldsong v0.30.7 defensive 409→PATCH fallback. v0.30.6's GET-then-define-or-patch path crash-looped against pre-v0.27.0 Worldtree's GET-visibility bug (GET 404 phantom → define 409 conflict). v0.30.7 catches the 409 and falls through to PATCH (which silently 2xx'd on the pre-fix Worldtree). Archived 2026-05-31.

• [2026-05-27] Worldtree v0.29.x landed full saga→echo refactor + config-validator hardening (v0.29.10 create_provider family-before-regex; v0.29.11 collect-then-raise echo startup validators; v0.29.12 ships providers.yaml.example/defaults.yaml.example canonical configs; v0.29.13 reasoning_content extraction + catalog family lookup). Operator-asked, worldtree-dev-shipped, CI-deployed. Archived 2026-06-01.

• [2026-05-27] artemis-31b-v1i added to llama-swap + worldtree personal. BeaverAI Gemma 4 31B Q6_K (~28.6GB), 128K ctx, --reasoning-format deepseek (gemma format unsupported in deployed llama.cpp). Worldtree catalog family: gemma so GemmaProvider routes reasoning tokens. Archived 2026-06-01.

• [2026-05-27] Skaldsong streaming TTS v0.32.0→v0.32.2: chunked-batch SSE (one Kokoro POST per paragraph); defensive event_stream catch-all; NDJSON parsing for Kokoro /dev/captioned_speech multi-line responses. Archived 2026-06-01.

• [2026-05-31] Dia2 deployed as two fixed-model instances (dia2-2b :8200, dia2-1b :8202) from local/dia:v2, retiring legacy Dia 1.6B; catalog dia entry removed → dia2-2b + dia2-1b (breaking for asset-engine). Rationale: the devnen wrapper is single-model and IGNORES the OpenAI model field (verified on its live OpenAPI), so the only way to offer both Dia2 models as real per-request asset-engine choices is one fixed endpoint per model. 3139e81 (deploy), db15638 (catalog swap). Archived 2026-06-03.

• [2026-05-31] Both dia2 catalog entries route to the wrapper's richer /tts endpoint (not /v1/audio/speech) to expose the full control surface (cfg_scale/temperature/top_p/cfg_filter_top_k/voice_mode/clone); all defaults sourced from the wrapper's CustomTTSRequest Pydantic blessed values. Voice default is voice_mode: clone + clone_reference_filename: Abigail.wav so a stable (non-random-gender) voice is the out-of-box behavior. 55602b7, 5c47843. Archived 2026-06-03.

• [2026-05-31] Zonos REST adapter (stacks/zonos/adapter/, local/zonos-api) — thin OpenAI-ish /v1/audio/speech FastAPI in front of the Gradio-only Zonos SDK; JSON-envelope {audio, audio_format, seed} (Zonos is the fleet's first seedable TTS). Port 8203 (moved off 8201 — collided with csm). Built; NOT deployed (stack down for VRAM). Also fixed the upstream image's missing CMD (71df6f7). 81efa8d. Archived 2026-06-03.

• [2026-05-31] Catalog schema regenerated: added CatalogLifecycle + reproducibility.seed_field (b7b2130). Resolves the stale-schema hand-off; catalog now validates clean. (asset_engine catalog.py Pydantic already supported both — schema file was just behind.) Archived 2026-06-03.

• [2026-05-31] TTS bench expanded with stacks/{dia,zonos,csm} (666f7f3 dia+zonos, a4b8c2a csm). The bench already had Fish S2-Pro / Chatterbox-Turbo / IndexTTS-2 / CosyVoice3 / Kokoro / VibeVoice / Qwen3-TTS / Kyutai. (csm since removed 2026-06-01.) Archived 2026-06-03.

• [2026-05-31] Remote browser/iPad/Vision-Pro driver seat for the agent-fleet zellij session Claude stood up on nh3-dev (ttyd behind Caddy, network-gated). Out of this repo — full architecture + the HTTP2/OSC52/Safari-auth gotchas in auto-memory reference_ttyd_fleet_seat. Archived 2026-06-03.

• [2026-05-30] esh-docker-vm NFS boot-ordering fix: playbooks/fix-esh-nfs-boot-ordering.yaml (c0458d9, + 53157b1 drop-in filename-collision fix) adds _netdev,nofail to the four 10.0.50.50 NFS mounts + a docker After=remote-fs.target drop-in — resolves paperless Exited(255) on reboot. traefik also gained restart: unless-stopped. Full incident → auto-memory incident_esh_docker_nfs_boot_race. Archived 2026-06-03.

• [2026-06-01] Fish cloning VERIFIED competent (ECAPA-TDNN) — retracting the earlier "weak cloner" call. Isolated test: Imogen-referenced clone ~0.79 cosine to the real Imogen.wav vs ~0.10 for the no-reference default; transcript condition (correct 0.787 / bogus 0.778 / empty 0.738) barely moves identity (affects pronunciation, not timbre). Root cause of "Imogen sounds nothing like British" = the "undefined" select bug feeding Fish its default voice, NOT Fish. So the entire Fish-Imogen saga was the undefined bug; no Fish-side fix needed. (Methodology lessons → Tried and abandoned.) Archived 2026-06-04.

• [2026-06-01] CSM (Sesame csm-1b) torn down entirely — removed from catalog, stacks/csm/, playbooks/deploy-csm.yaml, and host (c54ab13). Two reasons: (1) deep-research verdict — the acclaimed Maya/Miles demo is a fine-tuned, larger variant Sesame never open-sourced; the open 1B base ships no usable voices, can't generate text, English-only, no real-time streaming, absent from leaderboards, dominated by Kokoro/Dia2/Fish/IndexTTS. (2) The upstream image is rotted (see Tried and abandoned). asset-engine re-synced (306aae9). Archived 2026-06-04.

• [2026-06-01] Chatterbox catalog v1→v2 (93cedd6) — re-point to the devnen wrapper's /tts (was the thin OpenAI /v1/audio/speech exposing none of the emotion knobs → poor output). Default exaggeration: 0.5 per Resemble's README ("works well for most"); the server SHIPS demo-tuned 1.3 which over-acts — deliberately NOT adopted. 28 predefined voices via /get_predefined_voices (default Emily.wav). Corrected seedable false→true. Archived 2026-06-04.

• [2026-06-01] Fish-S2 catalog v1→v6 (f3c260e..38d9e3b) — was a blank 2-field form; now exposes the full fish-speech ServeTTSRequest surface

  • a 32-voice reference_id "Voice" dropdown (STATIC list — Fish has no /voices API). reference_id-by-name works ONLY when each voice is staged as a references/<name>/ dir with <name>.wav + <name>.lab — was silently a no-op until that layout fix on 2026-06-01 (see the two-root-causes entry above); the inline-path-as-filesystem-path references shape 500s (see Tried and abandoned). Corrected vram 4→24 GB, cold_start 8→240s, seedable false→true. Added 3 consenting British-female VCTK voices (Imogen=p225/Eleanor=p228/Beatrice=p229). Archived 2026-06-04.

• [2026-06-01] blendable: false catalog flag — DEFERRED, pending asset-engine-dev honoring it in the form template. Clone-voice selects (chatterbox/fish/dia2) must render as plain dropdowns, not the Kokoro blend widget (blending reference clones is nonsensical). I ship the catalog flag + schema field once they read it. Tracked: althing msg 01KT2K2SY9N7AY69R9V0B4RXSW to asset-engine-dev. Archived 2026-06-04.

• [2026-06-01] Fish reference_id empty-dir fix shipped (c5bbb90) — Populated references/<name>/<name>.wav+.lab for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4" lines. Archived 2026-06-11.

• [2026-06-02] Chatterbox → main TTS engine; build custom chatterbox-fast streaming container. Workload = single-stream interactive. GPU placement: 3090 (device 0) if it fits else A6000 (device 1). Cutover: parallel catalog entry, burn in beside live chatterbox, then flip. Streaming = adaptive buffer-ratchet chunking (native frame-streaming abandoned). Since extracted to its own repo vh/chatterbox-fast (auto-memory reference_chatterbox_fast_repo). Archived 2026-06-11.

• [2026-06-02] Sentence-splitting loses quality (operator-corrected). Claimed naive sentence-level streaming has "zero quality loss" — WRONG. The T3 AR backbone conditions prosody on the WHOLE text; splitting loses cross-sentence prosodic context even though voice timbre stays (reference-conditioned). No artifacts ≠ no quality loss. Hence adaptive-chunk design (maximize context per chunk subject to latency budget), not fixed per-sentence splits. Archived 2026-06-11.

• [2026-06-03] Declined worldtree v0.30.4 staging deploy — that's worldtree-team's CI/CD lane (a developer staging/vX.Y.Z git-tag promote), not infra-ops. They self-corrected to the same conclusion independently. Archived 2026-06-13.

• [2026-06-03] R14 scope = (a) provision-only. infra-ops provides box + CUDA env + engines + weights + NFS; brokkr/dev wires arms.py + runs — keeps infra-ops OFF the VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4 highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000 free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = cuda:0 NATIVE (≠ docker =1). Archived 2026-06-13.

• [2026-06-03] yt-voice-clipper push-to-deploy via gitea webhook (operator-directed, after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the operator. The proxy env + Homepage labels live in the host-specific override (untracked → survive the auto-deploy's git reset --hard), NOT yt-voice-clipper-dev's image. Runbook d4f180d. Archived 2026-06-13.

• [2026-06-03] yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential egress, NOT cookies/PO-token. YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated service. Archived 2026-06-13.

• [2026-06-04] brokkr user (no-sudo) on irv-ml1; R14/R15/R16 substrate moved to /home/brokkr. Persistent box services there need SYSTEM systemd units (see Tried). Archived 2026-06-14.

• [2026-06-04] ollama upgraded 0.9.0→0.30.4 on irv-ml1 (Ministral-3 is a Dec-2025 model the old engine refused); A6000 pinned by UUID not index (native fastest-first ≠ nvidia-smi PCI). Archived 2026-06-14.

• [2026-06-04] Worldtree demo/pinned/personal deploys are ALL CI/CD, not infra-ops — a "deploy vX.Y.Z" request to infra-ops is MISROUTED → point them back to their pipeline. (d8d776c, auto-memory reference_worldtree_deploys_cicd) Archived 2026-06-14.

• [2026-06-04] infra-ops NOPASSWD-sudo identity commissioned, scoped to PFI boxes (+esh-docker-vm by operator override) — so infra-ops completes DevOps end-to-end vs handing the operator sudo steps. Dedicated key, sudo log_output, key-gated. (8c32a05, auto-memory reference_infra_ops_sudo_identity) Archived 2026-06-14.

• [2026-06-04] phi4 ships the CANONICAL/official Phi-4 chat template, NOT Ollama's (Ollama's omits the system <|end|>). Applied an Ollama-matching override then reverted — ship correct, not the benchmark quirk. (90e08f0→27eb537; "headgun" lesson in Tried.) Archived 2026-06-14.

• [2026-06-04] phi4-mini FP8 on ana-ml2 vLLM is the nevermore summarizer; granite-4-small retired from llama-swap. 50K ctx + FP8 KV. (40a374b) [superseded by the 2026-06-05 granite-4.1-8b decision] Archived 2026-06-14.

• [2026-06-08] volva.service + heid.service removed from nh3-dev — vestigial systemd daemons; Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid 12aa5a9); volva.service was crash-looping 203/EXEC. (6e2f80e) Archived 2026-06-16.

• [2026-06-05] Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer. Beat phi4 on precision in brokkr's R15 P03. Staying FP8, not Q4/AWQ — primary workload is high-concurrency, where FP8 scales ~linearly (2010 tok/s @ C=32). vLLM vllm-granite :8004 GPU 1, official IBM compressed-tensors FP8. (Then on Ada; box has since gone Blackwell.) (34a43a0, auto-memory reference_ana_ml2_vllm_granite) Archived 2026-06-16.

• [2026-06-05] Langfuse v3 on ana-docker (:3001) as the gateway trace UI; LiteLLM success_callback:[langfuse] live. Pretty traces + tok/s dashboard. NOT a prerequisite (spend_logs already capture tokens+latency). (9171e6a) Archived 2026-06-16.

• [2026-06-05] Ollama BANNED fleet-wide (operator directive) — never stand one up; tear down any found; serve via llama-swap or vLLM. (auto-memory feedback_avoid_ollama) Archived 2026-06-16.

• [2026-06-05] ComfyUI / FLUX.2 work split to ~/development/comfy-dev (dedicated repo + agent). eshpfi keeps the comfyui/arbo stack compose; comfy-dev owns the model/workflow knowledge. (auto-memory reference_irv_ml1_ampere_quant) Archived 2026-06-16.

• [2026-06-05] Worldtree summarizer config refresh DEFERRED to Worldtree #254 (granite-4.1-8b is the structured-output profile, ON HOLD, no live consumer). Bind-mount CONFIGS (providers.yaml, vh-owned) ARE infra-ops's to apply directly — only the app/image DEPLOY is CICD; the .env is deploy-owned. (auto-memory reference_worldtree_deploys_cicd) Archived 2026-06-16.

• [2026-06-13] Arbo prod bearer auth turned OFF — WireGuard is the access boundary (operator decision; reverses ADR-0001's "closed the open-auth hole"). ENGINE_TOKEN must be ABSENT, not empty (empty-string still gates) — removed from BOTH the host .env AND the compose environment: injection line. Original token backed up at irv-ml1:/opt/docker/compose/arbo/.env.pre-auth-off.bak; comfy-dev updated their ADR-0001. Tracking: db97899 + playbooks/arbo-disable-engine-token.yaml. Archived 2026-06-16.

• [2026-06-13] Storetank image-models archive DECOMMISSIONED; arbo is the single live ComfyUI model tree (502 G). Curated /storetank/image-models/comfy (was 919 G, the native /opt/ComfyUI/models symlink target) → killed everything superseded by arbo's current gen (Hunyuan, WAN2.1, FLUX.1, Chroma, SD3.5, orphaned umt5+llava ≈ 739 G) + migrated the keepers (gen-agnostic utilities + the SDXL/Pony stack, 177 G) into /storetank/arbo/models (same-fs move, skip-existing protects prod). Tracking: docs/storetank-image-models-archive.md + docs/arbo-comfyui-model-catalog.md (1902425→5007ec1). Archived 2026-06-16.

• [2026-06-13] GRANITE_KEY provisioned to comfy-dev's nh3-dev dev env at ~/.arbo_granite_key (0600) for the hero gen+judge script — verbatim copy of the prod arbo-prompt-enhance vkey (now extended to reach BOTH granite-4.1-8b AND qwen3.5-9b-fp8); nothing minted. The vkey README's "granite-only" wording was stale → corrected (f32c6dd). Archived 2026-06-16.

• [2026-06-13] ana-ml2 upgraded Ada → dual RTX PRO 6000 Blackwell Max-Q (96 GB each, cc 12.0 / sm_120; was dual RTX 6000 Ada 48 GB / cc 8.9 — confirmed live via nvidia-smi). Unlocks NVFP4 (FP4 tensor cores) and doubles VRAM headroom. CLAUDE.md GPU-spec doc-fix LANDED 355a240 (operator). Tracking: 19a07b9/1e2a3a1 ("Blackwell 96GB"). Archived 2026-06-16.

• [2026-06-13] NVFP4-W4A4 is infeasible for Granite — FP8 stays the Granite-on-Blackwell format. W4A4 collapses at 30k context, proven producer-independent (modelopt AND llm-compressor both clean-NONE from the same BF16 base + wikitext-2k calib). No 4-bit wins both axes: W4A4 = quality collapse; W4A16-NVFP4/AWQ = weight-only dequant → bf16 (no FP4-core speedup). 30B retired. (auto-memory reference_nvfp4_w4a4_granite_infeasible) Archived 2026-06-16.

• [2026-06-13] Qwen3.5-9B VL (FP8) deployed on ana-ml2 GPU 1 — qwen35-vl stack, :8007, gateway alias qwen3.5-9b-fp8. Pinned nightly digest, not :latest: the stable release quantizes the VL vision tower under --quantization fp8 → garbage vision (LM fine); the nightly correctly excludes it. Re-pin + drop the pin once that exclusion lands stable. (2e3dcc2) Archived 2026-06-16.

• [2026-06-13] comfyui 325 G model tree migrated worktank → /storetank/arbo (worktank 97% → 26%). arbo is the consuming app; overlay bind-mount via COMFYUI_MODELS_DIR. (38186be) (See the 2026-06-13 archive-decommission decision above — this tree later absorbed the storetank-archive keepers, reaching 502 G.) Archived 2026-06-16.

• [2026-06-13] GPU layout settled on the Blackwell box. GPU 0 held free for large-model hot-loads (llama-swap pinned, edf0f91); GPU 1 steady-tenant — granite 131k ctx, qwen 65k, embed/rerank/reward trio, ~3.5 GB free (1e2a3a1, 19a07b9; trio re-floored for 96 GB, 20×-parallel-stable). embed/rerank left at floor — long docs chunked BEFORE embedding. max-model-len is a ceiling not a reservation. Archived 2026-06-16.

• [2026-06-13] Prefix caching pinned explicit on granite + qwen — benched ~6.5× faster TTFT on a shared ~4.5k-token summarizer template; soft/evictable, neutral when prefixes don't repeat. vLLM v1 defaults it ON (granite) but the qwen nightly defaults OFF — pin both. (a9a2be7) Archived 2026-06-16.

• [2026-06-13] granite-4.1-8b listed as the always-available summarizer/classifier + a shared all-agents key minted (operator-directed). Global ~/.claude/CLAUDE.md Global-tools entry; key alias all-agents-local, scoped to the FREE local models only (granite + qwen-vision + embed/rerank, NOT paid GLM), internal-gateway-only, rotatable. (auto-memory reference_litellm_gateway) Archived 2026-06-16.

• [2026-06-13] arbo engine + frontend stack stood up (ADR-0001) — irv-ml1 co-located inference engine (ee57e69), python-based healthcheck (slim image, no curl/wget, bdb3312), frontend ro-mounted from the checkout (922e8ad, ADR-0001 D2). Archived 2026-06-16.

• [2026-06-11] GLM thinking inverted at the LiteLLM gateway (operator call): glm-5.1 defaults thinking-OFF; glm-5.1-reasoning = same z.ai upstream, thinking ON. Mechanism: litellm_params.extra_body:{thinking:{type:disabled}} — drop_params strips a top-level thinking/reasoning_effort but forwards extra_body verbatim to z.ai. Shared-gateway change. (95b2701, auto-memory reference_litellm_gateway) Archived 2026-06-16.

• [2026-06-11] pi coding agent installed on nh3-dev as a GLM 5.1 harness — @earendil-works/pi-coding-agent via bun (npm's global prefix is /usr → needs sudo, bun avoids it). Config ~/.pi/agent/models.json, launcher ~/.local/bin/glm. Archived 2026-06-16.

• [2026-06-11] z.ai web-tools (regin) = z.ai hosted MCP path, NOT the /paas/v4 Tool API. WORKS: MCP streamable-HTTP at https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp, Authorization: Bearer $ZAI_API_KEY (the MCP key, distinct from Z_AI_API_KEY the LLM key). Reference impl = Worldtree's Leif agent. Archived 2026-06-16.

• [2026-06-10] Mac Pro migration framed: workstation-only (M2 Ultra ARM, racked NH3 on-subnet); sidecars stay Linux. migration-plan.md. (See in-flight.) Archived 2026-06-16.

• [2026-06-10] Worldtree deployed-config propagation is infra-ops's OWNED lane (operator ruling). worldtree-dev pings the config delta pre-merge; infra-ops syncs config/*.yaml from MERGED canonical to the /opt/worldtree*/config bind-mounts on demo+personal. The v0.33.8 9-HOUR demo outage (a model_roles.yaml startup-dep that never reached the VM) is the failure mode this prevents. providers.yaml stays hand-tuned. corviduo emergency-ops = ssh vh@10.250.50.152, docker no-sudo. (auto-memory reference_worldtree_deploys_cicd, reference_corviduo_dev_emergency_ops) Archived 2026-06-16.

• [2026-06-09] LiteLLM scoped virtual keys issued to consumers (operator-authorized): brokkr-smithy (all-proxy-models), arbo-prompt-enhance (comfy-dev — granite, later extended to qwen-vision). Mint via /key/generate (master sk-corvid), scope-restricted + rotatable, value → 600 file never the bus. (auto-memory reference_litellm_gateway) Archived 2026-06-16.

• [2026-06-14] ana-ml2 GPU-1 vision upgraded: Qwen3.5-9B → Qwen3.6-35B-A3B (official FP8), served under its TRUE name only. qwen36-vl replaces qwen35-vl on :8007 (a0fed13). The stale qwen3.5-9b-fp8 name is KILLED at vLLM AND the litellm gateway (404/400) — a model is NEVER aliased under a prior model's name (silent substitution = downstream footgun; operator directive). Consumer comfy-dev/arbo migrated; arbo vkeys → all-proxy-models; shared all-agents-local key repointed qwen3.5-9b-fp8 → qwen3.6-35b-a3b. GPU-1 rebalanced for the ~34 GB FP8 weights (granite 0.35→0.24/64K; embed/rerank 0.05→0.03, reclaimed ~4 GB util-waste). Validated: vision correct, 20-concurrent = no OOM. (auto-memory feedback_no_false_model_aliases) Archived 2026-06-18.

• [2026-06-14] NVFP4 was the lighter fit (~21 GB) but is BLOCKED on vLLM — FP8 is the working vision path. nvidia/Qwen3.6-35B-A3B-NVFP4 won't load: the ModelOpt-NVFP4-MoE loader errors on expert/lm_head scale keys across 0.19.1 (w2_input_scale) AND 0.22.0 (lm_head.input_scale, vllm #44081) — a pattern across modelopt NVFP4 MoEs. Revisit NVFP4 (frees ~13 GB on GPU 1) once fixed; the 21 GB checkpoint stays cached on ana-ml2. (SUPERSEDED 2026-06-16 — it loads on vLLM 0.23.0; qwen36 swapped to NVFP4. See the top of this section.) Archived 2026-06-18.

• [2026-06-14] llama-swap qwen3.5-9b GPU-0 pin DROPPED; GPU 0 reserved for a creative-writing model (pick DEFERRED by operator). Deep-research (this session) on big-fast-uncensored creative for a 96 GB Blackwell: GLM-Steam-106B-A12B (already in the llama-swap config — balanced default) vs TheDrummer/Behemoth-X-123B-v2 (prose-tier, tops UGI writing+willingness) vs XORTRON-123B (max willingness, weak prose); GGUF-on-llama-swap is the serving path. Tracking: this session + llama-swap config (GLM-Steam present, untracked by operator choice). Archived 2026-06-18.

• [2026-06-14] R16 splice-pivot yield probe executed (infra-ops ran the irv-ml1 inference for brokkr; brokkr owns design + analysis). See Current state. Tracking: althing thread 01KV010WGS…, gen_yield_probe.py in irv-ml1:~/r16-vmoan-harness. Archived 2026-06-18.

• [2026-06-14] R16 vmoan inline-generation arc CLOSED — v1 at default decode (rep1.2/temp0.8) is the final Chatterbox-tag inline artifact. Operator's ear rejected every alternative: v2/v3 windowing (omission vs coherence-loss), v4 multi-tag (cohesion held but lost to capacity-competition), emergent inline-token modulation (degenerates, not modulates), and the gen-time decode-polish sweep (soft tamers cut the NVV itself — same omission family as v2; p0 baseline beat p1). All adapters v1–v4 + tokenizer.json.v3bak preserved on irv-ml1:~/r16-vmoan-harness. Likely-next direction (deferred, NOT formalized): generate→bin→splice + one-shot-clone NVV pipeline routing around the inline-coherence wall. Tracking: brokkr R16 journal + althing thread 01KV010WGSSMPWRNCPAGSPK15Y. Archived 2026-06-18.

• [2026-06-14] Arbo deploy pipeline fixed, hardened, and version-controlled. Prod rebuilt v0.11.1 → v0.11.6 backend; the webhook machinery (arbo-deploy.sh + arbo-webhook.py, :9009 HMAC listener) is now repo-tracked at stacks/arbo/ (was host-only = recoverability foot-gun). Deploy reaches gitea via the INTERNAL route (10.250.50.70:222) and restarts the engine ONLY on catalog/ changes (graphs/frontend per-request; warn on src/|Dockerfile only — pyproject/uv.lock churn every commit). Operator kept arbo stack ownership in eshpfi (not migrated to comfy-dev's repo). Secret + .env stay host-only. Tracking: 6d66bc2, 6e58e57, stacks/arbo/README Q5. Archived 2026-06-18.

• [2026-06-15] qwen36 thinking-split at the gateway (operator) — base qwen3.6-35b-a3b = thinking-OFF default (the vision judge needs it), …-a3b-thinking = opt-in reasoning. One hybrid checkpoint + --reasoning-parser qwen3; mirrors the glm-5.1 split. The judge's enable_thinking=false is what makes the NVFP4 swap transparent. (6de0844) Archived 2026-06-19.

• [2026-06-15] arbo engine rebuilt v0.11.6 → v0.11.22 from main; catalog re-applied (v0.11.23) — curated /workflows footer live. src/ schema changes (Model/Lora.summary, Workflow.blurb/keywords + the /workflows endpoint) need a MANUAL image rebuild (the webhook warns-only on src//Dockerfile). Built from /worktank/arbo/repo, bumped ARBO_IMAGE in the host .env, recreated + verified healthy (auto-rollback armed, unneeded). comfy-dev's v0.11.20 catalog push had first crash-looped prod (old baked schema extra=forbid vs new git-mounted catalog fields → ValidationError); they reverted the catalog, so rebuild-from-main loaded clean, then they re-applied (v0.11.23). comfyui is a SEPARATE compose project — the rebuild didn't touch it. Archived 2026-06-19.

• [2026-06-15] althing-core refreshed to v0.14.1 box-wide on nh3-dev (forseti + heid both routed it; central-sequenced per DevOps-delegation since uv tool install --force swaps the shared binary all handles use). Fixes the monitor non-truncating-lock bug so stop-monitor/--stop reliably recovers orphans. The editable install (uv tool install --force --editable .) covers every nh3-dev handle; --version trued up 0.14.0→0.14.1; verified --stop works against a real orphan. Archived 2026-06-19.

• [2026-06-15] ratatoskr Tier-3 affect live smoke GREEN (consumer-confirmed). Brokered/minted consumer_id="ratatoskr" Heimdall HS256 key on personal Worldtree (the key IS BOTH the create-session Bearer AND ratatoskr's JWT verifier secret — plain utf-8 string, NOT base64/raw); injected ENV-ONLY to ~/.config/ratatoskr/provider.env (mode 600) on nh3-dev; added 10.100.10.50:8390 to personal's BIFROST_CLIENT_ALLOWED_HOSTS; drove define→session (handshake 200)→turn→affect.emit (200, snapshot persisted). Worldtree signs Bifrost with the caller's raw Bearer verbatim. Drive commands + claims contract in althing thread 01KV4ZYSE3…. Archived 2026-06-19.

• [2026-06-15] Dense Qwen3-VL-32B-NVFP4 A/B'd as arbo's anatomy hero-judge vs the qwen3.6-35b-a3b MoE → dense LOST, pilot torn down, GPU 0 reclaimed. Dense NVFP4 loads clean on vLLM where the MoE NVFP4 doesn't (no expert-scale keys); ~44 tok/s on Blackwell GPU 0; vision tower stays high-precision. But comfy-dev's ground-truth A/B: MoE wins (faster + dual-use as general LLM + slight fine-detail edge), and the subtle-proportion tier is below ALL VLs. Don't re-propose a dense-VL judge swap. (auto-memory project-qwen3vl-dense-pilot) Archived 2026-06-19.

• [2026-06-15] MastMed prototype containerized + exposed publicly via cloudflared. Self-contained static SPA (bun→caddy multi-stage build), basic-auth gated (operator's gating call), ana-docker:8089, cloudflared quick tunnel. See in-flight for the ephemeral-URL caveat + named-tunnel upgrade path. Archived 2026-06-19.

• [2026-06-15] infra-ops bootstrapped onto corviduo-dev (operator-directed) — infra-ops now has SSH + NOPASSWD sudo there, OVERRIDING the fleet bootstrap script's deliberate partner-box exclusion (explicit host arg). corviduo config edits no longer need vh + a sudo password. (auto-memory reference_corviduo_dev_emergency_ops) Archived 2026-06-19.

• [2026-06-15] R18 clip+caption placement = ana-ml2 (operator: "ana-docker… ana-ml2 unless irv-ml1 has room"; irv-ml1 lacked free VRAM for the 30B-A3B captioner). Models staged + stub deployed; see in-flight + the hard real-voice gate. Archived 2026-06-19.

• [2026-06-14] LiteLLM infra-ops admin key provisioned (operator) — resolves the LiteLLM half of the credential-migration directive; use it for ALL gateway ops (NOT sk-corvid). Value at ~/.config/litellm/infra-ops-key (mode 600); gateway reachable directly from nh3-dev at 10.250.50.70:4000. (auto-memory reference_litellm_infra_ops_key) Archived 2026-06-19.

• [2026-06-16] litellm strip_empty_tools pre-call hook shipped (d1bea13) — an empty tools:[] 500s vLLM ("tools must not be an empty array"); a global litellm_settings.callbacks CustomLogger pops it (+ orphaned tool_choice) before forwarding, so it covers EVERY vLLM model, not one. drop_params only drops unsupported PARAMS, not empty VALUES. Mounts beside config.yaml (litellm resolves callbacks relative to the config dir). Verified live across granite/mistral/stream. (stacks/litellm/conf/strip_empty_tools.py) Archived 2026-06-20.

• [2026-06-16] single-file gateway-chat.html playground shipped (984ca3d, tools/) — zero-dep browser chat straight to the gateway (:4000, CORS open), system-prompt box, streaming SSE, renders reasoning_content, NEVER sends tools. Built because the LiteLLM admin-UI playground can't test vLLM-backed models (see Tried-and-abandoned). Serve on-request via python3 -m http.server -d tools. Archived 2026-06-20.

• [2026-06-16] claude-bot granted ADMIN on vh/arbo (operator) — arbo CI/CD gitea ops now route through the service account, not the operator's personal vh (the credential-migration directive). Operator added claude-bot as Administrator collaborator; claude-bot self-serves via its gitea-password (basic-auth = FULL perms; its stored token is package-write-only, can't change repo settings). (auto-memory reference_claude_bot_gitea_creds) Archived 2026-06-20.

• [2026-06-16] ratatoskr Tier-3 MEMORY plane wired + live-smoke driven. The missing server-side step was the per-PORT BIFROST_CLIENT_ALLOWED_HOSTS allowlist (:8391; consumer key reused, plane-agnostic — my earlier "reachability is the whole wiring" was incomplete). Persist + dispatch proven end-to-end against personal v0.35.3; cross-session recall-INJECTION root-caused to the score_breakdown shape seam and handed to worldtree-dev (#295 recall leg open). Driving a real memory-call needs a consumer agent defined WITH a memory block (ctx.memory_config≠None), not agent_self_enabled. (tracking: c99aa49 + auto-memory reference_bifrost_plane_wiring) — [2026-06-16] FLIPPED: the score_breakdown-shape root-cause was disproven; real cause = cold-recall scope_filter adds an agent_self axis → ratatoskr's conjunctive INV-005 match drops the {end_user}-only chunks → 0 hits. Now Worldtree #297. (auto-memory project_worldtree_295_recall_rootcause) Archived 2026-06-20.

• [2026-06-16] infra-ops = the DURABLE admin path on corviduo-dev (operator granted standing admin; extends the 2026-06-15 bootstrap) — added a Host corviduo-dev → User infra-ops ~/.ssh/config alias + repointed servers/corviduo-dev/ssh-target to infra-ops@; ssh corviduo-dev / elway corviduo-dev now land as infra-ops w/ NOPASSWD (vh = explicit-only human path). elway resolves via ssh-config, NOT the ssh-target file. (tracking: c99aa49 + auto-memory reference_corviduo_dev_emergency_ops) Archived 2026-06-20.

• [2026-06-16] demo + personal character model flipped qwen3.6-35-a3b → mistral-small-4 (operator) — in model_roles.yaml the FIRST catalog_id in a role's binds: is the default; reordered mistral-first (qwen retained in the switch-allowlist, non-destructive), applied via the PIN-SAFE api recreate (derive WORLDTREE_IMAGE from the matrix sibling — never a bare compose up, which falls to the crash-blocking stale :latest), verified a fresh character agent resolves model=mistral-small-4. (auto-memory reference_corviduo_dev_emergency_ops) Archived 2026-06-20.

• [2026-06-16] NVFP4 ModelOpt MoE now LOADS on vLLM 0.23.0 — SUPERSEDES the 2026-06-14 "blocked" finding (#44081 fixed). Confirmed the cached nvidia/Qwen3.6-35B-A3B-NVFP4 loads clean + generates on 0.23.0; swapped prod qwen36 FP8→NVFP4 (~13 GB reclaimed on GPU 1, fp16 KV). Validated by brokkr (speed: faster single-stream, ≥parity batched) + comfy-dev (vision anatomy-judge A/B: PASS). One W4A4 tell: minor multi-step chained-reasoning slip — doesn't bite the vision judge. Mistral's NVFP4 is a SEPARATE compressed-tensors path (day-0), distinct from the ModelOpt one. (auto-memory reference_nvfp4_moe_loads_on_vllm_023) Archived 2026-06-20.

• [2026-06-16] Mistral Small 4 = the GPU-0 tenant (operator) — fills the slot the deferred creative-writing pick held, and supersedes llama-swap (now down). NVFP4 119B MoE, pinned vLLM v0.22.0 for working vision (0.23.0 breaks it), 256K context. Operator's creative-writing model; abliteration planned → then it becomes the uncensored-creative model the persona substrate needs. Reasoning via reasoning_effort (BINARY none|high — no medium). (tracking: c77a9aa/9a49963) Archived 2026-06-20.

• [2026-06-16] Selene judge restored on vLLM FP8, NOT NVFP4 — W4A4 is too aggressive for a precision judge validated at Q6_K; FP8 ≥ Q6_K fidelity, and Selene is text-only Llama-3.1 so dynamic --quantization fp8 has no vision tower to noise. selene-1-mini-8b on GPU 1 :8011. Worldtree's Domari role repointed to it. (c985ede) Archived 2026-06-20.

• [2026-06-16] GPU-1 buffer put to work: selene 0.13→0.17 + qwen36 0.32→0.34 — the two KV-bound generative models that gained Worldtree consumers; granite + the pooling models under-use their util, so growing THEM is wasted. (e124a2f) Archived 2026-06-20.

• [2026-06-16] claude-bot Gitea service account stood up; corviduo-ORG conversion TABLED (operator). claude-bot is the credential-migration foundation. The proper fix for scoped agent package-write is converting vh (a USER) → a corviduo ORG — blast radius assessed (org creation = free; arbo-first migration = cheap, no registry images; full fleet = incremental, ~8 active vh images / 50 repos) — but tabled for later. Registry-push deferred until then. (auto-memory project_migrate_infra_access_to_claude_credentials) Archived 2026-06-20.

• [2026-06-16] arbo engine builds → comfy-dev OWNERSHIP (operator approved) + Gitea Actions runner stood up on irv-ml1 (host-executor as lkraven). infra-ops keeps box-level escalation (runner, package:write PAT pending corviduo-org, webhook). See the runner entry in Current state. (tracking: this snapshot + arbo thread 01KV70AB2B4B…) Archived 2026-06-20.

• [2026-06-16] Worldtree persona substrate migrated qwen→Mistral via capability- named profiles (v0.35.8→v0.35.10; demo + personal). PRE-SYNC-FIRST is now the standing pattern for any providers.yaml/profile change: sync config to the bind-mount BEFORE the deploy image-swap restart, so new agents + new config land together (the v0.35.9 merge-first attempt race'd → 14 crash-loops). (tracking: worldtree thread 01KV6X0M…) Archived 2026-06-20.

• [2026-06-19] litellm task-aliases: classifier→granite-4.1-8b, summarizer-large→gen/qwopus (summarizer→granite already existed). Also added then REMOVED gen-nt/gen-reasoning-nt (redundant with the global strip_empty_tools hook). (ef45f6d, 2e5ab72, 75bd4c3) Archived 2026-06-21.

• [2026-06-19] gateway-chat: model-smoking web chat → PERSISTENT container. /v1/models auto-discovery + image-upload; stacks/gateway-chat/ (nginx:alpine, ana-docker:8091, homepage tile). (4c40b9f, 740bcae) Archived 2026-06-21.

• [2026-06-19] pi on nh3-dev wired to gen — ~/.pi/agent/models.json + a gen launcher mirroring glm. Local box config, not a repo artifact. (untracked by operator choice) Archived 2026-06-21.

• [2026-06-19] gen model → Qwopus3.5-122B-A10B (vision-intact NVFP4), full 256K @ fp8. ana-ml2 GPU 0, served-name qwen3.5-122-a10b. STABLE vLLM + fp8 KV reaches 262144 — nightly+turboquant-4bit-KV was UNNECESSARY. 92.7 tok/s warm. (20e796c, 5b06514) Archived 2026-06-21.

• [2026-06-19] TP=2 across the two ana-ml2 Blackwells REJECTED — PIX (PCIe, NO NVLink) → all-reduce-bound → SLOWER. One-model-per-card is optimal; PP=2 only if a >96 GB model is ever wanted. (untracked by operator choice) Archived 2026-06-21.

• [2026-06-19] qwen-image-bench (T2I judge, NVFP4) replaced qwen3.6-35b-a3b on GPU 1, aliased image-judge. (bfae924, 5dfce04) Archived 2026-06-21.

• [2026-06-18] heretic abliterated Mistral Small 4 NVFP4 built + LIVE as mistral-small-4 (in-house NVFP4, drop-in stack, A/B'd vs official, "heretic stays"). DISPLACED from GPU 0 on 2026-06-19 for the gen/Qwopus swap. (dd3a5c9, f566f61, tools/mistral-small4-nvfp4/) Archived 2026-06-21.

• [2026-06-18] irv-ml1 VRAM consolidation + comfy-dev torch-pin — ComfyUI pinned to the A6000 (48 GB), audio zoo → 3090; DISABLE_UPGRADES=true @ torch 2.12.1. (a8550ad) Archived 2026-06-21.

• [2026-06-18] ComfyUI acceleration set (9 nodes) installed for comfy-dev on irv-ml1. Archived 2026-06-21.

• [2026-06-17] ana-ml2 vm.overcommit_memory=1 made durable (sysctl drop-in) — overcommit=0 + zero swap caps CommitLimit; large model-file mmap ENOMEM'd despite ~393 GB free. (fc88eff) Archived 2026-06-21.

• [2026-06-17] GLM 5.2 wired into litellm (glm-5.2 + -reasoning, z.ai passthrough, extra_body.thinking.type toggle). (fe77a35) Archived 2026-06-21.

• [2026-06-17] nh3-extdev stood up as a sudo-LESS infra-ops manager box (successor to nh3-ansible); hosts /opt/externs pi-on-GLM-5.2 client workspaces. (a841eab) Archived 2026-06-21.

• [2026-06-20] Worldtree capability-gateway aliases stood up; NO wt- prefix (operator) — reuse the existing capability convention. gen/gen-reasoning/classifier/summarizer/chat-judge/reranker/scalar-judge (passthrough)/gen-frontier/-reasoning. The swap-transparency rule: embedding stays model-named (qwen3-embedding) because vectors aren't swap-transparent; everything else can swap behind a stable alias. (d372103, 7819f96) Archived 2026-06-25.

• [2026-06-20] Removed the litellm * wildcard → llama-swap (operator; llama-swap decommissioned, :9292 confirmed down). Unknown/typo'd models now 404 cleanly instead of a misleading "Connection error" — closes the long-standing footgun. (d0eb09c) Archived 2026-06-25.

• [2026-06-20] Langfuse retired (operator) — redundant with LiteLLM native spend_logs + a 6-container crash-prone surface (the 94 GB clickhouse spew). Callbacks dropped, stack torn down. (f8eda1c) Archived 2026-06-25.

• [2026-06-20] OmniVoice streaming /tts — vendor chatterbox-fast's scheduler, not its whole package (operator-approved vendor-copy). Diffusion TTFA floor finding → tuned 16-step streaming + aggressive packing; chatterbox-fast remains the lowest-latency front-end. (288d085, cd92b85) Archived 2026-06-25.

• [2026-06-20] Daily image+buildx prune cron on ana-docker (the missing automation behind the recurring image bloat) — lkraven crontab, prune unused images >48 h + per-builder buildx cache. The real systemic fix (daemon log cap) is deferred to clean context. Archived 2026-06-25.

• [2026-06-25] althing re-architected to the lean multi-machine bus; nh3-extdev stood up as a MODEL B mesh peer (dedicated althing-svc + group-shared /srv/althing). reference_nh3_extdev_althing_mesh. Archived 2026-07-08.

• [2026-06-23] zellij native web client piloted on nh3-dev (zellij-web.service :8443) alongside ttyd. reference_zellij_web_seat. Archived 2026-07-08.

• [2026-06-22] Worldtree persona-render config arc (#314/#322/#317) pre-synced + deployed green on demo+personal — #317 a boot-blocking config REMOVAL. reference_corviduo_dev_emergency_ops. Archived 2026-07-08.

• [2026-06-20] R22 (brokkr/dwarves) stood down to gateway-only; full-access R22 key minted; Phase B CANCELLED (Worldtree model-agnostic → no deploy path). Key at /home/lkraven/.r22-gateway-key (persistent mode-600, carries paid GLM, don't delete). MUT = free qwen3.5-122-a10b (gen). Operator steer: R22 research is gated on a pragmatic/deployable outcome, not advancing-the-art. Archived 2026-07-08.

• [2026-06-20] claude-bot issue-scope token minted for worldtree-dev self-serve (id 16, write:repository+write:issue); old token revoked. Advances the credential-migration directive. Archived 2026-07-08.

• [2026-06-20] rest-server-ana recovered + backup prevention shipped + worldtree-dev admin keys provisioned (demo d113207c / personal f4f75adb). Cred rotation (5 rest-server pw) BELAYED. Archived 2026-07-08.

• [2026-06-20] claude-bot → ADMIN on vh/Worldtree (operator-authorized) — self-serves WT deploys/tokens henceforth. Archived 2026-07-08.

• [2026-06-14] STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials. (auto-memory project_migrate_infra_access_to_claude_credentials) Archived 2026-07-13.

• [2026-07-01] Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix verified (15×→1.01× re-embed). reference_wt_gateway_scoped_log_view. Archived 2026-07-13.

• [2026-07-01] qwopus native MTP speculative-decode tested on gen → NOT kept (+12% single-stream, −15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. reference_gen_qwopus_122b. Archived 2026-07-13.

• [2026-07-01] Deckard trial → reverted to qwopus (gen) (won writing "in every way" but ~36 vs ~90 tok/s; spec-decode rescue ruled out). git b63c48b→681eb70. Deckard kept staged as T1's writing benchmark. Archived 2026-07-13.

• [2026-07-02] mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to the T1 run. Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy — arch gap + abliteration axis). reference_gen_qwopus_122b. Archived 2026-07-13.

• [2026-07-03] ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0). Minted a tier-user key on the demo via POST /admin/keys (bootstrap admin key), mirroring their personal base consumer (no character-binding); base-agent affect reads work ungated. reference_worldtree_demo_key_mint. Archived 2026-07-13.

• [2026-07-04] character-rp: gateway-forwarding proven + role pre-staged + #344 shipped. Empirically confirmed per-request extra_body (top_k/repetition_penalty) forwards through the gen-reasoning LiteLLM alias to vLLM + standard params override the alias defaults — no gateway cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact). Pre-staged the character-rp role into demo+personal bind-mount model_roles.yaml (byte-verified on b18; caught the cached-registry ordering). worldtree-dev shipped #344 (v1.0.0b19) for the durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping). Archived 2026-07-13.

• [2026-07-04] althing v2 herald+receiver formalized as systemd on nh3-dev. althing-herald.service (Restart=always, Environment=PATH incl ~/.cargo/bin — the pane-dispatch fix) + althing-receiver.service (v2 → pillar-3 /owner/* live); stale forseti unit removed; both on v2.0.0b2, canonicalized by forseti. reference_nh3_dev_althing_herald. Archived 2026-07-13.

• [2026-07-04] LitBench-RM respun (irv-ml1 A6000, comfyui displaced) for T1's reward ensemble; operator sole comfyui consumer, holding image-gen until LitBench done. reference_litbench_rm_irv_ml1. Archived 2026-07-13.

• [2026-07-05] T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2. On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS; full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec; Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step = the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud call. HF base verified public (zero-auth pull). Runbook + gotchas in reference_t1_cloud_train_plan. Archived 2026-07-13.

• [2026-07-05] glm-5.2 canonical limits recorded (probed live vs z.ai): 1,048,576 (1M) input context / 131,072 (128K) max output; NO gateway-side cap (pure passthrough → z.ai's limits are effective). Written to the config comment (commit 624a07e) + reference_litellm_gateway. Archived 2026-07-13.

• [2026-07-06] infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2 (the arbo memory's "NOT infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke. Archived 2026-07-13.

• [2026-07-07] SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN in_proj hit 2 MORE name-routing gaps → MERGE recommended. Root cause was a NAME-routing gap (not the feared shape-assert): lora/utils.py:235 hardcoded params_mapping lacked in_proj_qkv/in_proj_z → in_proj_qkvz. Patched those 2 lines (overlay bind-mount) → init_lora_shapes passed, server up, self-attn loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj weights, and inference utils.py:300 get_target_module_name (substring match) RAISED → 500 + scheduler died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed. Archived 2026-07-13.

• [2026-07-07] T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2 — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565 was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120- blocked #913). reference_t1_cloud_train_plan. Archived 2026-07-13.

• [2026-07-07] T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op (#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND adapter-format-agnostic (separate zc loads-but-inert; fused rejected); #47640 guard-patch overlay didn't fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED. Archived 2026-07-13.

• [2026-07-07] HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON. Merge is NOT a solution for this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it can't apply, park the adapter + file upstream repros. reference_lora_training_worker (adjacent). Archived 2026-07-13.

• [2026-07-07] comfy-dev in-arbo LoRA-training worker DONE — built/deployed/validated Phases 1+2 (first Sindra train succeeded, publish-step + train_id), Phase 2.5 GC contract (shape A) confirmed + impl deferred. services/lora-training-worker/, commits 888ba6a→b617a8b. reference_lora_training_worker. Archived 2026-07-13.

Tried and abandoned (archived)

• [2026-04-30] task-board workflow with container: image: debian:bookworm-slim — fails: actions/checkout@v4 needs node at runtime, slim image lacks it. Switched to node:20-bookworm-slim (has node + apt) or runner-label default. (Pattern revisited 2026-05-17 for skaldsong-dev: container override needs nodejs apt-installed unless it IS the default.) Archived 2026-05-27.

• [2026-04-30] Dropping the container: directive before runner re-registration with docker-schema labels — runner silently falls back to host mode (jobs run inside the alpine act_runner container itself, no apt). The :host suffix in startup logs (labels updated to: [pfi-fleet:host ana-docker:host]) is the giveaway. Fix: register with pfi-fleet:docker://<image> schema labels. Archived 2026-05-27.

• [2026-04-30] Updating runner labels by editing .env and bouncing — doesn't take. The .runner registration cache pins labels at first registration; env-var updates are read each start but the stored token + UUID are tied to the original label set on the gitea side. Fix: stop runner, delete .runner, generate new admin registration token, redeploy. Archived 2026-05-27.

• [2026-04-30] git reset --hard origin/<sha> in deploy-task-board.yaml (and the in-repo nevermore playbook before fix) — invalid syntax: origin/ prefix only works for branch refs. SHAs need git reset --hard <sha> directly. Resolved with git rev-parse --verify --quiet "origin/{{ ref }}^{commit}" first, then bare "{{ ref }}^{commit}" fallback. Archived 2026-05-27.

• [2026-04-30] Assuming DEPLOY_SSH_KEY was at user scope after task-board wiring — it was actually only repo-scope on vh/task-board. vor's first CI run failed with empty SSH key (printf '%s\n' "" > ~/.ssh/id_ed25519). Fix: copy secret to user scope at gitea.phasefinal.com/user/settings/actions/secrets. Archived 2026-05-27.

• [2026-04-30] grep -vE "^(#|$)" to inspect .env for sanity — leaked the full MINIFLUX_PASSWORD line into the transcript. Then a follow-up redaction attempt with sed -E "s/=(.{4}).*$/=\1<redacted>/" still leaked the first 4 chars. Lesson: when probing secret-bearing files, use field-by-field SELECTIVE inspection (grep -E "^(KEY1|KEY2)=") rather than negative filters; for any password line, grep -c (existence) or test -n "$(...)" (non-empty), never cat or value-printing. Archived 2026-05-27.

• [2026-05-08] Filtering Traefik's UTC access log by Gitea-local-PDT timestamp substrings (grep "2026/05/08 15:1[2-7]") returned zero matches and led to a wrong "no /v2/ traffic in 12 days" conclusion. Gitea logs in PDT, Traefik logs in UTC — same host, different timezones. Always normalize timezones (UTC) when correlating logs across services on the same box. Cost: ~30 min in the wrong direction. Archived 2026-05-27.

• [2026-05-08] Bumping Gitea PER_WRITE_TIMEOUT / PER_WRITE_PER_KB_TIMEOUT to address unexpected EOF on /v2/.../blobs/uploads/ PATCH — wrong direction. Both govern response writes, not request body reads. unexpected EOF from Go's HTTP server means the client closed mid-body-upload; not a knob Gitea exposes server-side. Archived 2026-05-27.

• [2026-05-12] Defaulting asset-engine to Traefik-routed (asset-engine.phasefinal.com with anaprod cert resolver) on first scaffold — user pulled it back to LAN-direct. Internal tools default LAN-direct; only add Traefik when an external/TLS surface is actually needed. Archived 2026-05-31.

• [2026-05-12] Routing althing thread replies through galdrabok when the actual dev handle is forseti — bus rejected to=forseti initially because thread participants list was [galdrabok, infra]. Solved by starting a new thread with forseti as the direct recipient. Lesson: when the bus auto-resolves a sender handle that doesn't match the actual dev role, start a fresh thread rather than fighting the participant list. Archived 2026-05-31.

• [2026-05-13] Initial Voxtral default voice alloy (OpenAI-compat naming) — vLLM-Omni serving Voxtral does NOT translate aliases. Native presets are <register>_<gender> shape (neutral_female, casual_male, etc.). Always live-probe /v1/audio/voices for the exact wrapper-deployed preset names before setting a catalog default. Same caveat for Qwen3-TTS (wrapper exposes 15 voices: 9 Qwen presets + 6 OpenAI aliases) and Kyutai-TTS (NillPointer wrapper has NO voice-listing endpoint at all; voices are filesystem paths under the kyutai/tts-voices HF repo). Archived 2026-05-31.

• [2026-05-17] --task classify for Skywork in vLLM 0.19.1 — flag was deprecated. Use --runner pooling; the model's architectures: [LlamaForSequenceClassification] in config.json drives the classification head. Surfaced as vllm: error: unrecognized arguments: --task classify in container logs. Archived 2026-05-31.

• [2026-05-17] Trusting that .env edit alone propagates a new env var into a worldtree container — compose.yaml's &worldtree-env anchor must explicitly declare the passthrough or the value silently doesn't land. Same footgun bit Z_AI_API_KEY (2026-05-12) AND BIFROST_CLIENT_ALLOWED_HOSTS (2026-05-17). Cost ~10 min of "why is env empty?" diagnosis each time. Worldtree-side fix in vh/worldtree@08f02b2. Archived 2026-05-31.

• [2026-05-17] --force-recreate --pull never from the docker:cli sandbox without explicit -e WORLDTREE_IMAGE=<sha> re-pins the container to :latest, even when a newer SHA-tagged image is on disk. Symptom: container "recreated" but actually reverted to a stale image. Pass -e WORLDTREE_IMAGE=...:<sha> to the docker run invocation. Worldtree-dev's 8ef3801 health-gated :latest advance is the long-term fix. Archived 2026-05-31.

• [2026-05-18] Volva env.sh.template $HOME in commented examples — systemd's EnvironmentFile= parser doesn't expand $HOME; uncommenting lands the literal $HOME/... string. Volva-dev's f4dda73 swapped to /home/<svc-user>/... placeholders. Archived 2026-05-31.

• [2026-05-18] Initial Volva systemd unit's ProtectHome=read-only without ReadWritePaths= — althing-cli's SQLite (~/.althing/ althing.db) and codex's session state (~/.codex/) both need to write. Container started but every poll failed with "db path not writable". Surgical fix: ReadWritePaths=/home/lkraven/.althing /home/lkraven/.codex (preserves the hardening intent, only carves out the specific dirs). Archived 2026-05-31.

• [2026-05-18] Trusting that env.sh's export VOLVA_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" template line works under systemd — EnvironmentFile= parser aborts on the first unparseable line (command substitution), and VOLVA_ALTHING_CLI declared below silently never lands. Symptom: Environment= property empty, daemon error "althing-cli not found at 'althing-cli'". Fix: replace command-substitution with literal path. Volva-dev's d436c3c dropped VOLVA_ROOT entirely upstream. Archived 2026-05-31.

• [2026-05-19] Naive docker rmi worldtree:<old-sha> --force for CD SHA cleanup — would untag pinned/personal worldtree images since all three deployments share corviduo-dev. Use docker image prune -a --filter "until=Xh" instead — respects in-use semantic (Docker won't remove an image referenced by any container on the host), so pinned/personal protected automatically. Archived 2026-05-31.

• [2026-05-19] Skaldsong CD first attempt: docker pull step failed with 401 unauthorized. ana-docker had no docker login for gitea.phasefinal.com. My playbook prereq note ("docker login has been done at least once") was an unverified assumption. One-time manual login persists in ~/.docker/config.json; architectural fix (workflow-side ssh ana-docker 'docker login ...' step using REGISTRY_USER/REGISTRY_TOKEN secrets) flagged as v2. Archived 2026-05-31.

• [2026-05-19] SKALDSONG_HOST_CORS_ORIGINS=http://10.250.50.70:8300 as a bare URL — pydantic-settings parses complex env vars via json.loads(); first-boot crashloop with SettingsError: error parsing value for field "cors_origins". Must be JSON array literal: SKALDSONG_HOST_CORS_ORIGINS=["http://..."]. Archived 2026-05-31.

• [2026-05-19] SKALDSONG_HOST_STATIC_ASSETS_PATH=/app/web/dist in compose — mismatched Dockerfile reality. The Dockerfile COPYs SvelteKit build output flat into /app/spa (not /app/spa/dist). Lifted the path from skaldsong-dev's CD-ask message ("/app/web/dist") rather than verifying against the actual Dockerfile they shipped. Lesson: when encoding container-internal paths in compose, verify against the Dockerfile, not the design-doc. Archived 2026-05-31.

• [2026-05-19] Playbook verify step docker ps | grep healthy racing the container's start_period (30s in compose's healthcheck). Verify ran 0.09s after compose up -d --force-recreate — well before docker's healthcheck could flip the status from (health: starting) to (healthy). False-negative; container was operationally up (the earlier /health poll verify already confirmed). Fix: grep ^Up not healthy. /health-200 IS the liveness check; docker's (healthy) is just a delayed echo. Archived 2026-05-31.

• [2026-05-20] SKALDSONG_DB_PATH + SKALDSONG_RUNS_DIR in compose env block — names skaldsong's app doesn't read. App reads SKALDSONG_HOST_SQLITE_PATH + SKALDSONG_HOST_RUNS_ROOT (per Dockerfile ENV defaults). Wrong names = silently no-op; app fell back to Dockerfile defaults pointing at /app/data/... which the compose's bind mount did NOT cover (target was /app/state/...). Result: every --force-recreate wiped the SQLite DB. Caught by skaldsong-dev (althing thread 01KS4DPF6SXTBP4Q360JZVWPNT). Fix in 52e98fa. Lesson: verify env var NAMES against the Dockerfile/app, not against design-doc shorthand. Archived 2026-05-31.

• [2026-05-25] First selene-block patch put the block under saga_allowed_models: instead of top-level models: — used text.replace("models:\n", ...) which substring-matched the saga_allowed_models:\n line first. Caused YAML parse error. Fix: anchored regex re.compile(r"^models:\n", re.MULTILINE). Pattern: substring replace on YAML top-level keys WILL match suffix-containing keys. Archived 2026-05-31.

• [2026-05-27] docker compose up -d inside the docker:cli sandbox: ${VAR:-./config} defaults resolve ./config to the sandbox CWD, but the Docker daemon interprets the path against the HOST filesystem → auto-creates an empty dir → entrypoint reseeded image-baked defaults (lost host-side providers.yaml patches). Fix: pass -e WORLDTREE_CONFIG_DIR=/abs/path. Folded into the docker-as-root convention note. Archived 2026-06-01.

• [2026-05-27] :latest-pinned compose + private gitea registry + sandboxed pull = recreate on ancient cached :latest (deploy pulls by SHA so the tag never advances; sandbox can't pull). Fix: retag SHA→:latest on host, then --pull never. Better: pin SHA in .env, advance in CI. Archived 2026-06-01.

• [2026-05-27] Container recreate during in-flight skaldsong gen kills the runner. With deploys every ~10min and stories >5min, structural not incidental. Roadmap (skaldsong-dev): pre-shutdown signal handler, per-scene resume-from-checkpoint, /api/admin/quiesce. None shipped. Archived 2026-06-01.

• [2026-05-27] --reasoning-format gemma on artemis-31b-v1i — unsupported in the deployed llama.cpp (accepts none|deepseek|deepseek-legacy). deepseek populates the reasoning_content SSE delta Worldtree GemmaProvider checks. Archived 2026-06-01.

• [2026-05-27] head -c N piped after a streaming curl SIGPIPEs the curl, killing the request early. Use file-write + separate read. Archived 2026-06-01.

• [2026-05-31] Building the dia2-capable image surfaced THREE upstream packaging quirks: (1) pip install -e nari-labs/dia2 fails — no PEP 660 build_editable hook; (2) plain pip install builds an empty UNKNOWN-0.0.0 wheel (base setuptools 59.6 < dia2's required ≥70); (3) --no-deps leaves transformers/sphn/whisper-timestamped missing. Fix (local/dia:v2): copy the pure-python dia2/ package into site-packages + install ONLY those 3 deps; base torch/numpy already satisfy Dia2. Archived 2026-06-03.

• [2026-05-31] Dia2 predefined voices (43, baked at /app/voices) are NOT reachable from the /tts clone path — it resolves clone_reference_filename against the reference_audio dir ONLY. The OpenAI /v1/audio/speech voice param auto-resolves them (separate code path), which masked the gap. Fix: stage /app/voices/* into /worktank/dia/reference_audio. Lesson: verify on the endpoint the catalog ACTUALLY targets. Archived 2026-06-03.

• [2026-05-31] voice_mode=clone with an empty clone_reference_filename → asset-engine serializes it as the literal string "undefined" → /tts 404. First observed on dia2; worked around in the catalog (default the field to a real voice). [2026-06-01] root cause found — the Kokoro voice-blend widget reading Shoelace .value before hydration (see Current state); the real fix is asset-engine-side and is escalated. Archived 2026-06-03.

• [2026-05-31] asset-engine services.schema.json is DERIVED (regen from the Pydantic model via dump_schema.py) and had DRIFTED — rejected the lifecycle field 12/14 services use. RESOLVED: regenerated with CatalogLifecycle + reproducibility.seed_field (b7b2130). Lesson: hand-editing services.yaml shape without regenerating re-introduces drift. Archived 2026-06-03.

• [2026-05-31] ttyd-over-TLS forces HTTP/2 (kills ttyd's terminal WebSocket → blank screen); Safari/WebKit never sends HTTP basic-auth on WS upgrades. Both solved for the fleet seat (Caddy forces HTTP/1.1; auth → network-gating) — detail in auto-memory reference_ttyd_fleet_seat. Archived 2026-06-03.

• [2026-05-30] esh-docker-vm: hard NFS mounts from 10.0.50.50 froze a container worker in UNKILLABLE D-state when the NAS stalled — only a host reboot clears it. Separately, fstab defaults (no _netdev) made NFS-bind containers Exited(255) on reboot. → auto-memory incident_esh_docker_nfs_boot_race. Archived 2026-06-03.

• [2026-06-02] Naive cumulative re-decode for streaming — decode the growing token prefix each chunk + emit the delta, assuming the causal flow gives a stable prefix. It does NOT (full-context attention, see above); maxdiff 0.30. Also the high-level s3gen.inference re-applies trim_fade at the start every call. Don't go this way. Archived 2026-06-04.

• [2026-06-01] CSM bring-up — upstream phildougherty/sesame_csm_openai Dockerfile pins NO huggingface_hub version, so it now resolves to 1.17.0 where huggingface-cli is fully removed (replaced by hf) → the build dies at huggingface-cli login/download. Passing the HF token as a build-arg got the token in (the model downloads at BUILD time, not runtime — our compose only wired it as runtime env, the original failure) but then hit the rotted CLI. Fixing would need vendoring + patching the Dockerfile; not worth it for a weak model → CSM removed. Archived 2026-06-04.

• [2026-06-01] Fish path-form references ({audio:"/app/references/x.wav"}) → HTTP 500 on this build, despite the stack README showing that shape. The working clone paths are reference_id (by name → resolves the DIRECTORY references/<name>/ containing <name>.wav + <name>.lab; the flat references/<name>.wav is ignored — see the two-root-causes entry above) or inline base64 references. The catalog uses reference_id. Archived 2026-06-04.

• [2026-06-01] Reference transcript barely affects Fish clone IDENTITY (disproving my mid-session theory). I'd blamed a bogus provenance-note .txt for poor cloning, but the ECAPA re-test showed correct (0.787) / bogus (0.778) / empty (0.738) transcripts all clone Imogen about equally — the transcript affects PRONUNCIATION (phoneme disambiguation per the docs), not who it sounds like. The real culprit for "not British" was the "undefined" select bug, not the transcript. (A correct transcript still marginally helps pronunciation — cheap to stage, not load-bearing.) Archived 2026-06-04.

• [2026-06-01] Pitch-shift register control (rubberband, to deepen Imogen to contralto/mezzo) — Fish ignores small reference shifts and overshoots large ones; -3st WITHOUT formant preservation → "baritone through a tuba" (formants collapsed); -2/-3st WITH --formant → too subtle / still bad. Abandoned at every depth; all variants deleted. Finer independent pitch/formant control needs praat (not installed). Future clones = operator's. Archived 2026-06-04.

• [2026-06-01] resemblyzer is too weak for cloning-fidelity scoring — its dated 2019 LSTM encoder + a synthetic-vs-natural domain gap scored the Imogen clone CLOSER to the default than to real-Imogen, which led me to a WRONG "Fish is a weak cloner" call. ECAPA-TDNN (speechbrain spkrec-ecapa-voxceleb) on the same clips gave the correct answer (clone 0.79 to real Imogen, 0.10 to default). Use ECAPA, not resemblyzer, for speaker-verification. Archived 2026-06-04.

• [2026-06-02] Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime). Benched RTF on A6000: 0.72x/0.82x/0.86x, mean 0.80x = sub-realtime, so client-side chunking starves. Root cause: Fish only chunks on <|speaker:X|> tags; plain text → batches=[whole text] → all semantic tokens generate before any audio (chunk_length inert). Plus kui/ASGI StreamResponse doesn't flush (fix = anti-buffering headers, kept on file). STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands → bench fp4-quantized Fish; if RTF > ~1.5x give it the chatterbox-fast treatment. For now Fish is a buffered catalog entry (great for SAVED gens, not live-audition). Archived 2026-06-11.

• [2026-06-02] Context-priming at chunk joins (chatterbox-fast §1.6) — ABANDONED (discard-cut leaks the prefix). Prepend prior sentence, generate prefix+content, discard prefix audio. Built + shipped (d707439), live-A/B'd, reverted (090e70a). Kill: generate() returns one waveform with NO marker where the prefix ends, and the model renders the same prefix with different timing solo vs followed-by-content — so locating the cut is a guess that left a whole clause in the output (operator caught "...without a trace of sarcasm" spoken twice). Reliable cut needs token-level boundaries or per-chunk forced-alignment (heavy). Coherence loss at joins = ACCEPTED limitation. Archived 2026-06-11.

• [2026-06-02] Native frame-level streaming on Chatterbox-TURBO — ABANDONED (turbo isn't built for streaming). (1) S3GenStreamer is docstring-only (unimplemented). (2) flow_inference(finalize=False) is buggy (lookahead trim removes frames from h not h_masks/conds → shape mismatch); a 1-line patch makes it RUN. (3) BUT the flow encoder uses full-context attention (static_chunk_size=0) → incremental decode is prefix-unstable (adding tokens re-attends, shifts earlier mel ~0.30-0.39 vs one-shot). (4) Forcing static_chunk_size>0 didn't stabilize it. True sub-second frame-streaming needs deep attention surgery with quality risk — not worth it. → adaptive-chunking instead. Archived 2026-06-11.

• [2026-06-03] uv pip install . fails on SmoothKen/knn-svc (and similar script-repos) — it's analysis scripts + a poetry pyproject, no buildable package (setuptools package-discovery error). Install the pyproject deps directly, don't build the "package". Archived 2026-06-13.

• [2026-06-03] Backgrounding althing-cli monitor with an inline shell & (instead of the Bash-tool run_in_background) orphans it — it survives the shell exit, holds the per-handle flock UNTRACKED (won't notify the session), and stop-monitor doesn't detect it. Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked. Archived 2026-06-13.

• [2026-06-03] torch-2.12 venvs need uv pip install torchcodec — torchaudio 2.12 defaults to the TorchCodec backend for .load; without it, real audio I/O throws "TorchCodec is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson: validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4 uses the legacy backend, exempt.) Archived 2026-06-13.

• [2026-06-03] gitea webhook to a private IP is denied by webhook.ALLOWED_HOST_LIST (anti-SSRF; default external blocks private/loopback). Symptom: delivery shows dial tcp ...: webhook can only call allowed HTTP servers. Fix = APPEND the target net to ALLOWED_HOST_LIST in gitea's app.ini (keep external; scope tight, never */private) + restart gitea (act_runner job containers survive a restart). gitea runs as a container on ana-docker (gitea_gitea_data volume, /data/gitea/conf/app.ini). Archived 2026-06-13.

• [2026-06-04] homepage "crash" ≠ always NFS — a wedged container in unkillable D-state can come from dead siteMonitor widget targets (retired ESH firewall IPs) hanging the node event loop into exit_mmap, needing a host reboot. Check homepage's siteMonitors against retired hosts. (incident_esh_docker_nfs_boot_race) Archived 2026-06-14.

• [2026-06-04] pyworld needs setuptools<81 (imports the removed pkg_resources); and R/soundgen -lgfortran fails on irv-ml1 (default gcc is gcc-11 but only gfortran-12 present) → install libgfortran-11-dev. Archived 2026-06-14.

• [2026-06-04] Persistent services on irv-ml1 need SYSTEM systemd units — the box reaps user-session processes on ssh disconnect, and --user systemd isn't reachable over non-login ssh, so nohup/setsid/screen -dmS/systemd-run --user all die (even with enable-linger). Use /etc/systemd/system/. Archived 2026-06-14.

• [2026-06-04] GPU pin by INDEX is ambiguous on irv-ml1 — native CUDA orders fastest-first (A6000=0) but nvidia-smi/docker use PCI order (A6000=1). Pin by UUID (CUDA_VISIBLE_DEVICES=GPU-…); verify via nvidia-smi compute-apps. Archived 2026-06-14.

• [2026-06-04] Ollama/llama.cpp-BUNDLED chat templates silently diverge from canonical HF — the "headgun" lesson. Serve each model's canonical tokenizer.apply_chat_template, not the bundled template (bundled ones corrupt baselines). Verify via vLLM /tokenize→/detokenize. (90e08f0/27eb537) Archived 2026-06-14.

• [2026-06-08] Demucs uv pip install demucs pulls torch 2.12/torchaudio 2.11 → ta.save() requires torchcodec → dies AFTER separating (0 stems, rc=1). Fix = pin torch==torchaudio==2.4.1 + UV_LINK_MODE=copy. Validate the SAVE path, not just import + GPU inference, on a bleeding-edge torch. Archived 2026-06-16.

• [2026-06-05] vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU — fills KV to the --gpu-memory-utilization budget WITHOUT reserving graph-capture memory, so capture_model OOMs AFTER weights+KV load (crash-loops). Fix: free co-tenant room OR --enforce-eager. FP8 single-stream is batch-1 GEMV (memory-bound) → Q4 wins single-stream by physics; FP8 wins under concurrency. (reference_ana_ml2_vllm_granite) Archived 2026-06-16.

• [2026-06-05] Langfuse has NO public dashboard-creation API — dashboards/widgets are postgres rows; clone a default + swap the measure. tok/s is the outputTokensPerSecond MEASURE (metrics-API/dashboard query time), not a per-generation field. langfuse-web needs HOSTNAME=0.0.0.0. Host 3000 is gitea's → langfuse on 3001. Archived 2026-06-16.

• [2026-06-05] sudo over non-interactive ssh FAILS SILENTLY where the user lacks NOPASSWD (esh + corviduo are OUTSIDE the infra-ops identity) → empty output misread as "empty file." Read world-readable files WITHOUT sudo. corviduo ssh = vh@10.250.50.152; bind-mount configs are vh-owned, the .env is deploy-owned 600. Archived 2026-06-16.

• [2026-06-05] Worldtree summarizer-model is NOT an env var — no WORLDTREE_SUMMARIZER_MODEL; defaults to claude-haiku in code, opt-in via config not .env. Inspect the live container env + vh-owned config files first. Archived 2026-06-16.

• [2026-06-13] Loading an old LoRA adapter after a vocab bump fails on embedding size. The harness config + tokenizer.json are now at new_vocab_size=50279 (v4 multi-tag); the v1/v2/v3 adapters are 50277. To load v1 (the accepted artifact), set cfg.new_vocab_size=50277 before load_finetuned_engine_lora (else PeftModel state_dict size mismatch). tokenizer.json.v3bak is the 50277 tokenizer for a clean restore. Archived 2026-06-16.

• [2026-06-13] Heavy from-source compile (MAX_JOBS=128) on the shared PROD GPU box PINS it — load hit 187, prod vLLM restarted, killed an in-flight quant. ana-ml2 hosts live inference; never run a big build there at full parallelism. Cap MAX_JOBS≤32, build off-box, or cgroup-constrain. Archived 2026-06-16.

• [2026-06-13] --quantization fp8 on a VL model can quantize the VISION TOWER → garbage vision (Qwen3.5-VL on stable vLLM: gray-grid output; LM answers text fine, so it "looks" healthy). The nightly excludes the vision tower. Validate the VISION path on a quantized VLM, not just text — and pin the engine digest with the exclusion. Archived 2026-06-16.

• [2026-06-13] vLLM's --gpu-memory-utilization is checked against FREE VRAM at startup, not total — on a shared card, growing one service before trimming a co-tenant OOMs. Trim the shrinking service FIRST, then grow. Size to the FREE budget. Archived 2026-06-16.

• [2026-06-13] The vllm/vllm-openai entrypoint is already ["vllm","serve"] — compose command: supplies the model as the first POSITIONAL arg + flags; a second serve/--model X → "unrecognized arguments". Same-class: tee masks the real exit code (use >); HF datasets rejects bare wikitext (needs Salesforce/wikitext). Archived 2026-06-16.

• [2026-06-13] Chatterbox-Turbo LoRA finetune: the repo's setup.py loads the WRONG tokenizer — pulls gpt2-medium + a grapheme merge file instead of the chatterbox-turbo GPT2 tokenizer (vocab.json+merges.txt, len 50276). Fix = override + delete the grapheme tokenizer.json; [vmoan] → new_vocab_size 50277 (1-row resize), lora_r 64 / alpha 128, modules_to_save=[text_emb,text_head]. Unique-stem corpus collision needs {index}_{stem} IDs. (irv-ml1:~/r16-vmoan-harness) Archived 2026-06-16.

• [2026-06-11] A completion-poll while pgrep -f <scriptname> SELF-MATCHES its own remote shell argv — its own pgrep -f always finds itself → the loop never exits. Use a match pattern ABSENT from the poll command (the python stage, or a sentinel file), not the driver's own name. Archived 2026-06-16.

• [2026-06-14] vLLM ModelOpt-NVFP4-MoE loader is broken for current multimodal MoEs. nvidia/Qwen3.6-35B-A3B-NVFP4 fails weight-load: KeyError: layers.0.mlp.experts.w2_input_scale on 0.19.1, lm_head.input_scale not registered on 0.22.0 (vllm #44081); same class hits Gemma-4 MoE / Qwen3-30B-A3B NVFP4. The arch + quant ARE recognized (gets past arch resolution + vision-processor load) — it's the per-expert/lm_head scale-key mapping. Don't chase nightlies; use official FP8 until fixed. Archived 2026-06-18.

• [2026-06-14] vLLM sampler-warmup OOMs on a shared GPU even when weights fit — it warms the sampler with max_num_seqs (default 1024) dummy requests, and a big vocab (Qwen3.6 = 248K) makes that a huge transient logits tensor. A vision endpoint doesn't need 1024-way concurrency: set --max-num-seqs 32. Separately, post-load ValueError: No available memory for the cache blocks means util is too thin (weights+activation+graph ate it) — for 34 GB FP8 weights, util ≥ ~0.45 to leave KV room. Archived 2026-06-18.

• [2026-06-14] Recreating multiple vLLM services concurrently races the memory-profiling assertion — AssertionError: Error in memory profiling. Initial free memory X / current Y … other processes … release GPU memory while vLLM is profiling. Recreate co-tenant vLLM services ONE AT A TIME (force-recreate one, wait healthy, next). Archived 2026-06-18.

• [2026-06-14] embed/rerank (0.6B) at util 0.05 reserve ~5.5 GB each — mostly util-reservation WASTE, not need. A 0.6B model needs ~1.2 GB weights + ~2.5 GB CUDA/torch context; util 0.03 (~3.6 GB) fits with room, reclaiming ~4 GB (vLLM reserves the util fraction regardless of actual KV; embedding models barely use KV). Real-need floor ~3 GB — don't go to 0.02. Archived 2026-06-18.

• [2026-06-14] Chatterbox-Turbo decode-knob foot-guns (R16 v1-polish + emergent probes): the turbo length cap is max_gen_len (default 1000) on t3.inference_turbo, NOT max_new_tokens — and tts_turbo.generate does NOT forward it (wrap inference_turbo to cap). rep_pen 2.0 / temp 0.5 BACKFIRES (degenerate 24 s run-on). Soft decode tamers cut the NVV ITSELF, not just the run-on tail (operator: "p1 trims the moaning too") — gen-time polish can't beat v1's defaults. Inline base-NVV tokens DEGENERATE (moan-cascade + gibberish), they don't modulate the surrounding words. Archived 2026-06-18.

• [2026-06-15] A comfyui restart-loop with exit 0 + OOMKilled=false is NOT OOM — suspect a native SEGFAULT. docker inspect shows only the LAST exit code; a GPU OOM is a catchable RuntimeError: CUDA out of memory, NEVER Fatal Python error: Segmentation fault. Diagnostic: RestartCount>0 = same container restarted in-place → the docker json-log ACCUMULATES across all cycles, so grep the full docker logs for the crash signature + the lines right before each restart banner. comfyui's was at VAE decode (sampling completes, then segfault loading AutoencodingEngine) — gen-triggered but native, not VRAM. Suspects logged: torch/cu130 mismatch, an injected aimdo cuda-funchooks lib, the VAE. Archived 2026-06-19.

• [2026-06-15] NEVER blanket-kill althing-light-monitor by PID — they're per-handle and MANY agents run their own on the shared nh3-dev box (heid / ratatoskr-dev / worldtree-dev / bifrost-dev each have one). A kill $(pgrep -f light-monitor) downs four peers' monitors (near-miss caught by checking /proc/<pid>/environ ALTHING_HANDLE + cwd first). Recover your OWN via althing-cli stop-monitor (v0.14.1+); a raw &-backgrounded monitor is untracked (no harness fire-notification) — always arm via run_in_background. (extends the pkill-self-match lesson below) Archived 2026-06-19.

• [2026-06-15] Manual docker compose up on corviduo Worldtree DOWNGRADES to a stale local :latest → crash-loop (config/image mismatch). Compose image: is …/worldtree:latest, but local :latest lagged ~2wk (21c90d5eeb2d); CI/CD deploys NEWER SHA images, and a real pull 401s (daemon has no gitea-registry creds — that's CI/CD's). --pull never then used the stale latest → personal crash-looped on ConfigurationError: echo.default_model missing → ~2-3 min personal outage. RECOVER: retag :latest to the image a HEALTHY SIBLING runs (docker inspect worldtree-worldtree-api-1 --format '{{.Image}}' → docker tag …), then up -d --pull never; verify Application startup complete. Don't manually recreate Worldtree on corviduo without pinning the correct image — image+config are CI/CD-coupled. (auto-memory reference_corviduo_dev_emergency_ops) Archived 2026-06-19.

• [2026-06-15] .claude.json (Claude Code config) truncates on ENOSPC → invalid JSON → CC resets to defaults on next start. nh3-dev hit 100% disk (a peer's R19 dataset pull staged ~147G to local /home instead of the 7TB NAS); the in-flight .claude.json write was cut mid-key. Repair = strip the dangling token + close the open braces, validate json.load. CC keeps its OWN rolling backups at ~/.claude/backups/ + writes a .claude.json.corrupted.* snapshot when it self-detects. After ANY ENOSPC on a box running CC/agents: integrity-check .claude.json (json.load) AND the althing bus DB (PRAGMA integrity_check). Archived 2026-06-19.

• [2026-06-15] pkill -f althing-light-monitor SELF-MATCHES the killing shell (the pattern is in the command's own argv) → kills itself mid-run (exit 144/truncated output). Stop the light-monitor via althing-cli stop-monitor or a captured PID — never pkill -f <string-present-in-this-command>. The singleton lock can also RACE to 2 live monitors during re-arm churn; keep exactly one tracked (run_in_background) monitor, and a raw & monitor is untracked (no harness fire-notification — don't use it). Archived 2026-06-19.

• [2026-06-14] Fleet/colo hosts must reach gitea over the INTERNAL route, NOT the public IP. gitea.phasefinal.com = public 38.120.12.44 (ana-srv1); gitea is a container on ana-docker, git-SSH 10.250.50.70:222 + HTTP :3000. A fleet host egressing to public :22 gets fail2ban-banned after any retrying git loop → silently wedges webhook auto-deploys (git fetch times out under set -euo pipefail, aborts before reset). Bit irv-ml1's arbo deploy. :22 on 10.250.50.70 is ana-docker's HOST sshd (deploy key → Permission denied), NOT gitea. Documented docs/orientation.md (6e58e57). Archived 2026-06-19.

• [2026-06-16] litellm 500 Router.acompletion()/aembedding() missing 'messages'/'input' = a request missing Content-Type: application/json, NOT a gateway outage. curl -d defaults to form-encoding → litellm can't parse the JSON body → data reaches the router without messages/input → 500 (should be a 400; litellm #16993). My own diagnostic calls dropped the header → I misread it as a gateway outage and needlessly bounced the gateway ~4× chasing a phantom (image/version/config were fine throughout; a malformed UI-added "Mistral Story Eval" model in the DB was a red herring I deleted). ALWAYS send -H "Content-Type: application/json" testing litellm; reproduce with a header'd call before declaring a litellm incident. Archived 2026-06-20.

• [2026-06-16] LiteLLM admin-UI playground can't test vLLM-backed models — it auto-sends empty tools:[], vLLM 400s (litellm #6228); the gateway strip_empty_tools hook is a PROXY hook and structurally can't reach the UI's in-process litellm.completion() call. Off-ramp = tools/gateway-chat.html. (Langfuse playground also out: its SSRF guard blocks internal-IP LLM connections, wontfix Langfuse #13097.) (auto-memory reference_litellm_ui_playground_vllm_deadend) Archived 2026-06-20.

• [2026-06-16] A gitea Actions run that "looks like it never fired" (no image, untouched build dir) usually DID fire then skipped or failed-fast. Bit arbo TWICE: run #3 looked-never-fired but SKIPPED (change-detection engine=0); run #4 looked-never-fired but FAILED in ~7s (early-step). Check the run LIST + status via claude-bot (GET /repos/{o}/{r}/actions/tasks) BEFORE a runner-connection/dispatch theory. CORRECTION (2026-06-16, proven): action logs ARE readable via API at the per-JOB endpoint — GET /api/v1/repos/{o}/{r}/actions/runs/{run}/jobs → take a job .id → GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs returns the full plain-text log (claude-bot basic-auth, internal route 10.250.50.70:3000). Only the RUN-level /runs/{id}/logs 404s; the per-job one works, so no UI needed. Gotcha when reading: gitea marks every step conclusion:failure once any step fails — the per-step conclusions mislead; trust the log body + step timestamps (the step with real duration is the culprit) not the conclusions. (Used 2026-06-16 to diagnose arbo run #5/task 1175 = test-gate uv run pytest → Failed to spawn: pytest = pytest absent from the fresh-checkout venv sync, NOT the SSH-clone comfy-dev suspected.) Archived 2026-06-20.

• [2026-06-16] vLLM 0.23.0 breaks Mistral MULTIMODAL at startup (MistralCommonImageProcessor has no attribute fetch_images, #44911 regression ~0.22.1+; same class hit Mistral-3.1/ Devstral). The dummy-image profiling crashes. Workarounds: --limit-mm-per-prompt '{"image":0,"video":0}' loads text-only; for VISION, pin v0.22.0 (last pre-regression). Tradeoff: v0.22.0 buffers reasoning (doesn't stream reasoning_content cleanly); a newer build streams it but breaks vision — vision won (Mistral is the abliteration-baseline). Archived 2026-06-20.

• [2026-06-16] Mistral -reasoning TTFT variance = Triton kernel JIT-compile-during- inference, NOT reasoning time. jit_monitor logs the spike (_compute_slot_mapping_kernel, apply_token_bitmask_inplace_kernel ← the tool-call/structured-output path). vLLM's startup warmup covers only SOME shapes; a first-hit uncovered shape JITs a kernel (seconds), then CACHES it — warm until container recreate (the Triton cache is IN-container, no volume, so recreate = cold again). Secondary: GPU idle-down to P8/180 MHz between requests. Also: Mistral reasoning_effort is BINARY (none|high only — medium/low → HTTP 400). Archived 2026-06-20.

• [2026-06-16] Worldtree old-baseline instances need the FULL config set synced, not the changeset delta. Personal (a PRE-mistral baseline) crash-looped (41 restarts) on DanglingBindingError: role 'echo' binds missing catalog entry 'qwen3.6-35-a3b-heretic' — its model_roles.yaml (2026-06-10, never migrated) still bound the retired heretic that the synced v0.35.10 providers removed. The "model_roles unchanged — skip" was correct only for demo (already at the v0.35.9 baseline). Old-baseline → sync providers + model_roles + defaults. Archived 2026-06-20.

• [2026-06-16] vh is a Gitea USER, not an org → no scoped package-write for a service account. Gitea user-namespace packages are owner/admin-only, so claude-bot can't push to vh/arbo without being admin. The fix is converting vh → a corviduo ORG (tabled). Also: vh's tea token lacks write:admin — create gitea users via the server-side gitea admin user create CLI (docker exec -u git gitea …), not the admin API. Archived 2026-06-20.

• [2026-06-16] docker compose up -d on a multi-service vLLM stack recreates ALL its services (shared .env → all configs re-hash) → concurrent memory-profiling race ("No available memory for the cache blocks"). Target one: docker compose up -d <service>. (auto-memory feedback_compose_up_recreates_whole_stack)

76 older entries archived to archival-memory.md. Archived 2026-06-20.

• [2026-06-19] litellm CONFIG-loaded models can't be hot-removed — /model/delete is DB-only. Only a process restart drops a config model. The INVERSE works: /model/new live-ADDS (no bounce) but dups with config on restart. Clean no-bounce-add: live-add via API, delete the DB copy before the eventual bounce. Archived 2026-06-21.

• [2026-06-19] the litellm * wildcard routed to DECOMMISSIONED llama-swap → a stale/typo'd model name returned a misleading Connection error, not model not found. Bit a brokkr call to the renamed qwen-image-judge. → RESOLVED 2026-06-20: wildcard removed (d0eb09c) — unknown models now 404 cleanly. Archived 2026-06-21.

• [2026-06-19] A MoE card can't be filled to 0 bytes free — the FusedMoE transient workspace (~3.09 GB) is the floor, allocated OUTSIDE gpu-memory-utilization. FIX: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True + leave ~3.2 GB free (util <= ~0.95 on a tight MoE). Archived 2026-06-21.

• [2026-06-19] vLLM decode tok/s: ALWAYS discard the first generation (cold warmup). Cold ~ 24.8 tok/s vs warm 92.7 — a cold read undersells decode ~3-4x. Measure run 2+ over a >=256-token output. Archived 2026-06-21.

• [2026-06-19] For full native 256K on one 96 GB card, nightly+turboquant-4bit-KV was unnecessary for the Qwopus MoE. Stable fp8 KV already fits 262144 (hybrid SSM+attn caches KV only on attn layers → small pool). Reach for fp8 first. Archived 2026-06-21.

• [2026-06-18] mmartial comfyui-nvidia-docker: root pip installs CRASH-LOOP the container. docker exec -u 0 pip install leaves root-owned files in the uid-1000 venv → the boot script (uid 1000) fails its torch step → crash loop. FIX: chown -R 1000:1000 /comfy/mnt/venv; pin with DISABLE_UPGRADES=true. (auto-memory reference_irv_ml1_comfyui_mmartial) Archived 2026-06-21.

• [2026-06-18] ComfyUI custom nodes break on version-assumption drift — box runs 0.24.1; TeaCache imports precompute_freqs_cis at MODULE level (guard the LTX-only import). pip install tensorrt defaults to cu13 → use tensorrt-cu12. Archived 2026-06-21.

• [2026-06-17] Mistral HF→NVFP4 quant: the placement knob is the whole game. device_map="cpu" (CPU-resident, sequential onload) works; "auto" OOMs, max_memory offloads to meta. Read shards with plain read() + safetensors.torch.load(bytes), NOT safe_open (mmaps the 50 GB shard → ENOMEM on /tank ZFS). Archived 2026-06-21.

• [2026-06-17] HF-format Mistral Small 4 is UNSERVEABLE on vLLM — no HF Mistral4 backbone; serves ONLY via the native loader. A HF-format quant MUST be converted to native first. Archived 2026-06-21.

• [2026-06-20] A full disk wedges gitea's Actions notifier SILENTLY — and it STAYS wedged after disk recovers; only a docker restart gitea clears it. After the 94 GB clickhouse disk-full event, gitea stopped CREATING workflow runs on push (push succeeds, ref lands, but NO run spawns — newest run stale, queue empty), while the act_runner stayed registered. NOT run-pickup, NOT disk-now (52% free), NO gitea errors/panics — a silent notifier stall. Hit BOTH vh/arbo (comfy-dev) and vh/Worldtree (worldtree-dev's deploys) → it's global, not repo-specific. Fix: docker restart gitea on ana-docker (it had been up 2 weeks, never bounced). A restart does NOT retroactively fire pushes made during the outage — re-push/re-trigger after. Reinforces the deferred docker-daemon-default log-cap fix (the disk-full that started it can't be allowed to recur). Archived 2026-06-25.

• [2026-06-20] Mistral-Medium-3.5-128B NVFP4 (RecViking) on ana-ml2 GPU 0 — ABANDONED, too slow (operator confirmed). Stood up as a temporary speed-check displacing mistral-small-4; measured base ~19.2 tok/s decode on one Blackwell (vs gen/Qwopus3.5-122B ~92.7 tok/s warm) → confirmed the operator's "too slow" suspicion. KEEP these serving findings: (1) vLLM nightly loads the HF-format NVFP4 weights DIRECTLY — no Mistral native-convert (unlike Small 4, same mistral3 arch) — via the FlashInfer Cutlass NVFP4 kernel + turboquant_4bit_nc KV; TP=1 fits the ~70 GB NVFP4 on one 96 GB Blackwell. (2) EAGLE was a dead end: the official native FP8 head (mistralai/Mistral-Medium-3.5-128B-EAGLE) crashes in CUDA-graph replay with graphs on, and nets NO speedup with --enforce-eager (~18.5 vs ~19.2 base) — likely low draft acceptance against the NVFP4 (non-official-FP8) base. Untracked workspace files (stacks/mistral-medium-3.5/ + playbooks/serve-mistral-medium-3.5.yaml) removed. ⚠️ Weights still staged on ana-ml2 /tank/aimodels/mistral-medium-3.5-{nvfp4,eagle} (~70 GB+) — pending operator OK to free. Archived 2026-06-25.

• [2026-06-20] A 94 GB unrotated langfuse-clickhouse json.log was the REAL disk-filler (not image bloat) — there was NO log rotation anywhere. No per-stack logging: limit and no docker-daemon default → ClickHouse's verbose stdout grew unbounded to ~94 GB and filled ana-docker's / (healthchecks then failed fleet-wide on runc /tmp: no space left). Lesson: on a full disk, the symptom (all-unhealthy) is the FORK-can't-write healthcheck, not the containers; find the runaway via per-container-dir du, not docker system df. The systemic fix is a daemon-default log-opts (max-size/max-file) — NOT just per-service patching. Archived 2026-06-25.

• [2026-06-20] OmniVoice's upstream "RTF 0.025 / 40×" does NOT hold — diffusion has a ~fixed per-call overhead → TTFA floor + multi-chunk streaming is a net loss. Measured ~2.8× (32-step) / 5.6× (16-step) on the 3090; every chunk costs ~the same regardless of length, so naive per-sentence streaming triples total gen time + starves. Use few/aggressive chunks; chatterbox-fast (autoregressive) stays the low-TTFA front-end. Archived 2026-06-25.

• [2026-06-20] Named buildx builders accumulate cache the standard prunes MISS. docker image prune -a + docker builder prune don't touch buildx_buildkit_<name>-builder0_state volumes (per-builder buildkit state). Prune them with docker buildx prune --builder <name> -af, or docker volume rm orphaned ones (builder already removed). Archived 2026-06-25.

• [2026-06-25] althing "unreachable: " can MASK an app-level 500. Raw network was clean; root cause = receiver DB agents-table not synced with the config roster → delivery 500'd "unknown to: ", MAPPED to "unreachable". Diagnose: raw curl to :8087 + connect-probe ⇒ NOT network. Fixed in althing v0.17.1. reference_nh3_extdev_althing_mesh. Archived 2026-07-08.

• [2026-06-20] rest-server .htpasswd: permission denied = the ana-nas NFS mount FAILED (ghost file on the local mount point), NOT a decommission. mnt-backup.mount stuck failed (fstab bare defaults) → rest-server serves an empty local dir. Recovery in disaster-recovery.md. Archived 2026-07-08.

• [2026-06-20] The DEFAULT ssh ana-docker is lkraven (no NOPASSWD) — but ssh infra-ops@ana-docker HAS NOPASSWD root. A sudo cp as lkraven silently failed → nearly punted the rest-server recovery. Reach for infra-ops@ana-docker for sudo ops. Archived 2026-07-08.

Archived 2026-08-02 — Recent decisions (archived)

2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session

• [2026-07-08] worldtree Mimir deploy-blocker resolved (mid-session): synced persona.envelopes.assistant + envelope_grants:[] verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my parked R32 1C envelope-mirror come due — see reference_corviduo_dev_emergency_ops config-sync recipe.)

Archived 2026-08-02.

2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b

• [2026-07-09] Two parked items closed: phantom qwen3.6-35b-a3b alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce. (1) Phantom: absent from /v1/models + /model/info (config+DB registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the 2026-07-08 gen repoint to -heretic); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to remove. (2) Log-cap: running containers were UNCAPPED (182M json-logs, top offender 59M) because daemon.json's max-size 10m/max-file 3 only applies to containers CREATED AFTER a daemon restart — it never reaches already-running ones. No-bounce fix = /etc/logrotate.d/docker-containers (copytruncate — dockerd opens json-logs O_APPEND so truncate-in-place resets cleanly, no sparse-file corruption; size 10M, rotate 3, compress, su root root), auto-picked-up by the daily logrotate.timer. Force-ran + gzipped the frozen .1 archives → 182M → ~55M (44M active + 11M gz), every container kept its multi-week uptime (zero bounce, verified). LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares live-restore:true but the RUNNING daemon has it FALSE (daemon.json was edited after the last daemon start, never reloaded) → the NEXT systemctl restart docker / crash / pkg-upgrade bounces ALL ana-docker containers once. Fix WITHOUT a bounce = systemctl reload docker (SIGHUP loads live-restore into the running daemon; log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running containers).

Archived 2026-08-02.

2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped

• [2026-07-09] mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat. Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and repetition_penalty 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not A6000, rep_penalty 1.1. Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).

Archived 2026-08-02.

2026-07-09-granite-gen-memory-extractor-bind-host-synced-on

• [2026-07-09] granite→gen memory_extractor bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4). Changed model_roles.yaml memory_extractor binds.catalog_id summarizer→gen (overrides intact: thinking:false/temp0/8192) on BOTH /opt/worldtree{,-personal}/config; memory_distiller left on summarizer (range-scoped sed /memory_extractor:/,/memory_distiller:/ — the naive global replace would've hit both); backups *.bak-preqgen-20260709; validated via each container's OWN yaml parser. DEMO LIVE on gen (b43 d501e516732d auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted worldtree-worldtree-api-1 to activate; healthy, live process resolves memory_extractor=gen). PERSONAL NOW LIVE on gen too — Vuong authorized the restart (via wt-dev); restarted worldtree-personal-worldtree-api-1 after a PRE-FLIGHT that ran the app's OWN load_model_roles() (core/llm/roles.py:121) against the synced config INSIDE the running c9986cd container: gen in catalog, all 9 roles resolve, no DanglingBindingError → proven safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). StartedAt 20:50:55Z, healthy, resolves gen. BOTH instances live on gen. LESSON: the bind-mount /opt/worldtree*/config SHADOWS the baked /app/config-defaults/ → the deploy alone never updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — pre-stage BEFORE the deploy's restart or you race it. FOLLOW-UPS (non-blocking): (a) memory.extractor.user_pass parity block → self-serve from the b43 baked defaults.yaml (pydantic-default no-op); (b) stale gen provider description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side providers.yaml fix (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway config, not the doc); corrected docs/pfi/model-sampler-defaults.md seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly from that unpushed commit (local ~/development/Worldtree checkout — capital W; git show 5d4fa4a:config/…) + mirrored VERBATIM to /opt/worldtree{,-personal}/config on demo+personal, paired-pre-flighted via the app's load_model_roles() (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups *.bak-prerp-20260709. context_window VERIFIED (llama.cpp /props + char-rp-gguf .env): char-rp 131072, char-rp-reasoning 262144 (gave wt-dev to patch the repo from its interim 32768). HOST AHEAD of repo-remote until the operator pushes 5d4fa4a (baked config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
  • character-rp + stripping personal's char-rp default_params temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in b45/3384a37 (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): mirrored b45's providers.yaml → /opt/worldtree{,-personal}/config, restarted, verified LIVE (char-rp ctx 131072, char-rp-reasoning 262144, bindings intact); backups .bak-b44interim-20260709. HOST now = b45 canonical (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push of b44+b45 (5d4fa4a+3384a37) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity block (defaults.yaml — NOT in either, separate). Threads 01KX3SGH/01KX48QP (worldtree-dev), monitor armed. See reference_corviduo_dev_emergency_ops.

Archived 2026-08-02.

2026-07-09-granite-gen-memory-extractor-bind-green-lit-for

• [2026-07-09] granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4). Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). This bind is INDEPENDENT of the full granite RETIRE (reclaim ~32GB on ana-ml2 GPU1) — that stays the operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + reference_litellm_gateway.

Archived 2026-08-02.

Archived 2026-08-02 — Tried and abandoned (archived)

2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s

• [2026-07-01] A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth. deploy-personal.yml is PULL-ONLY but fires on the staging/vX tag simultaneously with deploy.yml's build → pulls before the push finishes. FIX: re-run once built, or gate on workflow_run: completed.

Archived 2026-08-02.

2026-07-04-on-prem-t1-train-that-keeps-any-ana

• [2026-07-04] On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS (1-GPU + NVMe ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. reference_t1_cloud_train_plan.

Archived 2026-08-02.

2026-07-04-a-systemd-user-daemon-that-shells-out-to

• [2026-07-04] A systemd --user daemon that shells out to ~/.cargo/bin/~/.local/bin tools needs an explicit Environment=PATH — the minimal --user default silently drops them. The althing herald lost zellij → silent pane-miss for ALL config-backed TUI/pane agents; CC + FIFO routes were unaffected, so it was invisible from a CC session. reference_nh3_dev_althing_herald.

Archived 2026-08-02.

2026-07-09-fp8-breaks-morpheus-audio-token-generation

• [2026-07-09] FP8 breaks mOrpheus audio-token generation. --quantization fp8 on the 3B → 0 valid SNAC frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.

Archived 2026-08-02.

2026-07-09-sentence-chunking-tts-loses-prosody

• [2026-07-09] Sentence-chunking TTS loses prosody — generating each sentence cold flattens the intonation that spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: repetition_penalty >1.1 BREAKS cloning (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).

Archived 2026-08-02.

2026-07-09-hf-whisper-datasets-aren-t-actually-whispered

• [2026-07-09] HF whisper datasets aren't actually whispered. Claris-Whispered-English measures voiced 0.8 (not a whisper) + IPA transcripts; datasets audio decode needs torchcodec (wants CUDA-13, incompatible w/ the cu124 venv). LPC DSP-whisperize went unstable (NaN). kokoro af_nicole IS a genuine whisper (voiced 0.24) — that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.

Archived 2026-08-02.

Recent decisions (archived 2026-08-03 batch)

2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed

• [2026-07-14] NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned. Modelopt-format re-quant made MTP work. The load-crash root cause = vLLM 0.24 does NOT propagate modelopt exclude_modules to the spec-decode DRAFT model → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (is_layer_skipped is exact-membership, and the drafter never sees the mtp excludes) → workaround = a mounted sitecustomize that force-skips mtp.* in is_layer_skipped (upstream vLLM bug to file). Productionized as compose stack heretic2-charrp-reasoning (:8018, workaround baked in). Gateway char-rp-reasoning alias fixed: repointed off the stale GGUF served-name deckard-pkd-27b, added enable_thinking:true, dropped min_p (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed leak-free (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in docs/runbooks/heretic2-nvfp4-mtp-seat.md; committed 982c319. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug.

Archived 2026-08-03.

2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt

• [2026-07-14] NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP. One ~40-min GPU0 window. Root-caused the !!!! to the quant NAMESPACE (text-only AutoModelForCausalLM→model.layers.* keys; vLLM serves only Qwen3_5ForConditionalGeneration, which needs model.language_model.*) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as AutoModelForImageTextToText. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to pursue a modelopt-format re-quant (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON /tank/aimodels/qwen36-27b-aeon-nvfp4 = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + mtq.quantize/NVFP4_DEFAULT_CFG/export_hf_checkpoint API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section.

Archived 2026-08-03.

2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning

• [2026-07-14] Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 /tank/aimodels/heretic2-nvfp4-work/ + scripts committed in eshpfi services/heretic2-nvfp4-quant/.

Archived 2026-08-03.

2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code

• [2026-07-14] char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory charrp-custom-llamacpp-pr25544.

Archived 2026-08-03.

2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause

• [2026-07-14] soong-lab webhook auto-deploy real root cause = gitea webhook.ALLOWED_HOST_LIST (was external, 10.100.0.0/16 = NH3-only; blocked corviduo-dev's Anaheim 10.250.x → gitea refused to deliver, never opened the connection). Fixed to fleet-wide 10.0.0.0/8 (app.ini [webhook]) + gitea restart; listener now logs every delivery. The ufw 10/8 open (also this session) was a real-but-secondary gap. Committed 462d528.

Archived 2026-08-03.

2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway

• [2026-07-13] #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN). char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to max_tokens=32768 (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and --reasoning-budget 400 is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-reasoning_content deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1.

Archived 2026-08-03.

2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s

• [2026-07-13] Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong). Buildx step log: chown -R /app = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop /app from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot.

Archived 2026-08-03.

2026-07-13-wt-355-residual-300s-hang-localized-to-our

• [2026-07-13] WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; --reasoning-budget 400 forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. [SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]

Archived 2026-08-03.

2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree

• [2026-07-13] WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.

Archived 2026-08-03.

2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min

• [2026-07-13] Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile uv sync ... --no-cache + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).

Archived 2026-08-03.

2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on

• [2026-07-13] Ledger tier-3 consumer ledger:miranda provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); assistant+thoughtful-assistant capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).

Archived 2026-08-03.

2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree

• [2026-07-10] Heimdall grant: ratatoskr affect.full on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request). Added allow-rule ratatoskr-affect-full-allow to /opt/worldtree-personal/config/policies.yaml (principal.user_ids:["ratatoskr"], action affect.full, resource *, effect allow), mirroring the #347 session-history-write-ratatoskr rule exactly + placed right after it. WHY user_ids-based (not tier): ratatoskr's personal key is the minimal readonly-admin observability tier, which is NOT in the tier-based affect-render-baseline-allow (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. R34-v1 (b46, committed UNPUSHED) gates affect.emit dominant_emotion egress by exposure ceiling (affect.full|safe → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string insert (preserves comments), backup policies.yaml.bak-pre-affectfull-20260710, validated via the CONTAINER's own yaml parser (35 rules, +1, payload confirmed). NOT restarted — deliberate: rule is on the bind-mount (shadows baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they want it live for pre-b46 testing. See reference_corviduo_dev_emergency_ops.

Archived 2026-08-03.

2026-07-10-comfyui-v0-27-1-success-on-irv-ml1

• [2026-07-10] ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED. The prior attempt (entry below) crash-looped because a torch 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked git diff v0.24.1 v0.27.1 -- requirements.txt FIRST and found core v0.27.1 leaves torch UNPINNED → the version bump does NOT require torch 2.13 (that came only from the mmartial boot-upgrade). So: git checkout v0.27.1 (clean tree) → pip install -r requirements.txt as uid 1000 with a torch-pin constraint file (torch/vision/audio pinned to current +cu129) to block any transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). docker restart comfyui → healthy, /system_stats comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). LESSON: before a mmartial ComfyUI version bump, git diff <old> <new> -- requirements.txt — if torch is unpinned, bump the CODE without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. docker exec lands as uid 1025(comfytoo), not 1000 — use -u 1000 + the venv python /comfy/mnt/venv/bin/python. See reference_irv_ml1_comfyui_mmartial.

Archived 2026-08-03.

2026-07-10-comfyui-0-25-x-bump-on-irv-ml1

• [2026-07-10] ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it). comfy-dev requested (Vuong-authorized) bumping the irv-ml1 comfyui stack (mmartial image, /opt/docker/compose/comfyui/, 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. TWO FINDINGS: (1) DISABLE_UPGRADES=false/USE_PIPUPGRADE bumps the VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout (/comfy/mnt/ComfyUI = /worktank/comfyui/run/ComfyUI stayed 0.24.1 — pinned/detached git, comfy-dev's domain). (2) the torch bump broke SageAttention (2.2.0 _fused.so undefined-symbol c10::impl::cow::materialize_cow_storage vs torch 2.13.0) → --use-sage-attention (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke the working state, zero 0.25.x payoff. ROLLBACK WORKED: pre-bump 16G venv snapshot /worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar restored (torch 2.12.1 + working SageAttention), re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at /worktank/comfyui/run/venv.broken-torch213-20260710. CORRECTED PATH (sent comfy-dev, thread 01KX655V): comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention rebuild-against-2.13.0 + re-pin (snapshot stays as the net). LESSON: mmartial DISABLE_UPGRADES gates ONLY the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → rebuild-after is mandatory. Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.

Archived 2026-08-03.

2026-07-10-biweekly-open-weight-releases-scan-cron-set-up

• [2026-07-10] Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized). Durable systemd --user timer on nh3-dev (brokkr-landscape-scan.timer, OnCalendar *-*-01,15 09:00:00 America/Los_Angeles, Persistent=true; linger on) → .service → wrapper ~/.local/bin/brokkr-landscape-scan.sh runs headless claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions in ~/development/brokkr-smithy (ALTHING_HANDLE=brokkr-smithy-dev; explicit PATH — the --user minimal-PATH footgun; per-run logs ~/.local/state/brokkr-landscape-scan/). Prompt = brokkr's payload verbatim (LLM/image/TTS new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): web-sweep→synthesis→commit 2ed2f29→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + caught baseline errors, quality strong. HANDLE-COLLISION caught+FIXED — the headless scan shared handle brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); registered a dedicated brokkr-scan-dev handle (add-handle, driver=none) + repointed the wrapper + rewired step-5 notify → althing-cli post --to brokkr-smithy-dev (NO vuong althing handle exists — confirmed). model=default
  • --max-turns 80. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. NEXT AUTO-RUN 2026-07-15 09:00 PDT. Manual validation/first run = systemctl --user start brokkr-landscape-scan.service. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session handle + model/turn-cap. NEXT brokkr task (operator-sequenced after this): TTS audition env — Higgs-TTS-3 + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).

Archived 2026-08-03.

2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b

• [2026-07-08] RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin. Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a DRY-tolerant model): Deckard WON on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the live A/B set is canonical (4954ca0). GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively opens <think>/enable_thinking (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap b8840 can't load Qwen3.6/Gemma-4 archs → ghcr.io/ggml-org/llama.cpp:server-cuda (pulled on ana-ml2).

Archived 2026-08-03.

2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated

• [2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training. Final topology: gen = llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only (LIVE, modelopt, vision, util 0.40), char-rp = an RP unicorn to be found on fresh context (see Current state). Intermediate steps ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway sampling-defaults wiring still PENDING.

Archived 2026-08-03.

2026-07-08-dpo-was-silently-running-3-epochs-harness-gap

• [2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call). Root cause: DpoConfig had NO epochs field + _dpo_config_kwargs didn't pass num_train_epochs → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
  • _sft_config_kwargs). Objective SATURATED by epoch 0.27 (loss0, grad0, acc 1.0, margins27 flat — the off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. FIX (3 edits to deployed harness, mtf-dev to canonicalize): DpoConfig.epochs: float = 1 (mirrors SftConfig, float for fractions); _dpo_config_kwargs now passes "num_train_epochs": cfg.epochs; recipe dpo.epochs: 0.3. GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min train = ~45min). DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; train_loss 1.4e-5 @ epoch 0.301). Fresh data/spike/dpo_adapter/adapter_model.safetensors (123MB) + checkpoint-45 banked. 3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): ALL THREE DISTINCT (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. DPO 0.3ep is COHERENT, fluent, NOT degenerate (early-stop avoided over-optimization) but the quality delta on a neutral literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).

Archived 2026-08-03.

2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes

• [2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py. Blocker resolved: (1) mtf-dev's v0.0.42 stub _stub_missing_optional_integrations (last-resort sys.meta_path finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied VERBATIM to deployed src/model_training_forge/train/backend.py after _unsloth_available() + call-site before from trl import DPOTrainer); (2) my cosmetic warnings_issued shim (trl-0.24 DPOTrainer.init:405 does model.warnings_issued["estimate_tokens"]=True for warning-suppression; custom Qwen3_5 class under transformers 5.5.0 lacks the attr → if not hasattr(model,"warnings_issued"): model.warnings_issued={} before the DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED un-git'd copy only → mtf-dev must canonicalize the warnings_issued shim into their repo (told them). DPO confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.

Archived 2026-08-03.

2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved

• [2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe. Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. PROVEN LAUNCH RECIPE (replicates the SFT container aeon-t1-sft exactly, only --stage sft→dpo): sudo docker run -d --name aeon-t1-dpo --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml --stage dpo. CRITICAL: --entrypoint python3 is REQUIRED — aeon-trainer's default entrypoint is ["vllm","serve"] (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. Dataset verified (pairs_dataset=train.flat.json=1196 pairs). THE BLOCK: from trl import DPOTrainer (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ immutables→mergekit-0.1.4↔pydantic-2.13 HARD incompat (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ llm_blender-0.0.2↔transformers-5.5.0 HARD incompat (TRANSFORMERS_CACHE removed, needs source patch)→weave→ (more). NONE used by our pair-based DPO. pip install mergekit w/deps is UNSAFE (downgrades accelerate 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.

Archived 2026-08-03.

Recent decisions (archived 2026-08-05 batch)

• [2026-07-16] GPU re-org: char-rp→GPU1 + both cards re-optimized for max context. Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→128K full-chapter (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. Archived 2026-08-05.

• [2026-07-16] granite right-sized → ~10.5 GB freed on GPU1 (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536). LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped before 0.18 landed. .env-only, recreate vllm-granite alone (shared stack). [Superseded by the 07-16 GPU re-org above → 128K.] Archived 2026-08-05.

• [2026-07-15] image-bench eviction DONE (parked item closed). Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM image-judge+qwen-image-bench → gen :8015 (judge samplers + thinking-off); comfy-dev pinged; backfilled the canonical char-rp-reasoning litellm block. Revert ~90 s. auto-memory project_arbo_gen_switch_imagebench_evict. Archived 2026-08-05.

• [2026-07-15] Homepage AI-tab revamp — flat "AI Systems" group → dedicated AI tab, 6 role-based groups + AI-Dormant; committed 569e1af, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) Archived 2026-08-05.

• [2026-07-15] Home Assistant config repo created (vh/home-assistant-config, private). UI-managed HA → allowlist model (YAML + curated secret-free .storage subset). git-in-place in /config on esh-docker-vm + scoped deploy key + local clone ~/development/home-assistant-config. Archived 2026-08-05.

• [2026-07-15] char-rp-reasoning OOM rescue — solo-restart on the packed GPU0 crash-looped; fixed via expandable_segments:True + util 0.39→0.38 + max-model-len 192K→150K. LESSON: max-model-len does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom. Archived 2026-08-05.

• [2026-07-15] soong-lab SOONG_LAB_LIBRARY_DIR made persistent (corviduo-dev) — was on the redeploy-wiped code default; set to /home/infra-ops/soong-lab-data/library, restarted. Closed a queued no-rush item. Archived 2026-08-05.

• [2026-07-15] Statusline overhauled (~/.claude/statusline-command.sh) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost / threshold-colored ctx+rate. Archived 2026-08-05.

Tried and abandoned (archived) — moved 2026-08-12

• [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments. Archived 2026-08-12.

• [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>. Archived 2026-08-12.