vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6332f14af5f038c29e5b08b23c1536f777cb3c06
esh-pfi-infrastructure/persistent-memory.md
T
vh 6332f14af5 memory: chatterbox-fast tail-degradation diagnosed + fixed (max_chunk_chars=250 cap) 
Long operator-driven diagnosis: the 'broken/German/dead-air' was the Turbo T3
model over-running its generation tail (garble in final ~2-3s, worse with
length + tight sampling), NOT a language leak or OOM. Fix = server-side
max_chunk_chars=250 cap (:v2), keeping 3-4 sentence clean chunks with smooth
joins. Method (amplitude-gated voiced-ZCR), foot-guns (tail-trim unreliable,
build-context/image drift), and the flat-vs-repo divergence recorded.
2026-08-07 11:40:05 -07:00

75 KiB
Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-08-07

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo Role CI status
vh/task-board MCP + web dashboard for assistant task state (port 7878) push-to-main → CI deploys (2026-04-29)
vh/vor Inquisitor UI sidecar (port 7879) push-to-main → CI deploys (2026-04-29)
vh/nevermore Twice-daily LLM-curated briefing (port 8181, replaces news-digest) push-to-main → CI deploys (2026-04-30)
vh/asset-engine Internal control plane over inference services (port 8200, LAN-direct) push-to-main → CI deploys (2026-05-12)
vh/althing Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API /owner/* / althing-mcp stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. per-box uv tool install (NOT CI-deploy); nh3-dev = the DEV box (editable install of ~/development/althing, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared /srv/althing)
vh/mead-hall Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) push-to-main → CI deploys (2026-05-16)
vh/skaldsong Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) push-to-main → CI deploys (2026-05-19)
vh/Worldtree Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. push-to-main → CI build-and-deploy (runner on ana-docker)
vh/yt-voice-clipper YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see docs/runbooks/ytvc-autodeploy.md
vh/arbo Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
vh/zonos-gateway OpenAI-compatible TTS gateway over stock ZONOS2 (:8890 irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM ext-tts alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (resolve_preset(name,voice); angry/happy/startled_happy per-voice). 8 voices incl. 4 clones pushed to gitea (main 8f1885b/v0.2.1); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec docs/EMOTION-DIALS-SPEC.md; host-managed voices bind-mount (./voices:/app/voices, drop wav + restart, no rebuild)
vh/soong-lab Noonien Soong character-design studio (SPA + /api + WT /bifrost/tool-call); containerized 2026-07-18, LIVE on corviduo-dev :8443 (image vh/soong-lab:latest). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as deploy, compose pull && up -d from /opt/soong-lab, health-gated on /api/version). Manual redeploy sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'. → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md
model-training-forge (mtf-dev) Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward) training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

  • Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)

  • pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.

  • Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.

  • Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).

  • Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.

  • gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.

  • docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.

  • scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.

  • Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-08-07 — heavy overnight session landed; b182 deployed and the operator-authorized personal-KB orphan sweep EXECUTED + verified this (post-/clear) session. The night's four arcs are all landed or handed off:

  • FLEET RERANKER CUT OVER (the big one) — DONE + VERIFIED. Brokkr R43 bake-off replaced the fleet reranker (Qwen3-Reranker-0.6B, measured HARMING 80/90 queries fleet-wide — no-reranker beat it 89/90 vs 56/90) with A3 = BAAI/bge-reranker-v2-m3. LiteLLM reranker alias repointed incumbent→A3 (boundary 2026-08-06T17:37:48Z, config-edit on ana-docker litellm + ~52s gateway restart); Brokkr independently VERIFIED (maxdiff 0, single backend) and the R42 v13 acceptance gate PASSED — first ever (main+kb 56/90→90/90, evictions 33→0). Bench wound down: A2 (control) torn down; A3 = PROD on ana-ml2 GPU1 :8013; A4 (gte-modernbert) = warm fallback :8014; incumbent still up :8002 for rollback (via the untouched qwen3-reranker alias — rollback = revert one config line + a SECOND ~52s gateway restart). A3/A4 restart=unless-stopped. Full arc + rollback runbook: docs/pfi/reranker-selection-ledger.md (git-tracked; commits ad2df89/2c11748/377f8a4, UNPUSHED). Passive watch: A3 caps ~34 req/s (vs incumbent's ~55) — queues gracefully; if real p99 bites, levers = A4 swap / bump A3 gpu-util / 2nd A3 replica (Brokkr re-runs the grid, no intuition swap). Non-urgent follow-up: promote A3 from throwaway docker run to a stacks/vllm/ compose service (a recreate briefly drops the alias → needs a window); retire incumbent when the post-cutover watch closes.

  • b182 DEPLOYED + personal-KB orphan sweep DONE (2026-08-07 ~14:0xZ). personal-api recreated onto b182 (gitea…/worldtree:d6e5d6ec7b00, git d6e5d6e, image sha256:3f82b147…, healthy, recreated 13:55:16Z; identical to the image demo serves — worldtree-dev re-tagged staging/v1.0.0b182 onto d6e5d6ec7b00 after the mid-train-memory-commit staging-tag miss). First-boot kb held at 8,230 (no re-sweep — the #394 fix works). All 3 preconditions verified (image/kb/orphan) → per operator's AUTO-EXECUTE ruling (no final-GO pause) ran reconcile --repair (uid 1000, index-rows-only): kb 8,230 → 2,876 (−5,354 orphans); fresh reconcile confirms orphan=0/missing=0/stale=0. flag-before-personal-redeploy commitment RETIRED (b182 is the fix that guard protected). Report posted to worldtree-dev thread 01KZE6TGRHAW (msg 01KZE8WT6W…). ⚠ --repair exited rc=1 on a NON-blocking worldtree-side bug: after the sweep, reconcile.py:166 ws._repo.index.add(report.uncommitted_in_git) threw FileNotFoundError on phantom path docs/specs/saga-taxonomy-redesign-plan.md (a tracked-but-deleted entry in the kb wing's 1,847 uncommitted set; GitPython can't lstat it). No commit ran, wing git state stable (uncommitted still 1,847). Flagged to worldtree-dev as (1) reconcile robustness gap + (2) pre-existing kb-wing git inconsistency — their code/data call, I did NOT touch reconcile.py or the wing. Sweep GATES Brokkr's 16-subject floor re-run + ratatoskr's #393 baseline (worldtree coordinates).

  • kb-CONTAMINATION INCIDENT (WT #394) — diagnosed, fix shipped (b182), attribution CLOSED UNRESOLVED. A WingStore._embed reconcile full-tree walk (kb wing fs_root = KB_PATH root, sibling wings nested under it) swept 5,354 fiction+main rows into personal's knowledge_base (incl. 2 SUPERSEDED generations served as current). Fixed by worldtree #394 (aca39a1, kb walks exclude sibling wings). Trigger un-attributable — a peer reconcile via the SHARED infra-ops identity leaves no fingerprint (0 dockerd/containerd exec-logging). Durable finding → auto-memory infra_ops_shared_identity_attribution_gap, PARKED (operator ruled A) into the identity-hygiene arc project_migrate_infra_access_to_claude_credentials. EVIDENCE HOLD — INDEX-ROW half LIFTED + EXECUTED 2026-08-07: operator authorized (relayed via worldtree-dev thread 01KZE6TGRHAW); the 5,354 swept index rows are now DELETED via reconcile --repair post-b182 (kb 8,230→2,876, orphan=0 — see the b182 sweep entry above). The FILE half of the hold STILL STANDS: do NOT delete the on-disk generation dirs (fiction/rex390-dcc, rex392-dcc, b59c147c5ce0) — file-retention hold is separate and unlifted.

  • gen-reasoning empty-content (WT #392) — serving EXONERATED, no infra action. ~2-3% zero-concept sections on the personal concepts seat = thinking-model channel behavior + broken JSON string-escapes, NOT serving config (vllm-aeon-gen :8015 runs --reasoning-parser qwen3, no reasoning-budget). rex393 raw data confirmed; worldtree's b179 retry self-heals at zero data loss. The thinking-off lever is Worldtree-court AND benched-mandatory (reasoning needed for the significance filter per #385), so no flip.

  • PERSONAL-KB SNAPSHOTS on nh3-dev (read-only pulls for worldtree probes, provenance-marked, do NOT auto-delete): ~/snapshots/r42-gate-index/ (#389, operator ruled KEEP), ~/snapshots/rex393-fiction-index/ (#393, full multi-collection persist store — fiction 1578 / main 3224 / kb 8230; clear when worldtree signals #393 done). The conversations.db attribution copy was SHORT-LIVED and is ALREADY DELETED.

  • muninn-dispatch 0.1.6 PUBLISHED to vh Gitea PyPI (2026-08-07) — worldtree-dev's carried 08-05 ask, DONE. Comment-only bump. Built from a clean git archive origin/main packages/muninn-dispatch (commit 578f8fc "release 0.1.6 — provenance sweep of shipped-code comments", on the b182 line, origin/main tip d6e5d6e) — NOT the operator's dirty working tree; hatchling, deps=[] stdlib-only (INV-377-5). uv publish via the vh-token; index lists 0.1.6 (whl+sdist), clean-venv install + import + importlib.metadata==0.1.6 all verified. Pinged muninn-dev (lift the 0.1.5 pin) + worldtree-dev (done). Runbook note: monorepo subpackage → source via git archive of the Worldtree subpath (the sdk runbook's API-archive is for standalone repos); otherwise identical to reference_infra_ops_vh_gitea_token_and_sdk_publish. Other standing/parked: CI-flip runner-auth research (Recent decisions 08-05 detail file); gitea REQUIRE_SIGNIN_VIEW=false standing; #363 research-wing ingest (no deadline); zonos emotion + zonos-gateway CI-wire follow-ups.

althing monitor ARMED (handle infra-ops; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after a real FIRE (<task-notification> completed rc0), never after a plain operator turn (bounces rc3); spawn althing-wake-listener as its OWN run_in_background task, NEVER chained with &/&&.

eshpfi push state: origin/main is behind by 7 commits (08-05 booth/memory + this session's 3 reranker commits ad2df89/2c11748/377f8a4 + this snapshot). Push is the operator's call. graphify-out/GRAPH_REPORT.md = graphify-hook artifact (churns every commit, ignore); stacks/heretic2-charrp-reasoning/ UNTRACKED (pre-existing, operator's).

Recent decisions

  • [2026-08-07] chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed). Long saga, operator-driven clean diagnosis. Symptom: ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. NOT German-leak (Turbo generate() has NO language param — plain AutoTokenizer, no language_id; the multilingual language_id="en" lever lives only in the separate ChatterboxMultilingualTTS), NOT OOM alone. Real cause: the Chatterbox Turbo T3 model OVER-RUNS its generation tail — a long single generate() degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens <6561 + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). Diagnosis method (deterministic, no ears-only): single-shot length sweep + amplitude-gated voiced-ZCR (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. FIX: server-side max_chunk_chars=250 cap on the scheduler (:v2 image, CBF_MAX_CHUNK_CHARS=250 env) — bounds each generation to just under the ~300-char onset → clean 3-4 sentence chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; chatterbox's low emotiveness keeps chunk joins smooth (the harsh joins that got Zonos rejected are absent — operator's key call). ratatoskr TODO (relayed msg 01KZER9X7S): revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. Cap value tunable per-request (max_chunk_chars) + env. Deeper prosody (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). ⚠ FOOT-GUNS: (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) build-context vs image drift — the :v2 image was built from cap source, but after a :v1 rollback the build context held :v1 source → a docker compose build would've silently produced a cap-less :v2; re-synced the flat cap source to /opt/docker/compose/chatterbox-fast/ (rebuild-verified). ⚠ DIVERGENCE (follow-up): deployed build context is FLAT (app.py/scheduler.py, from scheduler import, thin-overlay FROM local/chatterbox:v1, cap-only) vs the vh/chatterbox-fast REPO which is PACKAGE-layout (chatterbox_fast/, from chatterbox_fast.scheduler, self-contained Dockerfile) + has norm_loudness (repo commit 6bc7bf0 = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: .bak-cap-20260807-104850 backups on irv-ml1 + :v1 image both retained. reference_chatterbox_fast_repo reference_zonos_tts_stack

  • [2026-08-07] Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY. Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. ⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init. GPU memory was held by the --multiprocessing-fork CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). RESTORE CMD (from irv-ml1, user lkraven): cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 & then docker start zonos-gateway. Consumers that lost Zonos: asset-engine + gateway-chat (via LiteLLM ext-tts alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). reference_zonos_tts_stack

  • [2026-08-07] chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos). Operator-directed. Copied zonos-gateway/voices/Donut.wav → chatterbox /refs (/worktank/chatterbox/reference_audio/donut.wav — the reference_audio SUBDIR is lkraven-owned so no sudo despite /worktank root; container globs /refs live → NO restart), exposed as voice:"donut" (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at http://10.100.10.50:8090/b/donut-chatterbox/. Answered ratatoskr's 8-question contract ask from the live gateway (local/chatterbox-fast:v1) + source: NOT OpenAI-shaped (POST /tts; body text/voice/format/stream, not input/model/response_format); NO affect dials (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; Zonos stays the only fleet TTS with real emotion steering); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. FYI-worthy (operator): ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory reference_chatterbox_fast_repo enriched w/ the live contract. reference_zonos_tts_stack

  • [2026-08-07] Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43). The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM reranker repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); R42 v13 gate PASSED first-ever (56/90→90/90). Incumbent kept warm :8002 (rollback via qwen3-reranker alias), A4 fallback :8014. Full arc + rollback runbook docs/pfi/reranker-selection-ledger.md; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.

  • [2026-08-07] Personal-Worldtree kb-contamination incident (WT #394) diagnosed; attribution CLOSED UNRESOLVED. A reconcile WingStore._embed full-tree walk (kb fs_root=KB_PATH root, sibling wings nested) swept 5,354 fiction+main rows into personal's knowledge_base (2 superseded generations served as current). Fixed by WT #394 (aca39a1, kb walks exclude sibling wings; ships b182). Trigger un-attributable — peer reconcile via the SHARED infra-ops identity + 0 dockerd exec-logging = fingerprint-less. Durable finding → auto-memory infra_ops_shared_identity_attribution_gap, PARKED (operator ruled A) into project_migrate_infra_access_to_claude_credentials. Evidence hold on the 5,354 rows until operator sequences cleanup (w/ Brokkr, on #394's agenda).

  • [2026-08-05] Fleet CI resilience flip (DEFAULT_ACTIONS_URL=self) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now). 7 gitea action mirrors staged public+populated (orgs actions+astral-sh); the flip resolves uses: correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); REQUIRE_SIGNIN_VIEW=false KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → persistent-memory.d/2026-08-05-ci-flip-parked.md

  • [2026-08-05] Booth — 3 features shipped, live on :8090 + tagged. (1) verbatim-index.html booths get a floating top-right "‹ all booths" chip + inherited favicon, doctype/charset-safe byte-injection (booth-v0.1.5, 8577e7e); (2) .md renders + .txt/.log view in-booth without downloading via the /b/<n>/view route + a markdown dep + doc.html (booth-v0.1.6, 315faac); (3) prev/next arrows in the image zoom viewer — wrap-around + keyboard ←/→, hidden for single-image booths (booth-v0.1.7, c37a425). Canonical services/booth/; deploy = systemctl --user restart booth.service on nh3-dev (runs from the checkout's .venv; uv pip install new deps into it first); 47 tests. uv.lock gitignored (348c5c1).

  • [2026-08-05] worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (d5d33df, deployed on nh3-dev). herald.py:363 rendered the wake command from the empty fresh mail set on the re-nudge path (should be deliver_msgs) → messages[0] IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + render_command empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. nh3-extdev herald 2.1.2 upgrade DEFERRED (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev /tmp (sha256 003508…cef27) — uv tool install --force + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory reference_nh3_dev_althing_herald.

  • [2026-08-03] worldtree b168/#384/#385 arc COMPLETE — providers.yaml boot-gate pre-sync → b168 deploy → DCC+P&P re-ingest (705+667 concepts, 0 truncations, #385 budget fix validated vs April's 785 control) → #381 restart → operator-approved production dedup sweep (785 April orphans deleted from main, 4009→3224). Fiction wing 166→1,372 concepts; consumer verify 0/5→5/5→saturated. Three of MY foot-guns hardened into fleet runbook rules (mv -t, docker exec -u 1000, shared-containerd pull-race — see Tried-and-abandoned). Full runbooks (deploy-wt-config, Chroma-verify, config-delta pre-sync rule, #381, sweep) → persistent-memory.d/2026-08-03-worldtree-b168-384-385-arc.md

  • [2026-08-02] mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice. mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live 8ece117); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and verified 3/3 by ratatoskr-dev; muninn-gate → dispatch 0.1.5; donut voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → persistent-memory.d/2026-08-02-mimir-inbox-arc.md

  • [2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090. First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → persistent-memory.d/2026-07-31-muninn-gate-deploy.md

  • [2026-07-31] worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred. memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on vh/worldtree-sdk (source pulled via the Gitea API archive — git-HTTP 403s on that repo); publishing to the vh USER namespace can't be delegated (401 reqPackageAccess even with write:package) so it needs an owner token — operator saved a FULL vh site-admin token at ~/.config/gitea/vh-token (0600) for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory reference_infra_ops_vh_gitea_token_and_sdk_publish.

  • [2026-07-31] kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap. heid's cross-frontier panel was silently degraded (empty content, finish_reason: stop). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. Proven on the wire against heid's real 500KB bundle: reasoning_effort: low drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send reasoning_effort via extra_body (LiteLLM drop_params: true strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = allowed_openai_params on the route. → persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md

  • [2026-07-27] Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end. Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via /v1/completions. Deep-research (106-agent workflow) picked Qwen/Qwen2.5-Coder-1.5B (BASE, Apache-2.0; native FIM <|fim_prefix|>/<|fim_suffix|>/<|fim_middle|> IDs 151659/60/61; Zed prompt_format:"qwen"). Runner-up 3B = non-commercial Qwen-Research license; no small dense Qwen3-Coder exists (all MoE, smallest 30B). Stood up vllm-coder on ana-ml2 GPU1 :8020 (served-name qwen2.5-coder-1.5b, 8192 ctx, util 0.06, fp8 KV). To fit, shrank granite (phasing out, operator-directed): util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). LiteLLM alias coder-fast → :8020 (mode: completion). Minted a coder-fast-SCOPED virtual key (verified 403 on gen — the real blast-radius bound). Built zed-fim-proxy (ana-docker :4141, network_mode: host, stdlib-python, stacks/zed-fim-proxy): keyless POST /v1/completions, model-allowlist coder-fast, injects the scoped key → LiteLLM :4000; GET /ping anon liveness; wrong-model→403, wrong-path→404, /chat/completions rejected. Verified keyless FIM end-to-end ('a + b', finish stop). Zed api_url = http://10.250.50.70:4141/v1, model coder-fast, prompt_format qwen. source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten: Zed roams the operator's WireGuard 10.0.0.0/8, so a single-IP pin would break it. Blast-radius bound is the coder-fast-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the 10.0.0.0/8 CIDR would need CIDR support — deliberately not added.) Canonical: stacks/vllm (coder + granite shrink), stacks/litellm (coder-fast), stacks/zed-fim-proxy (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).

  • [2026-07-27] Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377). worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a worldtree-muninn compose sidecar to /opt/worldtree-personal/compose.yaml — <<: *worldtree-common anchor inherits the api's image + full env + config/state/kb mounts; command: python -m core.muninn --watch; restart: unless-stopped; stop_grace_period: 1h (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). Pinned to the running SHA 773866084af9 (b146, ≥ b143 — dodges both the :latest trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at {ingestion_root=/data/state/ingestion}/.watcher-heartbeat (poll 30s). Container worldtree-personal-worldtree-muninn-1; backup compose.yaml.bak-muninn-20260727-081920. DURABILITY RESOLVED (worldtree-dev, same day): Q1 was a LIVE FOOTGUN — deploy-personal.yml scp's the REPO compose.yaml over the box's + runs up -d --remove-orphans, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a muninn compose profile (commit 5d7f6bd) — shared compose stays instance-identical, .env COMPOSE_PROFILES differentiates (demo watcher-less). My action: added COMPOSE_PROFILES=muninn to /opt/worldtree-personal/.env (backup .bak-muninn-profile-20260727-082541; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy up -d's the whole stack w/ WORLDTREE_IMAGE exported → sidecar version-tracks the api, no drift. CONFIG-AS-CODE EXTENSION: mirrored the non-secret delta as personal/env.public in vh/worldtree-instance-configs (repo a9d091e) — FIRST extension beyond config.yaml files to env-level config; the secret-laden .env stays box-only, env.public records only non-secret infra-ops-owned env deltas (record, not a deploy source — deploy-wt-config globs *.yaml). BOUNDARY CLARIFIED: compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance .env = infra-ops's differentiator. Deploy step of the #363/#377 arc. #377 CLOSED — acceptance PASSED 2026-07-27: worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — output.kb_notes=false ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) ⚠ OPERATOR-SURFACE (open): the env.public overlay mechanism is a repo-scope call to bless/adjust. reference_worldtree_deploys_cicd reference_worldtree_instance_configs_repo project_worldtree_research_wing_ingest

  • [2026-07-27] jackdaw-compose.service DECOMMISSIONED (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev :8787 user service (no client calls it — ai/server/AiChat deleted from main, /compose proxy removed); unit archived not deleted → ~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727 (revival = rename + daemon-reload). No credential revoked — the unit used the SHARED all-agents LiteLLM key (sk-eA_XOd…, model gen), not a dedicated one. Code preserved on jackdaw origin/ai-composer-preserved; treat as permanent. The :4500 HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)

  • [2026-07-26] Demo BIFROST_CLIENT_ALLOWED_HOSTS += 10.100.10.50:8391 (wyrd-dev's bifrost memory-store provider; operator-approved). First live exercise of the #376 config-as-code boundary working as designed — worldtree-dev routed the delta to infra-ops instead of hand-editing /opt/demo. Appended to /opt/worldtree/.env:25 (now 4 netlocs), recreated ONLY worldtree-api (the gated conv-api path), health-gate green, container env verified. REUSABLE FOOT-GUN: an env-var change needs a container RECREATE, not docker restart (env is baked at create); and the demo .env defaults WORLDTREE_IMAGE=:latest while the box runs a specific SHA — so a naive compose up risks the documented stale-:latest crash. FIX = capture the running image live (docker inspect …Config.Image → …:9eff09f007ba) and sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api. Backup /opt/worldtree/.env.bak-bifrost-20260726-221602. BOUNDARY SEAM: this was a compose-.env var, NOT a config.yaml file in vh/worldtree-instance-configs — the .env holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config files are versioned, compose env vars aren't). reference_worldtree_instance_configs_repo

  • [2026-07-25] nh3-extdev herald installed — box is now a full v2 push participant. forseti flagged (relaying operator): extdev had the althing-herald binary (/usr/local/bin/) but NO unit (skipped the whole v2 arc), so herald-status = "notifications suspended" and ldp-dev ran on the althing-light-monitor poll fallback. Installed /etc/systemd/system/althing-herald.service as a SYSTEM unit mirroring the receiver (User=althing-svc, Group=althing, Environment=ALTHING_ROOT=/srv/althing, ExecStart=/usr/local/bin/althing-herald --poll 5, enabled) via the lkraven@ NOPASSWD path (used under the then-mistaken belief infra-ops was sudo-less — CORRECTION 2026-08-03: infra-ops has had full NOPASSWD sudo on extdev since 2026-06-25 per reference_nh3_extdev_althing_mesh; future extdev installs can self-serve as infra-ops without the lkraven@ hop). Verified: active / 0 restarts / herald-status flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.

  • [2026-07-25] Booth v0.1.4 — booths are downloadable. Verbatim index.html booths (e.g. edict-design-brief) were served raw with no download affordance. Added /b/<name>/?download=1 (streams the whole booth as <name>.zip, attachment) + ?dl=1 on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. zip_booth() helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi 91a031f / tag booth-v0.1.4.

  • [2026-07-25] bil-smithy-dev wired as an althing zellij-window-ping (pane route). She's a driver: human dwarf peer (pane bil-smithy already live alongside eitri/dvalin/regin-smithy in the Claude zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. Mechanism (reusable for any pane-route handle): ~/.althing/config.yaml → zellij_sessions.Claude.agents[] maps handle → target (a zellij pane TITLE, matched via list-panes -j in althing/zellij.py:resolve_pane_id) → command (herald write-chars + CR into that pane). The herald loads config ONCE at startup (herald.py main()), so systemctl --user restart althing-herald.service after editing. Added bil (target: bil-smithy), restarted, verified: herald delivered the pending smoke 01KYD7W7CF… (available→attempted→delivered). ⚠️ Noticed pre-existing pane-route errors on worldtree-codex + eitri-smithy-dev ("route-error: list index out of range", empty msg_ids — likely render_command messages[0] on an empty list; NOT caused by this change, bil works) — worth a herald look.

  • [2026-07-25] Kimi K3 wired into the LiteLLM gateway — CODING endpoint (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). Primary model_name: kimi-k3 → openai/k3 @ https://api.kimi.com/coding/v1 (Kimi Code / Vivace membership; key KIMI_CODE_API_KEY). A general-endpoint variant kimi-k3-gen-api → openai/kimi-k3 @ https://api.moonshot.ai/v1 (key MOONSHOT_API_KEY) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server .env (NOT committed) + .env.example. Both verified live through the gateway :4000 (17+25→"42", "PONG"). k3 constraints on BOTH endpoints (config-pinned + commented): accepts ONLY temperature=1 (else 400 "only 1 is allowed"); REASONING model (CoT in reasoning_content, answer in content → tiny max_tokens returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries k3-256k / kimi-for-coding / kimi-for-coding-highspeed (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi edaa9a9 (gen wiring) + 9e2f787 (coding correction). OPEN: Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing 01KYD63ZBY…).

  • [2026-07-25] infra-ops Worldtree config-as-code repo SHIPPED — vh/worldtree-instance-configs (private) built, pushed, validated. Dir-per-instance (demo/, personal/; pinned/ = README stub, out-of-scope — no bind-mount, config frozen in image 446e5807). Seeded byte-exact from live /opt/<instance>/config; 5 files each (defaults/policies/model_roles/providers/matrix). scripts/deploy-wt-config = diff / deploy / capture, with in-run host backup → install(vh:vh,644) → restart api+matrix → health-gate api /health → auto-rollback. All verbs live-tested (in-sync, capture round-trips zero-diff, pinned refused, dry-run no-ops). Gitea repo created via ana-docker localhost API with vh creds (operator-authorized one-time); pushed over internal git-SSH 10.250.50.70:222 (nh3-dev 403s gitea HTTP). Boundary AGREED by worldtree-dev (althing 01KYCAECRW…): they stop hand-editing /opt/<instance>/config, route config deltas to infra-ops; three-layer model (image baseline → repo per-instance truth → host bind-mount deploy target); carve-out = their admin-API DB mutations (key mint / tier / retirement) stay in-band, not config edits. By-design deltas (personal agent_architect + ratatoskr-affect-full-allow; demo #308 metrics + grants) preserved verbatim. → persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md, auto-memory reference_worldtree_instance_configs_repo

  • [2026-07-23→25] Worldtree #376 config-divergence arc CLOSED — per-instance config ruled BY DESIGN. wyrd session.history.write demo grant was the one real bug (demo-intended grant not on demo; fixed via wholesale policies.yaml replace + restart). The b131 drift guard then surfaced broader divergence = legitimate live-bridged per-instance deltas; operator ruled deltas are the design not rot; guard demoted to INFO (b132); infra-ops drift-watcher built then retired same day. → persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md, auto-memory reference_worldtree_perinstance_config

  • [2026-07-20→25] The Booth SHIPPED (v0.1.3) — ephemeral media drop board for CC sessions. New fleet tool: user-systemd on nh3-dev :8090 (services/booth/, FastAPI+Jinja2, Corviduo "Australis" theme, 34 tests), Homepage-linked (Apps). Drop a folder in ~/booth-data/<name> → browsable "booth" (auto-gallery of images/webm/audio, or a folder's own index.html verbatim), 24h TTL. Added across the session: browser/curl upload-for-pickup with human-readable ids (4-wombat), image viewer (Fit/1:1, conditional toggle), copy-id button (HTTP-LAN execCommand fallback). Registered in global CLAUDE.md tools. auto-memory reference_booth_media_board.

  • [2026-07-23] jackdaw-compose backend deployed as a persistent nh3-dev service (:8787). Hosted for jackdaw-dev: thin stateless bun server/index.ts (from ~/development/jackdaw) → LiteLLM gen, Origin-gated (INV-BK04/05), reached same-origin via their :4500 bench's /compose proxy. jackdaw-compose.service (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (cloudflared now installed at ~/bin). In the nh3-dev README inventory (cd4d52e).

  • [2026-07-19] irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE). RTXVideoSuperResolution node + nvidia-vfx dep were manual installs; documented both in the canonical stacks/comfyui/README.md runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the node lives in basedir/custom_nodes (persistent, restic-included → durable) but the nvidia-vfx wheel lives in the venv under run/ (disposable, restic-excluded → dropped by any rm -rf run/* fresh-bootstrap), so the pip step must re-run after every venv rebuild. Both steps run as uid 1000 (root install → venv-ownership crash-loop, reference_irv_ml1_comfyui_mmartial); --extra-index-url https://pypi.nvidia.com kept scoped to the nvidia-vfx install, deliberately NOT a global compose PIP_EXTRA_INDEX_URL (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.

  • [2026-07-19] vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned. Consuming agents read the internal vh PyPI (https://gitea.phasefinal.com/api/packages/vh/pypi/simple/) with a shared read-only token (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated read:package-scoped PAT off claude-bot (POST /users/claude-bot/tokens, name vh-pypi-read-consumers; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = UV_INDEX_GITEA_USERNAME=claude-bot + UV_INDEX_GITEA_PASSWORD=<token> (or ~/.netrc); pyproject uses [[tool.uv.index]] name=gitea … explicit=true + [tool.uv.sources] <pkg> = { index = "gitea" } (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. reference_claude_bot_gitea_creds

  • [2026-07-18] soong-lab auto-redeploy WIRED + validated (queued item CLOSED). Added a WT-style CI-deploy step to build-and-push.yml: after build+push, the pfi-fleet runner SSHes corviduo-dev as the deploy user and runs docker compose pull && up -d from /opt/soong-lab, health-gated on /api/version (120s, fails loud). Reused WT's deploy account (uid 1001, docker-group → no sudo); relocated the deploy dir /home/infra-ops/soong-lab-deploy → /opt/soong-lab (deploy-owned; old dir retired .retired-20260718). Minted a dedicated soong-only ed25519 deploy key, pubkey on deploy's authorized_keys (fp SHA256:MG7M3Ri…). First dispatch FAILED on a bad DEPLOY_SSH_KEY paste (error in libcrypto — unparseable key bytes; build+push were fine, live Soong untouched); repo secrets are vh-owner-only (claude-bot token = write:package only → 403; the vh package-scoped PAT also 403 on secrets), so operator re-set DEPLOY_SSH_KEY/HOST/USER. Re-dispatch run #5 GREEN: live container recreated ...541f7730 → ...07526a08, health 200. soong-dev pinged to sync DEPLOY.md's redeploy path (/opt/soong-lab) + close the "auto-pull open follow-up". → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md

  • [2026-07-18] worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag python-v1.0.0 (clean worktree), uv publish → https://gitea.phasefinal.com/api/packages/vh/pypi; acceptance uv pip install worldtree-sdk==1.0.0 (vh index as extra-index-url) resolves + imports, version 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via [[tool.uv.index]] name=gitea). Publish cred = the vh write:package PAT the operator had already handed over (in worldtree-sdk/.npmrc _authToken) — Gitea write:package is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). reference_worldtree_demo_key_mint

  • [2026-07-18] soong-lab auto-redeploy APPROVED — QUEUED for next session (deferred, not started) — Vuong approved (via soong-dev thread 01KXT3A6C3908TA4V9THV3AMH7); mechanism = WT-style CI-deploy step (runner SSHes corviduo-dev → compose pull && up -d + health-gate); blocked on a vh-owned runner→corviduo-dev deploy SSH-key secret (reuse WT's demo-deploy key). Operator: "do soong on fresh context." → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md

  • [2026-07-18] nh3-dev /tmp auto-clean enabled — Debian ships /tmp with no tmpfiles age (D /tmp 1777 root root - → never cleans); this high-churn agent box had accreted ~190k stale temp dirs / 25G. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared /tmp/claude-* by name + anything ≤1d). Then /etc/tmpfiles.d/tmp.conf = D /tmp 1777 root root 3d (daily systemd-tmpfiles-clean.timer removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.

  • [2026-07-18] soong-lab containerize cutover COMPLETE + LIVE — systemd→container on corviduo-dev :8443 (image vh/soong-lab:latest v0.3.24), data migrated (Sindra + portraits) + backed up, old service+webhook retired, Homepage tile added, operator functional-confirmed. Deploy /home/infra-ops/soong-lab-deploy/; no proxy (co-located WT, plain-http callback). → persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md

  • [2026-07-18] zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) — resolve_preset(name,voice) → per-voice axes cell (angry/happy/startled_happy + aliases); NOT a global preset (BrF named-angry→fear). Docs on /docs + /v1/dials + repo spec. Pushed main 8f1885b/tag v0.2.1 (after reconciling two-unrelated-git-histories). → persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md

  • [2026-07-18] Fleet Gitea-Actions build recipe + the vh-is-a-USER package-write constraint (reusable for any fleet CI image build / package publish) — runner job image node:20-slim has no docker/git → use container: docker:24.0.7-cli + apk add git nodejs + RAW buildx (not the JS docker/* actions); vh is a user so its packages are OWNER-WRITE-ONLY (claude-bot can't push/publish/set-secrets — CI must auth AS vh); GITEA_ secret-prefix is reserved. → persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md

  • [2026-07-18] Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed. Wyrd: demo Worldtree user-tier key (key_id da7a0bdf, user_id wyrd-dev) minted via docker exec worldtree-worldtree-api-1 /admin/keys (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh write:package PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to vh/npm/. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. reference_worldtree_demo_key_mint

  • [2026-07-18] Axes sweep RESCUED angry; surprised-class dead but startled-happy ships. Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. ANGRY rescued (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0) — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). SURPRISED-class DEAD (max 0.047 across all 84 cells) but startled-happy (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a +0.17–0.20 identity LIFT over the named-surprised route (named hits happy1.0 but at id0.57–0.61, under floor; axes hits happy1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling ~/development/zonos-tools/axes_sweep.py; per-clip JSON was irv-ml1:/tmp/axes_sweep_results.json (ephemeral). Sent dvalin msg 01KXT2ZB8G…. NEXT = operator ear-confirm → bake presets. reference_zonos_tts_stack

  • [2026-07-18] Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline — 4 chars cloned (Emmie/Penny/Natalie/Miranda), host-managed gateway voices, two-regime accurate/expressive policy, happy/sad usable + angry-weak/surprised-dead on named directions, dvalin-synthesized; axes sweep is the NEXT experiment. Studio + sweep tooling at ~/development/zonos-tools/. → persistent-memory.d/2026-07-18-zonos-emotion-canonical.md

  • [2026-07-18] yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy. Fixed a latent misconfig — the host override said "pin worker to A6000" but NVIDIA_VISIBLE_DEVICES was "0" (the 3090); re-pinned worker+api to the A6000 by UUID (GPU-9672f0d5, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (docker compose up -d --build; SPA+Python; max_gap 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job f3ff746dbae9494d). yt-voice-clipper-dev thread 01KXT0T6GYHB. reference_ytvc_autodeploy

  • [2026-07-17] Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub. #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in lofn.chroma (old-code saga-v1 seeding + legacy contamination), and a clean-slate marker test proved current b125 code isolates character-session extraction correctly — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup /opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. reference_corviduo_dev_emergency_ops

  • [2026-07-17] Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets. The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native emotion_directions/ (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). zonos-gateway:0.2.0 (:8890) already wires it: simplest caller path = POST /v1/audio/speech {preset:"…"} — presets neutral/warm/excited/sad/intense/whisper (defined in ~/zonos-gateway/src/zonos_gateway/dials.py), reached via the LiteLLM ext-tts alias (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified 0.625 (+20%, still realtime). Captured the live gateway stack → stacks/zonos-gateway/ (compose+env+README); ⚠️ gateway SOURCE at ~/zonos-gateway on irv-ml1 is NOT in gitea (backup gap, follow-up); stacks/zonos (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper direction; escalation for hard affects = custom directions via scripts/build_emotion_directions.py or emotional-ref cloning speaker_audio_base64). Harnesses in scratchpad (not yet landed). reference_zonos_tts_stack

  • [2026-07-17] Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock. Config captured (14a0004, unpushed); build = cu128 base + uv sync vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md

  • [2026-07-15] arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md

  • [2026-07-15] esh-docker-vm NFS fstab fix = x-systemd.before=docker.service → persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md

175 older entries archived to archival-memory.md.

Tried and abandoned

  • [2026-08-03] ComfyUI --enable-triton-backend on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3. adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to COMFY_CMDLINE_EXTRA, recreated) → triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5") in comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8, failing at node 5 CLIPTextEncode. Triton's fp8 dequant kernel targets fp8e4nv (Hopper/Ada e4m3); sm_86 Ampere (A6000) lacks hardware e4m3 → the JIT compile dies. With triton on it grabs the global --fp8_e4m3fn-text-enc dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged sha256:94afb8ca, sage intact, prod restored). The parked cu130 rebuild won't fix it (e4m3 = hardware format, not CUDA version). DEFERRED to the Ada refresh (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). Mechanics: --enable-triton-backend is a compose environment: var, so toggling it needs docker compose up -d (recreate), NOT docker restart (reuses the baked env, no-ops silently). Full: auto-memory parked_triton_backend_ampere_fp8.

  • [2026-08-03] corviduo-dev shared containerd: a concurrent-pull race fails ONE instance's deploy; DON'T "prune to fix" — the image is in-use by the instance that won the race. b169 personal deploy failed at docker compose pull (Lchown … no such file or directory on the big torch layer → looked like disk pressure / corrupt snapshot). ACTUAL: NOT disk (56G free, inodes 7%). demo + personal + pinned share ONE /var/lib/containerd on corviduo-dev; demo (from main) and personal (from staging tag) extracted b169's shared torch layer simultaneously → personal's hit a partial snapshot mid-race and aborted while demo's completed. The image 6e34a87 was FULLY VALID — demo was RUNNING it healthy. Fix = just re-run the failed deploy (image already materialized; compose pull finds it present). NEAR-MISS: worldtree-dev's suggested "prune unused images/snapshots" would have rmi'd 6e34a87 = the image the running demo depends on → demo outage. Lesson: before any prune/rmi "cleanup," docker ps the running images — an "unused" image may be a co-tenant's live one; and verify the failure's REAL cause (disk? inode? in-use? race?) before applying the suggested remedy. (Pipeline fix, deferred: serialize demo-from-main + personal-from-staging, or a per-image pull lock, to avoid the shared-layer extraction race.)

  • [2026-08-02] docker exec into worldtree containers defaults to ROOT — root writes contaminate the uid-1000 (vh) KB tree. My sudo docker exec … --reindex on personal ran as ROOT (muninn app = uid 1000); its wing git-commit + atomic note-swap left root-owned files in the worldtree-personal_worldtree-kb volume: a root-owned .old-<job> backup dir (blocked the uid-1000 retry's rmtree → Errno 13, because unlink needs write on the DIR and it was root:root 755) AND 60 root-owned loose git objects in .git/objects/. Fix (host-side, corviduo-dev): sudo rm -rf the superseded .old- dir (tar'd aside to /tmp first) + sudo find … -user 0 -exec chown 1000:1000 the objects (ownership-only, git-content-safe; the .git/objects/XX/ dirs were vh-owned so these weren't a hard blocker, but violated "clean tree"). RUNBOOK RULE (worldtree-dev, ADOPTED): any docker exec into worldtree containers that WRITES pipeline state runs -u 1000, never default-root — same genus as the mv footgun (acting without matching the target's constraints; 3rd such slip in one session). GOTCHA that hid the scope: find … -user 0 | head -20 TRUNCATED (the .old- dir alone had 153 files, so the first page was all .old-) → I "verified clean" off a partial list. Never head a scope-defining find; count first (| wc -l). Related blind-spot (muninn-dev): a root-owned job SUBDIR passes every requeue guard (job_row/dispatch/list_jobs render fine) AND /health (contract's os.access(ingestion_root, W_OK) tests only the ROOT dir, so a foreign-owned subdir under pending/ still reports ingestion_root_writable: true) — then the uid-1000 gate can't write into it. "Clean board + green /health + failure at next mutation." muninn-dev added an OWNERSHIP column to the standing post-move check to catch it; two green signals both miss a foreign-owned subdir otherwise.

  • [2026-08-02] mv <job> complete/ → failed/ RENAMED the job to failed because failed/ didn't exist. worldtree-dev's round-2 unblock command (mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/) assumed failed/ existed; on PERSONAL muninn it did NOT (fresh instance — root was active/ complete/ pending/ sources/, no failed/). mv src nonexistent/ renames src→nonexistent, so job1 became the failed dir and job2 nested inside it. Caught on post-move ls (failed/ held job contents, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt failed/ (worldtree:worldtree 755) — NO data loss. Lessons: (1) before mv X into-dir/, verify the dir EXISTS ([ -d dir ]) — an empty ls dir/ 2>/dev/null is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is mv -t <targetdir> <src> (--target-directory): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a trailing slash does NOT protect — mv src failed/ with failed/ missing STILL silently renames to failed (rc=0); "just add the slash" is a false guard. (mkdir -p failed/ first also works, but mv -t inverts the failure from silent-wrong to loud-safe in one flag.) Container sh is dash — no ( in echo strings. SILENT failure mode (muninn-dev carry-forward): a misplaced ingestion-state move doesn't crash anything — list_jobs() stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (job_row→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (job_row found + guards pass), don't trust mv exit codes — and confirm job.dispatch.json survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. DON'T TIDY (round-2 pending): both DCC + P&P jobs currently REST in personal failed/ with manifests reading state: complete until round-2 requeue runs — deliberate + load-bearing (requeue keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. Round-2 sequencing: the requeue is mimir-dev's browser flow (pending their operator's board-vs-API ruling); muninn-dev is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); infra-ops = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. ARC COMPLETE (2026-08-03 ~05:49): both books terminal — DCC mimir-6351554e8e8f 705 concepts + P&P mimir-f3887c9b97b7 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); #381 restart-after-ingest FIRED (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. Delete-sweep precondition NOW MET — the stale DCC rows in main are genuine duplicates of live fiction rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).

  • [2026-08-02] donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED. Folded the onyx-58 bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth donut-onyx58) showed the original single-clip seg000 (16.3s) sounds better — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). Two durable lessons: (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) Emotion steering pulls the output AWAY from the cloned voice fast (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an emotion_*/preset dial is explicitly sent, so bare {input,voice} calls stay pure-clone. seg148 was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth onyx-58 (24h TTL — stash to /mnt/smithy/voice_clones/ if a future middle-ref experiment is wanted).

  • [2026-08-02] Verifying the INDEX is not verifying GROUNDING (#382). A search_library returning wing=fiction hits proves the content is retrievable; it does NOT prove the agent (Mimir) trusts and uses those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-looking answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."

  • [2026-07-30] brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom microsoft/Mage-Flow-{Base,Turbo,Edit} repo IDs. brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF registry API is ground truth — an unauth 401 ≠ exists ({"error":"Invalid username or password"} masks private/gated/nonexistent alike), an authed 404 = phantom, and author=X&search=Y refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory reference_verify_hf_repo_ids_before_pull.

  • [2026-07-30] magpie TTS serving — evaluated, ABANDONED. Pulled magpie_tts_multilingual_357m (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-main container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; magpie-nemo torn down. .nemo KEPT on NFS as brokkr's fine-tuning base. auto-memory project_magpie_tts_eval_rejected.

  • [2026-07-25] Chaining the althing wake-listener arm orphans it. reply && althing-wake-listener & (or spawning althing-wake-listener with & inside a run_in_background task) → the &-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a plain operator turn (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn althing-wake-listener as its OWN run_in_background task, and re-arm ONLY after a real fire (<task-notification> completed rc0). Reclaim an orphan with althing-cli stop-monitor then re-arm.

  • [2026-07-25] Peer green-light ≠ operator consent for a managed-box mutation. Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the operator's yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)

  • [2026-07-18] Fleet Gitea CI foot-guns (3 failed soong-lab builds): the pfi-fleet runner's node:20-slim job image has no docker/git so actions/checkout + docker/* marketplace actions all fail; vh is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); GITEA_-prefixed secret names are reserved/illegal. Fixes in → persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md

  • [2026-07-18] zonos-gateway local clone had NO git remote + a history unrelated to gitea's — "committed to vh/zonos-gateway" was never pushed from that clone; two separate git init lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check git remote -v + git merge-base before assuming a clone is wired.

  • [2026-07-15] docker.service After=remote-fs.target does NOT wait for nofail NFS mounts → persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md

  • [2026-07-15] The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md

  • [2026-07-15] vLLM max-model-len does NOT free GPU VRAM → persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md

  • [2026-07-15] Claude Code statusline .cost.total_cost_usd is per-SESSION → persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md

  • [2026-07-14] MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md

  • [2026-07-14] AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md

  • [2026-07-14] NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md

  • [2026-07-14] NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md

  • [2026-07-14] MTP graft via top-level mtp.* tensor names does NOT survive AutoModelForCausalLM.from_pretrained → persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md

  • [2026-07-14] gitea "test-delivery 204" is NOT proof a webhook works → persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md

  • [2026-07-13] Relaying a peer's diagnosis as fact without confirming it against raw data → persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md

  • [2026-07-13] althing-cli reply <THREAD_id> (thread id, not a MESSAGE id) → "unknown message_id"; and reply to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use post --to <peer>. Bit me several times this session.

  • [2026-07-09] vllm/vllm-openai:latest crashes on Ampere IMPORT — Blackwell-only kernels (oink/aiter, has_device_capability(100)) die during import on the 3090/A6000. Pin v0.23.0 on irv-ml1's Ampere GPUs. (vllm/vllm-omni:v0.18.0 has a different entrypoint — don't use it either.)

  • [2026-07-09] Per-frame CPU SNAC decode is too slow for streaming — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = windowed chunk decode (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).

  • [2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md

  • [2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24) → persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md

  • [2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md

  • [2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md

  • [2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md

  • [2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON → persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md

  • [2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for → persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md

  • [2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session → persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md

  • [2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md

  • [2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.

  • [2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

107 older entries archived to archival-memory.md.

Reference in New Issue View Git Blame Copy Permalink