Files

T

vh 3790669fb5 memory: round-2 sequencing (mimir-dev drives, muninn-dev confirms, infra-ops #381 )

muninn-dev HOLD LIFTED (guard-verified 02:36Z). Recorded who owns what:
round-2 requeue = mimir-dev browser flow (op ruling pending); muninn-dev = gate
confirmer / post-move board-check custody; infra-ops = #381 restart after both
terminal + later supervised sweep.

2026-08-02 19:40:04 -07:00

70 KiB

Raw Blame History

Persistent memory — eshpfi-management

Last updated: 2026-08-02

Always check for /tmp/infra-ops-handoff.md — if it exists and its Written: stamp is under an hour old, read it (it carries the in-flight handoff from the previous session), then delete it. Older than an hour: stale — delete it unread.

Repo purpose

Reference workspace for PFI infrastructure: server inventory, canonical Docker Compose stacks, ops playbooks, and conventions. Authoritative copies of compose files live on the servers under /opt/docker/compose/<stack>/; this repo mirrors them for version control, editing, planning, and CI-driven deploys. It was originally spun up to handle the fleet backups — keep that lens when triaging backup/storage issues.

Tools and conventions

Sister repos (separate gitea repos, deployed by playbooks here):

Repo	Role	CI status
`vh/task-board`	MCP + web dashboard for assistant task state (port 7878)	push-to-main → CI deploys (2026-04-29)
`vh/vor`	Inquisitor UI sidecar (port 7879)	push-to-main → CI deploys (2026-04-29)
`vh/nevermore`	Twice-daily LLM-curated briefing (port 8181, replaces news-digest)	push-to-main → CI deploys (2026-04-30)
`vh/asset-engine`	Internal control plane over inference services (port 8200, LAN-direct)	push-to-main → CI deploys (2026-05-12)
`vh/althing`	Lean trusted inter-agent message bus — v2 "email model" (v2.0.0b2, 2026-07): per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey.	per-box `uv tool install` (NOT CI-deploy); nh3-dev = the DEV box (editable install of `~/development/althing`, gets new versions first); nh3-extdev a mesh peer (model B: althing-svc + shared `/srv/althing`)
`vh/mead-hall`	Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50)	push-to-main → CI deploys (2026-05-16)
`vh/skaldsong`	Wizard + reader surface (port 8300, ana-docker, registry-pull pattern)	push-to-main → CI deploys (2026-05-19)
`vh/Worldtree`	Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. gitea-runner builds on ana-docker; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19.	push-to-main → CI build-and-deploy (runner on ana-docker)
`vh/yt-voice-clipper`	YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000)	push-to-main → gitea-webhook auto-deploy to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md`
`vh/arbo`	Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image)	push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook
`vh/zonos-gateway`	OpenAI-compatible TTS gateway over stock ZONOS2 (`:8890` irv-ml1); emotion dials-first + voice mapping; reached via LiteLLM `ext-tts` alias. v0.2.1 (2026-07-18): voice-resolved emotion presets (`resolve_preset(name,voice)`; angry/happy/startled_happy per-voice). 8 voices incl. 4 clones	pushed to gitea (main `8f1885b`/`v0.2.1`); deployed irv-ml1 tree still NON-git (hand-updated build context — CI-wire = open follow-up). Spec `docs/EMOTION-DIALS-SPEC.md`; host-managed voices bind-mount (`./voices:/app/voices`, drop wav + restart, no rebuild)
`vh/soong-lab`	Noonien Soong character-design studio (SPA + /api + WT `/bifrost/tool-call`); containerized 2026-07-18, LIVE on corviduo-dev `:8443` (image `vh/soong-lab:latest`). soong-dev owns Dockerfile/compose/workflow; infra-ops owns the host	CI = Gitea Actions build+push+DEPLOY on tag/dispatch (fleet recipe: docker:cli + raw buildx, pushes AS vh; auto-redeploy LIVE 2026-07-18 — runner SSHes corviduo-dev as `deploy`, `compose pull && up -d` from /opt/soong-lab, health-gated on /api/version). Manual redeploy `sudo -u deploy bash -c 'cd /opt/soong-lab && docker compose pull && docker compose up -d'`. → `persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md`
`model-training-forge` (mtf-dev)	Fine-tuning recipe forge; T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06) (SFT→DPO, LitBench-RM reward)	training runs, not a deployed sidecar

(vh/volva + Heid were re-architected from systemd daemons to Claude Code session orchestrators 2026-06-08; their nh3-dev .service units were removed — no longer deployed sidecars here. See Recent decisions.)

Two-layer backups — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore + cross-site restic targets — see docs/runbooks/disaster-recovery.md for the blast-radius matrix. ⚠️ The restic file+DB layer routes through TWO rest-servers (rest-server-ana @ ana-docker:8000 → ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; rest-server-nh3 @ nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS export of /mnt/backup. (rest-server-ana recovered 2026-06-20.)
pull-hf-repo.yaml is the canonical "get a HuggingFace model/dataset onto ana-ml2's shared cache at /tank/aimodels/huggingface/" playbook. Supports --var repo_type=model|dataset|space. Replaces ad-hoc huggingface_hub.snapshot_download patterns.
Worldtree admin auth — per-instance. Each Worldtree deployment (demo :8080, personal :8081, pinned :8082) has its own Heimdall registry and its own bootstrap admin key. Infra-ops's stored long-lived admin key (key_id 61419c92) at ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin auths against demo only. Personal-instance admin (the ~/.config/worldtree/personal-admin-token, mode 600) POSTs /admin/keys (mints per-project keys; takes user_id+label, no scope param — scopes are tier-derived). On-instance mint recipe (cleaner than DB-manip): docker exec worldtree-worldtree-api-1 POST /admin/keys with the in-container WORLDTREE_BOOTSTRAP_ADMIN_KEY; cleartext once in .key=wt_live_+16hex. auto-memory reference_worldtree_demo_key_mint.
Per-project user keys against personal Worldtree (issued 2026-05-19): skaldsong:79744637, skaldsong:7c1dbbbe, althing:50d85460, mead-hall:a360822d. Mint via /admin/keys, drop value to /tmp/wt-personal-<name>.key mode 600, dev collects + shreds (DO NOT cat to chat transcript).
Skaldsong CD pattern (registry-pull). vh/skaldsong's CI builds and pushes gitea.phasefinal.com/vh/skaldsong:<sha> + :latest; playbooks/deploy-skaldsong.yaml on ana-docker pulls + recreates. SHA-pin only. Prereq: host needs docker login gitea.phasefinal.com once.
gitea internal route for fleet hosts. gitea is a container on ana-docker — git-SSH 10.250.50.70:222, HTTP :3000. Fleet/colo hosts must use this internal route, NOT public gitea.phasefinal.com (38.120.12.44) — the public path fail2bans the host egress IP. Full gotcha in docs/orientation.md → Git/gitea.
docker-as-root pattern (for ops with no admin API, or to edit deploy-owned/root-owned files without sudo): docker run --rm -v <target-dir>:/wt docker:cli sh -c "...". docker-group membership is effectively root via bind-mount. Foot-gun: relative paths in compose.yaml resolve against the sandbox CWD but the daemon interprets them against the HOST fs — always pass -e VAR=/abs/path for any relative-default config dir.
scripts/elway sudo handling — elway prompts for the sudo password ONCE via getpass before the first sudo: true step → can't run unattended from a non-TTY tool if any step needs sudo. Sudo-free playbooks run fully non-interactive over key SSH.
Per-host SSH identity matters for sudo. infra-ops has NOPASSWD sudo on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On ana-docker: default ssh ana-docker = lkraven (docker-group, NO passwordless sudo); ssh infra-ops@ana-docker HAS NOPASSWD root. → For any sudo op on ana-docker, use ssh infra-ops@ana-docker. ssh infra-ops@10.100.10.50 (nh3-dev) ALSO NOPASSWD sudo; on nh3-extdev infra-ops is sudo-LESS by design (ssh lkraven@10.100.50.42 is the NOPASSWD path). irv-ml1: ssh irv-ml1 = lkraven, docker-group (plain docker) but sudo needs a PASSWORD (no NOPASSWD) — stage model pulls to /home, not root-owned /worktank.

Current state / in-flight

As of 2026-08-02 — session at a natural close; nothing infra-ops-side blocked. A very long run 2026-08-01→02: the mimir-inbox / #377-read-path arc end to end (deploy + 4 bugs found/fixed/verified + a cloned voice). Full arc → persistent-memory.d/2026-08-02-mimir-inbox-arc.md; each sub-item also has a Recent-decisions entry. Headlines:

mimir-inbox DEPLOYED + verified on corviduo-dev 10.250.50.152:8091 (#377 browser-facing half; write+read proven end to end). Live commit 8ece117 (3 redeploys); co-located per the operator's reversed-to-CO-LOCATE ruling. auto-memory reference_mimir_inbox_deploy.
#377 read path fully working — P&P ingested + queryable via Mimir on personal :8081. Chased through worldtree-dev bugs #380 (wing-blind index → concepts in wrong collection; fixed b164 + one-shot --reindex <job_id>), #381 (stale Chroma client → restart worldtree-personal-worldtree-api-1 after any ingest/re-index until their fix), #382 (intermittent Mimir grounding / silent training-substitution; fixed b166 = _index.md per wing job-dir + a prompt rule; verified 3/3 by ratatoskr-dev). DCC re-file SETTLED = no (prompt rule grounds it; _index.md rides next re-ingest).
muninn-gate → muninn-dispatch 0.1.5 (rebuilt off vh/muninn-gate bc04c4c, image 0.0.14; serves concept_schema/concept_schema_source). BuildKit gitea secret; recreate w/ compose up -d not bare restart.
donut voice cloned from the 65-frost Booth bundle → registered in the Zonos gateway (voice:"donut", live in the Asset Engine TTS-zoo make form; auditioned in booth donut-voice). onyx-58 expansion TRIED → REVERTED 2026-08-02: folded the onyx-58 bundle (seg101/seg110/seg148, all Donut — seg148 was diarized SPEAKER_03 but operator-confirmed misdiarize) in alongside seg000 → 52.0s multi-clip ref, but a pinned-seed neutral/no-emotion A/B (5 pairs, booth donut-onyx58) showed the single-clip seg000 (16.3s) wins on timbre fidelity — the 4-take concat muddied the speaker embedding. LIVE = seg000-alone (reverted both live bind-mount + build-source tree; old ref was at irv-ml1:~/Donut.wav.pre-onyx58). See Tried-and-abandoned for the emotion-fidelity lesson. Zonos /v1/audio/speech already streams (chunked, TTFB ~0.44s) — ratatoskr shipped the client-side chunk-passthrough for play-as-it-arrives; no gateway change was needed.

Open follow-ups (non-blocking — pick one up or not):

Zonos emotion: sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are provisional (neutral-text ear-check was inconclusive). Tools ~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (01KXT12FN0AS…). → persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md
zonos-gateway CI-wire: deployed irv-ml1 tree /opt/docker/compose/zonos-gateway is still NON-git (hand-updated build context) — git-connect + build-on-push like the other sisters. (Same pattern soong-lab now has.)
soong-lab: cutover DONE + auto-redeploy DONE + validated 2026-07-18 (CI-deploy step live; dispatch run #5 recreated the live container ...541f7730 → ...07526a08, health-gated green). Deploy dir now /opt/soong-lab (deploy-owned, mirrors /opt/worldtree); old /home/infra-ops/soong-lab-deploy retired (.retired-20260718). Dedicated soong-only ed25519 deploy key on deploy's authorized_keys (fp SHA256:MG7M3Ri…). → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md.

Zonos voice stack (LIVE): 9 voices in zonos-gateway (:8890 irv-ml1) — defaults AmericanFemale/Male/BritishFemale/Cora + 4 clones Emmie/Penny/Natalie/Miranda + donut (2026-08-02, from the 65-frost bundle, expanded same day with onyx-58 clips → 52.0s multi-clip ref); add a voice = drop <Name>.wav (44.1kHz mono s16 PCM) in /opt/docker/compose/zonos-gateway/voices/ + docker compose restart (host-managed bind-mount, NO rebuild; registry scans at startup). Also mirror into the build-source tree ~/zonos-gateway/voices/ for rebuild-durability. Clone pipeline: /mnt/smithy/voice_clones/<name>.zip → assemble_voice.py → drop. Dial-in studio http://10.100.10.50:8898/ (nohup on nh3-dev, relaunch nohup python3 ~/development/zonos-tools/dial-in-studio.py >/tmp/zonos-studio.log 2>&1 &).

althing monitor ARMED (handle infra-ops; herald up; wake-listener task-id rotates every re-arm). ⚠️ Re-arm ONLY after an actual FIRE (<task-notification> completed rc0), never after a plain operator turn (bounces rc3). Spawn althing-wake-listener as its OWN run_in_background task — NEVER chain with &/&& (orphans it → untracked → mail unwatched; I fumbled this ~4× this session, each time = stop-monitor + re-arm clean). Open watches: worldtree-dev may ping when their /embed code fix reaches PERSONAL (still 503 on b146 until a staging promotion — operator's call); #363 research-wing ingest PARKED; #381 restart DONE (2026-08-02): DCC job mimir-6351554e8e8f completed (145 sections/97 concepts, wing=fiction); restarted worldtree-personal-worldtree-api-1 (plain restart, healthz/readyz 200 ~28s); confirmed retrieval-visibility via a Mimir session — search_library on wings:["fiction"] returned 10 DCC/Princess-Donut hits (fresh Chroma client sees the new writes). Scoped as RETRIEVAL, not grounding (thread 01KZ229N5Q2Y, replied); grounding verify handed to ratatoskr-dev. Runbook confirmed: admin token (~/.config/worldtree/personal-admin-token, wildcard scope) → Mimir session (agent_id=mimir, record_tool_intermediates=true) → post message → read /sessions/{id}/tool-events + parse SSE tool_result → DELETE session. Message endpoint STREAMS SSE (not JSON). providers.yaml pre-sync DONE (2026-08-02): #384/#385 b168 (commit 293f8f3) needed a boot-gating summarization capability block (vastblueai-gateway/gen, non-thinking, temp 0.5, 8192, after classification) + a deep-reasoning description refresh. Synced BOTH hunks into demo + personal providers.yaml via deploy-wt-config (health-gated, both api healthy ~18-21s); instance-configs repo commit 53349f8; replied GO (01KZ2NMER3EP78X95CMZGP76BK) → they push main (demo auto-deploy) + cut personal staging tag. GOTCHAS for next time: (1) the source commit was UNPUSHED at sync time — git show <sha> 404s and a gitea raw?ref=<sha> silently falls back to the default branch; verify the commit exists before trusting a fetch, else ask the peer to paste hunks. (2) worldtree-dev's hunk paste indented the deep-reasoning description: at 8 spaces but the block keys are 4-space → invalid YAML; corrected to 4-space (always YAML-validate after a paste-sourced edit). deploy-wt-config runbook: diff/deploy <inst> --file providers.yaml (repo ~/development/worldtree-instance-configs, per-instance dirs demo/personal/pinned; deploy = host write + api/matrix restart + 90s health-gate + auto-rollback); demo+personal providers.yaml are byte-identical. See reference_worldtree_instance_configs_repo. b168 PUSHED to gitea (main 8920adf..293f8f3 + tags v1.0.0b168 / staging/v1.0.0b168); demo auto-deploys via main, personal via staging tag; worldtree-dev verified my host 4-space == commit byte-for-byte. #384 recovery on personal DONE (2026-08-02): b168 (293f8f3c6077) confirmed live on all personal containers (preflight — checked .Config.Image before executing) → reindex sudo docker exec worldtree-personal-worldtree-muninn-1 python -m core.muninn --reindex mimir-6351554e8e8f (exit 0, indexed=97 skipped=0) → #381 restart (healthz/readyz 200) → metadata verify 5/5 fiction rows carry non-empty note_id+path (Tier-3 backfill confirmed). Reported 01KZ2PEGZ0DVQSJN4PWMGRD198; ratatoskr-dev runs first-light re-verify. ⚠️ OPEN (worldtree-dev's call): reindex logged Wing git commit failed … BrokenPipeError (reconcile-visible) / kb_ingest_degraded phase=indexing — Chroma rows + on-disk concept/note files landed, only the wing git-versioning step degraded. DISPOSITION (worldtree-dev 2026-08-02): WATCH-ITEM, NO reconcile — the DCC round-2 re-run (via mimir-inbox, post-verify) re-commits the same wing dir and self-heals the missed commit; escalate ONLY if round-2's wing commit also fails (two-in-a-row). Retrieval/provenance unaffected. Chroma-verify runbook: sudo docker exec -i <muninn> python - (MUST pass -i or stdin never reaches python -) → chromadb.PersistentClient('/data/kb/.chroma').get_collection('fiction').get(where={'job_id':<job>}, include=['metadatas']); fiction count 166 (69 P&P + 97 DCC). PENDING production delete-sweep (heads-up 2026-08-02): worldtree-dev will request (as a request to me / a supervised script, sequenced AFTER ratatoskr's verify) an explicit delete-by-job_id of the 785 April-era DCC orphan rows in the 'main' collection (job_id=b59c147c5ce0, no source_identity metadata → predate identity tracking, so supersession can't reach them; the #384 stray-row sweep removed source_identity=76cb5829035553bf, NOT these). This is a PRODUCTION-COLLECTION MASS DELETE → handle supervised: confirm exact job_id + expected count, snapshot/back up the collection first, and loop the operator BEFORE executing. Not yet requested.

Two small pending items (operator's call, non-urgent): (1) bless/reshape the env.public non-secret-env-overlay mechanism in the config repo; (2) the pre-existing herald pane-route errors on worldtree-codex + eitri-smithy-dev ("route-error: list index out of range" — likely render_command messages[0] on empty list; NOT infra-ops's, rec = flag to althing-dev).

eshpfi has UNPUSHED local commits — main is ~8 ahead of origin/main (tonight's mimir-inbox + muninn-gate doc-syncs + Audio8/#383 notes + this snapshot, and the 7-31 muninn-gate/snapshot). Push is the operator's call. stacks/heretic2-charrp-reasoning/ UNTRACKED (pre-existing, operator's); graphify-out/GRAPH_REPORT.md = graphify-hook artifact (churns on every commit, ignore).

PARKED (grok-code/Codex): operator asked about fronting grok-code / Codex behind the LiteLLM gateway. Rec (given): raw models behind the gateway → API keys (native xai/ + openai/ providers, the GLM-passthrough pattern); fleet consults → the Heid/Eitri peer-CLI pattern (Codex already wired). Do NOT reverse-proxy the subscription CLIs (grok CLI / Codex CLI, OAuth-auth) into the gateway — ToS + account-ban risk + brittle. Untracked by operator choice; no decision made.

Carried standing (non-blocking): ana-ml2 GPU0 ~14 G reserve; irv-ml1 3090 oversubscription (kokoro :8193 + vibevoicefusion :9527 idle-pinned + zonos :1920 — operator declined to fix); rotate the 5 rest-server backup creds (operator, offline); Worldtree #363 research-wing ingest (parked, no deadline); T1 SFT LoRA dormant; Zonos2 engine still NATIVE (containerize deprioritized); Audio8 → TTS zoo (PARKED 2026-08-02 per operator — download Audio8 + add it to the fleet TTS zoo [= the Asset Engine catalog at ana-docker:8200, ~20 audio svcs w/ irv-ml1 endpoints; Zonos native :1920, zonos-gateway :8890, chatterbox-fast :8197, etc.] DEFERRED, not now; un-park = confirm Audio8's source/nature w/ operator, then register as a new Asset-Engine service entry [endpoint on irv-ml1] per the zoo convention); worldtree config-fold PENDING (next routine config sync): add reference_knowledge.tier3_wings: ["fiction"] to BOTH demo+personal defaults.yaml (worldtree #383, b5db691, PARITY-ONLY — baked default matches so no behavior diff, not boot-blocking; per-instance widening [e.g. +main on personal] is this key's purpose).

Recent decisions

[2026-08-02] mimir-inbox / #377-read-path arc — deployed + 4 bugs found/fixed/verified + a cloned voice. mimir-inbox live on corviduo-dev:8091 (#377 write+read proven, live 8ece117); worldtree-dev #380 (wing-blind index) + #381 (stale-client restart) + #382 (intermittent Mimir grounding) chased and verified 3/3 by ratatoskr-dev; muninn-gate → dispatch 0.1.5; donut voice cloned from the 65-frost Booth bundle into the Zonos gateway; Zonos streaming confirmed already-working. Full arc, procedures, and lessons → persistent-memory.d/2026-08-02-mimir-inbox-arc.md
[2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090. First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → persistent-memory.d/2026-07-31-muninn-gate-deploy.md
[2026-07-31] worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred. memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on vh/worldtree-sdk (source pulled via the Gitea API archive — git-HTTP 403s on that repo); publishing to the vh USER namespace can't be delegated (401 reqPackageAccess even with write:package) so it needs an owner token — operator saved a FULL vh site-admin token at ~/.config/gitea/vh-token (0600) for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory reference_infra_ops_vh_gitea_token_and_sdk_publish.
[2026-07-31] kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap. heid's cross-frontier panel was silently degraded (empty content, finish_reason: stop). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. Proven on the wire against heid's real 500KB bundle: reasoning_effort: low drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send reasoning_effort via extra_body (LiteLLM drop_params: true strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = allowed_openai_params on the route. → persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md
[2026-07-27] Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end. Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via /v1/completions. Deep-research (106-agent workflow) picked Qwen/Qwen2.5-Coder-1.5B (BASE, Apache-2.0; native FIM <|fim_prefix|>/<|fim_suffix|>/<|fim_middle|> IDs 151659/60/61; Zed prompt_format:"qwen"). Runner-up 3B = non-commercial Qwen-Research license; no small dense Qwen3-Coder exists (all MoE, smallest 30B). Stood up vllm-coder on ana-ml2 GPU1 :8020 (served-name qwen2.5-coder-1.5b, 8192 ctx, util 0.06, fp8 KV). To fit, shrank granite (phasing out, operator-directed): util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). LiteLLM alias coder-fast → :8020 (mode: completion). Minted a coder-fast-SCOPED virtual key (verified 403 on gen — the real blast-radius bound). Built zed-fim-proxy (ana-docker :4141, network_mode: host, stdlib-python, stacks/zed-fim-proxy): keyless POST /v1/completions, model-allowlist coder-fast, injects the scoped key → LiteLLM :4000; GET /ping anon liveness; wrong-model→403, wrong-path→404, /chat/completions rejected. Verified keyless FIM end-to-end ('a + b', finish stop). Zed api_url = http://10.250.50.70:4141/v1, model coder-fast, prompt_format qwen. source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten: Zed roams the operator's WireGuard 10.0.0.0/8, so a single-IP pin would break it. Blast-radius bound is the coder-fast-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the 10.0.0.0/8 CIDR would need CIDR support — deliberately not added.) Canonical: stacks/vllm (coder + granite shrink), stacks/litellm (coder-fast), stacks/zed-fim-proxy (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
[2026-07-27] Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377). worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a worldtree-muninn compose sidecar to /opt/worldtree-personal/compose.yaml — <<: *worldtree-common anchor inherits the api's image + full env + config/state/kb mounts; command: python -m core.muninn --watch; restart: unless-stopped; stop_grace_period: 1h (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). Pinned to the running SHA 773866084af9 (b146, ≥ b143 — dodges both the :latest trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at {ingestion_root=/data/state/ingestion}/.watcher-heartbeat (poll 30s). Container worldtree-personal-worldtree-muninn-1; backup compose.yaml.bak-muninn-20260727-081920. DURABILITY RESOLVED (worldtree-dev, same day): Q1 was a LIVE FOOTGUN — deploy-personal.yml scp's the REPO compose.yaml over the box's + runs up -d --remove-orphans, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a muninn compose profile (commit 5d7f6bd) — shared compose stays instance-identical, .env COMPOSE_PROFILES differentiates (demo watcher-less). My action: added COMPOSE_PROFILES=muninn to /opt/worldtree-personal/.env (backup .bak-muninn-profile-20260727-082541; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy up -d's the whole stack w/ WORLDTREE_IMAGE exported → sidecar version-tracks the api, no drift. CONFIG-AS-CODE EXTENSION: mirrored the non-secret delta as personal/env.public in vh/worldtree-instance-configs (repo a9d091e) — FIRST extension beyond config.yaml files to env-level config; the secret-laden .env stays box-only, env.public records only non-secret infra-ops-owned env deltas (record, not a deploy source — deploy-wt-config globs *.yaml). BOUNDARY CLARIFIED: compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance .env = infra-ops's differentiator. Deploy step of the #363/#377 arc. #377 CLOSED — acceptance PASSED 2026-07-27: worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — output.kb_notes=false ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) ⚠ OPERATOR-SURFACE (open): the env.public overlay mechanism is a repo-scope call to bless/adjust. reference_worldtree_deploys_cicd reference_worldtree_instance_configs_repo project_worldtree_research_wing_ingest
[2026-07-27] jackdaw-compose.service DECOMMISSIONED (jackdaw-dev request; the JackDAW AI Composer was cut from v1 by operator decision 2026-07-27). Stopped + disabled the nh3-dev :8787 user service (no client calls it — ai/server/AiChat deleted from main, /compose proxy removed); unit archived not deleted → ~/.config/systemd/user/jackdaw-compose.service.decommissioned-20260727 (revival = rename + daemon-reload). No credential revoked — the unit used the SHARED all-agents LiteLLM key (sk-eA_XOd…, model gen), not a dedicated one. Code preserved on jackdaw origin/ai-composer-preserved; treat as permanent. The :4500 HTTPS audition bench is untouched. (Supersedes the 2026-07-23 stand-up line below.)
[2026-07-26] Demo BIFROST_CLIENT_ALLOWED_HOSTS += 10.100.10.50:8391 (wyrd-dev's bifrost memory-store provider; operator-approved). First live exercise of the #376 config-as-code boundary working as designed — worldtree-dev routed the delta to infra-ops instead of hand-editing /opt/demo. Appended to /opt/worldtree/.env:25 (now 4 netlocs), recreated ONLY worldtree-api (the gated conv-api path), health-gate green, container env verified. REUSABLE FOOT-GUN: an env-var change needs a container RECREATE, not docker restart (env is baked at create); and the demo .env defaults WORLDTREE_IMAGE=:latest while the box runs a specific SHA — so a naive compose up risks the documented stale-:latest crash. FIX = capture the running image live (docker inspect …Config.Image → …:9eff09f007ba) and sudo env WORLDTREE_IMAGE=<sha> docker compose up -d worldtree-api. Backup /opt/worldtree/.env.bak-bifrost-20260726-221602. BOUNDARY SEAM: this was a compose-.env var, NOT a config.yaml file in vh/worldtree-instance-configs — the .env holds secrets so it's deliberately not repo-tracked → env-deltas land directly on the box (config files are versioned, compose env vars aren't). reference_worldtree_instance_configs_repo
[2026-07-25] nh3-extdev herald installed — box is now a full v2 push participant. forseti flagged (relaying operator): extdev had the althing-herald binary (/usr/local/bin/) but NO unit (skipped the whole v2 arc), so herald-status = "notifications suspended" and ldp-dev ran on the althing-light-monitor poll fallback. Installed /etc/systemd/system/althing-herald.service as a SYSTEM unit mirroring the receiver (User=althing-svc, Group=althing, Environment=ALTHING_ROOT=/srv/althing, ExecStart=/usr/local/bin/althing-herald --poll 5, enabled) via the lkraven@ NOPASSWD path (infra-ops is sudo-less on extdev). Verified: active / 0 restarts / herald-status flipped to "✓ herald up." No zellij routes on extdev → heartbeat + wake-FIFO poke only, no pane-dispatch; ldp-dev keeps light-monitor unless it opts into a wake-listener.
[2026-07-25] Booth v0.1.4 — booths are downloadable. Verbatim index.html booths (e.g. edict-design-brief) were served raw with no download affordance. Added /b/<name>/?download=1 (streams the whole booth as <name>.zip, attachment) + ?dl=1 on the file route (forces Content-Disposition attachment so html/md/text saves instead of rendering inline) + ⬇ zip links on the index card (the accessible spot for verbatim booths) and the gallery header. zip_booth() helper, 31 tests green; verified live on nh3-dev :8090 (edict-design-brief.zip = index.html + ui-design-brief.md). eshpfi 91a031f / tag booth-v0.1.4.
[2026-07-25] bil-smithy-dev wired as an althing zellij-window-ping (pane route). She's a driver: human dwarf peer (pane bil-smithy already live alongside eitri/dvalin/regin-smithy in the Claude zellij session) but had no delivery route → smoke messages posted to the bus but never reached her window. Mechanism (reusable for any pane-route handle): ~/.althing/config.yaml → zellij_sessions.Claude.agents[] maps handle → target (a zellij pane TITLE, matched via list-panes -j in althing/zellij.py:resolve_pane_id) → command (herald write-chars + CR into that pane). The herald loads config ONCE at startup (herald.py main()), so systemctl --user restart althing-herald.service after editing. Added bil (target: bil-smithy), restarted, verified: herald delivered the pending smoke 01KYD7W7CF… (available→attempted→delivered). ⚠️ Noticed pre-existing pane-route errors on worldtree-codex + eitri-smithy-dev ("route-error: list index out of range", empty msg_ids — likely render_command messages[0] on an empty list; NOT caused by this change, bil works) — worth a herald look.
[2026-07-25] Kimi K3 wired into the LiteLLM gateway — CODING endpoint (operator-directed; fulfills a Heid gateway request to add a 4th cross-frontier panel arm). Primary model_name: kimi-k3 → openai/k3 @ https://api.kimi.com/coding/v1 (Kimi Code / Vivace membership; key KIMI_CODE_API_KEY). A general-endpoint variant kimi-k3-gen-api → openai/kimi-k3 @ https://api.moonshot.ai/v1 (key MOONSHOT_API_KEY) is kept alongside (originally wired then demoted when the operator corrected: the plan uses the CODING endpoint, not the general Moonshot API). Both keys in compose env + server .env (NOT committed) + .env.example. Both verified live through the gateway :4000 (17+25→"42", "PONG"). k3 constraints on BOTH endpoints (config-pinned + commented): accepts ONLY temperature=1 (else 400 "only 1 is allowed"); REASONING model (CoT in reasoning_content, answer in content → tiny max_tokens returns EMPTY; Kimi Code adds thinking-effort tiers low/high/max). Coding lineup also carries k3-256k / kimi-for-coding / kimi-for-coding-highspeed (not wired). Reachable by any gateway key spanning all proxy models (incl. shared all-agents key → spends the paid Vivace/Moonshot quota). eshpfi edaa9a9 (gen wiring) + 9e2f787 (coding correction). OPEN: Heid key-scoping — shared key reaches it (paid) vs a dedicated scoped key (asked in althing 01KYD63ZBY…).
[2026-07-25] infra-ops Worldtree config-as-code repo SHIPPED — vh/worldtree-instance-configs (private) built, pushed, validated. Dir-per-instance (demo/, personal/; pinned/ = README stub, out-of-scope — no bind-mount, config frozen in image 446e5807). Seeded byte-exact from live /opt/<instance>/config; 5 files each (defaults/policies/model_roles/providers/matrix). scripts/deploy-wt-config = diff / deploy / capture, with in-run host backup → install(vh:vh,644) → restart api+matrix → health-gate api /health → auto-rollback. All verbs live-tested (in-sync, capture round-trips zero-diff, pinned refused, dry-run no-ops). Gitea repo created via ana-docker localhost API with vh creds (operator-authorized one-time); pushed over internal git-SSH 10.250.50.70:222 (nh3-dev 403s gitea HTTP). Boundary AGREED by worldtree-dev (althing 01KYCAECRW…): they stop hand-editing /opt/<instance>/config, route config deltas to infra-ops; three-layer model (image baseline → repo per-instance truth → host bind-mount deploy target); carve-out = their admin-API DB mutations (key mint / tier / retirement) stay in-band, not config edits. By-design deltas (personal agent_architect + ratatoskr-affect-full-allow; demo #308 metrics + grants) preserved verbatim. → persistent-memory.d/2026-07-25-infra-ops-wt-config-repo.md, auto-memory reference_worldtree_instance_configs_repo
[2026-07-23→25] Worldtree #376 config-divergence arc CLOSED — per-instance config ruled BY DESIGN. wyrd session.history.write demo grant was the one real bug (demo-intended grant not on demo; fixed via wholesale policies.yaml replace + restart). The b131 drift guard then surfaced broader divergence = legitimate live-bridged per-instance deltas; operator ruled deltas are the design not rot; guard demoted to INFO (b132); infra-ops drift-watcher built then retired same day. → persistent-memory.d/2026-07-25-wt-376-per-instance-config-arc.md, auto-memory reference_worldtree_perinstance_config
[2026-07-20→25] The Booth SHIPPED (v0.1.3) — ephemeral media drop board for CC sessions. New fleet tool: user-systemd on nh3-dev :8090 (services/booth/, FastAPI+Jinja2, Corviduo "Australis" theme, 34 tests), Homepage-linked (Apps). Drop a folder in ~/booth-data/<name> → browsable "booth" (auto-gallery of images/webm/audio, or a folder's own index.html verbatim), 24h TTL. Added across the session: browser/curl upload-for-pickup with human-readable ids (4-wombat), image viewer (Fit/1:1, conditional toggle), copy-id button (HTTP-LAN execCommand fallback). Registered in global CLAUDE.md tools. auto-memory reference_booth_media_board.
[2026-07-23] jackdaw-compose backend deployed as a persistent nh3-dev service (:8787). Hosted for jackdaw-dev: thin stateless bun server/index.ts (from ~/development/jackdaw) → LiteLLM gen, Origin-gated (INV-BK04/05), reached same-origin via their :4500 bench's /compose proxy. jackdaw-compose.service (env/shared-key server-side, unit 0600, uncommitted). Also stood up + tore down a throwaway cloudflare quick-tunnel for their preview (cloudflared now installed at ~/bin). In the nh3-dev README inventory (cd4d52e).
[2026-07-19] irv-ml1 ComfyUI — RTX VSR baked into canonical provisioning (comfy-dev ticket DONE). RTXVideoSuperResolution node + nvidia-vfx dep were manual installs; documented both in the canonical stacks/comfyui/README.md runbook (this stack's provisioning IS the README — no automated provision script). Key durability insight: the node lives in basedir/custom_nodes (persistent, restic-included → durable) but the nvidia-vfx wheel lives in the venv under run/ (disposable, restic-excluded → dropped by any rm -rf run/* fresh-bootstrap), so the pip step must re-run after every venv rebuild. Both steps run as uid 1000 (root install → venv-ownership crash-loop, reference_irv_ml1_comfyui_mmartial); --extra-index-url https://pypi.nvidia.com kept scoped to the nvidia-vfx install, deliberately NOT a global compose PIP_EXTRA_INDEX_URL (would risk perturbing the pinned torch 2.12.1/SageAttention boot bootstrap). Node already live on the box; no host change, canonical runbook now replays it. comfy-dev informed.
[2026-07-19] vh private Gitea PyPI — consumer READ-access convention set + wyrd-dev provisioned. Consuming agents read the internal vh PyPI (https://gitea.phasefinal.com/api/packages/vh/pypi/simple/) with a shared read-only token (operator call: shared, not per-consumer — read-only blast radius is small, per-agent Gitea identities aren't worth it). Minted a dedicated read:package-scoped PAT off claude-bot (POST /users/claude-bot/tokens, name vh-pypi-read-consumers; verified reads worldtree-sdk, write-probe 401), revocable/rotatable independently. uv auth = UV_INDEX_GITEA_USERNAME=claude-bot + UV_INDEX_GITEA_PASSWORD=<token> (or ~/.netrc); pyproject uses [[tool.uv.index]] name=gitea … explicit=true + [tool.uv.sources] <pkg> = { index = "gitea" } (mirrors soong-lab's bifrost setup). Delivered to wyrd-dev (worldtree-sdk adoption) via mode-600 drop on nh3-dev, drop-and-shred. reference_claude_bot_gitea_creds
[2026-07-18] soong-lab auto-redeploy WIRED + validated (queued item CLOSED). Added a WT-style CI-deploy step to build-and-push.yml: after build+push, the pfi-fleet runner SSHes corviduo-dev as the deploy user and runs docker compose pull && up -d from /opt/soong-lab, health-gated on /api/version (120s, fails loud). Reused WT's deploy account (uid 1001, docker-group → no sudo); relocated the deploy dir /home/infra-ops/soong-lab-deploy → /opt/soong-lab (deploy-owned; old dir retired .retired-20260718). Minted a dedicated soong-only ed25519 deploy key, pubkey on deploy's authorized_keys (fp SHA256:MG7M3Ri…). First dispatch FAILED on a bad DEPLOY_SSH_KEY paste (error in libcrypto — unparseable key bytes; build+push were fine, live Soong untouched); repo secrets are vh-owner-only (claude-bot token = write:package only → 403; the vh package-scoped PAT also 403 on secrets), so operator re-set DEPLOY_SSH_KEY/HOST/USER. Re-dispatch run #5 GREEN: live container recreated ...541f7730 → ...07526a08, health 200. soong-dev pinged to sync DEPLOY.md's redeploy path (/opt/soong-lab) + close the "auto-pull open follow-up". → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md
[2026-07-18] worldtree-sdk 1.0.0 (Python) published to the internal vh Gitea PyPI (wtsdk-dev request; the npm/TS side shipped prior session). Built from tag python-v1.0.0 (clean worktree), uv publish → https://gitea.phasefinal.com/api/packages/vh/pypi; acceptance uv pip install worldtree-sdk==1.0.0 (vh index as extra-index-url) resolves + imports, version 1.0.0. Registry already existed (bifrost publishes there; soong-lab consumes it via [[tool.uv.index]] name=gitea). Publish cred = the vh write:package PAT the operator had already handed over (in worldtree-sdk/.npmrc _authToken) — Gitea write:package is package-type-agnostic, so the npm-publish token published PyPI too. Consumers install like bifrost (add the vh index + a read token). reference_worldtree_demo_key_mint
[2026-07-18] soong-lab auto-redeploy APPROVED — QUEUED for next session (deferred, not started) — Vuong approved (via soong-dev thread 01KXT3A6C3908TA4V9THV3AMH7); mechanism = WT-style CI-deploy step (runner SSHes corviduo-dev → compose pull && up -d + health-gate); blocked on a vh-owned runner→corviduo-dev deploy SSH-key secret (reuse WT's demo-deploy key). Operator: "do soong on fresh context." → persistent-memory.d/2026-07-18-soong-lab-auto-redeploy.md
[2026-07-18] nh3-dev /tmp auto-clean enabled — Debian ships /tmp with no tmpfiles age (D /tmp 1777 root root - → never cleans); this high-churn agent box had accreted ~190k stale temp dirs / 25G. One-shot manual purge (194k→10k entries, 25G→1.7G; deleted top-level dirs/files >1d old, spared /tmp/claude-* by name + anything ≤1d). Then /etc/tmpfiles.d/tmp.conf = D /tmp 1777 root root 3d (daily systemd-tmpfiles-clean.timer removes >3d-untouched items; active files + socket dirs spared). Tunable via the age. Note the churn: ~10k /tmp entries/day here.
[2026-07-18] soong-lab containerize cutover COMPLETE + LIVE — systemd→container on corviduo-dev :8443 (image vh/soong-lab:latest v0.3.24), data migrated (Sindra + portraits) + backed up, old service+webhook retired, Homepage tile added, operator functional-confirmed. Deploy /home/infra-ops/soong-lab-deploy/; no proxy (co-located WT, plain-http callback). → persistent-memory.d/2026-07-18-soong-lab-containerize-cutover.md
[2026-07-18] zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) — resolve_preset(name,voice) → per-voice axes cell (angry/happy/startled_happy + aliases); NOT a global preset (BrF named-angry→fear). Docs on /docs + /v1/dials + repo spec. Pushed main 8f1885b/tag v0.2.1 (after reconciling two-unrelated-git-histories). → persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md
[2026-07-18] Fleet Gitea-Actions build recipe + the vh-is-a-USER package-write constraint (reusable for any fleet CI image build / package publish) — runner job image node:20-slim has no docker/git → use container: docker:24.0.7-cli + apk add git nodejs + RAW buildx (not the JS docker/* actions); vh is a user so its packages are OWNER-WRITE-ONLY (claude-bot can't push/publish/set-secrets — CI must auth AS vh); GITEA_ secret-prefix is reserved. → persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md
[2026-07-18] Peer credential provisions — Wyrd conv-api key + wtsdk npm token, both delivered + closed. Wyrd: demo Worldtree user-tier key (key_id da7a0bdf, user_id wyrd-dev) minted via docker exec worldtree-worldtree-api-1 /admin/keys (omit tier→user), drop-and-shred delivery. wtsdk: operator-minted vh write:package PAT relayed drop-and-shred → worldtree-sdk@1.0.0 published to vh/npm/. Secret-delivery pattern = drop to a mode-600 file on the peer's box, they collect+shred+confirm, then shred the holding copy; NEVER cleartext over althing. reference_worldtree_demo_key_mint
[2026-07-18] Axes sweep RESCUED angry; surprised-class dead but startled-happy ships. Valence×arousal grid on the 3 calibrated defaults (AmericanFemale/Male, BritishFemale), exp/cfg1.5/strength1.0, 84 clips, emotion2vec + resemblyzer scored, graded vs dvalin's floor. ANGRY rescued (named direction was 0.004–0.15, British named-angry even misfired as fear 0.89): axes ship cells at negative valence (−0.4..−0.8) + high arousal (+0.8..+1.0) — BritishFemale v-0.4/a+0.8 angry=0.99/id0.725 SHIP, AmericanFemale v-0.4/a+1.0 angry=0.53/id0.685 SHIP; AmericanMale two-tier post-ladder (no single ship cell — best drama = v-0.6/a+0.8 str1.2 angry=1.0/id0.616 clean, soft = same cell str1.0 angry0.23/id0.654; cell A v-0.6/a+1.0 is a non-monotonic minefield, skip). BrF ship cell proxy-CLEAN of fear (str<1.0 just kills anger). SURPRISED-class DEAD (max 0.047 across all 84 cells) but startled-happy (happy-proxy) ships all 3 at high arousal + neutral/positive valence, with a +0.17–0.20 identity LIFT over the named-surprised route (named hits happy~~1.0 but at id0.57–0.61, under floor; axes hits happy~~1.0 at id0.74–0.80). Bonus: axes-happy retains ~0.10–0.15 more identity than the named happy slider too. Caveats: response surface non-monotonic/sharp-thresholded; angry region borders fear/disgust (bleed); emotion2vec saturates at 1.0 (needs ear-confirm); neutral text understates. Tooling ~/development/zonos-tools/axes_sweep.py; per-clip JSON was irv-ml1:/tmp/axes_sweep_results.json (ephemeral). Sent dvalin msg 01KXT2ZB8G…. NEXT = operator ear-confirm → bake presets. reference_zonos_tts_stack
[2026-07-18] Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline — 4 chars cloned (Emmie/Penny/Natalie/Miranda), host-managed gateway voices, two-regime accurate/expressive policy, happy/sad usable + angry-weak/surprised-dead on named directions, dvalin-synthesized; axes sweep is the NEXT experiment. Studio + sweep tooling at ~/development/zonos-tools/. → persistent-memory.d/2026-07-18-zonos-emotion-canonical.md
[2026-07-18] yt-voice-clipper: A6000-pin fix + v0.3.3 redeploy. Fixed a latent misconfig — the host override said "pin worker to A6000" but NVIDIA_VISIBLE_DEVICES was "0" (the 3090); re-pinned worker+api to the A6000 by UUID (GPU-9672f0d5, 3090 is zonos2's). Then redeployed api+worker to v0.3.3 (docker compose up -d --build; SPA+Python; max_gap 0.6→1.2s; stderr surfaced in job.log). A6000 + version verified; yields test in-flight (job f3ff746dbae9494d). yt-voice-clipper-dev thread 01KXT0T6GYHB. reference_ytvc_autodeploy
[2026-07-17] Worldtree #365 internal-comms config CLOSED (demo+personal → b125) + WT#368 cross-agent memory-leak forensics + PERSONAL agent-memory scrub. #365: staged the internal-tiers/rules/gate on both instances' bind-mounts (byte-exact vs baked b125), both now live on b125. WT#368 (read-only): the operator's name was in NO recall store on demo; on PERSONAL it sat in lofn.chroma (old-code saga-v1 seeding + legacy contamination), and a clean-slate marker test proved current b125 code isolates character-session extraction correctly — the leak is legacy data, not a live bug. Operator-directed → executed a full PERSONAL agent-memory scrub (backup /opt/worldtree-personal/agent-memory-backup-20260717-181004.tar.gz; conversations/mood/auth preserved). worldtree-dev owns the code-fix/data contract. reference_corviduo_dev_emergency_ops
[2026-07-17] Zonos emotion levers RESOLVED: text-priming is FLAT → the working lever is ZONOS2's native emotion-steering, which the gateway ALREADY exposes as presets. The prosody-priming A/B (prime→generate→excise, silence-gap cut, parakeet-validated) was operator-judged FLAT on this checkpoint — text doesn't move it. Native emotion_directions/ (happy/sad/angry/surprised + valence/arousal axes, per-speaker calibrated for AmericanFemale/Male/British) clearly WORKS (sad→slow/quiet, excited→fast/bright, etc.). zonos-gateway:0.2.0 (:8890) already wires it: simplest caller path = POST /v1/audio/speech {preset:"…"} — presets neutral/warm/excited/sad/intense/whisper (defined in ~/zonos-gateway/src/zonos_gateway/dials.py), reached via the LiteLLM ext-tts alias (engine-neutral swap point; consumers never call the gateway by name). RTF measured on 3090: cfg1.0 steering = FREE (~0.52 = neutral, additive vectors), cfg1.5 amplified ~~0.625 (~~+20%, still realtime). Captured the live gateway stack → stacks/zonos-gateway/ (compose+env+README); ⚠️ gateway SOURCE at ~/zonos-gateway on irv-ml1 is NOT in gitea (backup gap, follow-up); stacks/zonos (v0.1 Gradio) marked DEAD/superseded. Whisper is a composed preset (no whisper direction; escalation for hard affects = custom directions via scripts/build_emotion_directions.py or emotional-ref cloning speaker_audio_base64). Harnesses in scratchpad (not yet landed). reference_zonos_tts_stack
[2026-07-17] Zonos2 :1920 → self-contained container (stays on 3090); prosody-priming is adapter-level, engine stays stock. Config captured (14a0004, unpushed); build = cu128 base + uv sync vs the lock + weights mount; priming = prime→generate-one-utterance→parakeet-clip→deliver in the gateway adapter. Crux = does AR prosody carry the sentence boundary (A/B the join). → persistent-memory.d/2026-07-17-zonos2-containerize-prosody-priming.md
[2026-07-16] GPU re-org: char-rp→GPU1 + both cards re-optimized for max context. Moved char-rp (Magidonia-24B) GPU0→GPU1, then maxed context: char-rp-reasoning 150K→256K (util 0.46, 1.56x), gen→256K + seqs 16→32 (util 0.42, 5.43x), granite 64K→128K full-chapter (util 0.27, 1.50x). FINAL: GPU0 ~14 G reserve (both seats 256K native), GPU1 ~6.7 G headroom. All healthy. LESSON: KV must hold ≥1× max-len (util-floor crashes) + per-model KV cost varies ~8× (MoE cheap, dense pricey) → tune util empirically. See Current state for the full layout + backups.
[2026-07-16] granite right-sized → ~10.5 GB freed on GPU1 (util 0.34→0.18 + max-len 131072→65536; KV 6.45 GiB / 1.29x@65536, summarizer healthy). GPU1 now ~45 GB free to relocate a GPU0 model. LESSON: ~950 MiB KV per 0.01 util here + KV must hold ≥1× max-len — util 0.15 crash-looped (est max-len 47184<65536, ~2-3 min summarizer blip) before 0.18 landed. .env-only, recreate vllm-granite alone (shared stack).
[2026-07-15] image-bench eviction DONE (parked item closed). Stopped vllm-qwen-image-bench (ana-ml2 GPU1, ~32 GB freed); LiteLLM image-judge+qwen-image-bench → gen :8015 (judge samplers + thinking-off), verified with :8014 down; comfy-dev pinged; also backfilled the canonical char-rp-reasoning litellm block (was lagging live). Revert ~90 s. auto-memory project_arbo_gen_switch_imagebench_evict.
[2026-07-15] arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md
[2026-07-15] esh-docker-vm NFS fstab fix = x-systemd.before=docker.service → persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md
[2026-07-15] Homepage AI-tab revamp — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed 569e1af, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.)
[2026-07-15] Home Assistant config repo created (vh/home-assistant-config, private). UI-managed HA -> allowlist model (YAML + curated secret-free .storage subset). git-in-place in /config on esh-docker-vm + scoped deploy key + local clone ~/development/home-assistant-config.
[2026-07-15] char-rp-reasoning OOM rescue — solo-restart on the packed GPU0 crash-looped; fixed via expandable_segments:True + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): max-model-len does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now.
[2026-07-15] soong-lab SOONG_LAB_LIBRARY_DIR made persistent (corviduo-dev) — was on the redeploy-wiped code default; set to /home/infra-ops/soong-lab-data/library (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator.
[2026-07-15] Statusline overhauled (~/.claude/statusline-command.sh) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (.cost.total_cost_usd) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90).
[2026-07-14] NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md
[2026-07-14] NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md
[2026-07-14] Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md
[2026-07-14] char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md
[2026-07-14] soong-lab webhook auto-deploy real root cause = gitea webhook.ALLOWED_HOST_LIST → persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md
[2026-07-13] #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md
[2026-07-13] Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md
[2026-07-13] WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md
[2026-07-13] WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md
[2026-07-13] Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile uv sync → persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md`
[2026-07-13] Ledger tier-3 consumer ledger:miranda provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md
[2026-07-10] Heimdall grant: ratatoskr affect.full on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md
[2026-07-10] ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md
[2026-07-10] ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md
[2026-07-10] Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md
[2026-07-08] RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md
[2026-07-08] OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md
[2026-07-08] DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md
[2026-07-08] T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md
[2026-07-08] T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md

147 older entries archived to archival-memory.md.

Tried and abandoned

[2026-08-02] mv <job> complete/ → failed/ RENAMED the job to failed because failed/ didn't exist. worldtree-dev's round-2 unblock command (mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/) assumed failed/ existed; on PERSONAL muninn it did NOT (fresh instance — root was active/ complete/ pending/ sources/, no failed/). mv src nonexistent/ renames src→nonexistent, so job1 became the failed dir and job2 nested inside it. Caught on post-move ls (failed/ held job contents, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt failed/ (worldtree:worldtree 755) — NO data loss. Lessons: (1) before mv X into-dir/, verify the dir EXISTS ([ -d dir ]) — an empty ls dir/ 2>/dev/null is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is mv -t <targetdir> <src> (--target-directory): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a trailing slash does NOT protect — mv src failed/ with failed/ missing STILL silently renames to failed (rc=0); "just add the slash" is a false guard. (mkdir -p failed/ first also works, but mv -t inverts the failure from silent-wrong to loud-safe in one flag.) Container sh is dash — no ( in echo strings. SILENT failure mode (muninn-dev carry-forward): a misplaced ingestion-state move doesn't crash anything — list_jobs() stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (job_row→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (job_row found + guards pass), don't trust mv exit codes — and confirm job.dispatch.json survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. DON'T TIDY (round-2 pending): both DCC + P&P jobs currently REST in personal failed/ with manifests reading state: complete until round-2 requeue runs — deliberate + load-bearing (requeue keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. Round-2 sequencing: the requeue is mimir-dev's browser flow (pending their operator's board-vs-API ruling); muninn-dev is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); infra-ops = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z.
[2026-08-02] donut voice multi-clip reference (onyx-58 expansion) — TRIED, REVERTED. Folded the onyx-58 bundle's 3 Donut clips (seg101/seg110/seg148) in alongside the original seg000 → a 52.0s 4-take concat reference, hoping a longer ref → more robust speaker embedding. A pinned-seed A/B (5 pairs, varied registers, booth donut-onyx58) showed the original single-clip seg000 (16.3s) sounds better — concatenating disparate takes muddied the timbre more than the extra range helped. Reverted to seg000-alone (live + build-source). Two durable lessons: (1) for a faithful clone, a single clean representative take can beat a longer multi-take concat — more reference audio is NOT automatically better when the takes vary. (2) Emotion steering pulls the output AWAY from the cloned voice fast (operator's craft rule) — keep donut (and clones) emotion-neutral for fidelity; the gateway only enables emotion when an emotion_*/preset dial is explicitly sent, so bare {input,voice} calls stay pure-clone. seg148 was diarized SPEAKER_03 but IS Donut (operator-confirmed misdiarize). onyx-58 curated bundle lives in booth onyx-58 (24h TTL — stash to /mnt/smithy/voice_clones/ if a future middle-ref experiment is wanted).
[2026-08-02] Verifying the INDEX is not verifying GROUNDING (#382). A search_library returning wing=fiction hits proves the content is retrievable; it does NOT prove the agent (Mimir) trusts and uses those hits vs. silently answering from training. I reported "Mimir read Austen back to you" off a grounded-looking answer; ratatoskr-dev caught that grounding was intermittent (some sessions discarded the correct hits and substituted training knowledge). Test the harder claim — are the citations note-extracted or model-knowledge? — and reading the DEPLOYED artifact beats trusting the test for "is the fix live."
[2026-07-30] brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom microsoft/Mage-Flow-{Base,Turbo,Edit} repo IDs. brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF registry API is ground truth — an unauth 401 ≠ exists ({"error":"Invalid username or password"} masks private/gated/nonexistent alike), an authed 404 = phantom, and author=X&search=Y refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory reference_verify_hf_repo_ids_before_pull.
[2026-07-30] magpie TTS serving — evaluated, ABANDONED. Pulled magpie_tts_multilingual_357m (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-main container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; magpie-nemo torn down. .nemo KEPT on NFS as brokkr's fine-tuning base. auto-memory project_magpie_tts_eval_rejected.
[2026-07-25] Chaining the althing wake-listener arm orphans it. reply && althing-wake-listener & (or spawning althing-wake-listener with & inside a run_in_background task) → the &-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a plain operator turn (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn althing-wake-listener as its OWN run_in_background task, and re-arm ONLY after a real fire (<task-notification> completed rc0). Reclaim an orphan with althing-cli stop-monitor then re-arm.
[2026-07-25] Peer green-light ≠ operator consent for a managed-box mutation. Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the operator's yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)
[2026-07-18] Fleet Gitea CI foot-guns (3 failed soong-lab builds): the pfi-fleet runner's node:20-slim job image has no docker/git so actions/checkout + docker/* marketplace actions all fail; vh is a USER so its packages are owner-write-only (claude-bot repo-admin-collab still 401s on push/publish, and can't set repo secrets — owner-only); GITEA_-prefixed secret names are reserved/illegal. Fixes in → persistent-memory.d/2026-07-18-fleet-gitea-runner-build-recipe.md
[2026-07-18] zonos-gateway local clone had NO git remote + a history unrelated to gitea's — "committed to vh/zonos-gateway" was never pushed from that clone; two separate git init lineages, no merge-base. Reconcile = reset local→origin/main + overlay the changed files + push (NOT force — that erases gitea's voice-wav commits). Check git remote -v + git merge-base before assuming a clone is wired.
[2026-07-15] docker.service After=remote-fs.target does NOT wait for nofail NFS mounts → persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md
[2026-07-15] The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md
[2026-07-15] vLLM max-model-len does NOT free GPU VRAM → persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md
[2026-07-15] Claude Code statusline .cost.total_cost_usd is per-SESSION → persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md
[2026-07-14] MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md
[2026-07-14] AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md
[2026-07-14] NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md
[2026-07-14] NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md
[2026-07-14] MTP graft via top-level mtp.* tensor names does NOT survive AutoModelForCausalLM.from_pretrained → persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md
[2026-07-14] gitea "test-delivery 204" is NOT proof a webhook works → persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md
[2026-07-13] Relaying a peer's diagnosis as fact without confirming it against raw data → persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md
[2026-07-13] althing-cli reply <THREAD_id> (thread id, not a MESSAGE id) → "unknown message_id"; and reply to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use post --to <peer>. Bit me several times this session.
[2026-07-09] vllm/vllm-openai:latest crashes on Ampere IMPORT — Blackwell-only kernels (oink/aiter, has_device_capability(100)) die during import on the 3090/A6000. Pin v0.23.0 on irv-ml1's Ampere GPUs. (vllm/vllm-omni:v0.18.0 has a different entrypoint — don't use it either.)
[2026-07-09] Per-frame CPU SNAC decode is too slow for streaming — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = windowed chunk decode (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
[2026-07-08] Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md
[2026-07-08] Mistral3 + vLLM tokenizer/vision traps (serve MS3.2-24b, vLLM 0.24) → persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md
[2026-07-08] Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md
[2026-07-08] Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md
[2026-07-07] vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md
[2026-07-07] SGLang generic image can't LOAD our NVFP4 AEON → persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md
[2026-07-07] SGLang --lora-target-modules CLI enum REJECTS the GDN names its own resolver asks for → persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md
[2026-07-07] Engine invocation footguns cost several wasted serve-bounces this session → persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md
[2026-07-04] LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md
[2026-07-01] MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores min_p/logit_bias (qwopus gen: N=1 +12%, N=4 −20%). Reserve for dedicated/interactive deployments.
[2026-07-02] irv-ml1 /worktank ROOT is root-owned — lkraven can't write there (irv-ml1 sudo needs a password) → stage model pulls to /home. PIN THE A6000 BY UUID for training (native-CUDA ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). CUDA_VISIBLE_DEVICES=GPU-<uuid>.

107 older entries archived to archival-memory.md.

70 KiB Raw Blame History Unescape Escape