memory: snapshot — session close (muninn-gate #377 deployed on corviduo-dev, wtsdk 1.1.0 published + vh token, kimi-k3 reasoning-cap fix relayed to heid, magpie TTS rejected)

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-07-30_
+_Last updated: 2026-07-31_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -109,11 +109,11 @@ no longer deployed sidecars here. See Recent decisions.)
 
 ## Current state / in-flight
 
-_As of 2026-07-30 — **session at a natural close; nothing actively in-flight.** A long multi-day infra session (2026-07-25→30); all work shipped, committed, and PUSHED (`main == origin/main`). Each item below has a full Recent-decisions entry:_
-- **Worldtree config-as-code repo** `vh/worldtree-instance-configs` built + PUSHED; boundary AGREED with worldtree-dev (they stop hand-editing `/opt/<instance>/config`; changes route to infra-ops as deltas). First live delta = demo `BIFROST_CLIENT_ALLOWED_HOSTS += 10.100.10.50:8391` (wyrd provider). Env-deltas tracked via `personal/env.public` (non-secret overlay — new mechanism, operator-blessing pending).
-- **Zed edit-predictions keyless FIM route** — deep-research picked `Qwen2.5-Coder-1.5B`; stood up `vllm-coder` (ana-ml2 GPU1 :8020), LiteLLM alias `coder-fast`, keyless `zed-fim-proxy` (ana-docker :4141). Granite shrunk to fit (phasing out). Zed `api_url=http://10.250.50.70:4141/v1`. LIVE + verified.
-- **Muninn watcher sidecar** on personal Worldtree (#377 CLOSED, acceptance passed); **Kimi K3** wired into LiteLLM (coding endpoint); **bil-smithy-dev** althing pane route; **nh3-extdev herald** installed; **Booth v0.1.4** (downloadable booths); **jackdaw-compose** decommissioned.
-- Demo `/embed` request (worldtree-dev, 2026-07-29) → turned out a Worldtree CODE bug (reader-with-no-writer at api.py:5474), not config — worldtree-dev fixed it (b154/b155), verified live. No infra action was needed; my check-before-acting prevented a pointless demo restart._
+_As of 2026-07-31 — **session at a natural close; nothing infra-ops-side actively blocked.** A long single-session run 2026-07-30→31 (model pulls, TTS eval, a gateway root-cause, an SDK publish, and the muninn-gate deploy). All shipped/committed; `main == origin/main` after the muninn-gate commit `786462a`. Each item has a Recent-decisions entry:_
+- **muninn-gate DEPLOYED + healthy on corviduo-dev `10.250.50.152:8090`** (#377 ingestion front door). `/ping` + `/health` (watcher:running:true ⇒ ingestion_root byte-identity proven) verified first boot. **SUBMIT path DEFERRED** until mimir-inbox (pre-implementation) + the real shared NFS mount exist → then repoint the staging bind (**+restart**, config is read-once), run muninn-dev's one-file path-agreement probe, then the acceptance run; **hand mimir-dev the `mimir-inbox` bearer key** (in the on-server `/opt/docker/conf/muninn-gate/muninn-gate.yaml`, 0600) when they implement. auto-memory `reference_muninn_gate_deploy`.
+- **kimi-k3 output cap root-caused; fix RELAYED to heid, NOT applied gateway-side.** It's a ~16384 **reasoning**-token ceiling (completion is uncapped); fix = `reasoning_effort: low` via **`extra_body`** (LiteLLM `drop_params` strips the top-level param). Proven on the wire against heid's real bundle (reasoning drops under the ceiling → content returns, both coding + general endpoints). heid to validate on a real round; **backstop if `extra_body` doesn't survive LiteLLM = add `allowed_openai_params: ["reasoning_effort"]` to the kimi-k3 route** (gateway change + restart).
+- **worldtree-sdk 1.1.0 (Python) PUBLISHED** to the vh Gitea PyPI (memory_context pass-through; unblocked wyrd-dev). claude-bot added as **write collaborator** on `vh/worldtree-sdk` (source via API archive — git-HTTP 403s on that repo). **Operator-directed cred: a FULL vh site-admin token is saved at `~/.config/gitea/vh-token` (0600)** for infra-ops SDK publishing — ⚠️ high blast radius; operator chose to keep it over a scoped one. auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
+- **magpie TTS evaluated → REJECTED** (Zonos wins expressive English; multilingual not needed). Not served; `magpie-nemo` torn down. **KEEP `/mnt/smithy/models/nvidia/magpie_tts_multilingual_357m/`** — brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
 
 **Open follow-ups (non-blocking — pick one up or not):**
 - **Zonos emotion:** sad axes/text pass on the 3 calibrated voices (only named-sad, untested); emotion-congruent-text pass (validates intensity, may rescue sad id); clone-char (Emmie/Penny/Natalie/Miranda) emotion rows use the mid-region fallback until measured. Presets are **provisional** (neutral-text ear-check was inconclusive). Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio,assemble_voice}.py` (run ON irv-ml1; dial-in studio = nohup :8898 on nh3-dev). dvalin thread at rest (`01KXT12FN0AS…`). → `persistent-memory.d/2026-07-18-zonos-gateway-0.2.1-emotion-presets.md`
@@ -134,6 +134,12 @@ _As of 2026-07-30 — **session at a natural close; nothing actively in-flight.*
 
 ## Recent decisions
 
+- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
+
+- `[2026-07-31]` **worldtree-sdk 1.1.0 (Python) published to vh Gitea PyPI + a durable infra-ops publish cred.** memory_context pass-through; unblocked wyrd-dev. claude-bot now a write-collaborator on `vh/worldtree-sdk` (source pulled via the **Gitea API archive** — git-HTTP 403s on that repo); publishing to the vh USER namespace **can't be delegated** (401 `reqPackageAccess` even with `write:package`) so it needs an owner token — operator saved a **FULL vh site-admin token at `~/.config/gitea/vh-token` (0600)** for it (⚠️ high blast radius, kept over a scoped one; org-namespace migration is the only real de-personalization, parked by wtsdk-dev). auto-memory `reference_infra_ops_vh_gitea_token_and_sdk_publish`.
+
+- `[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap.** heid's cross-frontier panel was silently degraded (empty content, `finish_reason: stop`). dvalin+bil researched (docs said deprecated-max_tokens); heid's live data refuted that (completion hit 18455) → it's a reasoning ceiling. **Proven on the wire against heid's real 500KB bundle:** `reasoning_effort: low` drops reasoning under the ceiling → content returns, on BOTH coding + general endpoints. Fix is CALLER-side (no gateway change): send `reasoning_effort` via **`extra_body`** (LiteLLM `drop_params: true` strips the top-level param — why heid's earlier attempt no-op'd). Relayed to heid to validate; backstop = `allowed_openai_params` on the route. → `persistent-memory.d/2026-07-31-kimi-k3-reasoning-cap.md`
+
 - `[2026-07-27]` **Zed edit-predictions: keyless FIM-completion route SHIPPED end-to-end.** Operator wants Zed's inline edit-prediction (which CANNOT send an auth header) to reach a FIM coder via `/v1/completions`. **Deep-research (106-agent workflow) picked `Qwen/Qwen2.5-Coder-1.5B`** (BASE, Apache-2.0; native FIM `<|fim_prefix|>/<|fim_suffix|>/<|fim_middle|>` IDs 151659/60/61; Zed `prompt_format:"qwen"`). Runner-up 3B = non-commercial Qwen-Research license; **no small dense Qwen3-Coder exists (all MoE, smallest 30B)**. **Stood up `vllm-coder`** on ana-ml2 **GPU1 :8020** (served-name `qwen2.5-coder-1.5b`, 8192 ctx, util 0.06, fp8 KV). To fit, **shrank granite (phasing out, operator-directed):** util 0.27→0.13, max-len 131072→16384, seqs 1024→256 (freed ~14 GB; the KV-≥-1×-max-len rule crash-looped it at util 0.12/32768 → settled 0.13/16384). **LiteLLM alias `coder-fast`** → :8020 (`mode: completion`). **Minted a `coder-fast`-SCOPED virtual key** (verified 403 on `gen` — the real blast-radius bound). **Built `zed-fim-proxy`** (ana-docker **:4141**, `network_mode: host`, stdlib-python, `stacks/zed-fim-proxy`): keyless POST `/v1/completions`, model-allowlist `coder-fast`, injects the scoped key → LiteLLM :4000; `GET /ping` anon liveness; wrong-model→403, wrong-path→404, `/chat/completions` rejected. Verified keyless FIM end-to-end ('a + b', finish `stop`). **Zed `api_url` = `http://10.250.50.70:4141/v1`, model `coder-fast`, prompt_format `qwen`.** **source-IP allowlist intentionally LEFT OFF (operator direction 2026-07-27) — do NOT tighten:** Zed roams the operator's WireGuard `10.0.0.0/8`, so a single-IP pin would break it. Blast-radius bound is the `coder-fast`-scoped key + model/path allowlist (keyless but coder-fast-only, internal-net-only). (The proxy does exact-IP matching; scoping to the `10.0.0.0/8` CIDR would need CIDR support — deliberately not added.) Canonical: `stacks/vllm` (coder + granite shrink), `stacks/litellm` (coder-fast), `stacks/zed-fim-proxy` (NEW). Server vllm compose.yaml has benign stale-comment drift vs canonical (didn't overwrite the newer canonical).
 
 - `[2026-07-27]` **Muninn ingestion-watcher sidecar deployed on PERSONAL Worldtree (#377).** worldtree-dev request (research-wing ingest arc, personal-only per the 2026-07-16 topology ruling); operator-approved. Added a `worldtree-muninn` **compose sidecar** to `/opt/worldtree-personal/compose.yaml` — `<<: *worldtree-common` anchor inherits the api's image + full env + config/state/kb mounts; `command: python -m core.muninn --watch`; `restart: unless-stopped`; `stop_grace_period: 1h` (INV-377-7: max 2 concurrent × worst-case job, SIGTERM-drains). **Pinned to the running SHA `773866084af9`** (b146, ≥ b143 — dodges both the `:latest` trap AND the "pre-b143 ref resurrects deleted dispatch.py from stale bytecode" warning). Verified: running / 0 restarts / flock sole-runner (no rc3) / heartbeat live at `{ingestion_root=/data/state/ingestion}/.watcher-heartbeat` (poll 30s). Container `worldtree-personal-worldtree-muninn-1`; backup `compose.yaml.bak-muninn-20260727-081920`. **DURABILITY RESOLVED (worldtree-dev, same day):** Q1 was a LIVE FOOTGUN — `deploy-personal.yml` scp's the REPO compose.yaml over the box's + runs `up -d --remove-orphans`, so the box-local sidecar would've been clobbered AND orphan-removed at the next staging tag. worldtree-dev fixed at source: moved the sidecar into their repo compose.yaml gated behind a **`muninn` compose profile** (commit 5d7f6bd) — shared compose stays instance-identical, `.env` `COMPOSE_PROFILES` differentiates (demo watcher-less). **My action:** added `COMPOSE_PROFILES=muninn` to `/opt/worldtree-personal/.env` (backup `.bak-muninn-profile-20260727-082541`; no-op vs the current unprofiled box-local sidecar → seamless handover at next deploy). Q2: their deploy `up -d`'s the whole stack w/ `WORLDTREE_IMAGE` exported → sidecar version-tracks the api, no drift. **CONFIG-AS-CODE EXTENSION:** mirrored the non-secret delta as `personal/env.public` in `vh/worldtree-instance-configs` (repo `a9d091e`) — FIRST extension beyond config.yaml files to env-level config; the secret-laden `.env` stays box-only, `env.public` records only non-secret infra-ops-owned env deltas (record, not a deploy source — `deploy-wt-config` globs `*.yaml`). **BOUNDARY CLARIFIED:** compose.yaml = worldtree-dev's (their repo, instance-identical, scp'd on deploy); per-instance `.env` = infra-ops's differentiator. Deploy step of the #363/#377 arc. **#377 CLOSED — acceptance PASSED 2026-07-27:** worldtree-dev enqueued a test job via muninn-dispatch 0.1.0 in a one-shot ephemeral container (no docker-exec); the sidecar claimed it within one 30s poll, drove it to terminal (structure→summarize→complete), zero restarts/rc3, heartbeat fresh throughout — whole loop (request→deploy→durability fix→acceptance) in <2h. (Pre-existing pipeline bug #379 surfaced — `output.kb_notes=false` ignored → 1 inert test note in the research wing — worldtree-dev owns it, nothing infra-ops-side.) **⚠ OPERATOR-SURFACE (open):** the `env.public` overlay mechanism is a repo-scope call to bless/adjust. [[reference_worldtree_deploys_cicd]] [[reference_worldtree_instance_configs_repo]] [[project_worldtree_research_wing_ingest]]
@@ -264,6 +270,10 @@ _142 older entries archived to archival-memory.md._
 
 ## Tried and abandoned
 
+- `[2026-07-30]` **brokkr's WebSearch "verification" CONFIRMED a hallucination — 3 phantom `microsoft/Mage-Flow-{Base,Turbo,Edit}` repo IDs.** brokkr-smithy-dev handed 3 gated-looking repo IDs for an operator-directed model pull; they don't exist (its own web-search fabricated an arXiv ID + project page, twice). Lesson: the HF **registry API is ground truth** — an unauth 401 ≠ exists (`{"error":"Invalid username or password"}` masks private/gated/nonexistent alike), an authed 404 = phantom, and `author=X&search=Y` refutes existence. API-verify every repo ID before a pull; LLM-summarized web fetches confabulate. auto-memory `reference_verify_hf_repo_ids_before_pull`.
+
+- `[2026-07-30]` **magpie TTS serving — evaluated, ABANDONED.** Pulled `magpie_tts_multilingual_357m` (the one real repo of brokkr's batch) to NFS, stood it up on irv-ml1 (ephemeral NeMo-Speech-`main` container — stock PyPI/NGC NeMo can't load v2607), A/B'd vs Zonos → Zonos wins expressive English decisively, multilingual not needed. Not served; `magpie-nemo` torn down. `.nemo` KEPT on NFS as brokkr's fine-tuning base. auto-memory `project_magpie_tts_eval_rejected`.
+
 - `[2026-07-25]` **Chaining the althing wake-listener arm orphans it.** `reply && althing-wake-listener &` (or spawning `althing-wake-listener` with `&` *inside* a `run_in_background` task) → the `&`-child reparents to init, UNTRACKED by the harness: no fire-notification, and re-arms bounce rc3 off a lock nothing services (mail silently unwatched). Compounding foot-gun: re-arming after a *plain operator turn* (not an actual fire) collides with the still-live prior listener (rc3). FIX: spawn `althing-wake-listener` as its OWN `run_in_background` task, and re-arm ONLY after a real fire (`<task-notification> completed rc0`). Reclaim an orphan with `althing-cli stop-monitor` then re-arm.
 
 - `[2026-07-25]` **Peer green-light ≠ operator consent for a managed-box mutation.** Auto-mode guard blocked a config-replace+restart on the Worldtree-team demo box that was authorized only by worldtree-dev's althing message — correctly: a persistent change to shared infra needs the *operator's* yes for that specific change, not a peer's. Surface it; don't route around the guard. (The operator then stood the whole change down — the guard's hold was the right call.)