memory: snapshot — session close (muninn-gate #377 deployed on corviduo-dev, wtsdk 1.1.0 published + vh token, kimi-k3 reasoning-cap fix relayed to heid, magpie TTS rejected)

2026-07-31 21:27:13 -07:00
parent 786462ac9c
commit e6eabc1dde
3 changed files with 51 additions and 6 deletions
@@ -0,0 +1,15 @@
+`[2026-07-31]` **kimi-k3 "output cap" root-caused = a ~16384 REASONING-token ceiling, not an output cap; fix relayed to heid, NOT applied gateway-side.**
+
+heid reported that `kimi-k3` (the primary route = Kimi Code coding endpoint `openai/k3` @ `api.kimi.com/coding/v1`) silently degraded its cross-frontier panel: on large/reasoning-heavy dispatches, `completion_tokens: 16381` **exactly**, `content` empty, `reasoning_content` ~64KB, `finish_reason: **stop**` (a truncation mislabeled as a clean stop). `max_tokens: 100000` in the request was not honored.
+
+**Investigation arc (a clean cross-frontier-triage + verify-on-the-wire case):**
+1. My first read: a flat ~16384 OUTPUT cap; fix = a LiteLLM `stop→length` relabel callback (heid's fallback ask). Confirmed the cap isn't in our LiteLLM config (no `max_tokens` clamp on the route).
+2. Operator routed a fix-research pass to **dvalin-smithy-dev + bil-smithy-dev** (independent). Both CONVERGED (docs-based): `max_tokens` is a deprecated alias on Kimi/Moonshot; the canonical field is `max_completion_tokens` (default 131072, max 1M); the coding endpoint defaults output to 16384; fix = send `max_completion_tokens` + `reasoning_effort` via `extra_body` (drop_params-safe).
+3. **heid's live data REFUTED the docs hypothesis:** a later dispatch hit `completion_tokens: 18455` (ABOVE 16384) cleanly, with `reasoning_tokens: 16198` (just under 16384) and content present. So COMPLETION is uncapped; the bound is on **REASONING at ~16384**. When a hard task's thinking exhausts that budget, nothing's left for content → empty answer under `stop`.
+4. **I proved it on the wire** — ran heid's real 500KB failing bundle direct at both endpoints (bypassing LiteLLM so `reasoning_effort` isn't dropped): default effort → 504/timeout (the failure); **`reasoning_effort: low` → reasoning ~12–13.5k (under the ceiling), content returns (6–7.6k chars)**, on BOTH coding AND general endpoints. So re-routing to the general endpoint buys nothing — the fix is the effort param, and it works on the wire.
+
+**THE FIX (caller-side, no shared-gateway change/restart):** send `reasoning_effort` via **`extra_body`** on kimi-k3 dispatches (`low` for large bundles). LiteLLM `drop_params: true` strips the top-level `reasoning_effort` — which is exactly why heid's earlier `reasoning_effort: low` was a no-op. `extra_body` survives drop_params (the house GLM-thinking pattern). Tradeoff: low effort = shallower reasoning, but a complete answer beats today's empty one.
+
+**Relayed to heid to validate on a real round** (the one unconfirmed hop is whether `extra_body` survives OUR LiteLLM). **Backstop if it doesn't:** add `allowed_openai_params: ["reasoning_effort"]` to the `kimi-k3` route in the gateway config — that IS a shared-gateway change + a ~10s restart (blips all consumers), so it needs a heads-up.
+
+Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/litellm/conf/config.yaml` (see Recent-decisions `[2026-07-25]` Kimi K3 wiring). No gateway change was made this session. Failing dispatch on record: `01KYTASKTY3T` (jackdaw-dev bug-hunt).
@@ -0,0 +1,20 @@
+`[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev `10.250.50.152:8090`.**
+
+WG-internal HTTP front door for the Muninn ingestion queue (`vh/muninn-gate`, muninn-dev's repo). The full provisioning ask (staging mount + closed-schema config + bearer keys + compose/WG bind) came after a 4-message discovery exchange with muninn-dev + a cross-team coordination with worldtree-dev; operator ruled the open architecture call (shared mount) and greenlit build+boot.
+
+**Deployment (eshpfi `stacks/muninn-gate/`):**
+- Image `muninn-gate:0.0.14` — no Dockerfile upstream, so infra-ops owns containerization. `python:3.11-slim` + `uv pip install .`; **`muninn-dispatch==0.1.4` from the internal Gitea index** (`[tool.uv.sources]`, `uv pip install .` honored the pin), token passed as a **BuildKit secret** (`--secret id=gitea_pw`) so it never lands in a layer. Built on corviduo-dev.
+- **`ingestion_root: /data/state/ingestion`** — the `worldtree-personal_worldtree-state` docker volume mounted at `/data/state`, byte-identical to the watcher's view. **Acceptance criterion (muninn-dev's): `/health` → `watcher.running: true` PROVES byte-identity** (the gate reads the heartbeat the watcher writes); `no_heartbeat` with the watcher up = root mismatch. Verified true first boot.
+- **`user: "1000:1000"`** — the ingestion dir is `vh:vh 0755`, so a non-root gate had to run as uid 1000 to WRITE the queue (my Dockerfile's `USER gate`/10014 would've been denied; the watcher itself runs as root and bypasses perms). This uid requirement was a genuine spec gap — muninn-dev added it to the contract (`084526e`, vh:vh 0755 + 1000:1000 as the worked example) so no future deployer re-derives it. `ingestion_root_writable: true` in `/health` is the post-deploy confirmation.
+- **staging `/mnt/muninn-staging/mimir-inbox`** — bound `:ro`, SAME absolute path in BOTH the gate AND the watcher (dispatch stores paths absolutely; the watcher opens them at claim time). worldtree-dev added the watcher-side bind (their image) in **b162** (`${MUNINN_STAGING_DIR:-…}:/mnt/muninn-staging/mimir-inbox:ro`). Currently a **LOCAL placeholder dir** on corviduo-dev.
+- config (single-writer) `/opt/docker/conf/muninn-gate/muninn-gate.yaml` (0600, 1000:1000). Schema CLOSED (unknown field = boot failure). 2 bearer keys minted: `mimir-inbox` [read,submit], `ops-curl` [read,submit,control]. `network_mode: host`; health probe = **`/ping`** (NOT `/health`, which is always-200 by design and would never restart the gate). Committed `786462a` (no secrets).
+
+**Verified boot:** `/ping` `{"service":"ok"}`; `/health` (ops-curl bearer, 200) `watcher.running:true` + `ingestion_root_writable:true`. muninn-dev independently poked the live gate — auth/route surface all held (401s w/ `WWW-Authenticate: Bearer`, the 4 FastAPI default routes gone, error-envelope-not-307 on trailing slashes = bug-hunt findings 5+6 confirmed outside pytest).
+
+**DEFERRED (the submit path) — the mimir-inbox era:** SUBMIT returns `not_found` against the placeholder staging (correct, not a defect — muninn-dev confirmed) until (a) the real **shared mount** is provisioned (needs mimir-inbox's host — pre-implementation) and (b) a mimir-inbox writer exists. Operator ruled **shared mount** (mimir-inbox stays off-box, writes to a shared/NFS mount both gate + watcher bind at the same path). Then: repoint the gate's staging bind + **restart** (config is read ONCE at boot, no reload), muninn-dev's **one-file path-agreement probe** (the only check that can catch a gate/watcher path divergence), then the real acceptance run. **Hand mimir-dev the `mimir-inbox` key** (retrievable via `sudo` from the on-server config).
+
+**Operational guard (no auto-check exists):** docker fabricates a MISSING bind source as an empty dir that passes every closed-config check → **confirm the host mount actually exists before wiring/repointing a bind** (`os.path.ismount` breaks on subdir roots; emptiness is normal pre-first-upload). This is why the gate/watcher path-agreement is an operational discipline, not a validated invariant.
+
+**Hardening candidate (flagged, not done):** the compose mounts the WHOLE `worldtree-personal_worldtree-state` volume at `/data/state` per muninn-dev's spec; a subpath mount of just `ingestion` → `/data/state/ingestion` would be tighter (gate only needs RW on ingestion). Confirm with muninn-dev before adopting.
+
+See auto-memory `reference_muninn_gate_deploy`, `reference_muninn_gate_staging_path`; [[2026-07-25-infra-ops-wt-config-repo]] (corviduo-dev boundary), and Recent-decisions `[2026-07-27]` muninn watcher sidecar (the other half of #377).