esh-pfi-infrastructure/persistent-memory.d/2026-07-31-muninn-gate-deploy.md

[2026-07-31] muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev 10.250.50.152:8090.

WG-internal HTTP front door for the Muninn ingestion queue (vh/muninn-gate, muninn-dev's repo). The full provisioning ask (staging mount + closed-schema config + bearer keys + compose/WG bind) came after a 4-message discovery exchange with muninn-dev + a cross-team coordination with worldtree-dev; operator ruled the open architecture call (shared mount) and greenlit build+boot.

Deployment (eshpfi stacks/muninn-gate/):

• Image muninn-gate:0.0.14 — no Dockerfile upstream, so infra-ops owns containerization. python:3.11-slim + uv pip install .; muninn-dispatch==0.1.4 from the internal Gitea index ([tool.uv.sources], uv pip install . honored the pin), token passed as a BuildKit secret (--secret id=gitea_pw) so it never lands in a layer. Built on corviduo-dev.
• ingestion_root: /data/state/ingestion — the worldtree-personal_worldtree-state docker volume mounted at /data/state, byte-identical to the watcher's view. Acceptance criterion (muninn-dev's): /health → watcher.running: true PROVES byte-identity (the gate reads the heartbeat the watcher writes); no_heartbeat with the watcher up = root mismatch. Verified true first boot.
• user: "1000:1000" — the ingestion dir is vh:vh 0755, so a non-root gate had to run as uid 1000 to WRITE the queue (my Dockerfile's USER gate/10014 would've been denied; the watcher itself runs as root and bypasses perms). This uid requirement was a genuine spec gap — muninn-dev added it to the contract (084526e, vh:vh 0755 + 1000:1000 as the worked example) so no future deployer re-derives it. ingestion_root_writable: true in /health is the post-deploy confirmation.
• staging /mnt/muninn-staging/mimir-inbox — bound :ro, SAME absolute path in BOTH the gate AND the watcher (dispatch stores paths absolutely; the watcher opens them at claim time). worldtree-dev added the watcher-side bind (their image) in b162 (${MUNINN_STAGING_DIR:-…}:/mnt/muninn-staging/mimir-inbox:ro). Currently a LOCAL placeholder dir on corviduo-dev.
• config (single-writer) /opt/docker/conf/muninn-gate/muninn-gate.yaml (0600, 1000:1000). Schema CLOSED (unknown field = boot failure). 2 bearer keys minted: mimir-inbox [read,submit], ops-curl [read,submit,control]. network_mode: host; health probe = /ping (NOT /health, which is always-200 by design and would never restart the gate). Committed 786462a (no secrets).

Verified boot: /ping {"service":"ok"}; /health (ops-curl bearer, 200) watcher.running:true + ingestion_root_writable:true. muninn-dev independently poked the live gate — auth/route surface all held (401s w/ WWW-Authenticate: Bearer, the 4 FastAPI default routes gone, error-envelope-not-307 on trailing slashes = bug-hunt findings 5+6 confirmed outside pytest).

DEFERRED (the submit path) — the mimir-inbox era: SUBMIT returns not_found against the placeholder staging (correct, not a defect — muninn-dev confirmed) until the real staging dir + a mimir-inbox writer exist. Operator ruled shared mount (mimir-inbox stays off-box, writes to a shared/NFS mount both gate + watcher bind at the same path). SUPERSEDED 2026-08-01 — operator REVERSED to CO-LOCATE: mimir-inbox runs ON corviduo-dev, alongside the gate + watcher, staging = a corviduo-dev-LOCAL dir (not NFS). Reason the off-box/NFS call fell: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; os.replace is inside ingestion_root), so staging's real constraint is path identity across writer/gate/watcher, which co-location buys outright — and it sidesteps the NFS failure modes (path-identity break, TOCTOU widening, stale handles, a hung mount blocking resolve(strict=True) — the last of which blocks mimir-inbox's event loop, not just a threadpool worker, since its staging check is in an async handler). Ruling relayed 3× (muninn-dev ×2 w/ msg-id citations, mimir-dev ×2) + operator in-session; mimir-inbox key handed over 2026-08-01 (bumped to [read,submit,control], 0600 drop on nh3-dev). Tail on the co-locate ruling: raise worldtree-dev (box-side provisioning + the watcher claim-semantics open Q) → provision the real corviduo-dev-local /mnt/muninn-staging/mimir-inbox (uid = mimir-inbox's runtime identity, rw-writer / ro-gate+watcher) → 0600 key drop on corviduo-dev → muninn-dev's one-file path-agreement probe → acceptance. NB gate submit surface = POST /jobs (path-addressed; NO POST /upload — upload deferred v0, gate never ingests bytes). staging_roots already allowlists the path (no gate-config change).

RESOLVED 2026-08-01 (worldtree-dev, from source core/muninn/runner.py:362-367): the watcher OPENS the staged file in place at claim (parse_document(file_path) on the dispatch-recorded absolute path) — it never moves/copies the source into the job dir (job dir holds DERIVED artifacts only). Consequences: (1) staging needs PATH IDENTITY only, so co-location is a CONVENIENCE, not a requirement — the parked multi-host option stays fully viable with a shared mount at the same absolute path on both hosts. (2) The real same-fs constraint is .enqueue-tmp/ → os.replace into pending/, same-fs with ingestion_root — never staging (confirms muninn-dev). (3) ⚠️ OPERATIONAL RULE for mimir-inbox lifecycle (worldtree-dev): open-in-place means the staged file MUST stay present+readable from submit until the job is TERMINAL (complete / failed-and-not-retried) — retry re-runs the structure phase, which re-opens the staged path. A cleanup that deletes on 201-submit kills every job at claim with a not-found that looks EXACTLY like the namespace-mismatch failure the bind exists to prevent. Relayed to mimir-dev for their cleanup design. Gate-side edge (muninn-dev): POST /jobs/{id}/retry returns 200 {requeued} even for a job whose staged source was deleted — muninn_dispatch.requeue validates job STATE not file existence, and admission isn't re-run on retry (nothing re-stats files) → a FALSE success that dies at claim. Gate deliberately unguarded (re-admit re-resolves under a new clock, still races; lifecycle is the writer's), recorded as a gate compatibility constraint. So the retention rule isn't just "avoid claim-fail" — it's "retry will LIE with a 200 if the source is gone."

worldtree-dev approved co-location (2026-08-01): another small infra-ops-managed LAN/WG-internal service on corviduo-dev in the gate's posture is fine at their OS/app layer; port/supervision/identity mine to shape; staging-dir ownership flip (mimir-inbox-writable, gate+watcher :ro — b162 watcher bind already :ro) at my convenience. NEXT: coordinate the mimir-inbox deploy inputs with mimir-dev (image/build recipe — likely infra-ops containerizes like muninn-gate; app config/env; port), then provision staging dir + stand up the service (uid 1000, matching the corviduo-dev muninn stack) + 0600 key drop on corviduo-dev + muninn-dev's path-agreement probe + acceptance.

Operational guard (no auto-check exists): docker fabricates a MISSING bind source as an empty dir that passes every closed-config check → confirm the host mount actually exists before wiring/repointing a bind (os.path.ismount breaks on subdir roots; emptiness is normal pre-first-upload). This is why the gate/watcher path-agreement is an operational discipline, not a validated invariant.

Hardening candidate (flagged, not done): the compose mounts the WHOLE worldtree-personal_worldtree-state volume at /data/state per muninn-dev's spec; a subpath mount of just ingestion → /data/state/ingestion would be tighter (gate only needs RW on ingestion). Confirm with muninn-dev before adopting.

See auto-memory reference_muninn_gate_deploy, reference_muninn_gate_staging_path; 2026-07-25-infra-ops-wt-config-repo (corviduo-dev boundary), and Recent-decisions [2026-07-27] muninn watcher sidecar (the other half of #377).