From 4b54a32d64babb3fd45028b92a610d336966990b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 1 Aug 2026 14:31:14 -0700 Subject: [PATCH] feat(mimir-inbox): book-ingestion UI stack on corviduo-dev:8091 (#377) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WG-internal FastAPI+HTMX front end for large-document ingestion into the Muninn KB, over the muninn-gate API (browser -> mimir-inbox -> staging -> path-addressed POST /jobs). Co-located on corviduo-dev with the gate (:8090) and the worldtree-personal muninn watcher per the operator's 2026-08-01 co-location ruling (reversing the earlier off-box/NFS plan; worldtree-dev approved the box placement). - Dockerfile: python:3.11-slim + uv sync --no-dev --frozen (--no-dev is load-bearing; the dev group's muninn-dispatch path source is absent in-image and INV-MI-7 forbids importing it). Single-stage by design — src/ stays in the final image (uv installs the project editable-linked to src/). - compose.yaml: uid 1000, host-net bind 10.250.50.152:8091, staging :rw, TCP-liveness healthcheck (deliberately not coupled to gate reachability). - Built from vh/mimir-inbox HEAD c8ab38f; deployed + healthy. Records the open-in-place claim semantics (worldtree-dev, runner.py:362-367) and the INV-MI-19 retention rule (staged files persist until job terminal; gate retry returns a false-200 on a swept source) in persistent memory. --- .../2026-07-31-muninn-gate-deploy.md | 6 +- stacks/mimir-inbox/.env.example | 33 ++++++++ stacks/mimir-inbox/Dockerfile | 37 +++++++++ stacks/mimir-inbox/README.md | 76 +++++++++++++++++++ stacks/mimir-inbox/compose.yaml | 59 ++++++++++++++ 5 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 stacks/mimir-inbox/.env.example create mode 100644 stacks/mimir-inbox/Dockerfile create mode 100644 stacks/mimir-inbox/README.md create mode 100644 stacks/mimir-inbox/compose.yaml diff --git a/persistent-memory.d/2026-07-31-muninn-gate-deploy.md b/persistent-memory.d/2026-07-31-muninn-gate-deploy.md index e4c4dd2..93e1178 100644 --- a/persistent-memory.d/2026-07-31-muninn-gate-deploy.md +++ b/persistent-memory.d/2026-07-31-muninn-gate-deploy.md @@ -11,7 +11,11 @@ WG-internal HTTP front door for the Muninn ingestion queue (`vh/muninn-gate`, mu **Verified boot:** `/ping` `{"service":"ok"}`; `/health` (ops-curl bearer, 200) `watcher.running:true` + `ingestion_root_writable:true`. muninn-dev independently poked the live gate — auth/route surface all held (401s w/ `WWW-Authenticate: Bearer`, the 4 FastAPI default routes gone, error-envelope-not-307 on trailing slashes = bug-hunt findings 5+6 confirmed outside pytest). -**DEFERRED (the submit path) — the mimir-inbox era:** SUBMIT returns `not_found` against the placeholder staging (correct, not a defect — muninn-dev confirmed) until (a) the real **shared mount** is provisioned (needs mimir-inbox's host — pre-implementation) and (b) a mimir-inbox writer exists. Operator ruled **shared mount** (mimir-inbox stays off-box, writes to a shared/NFS mount both gate + watcher bind at the same path). Then: repoint the gate's staging bind + **restart** (config is read ONCE at boot, no reload), muninn-dev's **one-file path-agreement probe** (the only check that can catch a gate/watcher path divergence), then the real acceptance run. **Hand mimir-dev the `mimir-inbox` key** (retrievable via `sudo` from the on-server config). +**DEFERRED (the submit path) — the mimir-inbox era:** SUBMIT returns `not_found` against the placeholder staging (correct, not a defect — muninn-dev confirmed) until the real staging dir + a mimir-inbox writer exist. ~~Operator ruled shared mount (mimir-inbox stays off-box, writes to a shared/NFS mount both gate + watcher bind at the same path).~~ **SUPERSEDED 2026-08-01 — operator REVERSED to CO-LOCATE:** mimir-inbox runs ON corviduo-dev, alongside the gate + watcher, staging = a corviduo-dev-LOCAL dir (not NFS). Reason the off-box/NFS call fell: muninn-dev's code-check showed staging is NOT same-fs-constrained (gate reads staging metadata + passes path strings; `os.replace` is inside `ingestion_root`), so staging's real constraint is **path identity across writer/gate/watcher**, which co-location buys outright — and it sidesteps the NFS failure modes (path-identity break, TOCTOU widening, stale handles, a hung mount blocking `resolve(strict=True)` — the last of which blocks mimir-inbox's *event loop*, not just a threadpool worker, since its staging check is in an async handler). Ruling relayed 3× (muninn-dev ×2 w/ msg-id citations, mimir-dev ×2) + operator in-session; **mimir-inbox key handed over 2026-08-01** (bumped to [read,submit,control], 0600 drop on nh3-dev). Tail on the co-locate ruling: raise worldtree-dev (box-side provisioning + the watcher claim-semantics open Q) → provision the real corviduo-dev-local `/mnt/muninn-staging/mimir-inbox` (uid = mimir-inbox's runtime identity, rw-writer / ro-gate+watcher) → 0600 key drop on corviduo-dev → muninn-dev's **one-file path-agreement probe** → acceptance. NB gate submit surface = **`POST /jobs`** (path-addressed; NO `POST /upload` — upload deferred v0, gate never ingests bytes). `staging_roots` already allowlists the path (no gate-config change). + +**RESOLVED 2026-08-01 (worldtree-dev, from source `core/muninn/runner.py:362-367`):** the watcher **OPENS the staged file in place** at claim (`parse_document(file_path)` on the dispatch-recorded absolute path) — it never moves/copies the source into the job dir (job dir holds DERIVED artifacts only). Consequences: (1) staging needs **PATH IDENTITY only**, so **co-location is a CONVENIENCE, not a requirement** — the parked multi-host option stays fully viable with a shared mount at the same absolute path on both hosts. (2) The real same-fs constraint is `.enqueue-tmp/` → `os.replace` into `pending/`, same-fs with `ingestion_root` — never staging (confirms muninn-dev). (3) **⚠️ OPERATIONAL RULE for mimir-inbox lifecycle (worldtree-dev):** open-in-place means the staged file MUST stay present+readable from submit **until the job is TERMINAL** (complete / failed-and-not-retried) — retry re-runs the structure phase, which re-opens the staged path. A cleanup that deletes on 201-submit kills every job at claim with a not-found that looks EXACTLY like the namespace-mismatch failure the bind exists to prevent. Relayed to mimir-dev for their cleanup design. **Gate-side edge (muninn-dev):** `POST /jobs/{id}/retry` returns `200 {requeued}` even for a job whose staged source was deleted — `muninn_dispatch.requeue` validates job STATE not file existence, and admission isn't re-run on retry (nothing re-stats files) → a FALSE success that dies at claim. Gate deliberately unguarded (re-admit re-resolves under a new clock, still races; lifecycle is the writer's), recorded as a gate compatibility constraint. So the retention rule isn't just "avoid claim-fail" — it's "retry will LIE with a 200 if the source is gone." + +**worldtree-dev approved co-location** (2026-08-01): another small infra-ops-managed LAN/WG-internal service on corviduo-dev in the gate's posture is fine at their OS/app layer; port/supervision/identity mine to shape; staging-dir ownership flip (mimir-inbox-writable, gate+watcher :ro — b162 watcher bind already :ro) at my convenience. **NEXT: coordinate the mimir-inbox deploy inputs with mimir-dev** (image/build recipe — likely infra-ops containerizes like muninn-gate; app config/env; port), then provision staging dir + stand up the service (uid 1000, matching the corviduo-dev muninn stack) + 0600 key drop on corviduo-dev + muninn-dev's path-agreement probe + acceptance. **Operational guard (no auto-check exists):** docker fabricates a MISSING bind source as an empty dir that passes every closed-config check → **confirm the host mount actually exists before wiring/repointing a bind** (`os.path.ismount` breaks on subdir roots; emptiness is normal pre-first-upload). This is why the gate/watcher path-agreement is an operational discipline, not a validated invariant. diff --git a/stacks/mimir-inbox/.env.example b/stacks/mimir-inbox/.env.example new file mode 100644 index 0000000..a7d4c34 --- /dev/null +++ b/stacks/mimir-inbox/.env.example @@ -0,0 +1,33 @@ +# mimir-inbox runtime config — copy to .env (gitignored on server) and fill the key. +# The 7 RUNTIME vars ONLY. The two DEV-ONLY drift-guard vars +# (WORLDTREE_CONFIG_PATH, MUNINN_GATE_REPO) MUST NOT be set here — they belong +# only to the test environment. All are validated at boot; any failure aborts +# startup with a secret-redacted message naming the variable. + +# muninn-gate front door — the co-located gate on corviduo-dev (host network). +MUNINN_GATE_URL=http://10.250.50.152:8090 + +# infra-ops-provisioned bearer (read+submit+control); SERVER-side only, never logged. +MUNINN_GATE_BEARER_KEY=__FILL_ON_SERVER__ + +# Shared staging root — RATIFIED value (not an example). Must exist, be a dir, and +# be writable by uid 1000; canonicalized at boot. +MIMIR_INBOX_STAGING_ROOT=/mnt/muninn-staging/mimir-inbox + +# The RETRY WINDOW (not a tidiness setting): staged files are retained until the +# job is terminal + this many days. Shortening it shortens retryability +# (INV-MI-19 / worldtree-dev open-in-place). Do not tune down without telling +# mimir-dev. +MIMIR_INBOX_STAGING_RETENTION_DAYS=7 + +# Wing allowlist — name:description pairs, ";"-separated (first ":" splits +# name/description; a name is one path component; order = select order). Names +# MUST be upstream Worldtree library.wings ids. Boot-fails if empty. +MIMIR_INBOX_WINGS=fiction:Novels, novellas and short stories;main:General knowledge — non-fiction, manuals and technical reference + +# Upload ceiling in MiB, enforced mid-stream. +MIMIR_INBOX_MAX_UPLOAD_MB=512 + +# Unclean-terminal float window (days) — how long a failed / complete_with_warnings +# job sorts above clean ones before settling into chronological order. +MIMIR_INBOX_FLOAT_WINDOW_DAYS=7 diff --git a/stacks/mimir-inbox/Dockerfile b/stacks/mimir-inbox/Dockerfile new file mode 100644 index 0000000..2deddbc --- /dev/null +++ b/stacks/mimir-inbox/Dockerfile @@ -0,0 +1,37 @@ +# mimir-inbox — UI front end for large-document / book ingestion into the Muninn +# KB, over the muninn-gate API (browser -> mimir-inbox -> staging -> path-addressed +# POST /jobs to the gate; bytes never reach the gate). No Dockerfile upstream +# (vh/mimir-inbox); this is infra-ops's, modeled on the muninn-gate one. +# +# All 7 RUNTIME deps are public PyPI — NO private index, NO BuildKit secret +# (unlike muninn-gate, which needed muninn-dispatch at runtime). +# +# docker build -t mimir-inbox:0.0.1 . +FROM python:3.11-slim + +RUN pip install --no-cache-dir uv==0.11.* + +WORKDIR /app +COPY pyproject.toml uv.lock README.md ./ +COPY src ./src + +# Install project + deps from the lockfile, EXCLUDING the dev group. --no-dev is +# load-bearing, not just leaner: the dev group carries muninn-dispatch via a path +# source ("../Worldtree/packages/muninn-dispatch") that is absent in the image, +# AND INV-MI-7 asserts runtime code never imports muninn_dispatch. Runtime deps +# resolve entirely from PyPI. +RUN uv sync --no-dev --frozen --no-cache + +# SINGLE-STAGE BY DESIGN — DO NOT convert to multi-stage that copies only .venv. +# `uv sync` installs the project EDITABLE-linked back to /app/src/mimir_inbox (not +# copied into site-packages), so mimir_inbox.__file__ — and the 16 templates + 442 +# static assets (vendored design CSS + htmx) it serves from — resolve out of src/. +# Dropping src/ in a final layer would kill the app at import; if it somehow got +# past import it would 404 every asset. src/ MUST remain alongside .venv. +# (mimir-dev preflight, 2026-08-01: `uv sync --no-dev --frozen` boots green with +# the path source absent; the alternative safe shape is `uv build --wheel` + +# install the wheel, which copies assets into site-packages.) +ENV PATH="/app/.venv/bin:$PATH" + +# The launcher (compose) owns --host/--port; no CMD here. Runs as uid 1000 (set +# in compose) — the sole writer to the staging root; /app is read-only to it. diff --git a/stacks/mimir-inbox/README.md b/stacks/mimir-inbox/README.md new file mode 100644 index 0000000..00f9082 --- /dev/null +++ b/stacks/mimir-inbox/README.md @@ -0,0 +1,76 @@ +# mimir-inbox + +UI front end for large-document / novel / book ingestion into the Muninn KB — +server-side FastAPI + HTMX over the **muninn-gate** API (#377). The submit path +is: **browser → mimir-inbox `/upload` (writes to staging) → path-addressed +`POST /jobs` to the gate**. Bytes never reach the gate; the gate references the +staged file by absolute path, and the watcher opens it **in place** at claim. + +- **Upstream:** `git@gitea.phasefinal.com:vh/mimir-inbox.git` (infra-ops owns + containerization — no Dockerfile upstream). +- **Host:** corviduo-dev `10.250.50.152:8091`, co-located with muninn-gate + (:8090) and the worldtree-personal muninn watcher. Operator ruled co-location + 2026-08-01 (reversing the earlier off-box/NFS plan); worldtree-dev approved + the box placement. +- **Posture:** LAN/WG-internal, no TLS, no public exposure (same as the gate). + No datastore, no migrations, no writable path other than the staging root — a + restart loses nothing. + +## Why co-located (not off-box/NFS) + +The watcher opens the staged file **in place** at claim (`core/muninn/runner.py`), +so staging's only hard constraint is **path identity** — the writer, gate, and +watcher must resolve the same absolute string to the same bytes. Co-location on +corviduo-dev buys that outright with one local filesystem and sidesteps the NFS +failure modes (path-identity drift, TOCTOU widening, stale handles, a hung mount +blocking the async staging check on mimir-inbox's event loop). Multi-host stays +viable later behind a same-path shared mount, but co-location is the cheapest +way to path identity. + +## Retention rule (load-bearing — INV-MI-19) + +Because the watcher opens the file in place, **a staged file must persist from +submit until the job is terminal** (complete, or failed-and-not-retried) — retry +re-runs the structure phase, which re-opens the same staged path. Deleting on +201-submit would kill every retried job at claim with a `not_found` that looks +exactly like a namespace mismatch. `MIMIR_INBOX_STAGING_RETENTION_DAYS` is the +**retry window**, not a tidiness knob; do not tune it down without telling +mimir-dev. + +## Config + +All runtime config is environment variables — no config file. The real values +(including the bearer-key secret) live in a gitignored `.env` next to +`compose.yaml` on the server (`0600`); the repo carries `.env.example`. Two +DEV-ONLY drift-guard vars (`WORLDTREE_CONFIG_PATH`, `MUNINN_GATE_REPO`) must +**not** be set in the container. + +## Deploy + +```bash +# 1. Fetch the source at the pinned tag/HEAD onto corviduo-dev, build the image +# (all deps public PyPI; uv sync --no-dev is load-bearing — see Dockerfile): +docker build -t mimir-inbox:0.0.1 . + +# 2. Provision the shared staging root (mimir-inbox-writable, gate+watcher :ro): +sudo install -d -o 1000 -g 1000 -m 0755 /mnt/muninn-staging/mimir-inbox + +# 3. Drop the real .env (0600) next to compose.yaml with the bearer key filled +# from the infra-ops key drop (~/.config/muninn-gate/mimir-inbox-env.sh). + +# 4. Bring it up: +docker compose up -d + +# 5. muninn-dev's one-file path-agreement probe, then the acceptance run. +``` + +**Operational guard:** docker fabricates a MISSING bind source as an empty dir +that passes checks — confirm `/mnt/muninn-staging/mimir-inbox` exists on the host +before `up -d` (step 2 does this). The container also fails boot if the staging +root is not writable (`os.access(W_OK)`), so a perms miss surfaces loudly. + +## Health + +`GET /health-banner` is the HTMX health surface (reflects gate reachability). +The container healthcheck is a **TCP-connect liveness** probe only — it must not +couple to the gate's state, so a gate outage never restarts mimir-inbox. diff --git a/stacks/mimir-inbox/compose.yaml b/stacks/mimir-inbox/compose.yaml new file mode 100644 index 0000000..f0bcf60 --- /dev/null +++ b/stacks/mimir-inbox/compose.yaml @@ -0,0 +1,59 @@ +# mimir-inbox — large-document / book ingestion UI over the muninn-gate API (#377). +# Deployed on corviduo-dev (10.250.50.152), co-located with muninn-gate (:8090) +# and the worldtree-personal muninn watcher. Operator ruled co-location 2026-08-01 +# (reversing the earlier off-box/NFS plan); worldtree-dev approved the box +# placement. infra-ops-managed stack, separate from the worldtree CI/CD compose. +# +# Image built out-of-band (all deps public PyPI, no secret needed): +# docker build -t mimir-inbox:0.0.1 . +# +# Runtime config (incl. the bearer-key secret) lives on the server in a gitignored +# .env next to this compose; repo carries the redacted .env.example. +services: + mimir-inbox: + image: mimir-inbox:0.0.1 + container_name: mimir-inbox + restart: unless-stopped + # uid 1000 (vh) — the SOLE writer to the staging root (INV-MI-3), which is + # provisioned 1000:1000. Matches the corviduo-dev muninn stack (gate is 1000). + user: "1000:1000" + # Host networking: the WG-internal bind lands directly on corviduo-dev, and + # MUNINN_GATE_URL=http://10.250.50.152:8090 reaches the co-located gate (whose + # own bind is host-network on that address). + network_mode: host + # The 7 runtime vars incl. the secret. The two DEV-ONLY drift-guard vars + # (WORLDTREE_CONFIG_PATH, MUNINN_GATE_REPO) are deliberately absent — they + # must not be set in the container. + env_file: + - .env + command: + - uvicorn + - --factory + - mimir_inbox.app:create_app + - --host + - "10.250.50.152" + - --port + - "8091" + volumes: + # Shared staging root — mimir-inbox is the SOLE WRITER; the gate + watcher + # bind this SAME absolute path :ro. A corviduo-dev-LOCAL dir (path identity + # is the real constraint; the watcher opens staged files IN PLACE at claim + # — worldtree-dev, core/muninn/runner.py:362-367 — so co-location buys path + # identity outright and nothing crosses a device boundary). + - /mnt/muninn-staging/mimir-inbox:/mnt/muninn-staging/mimir-inbox:rw + healthcheck: + # LIVENESS ONLY — a TCP connect proves uvicorn is bound, WITHOUT coupling + # container health to gate reachability. A gate outage must NOT restart + # mimir-inbox; the health banner surfaces that state instead (same lesson + # as the gate's /ping-not-/health probe). + test: ["CMD", "python", "-c", "import socket; socket.create_connection(('10.250.50.152',8091),3).close()"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + labels: + - homepage.group=Worldtree + - homepage.name=Mimir Inbox + - homepage.icon=mdi-book-arrow-right + - homepage.description=Large-document ingestion UI for the Muninn KB (#377) + - homepage.href=http://10.250.50.152:8091/