diff --git a/persistent-memory.d/2026-08-28-althing-v3-cutover.md b/persistent-memory.d/2026-08-28-althing-v3-cutover.md index b25ec32..60b6a6f 100644 --- a/persistent-memory.d/2026-08-28-althing-v3-cutover.md +++ b/persistent-memory.d/2026-08-28-althing-v3-cutover.md @@ -3,7 +3,7 @@ Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward. - post office ONE container on nh3-dev, http://10.100.10.50:8390 + post office ONE container on nh3-dev, http://10.100.50.40:8390 the only stateful component. SQLite + FTS + the typed API + the operator page. herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port, holds no state. Refuses to start if another herald holds the node. @@ -14,7 +14,7 @@ deleted, not deprecated.** No rollback was designed or tested; failures are fixe althing-light-monitor -> GONE althing-receiver -> GONE althing-herald -> althing-po-herald -**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and +**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.50.40:8390` and `ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office address** — a bare `postbox status` errors out rather than guessing. @@ -86,3 +86,65 @@ never-broadcast-unsolicited directive. ⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical (`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them. + +--- + +## `[2026-08-28, same day]` MOVED to nh3-docker — and the WAL nearly ate the mail + +Operator: *"I want it on the docker machine — that was always the goal."* The flag-day +deployment put the post office on **nh3-dev**, which was wrong on three counts: + +- our own server table calls nh3-dev **"not a Docker-stack host"**; NH-site non-GPU → nh3-docker +- nh3-dev had **three OOM events in fourteen days, interval halving**, and the confirmed hog is + CC sessions at 5-18 GB — the box's actual job. See [[2026-08-28-nh3-dev-oom-attribution]]. +- `mem_limit: 512m` protects the fleet **FROM** the post office. It does nothing to protect the + post office **from the box**: `oom_score_adj` was 0, an ordinary kill candidate, and the 08-28 + sweep took althing-herald and uvicorn. A sweep taking the post office takes all 73 handles. + + NOW http://10.100.50.40:8390 nh3-docker, oom_score_adj=-500, mem 512m verified honoured + WAS http://10.100.10.50:8390 nh3-dev (address now refuses) + canon stacks/althing-post-office/compose.yaml -> /opt/docker/compose/ on nh3-docker + +## ⚠ THE DURABLE FINDING — `docker stop` does NOT checkpoint the SQLite WAL + + post_office.db 155 KB mtime 15:09 + post_office.db-wal 4.1 MB mtime 16:56 <- every recent message lived HERE + +I expected a clean container stop to checkpoint. **It did not** — after `docker stop` the WAL was +still 4,124,152 bytes, unchanged. An explicit `PRAGMA wal_checkpoint(TRUNCATE)` was required, +after which the .db grew 155 KB → 163,840 B and the WAL/-shm vanished. + +**A `docker cp` of `post_office.db` alone would have produced a database that opens cleanly, +passes `PRAGMA integrity_check`, serves the full 73-handle roster — and is missing the day's +mail. Nothing would have errored.** Only a row count distinguishes those two outcomes. + +**Procedure for moving any WAL-mode SQLite service: stop → EXPLICIT checkpoint → verify counts → +copy → verify counts again on the far side, before deleting anything.** Not stop → copy. Verified +8 messages / 73 handles / 2 nodes / 8 recipients at source, in the staged copy, and after seeding. + +## Repoint list (everything that names the address) + + ~/.config/systemd/user/althing-po-herald.service nh3-dev herald (user unit) + /etc/systemd/system/althing-po-herald.service nh3-extdev herald (system unit) + ~/.claude/statusline-command.sh the hardcoded statusline fallback + every session's ALTHING_POST_OFFICE + re-arm althing-listen + +⚠ **Do NOT blind-sed `10.100.10.50:8390` across the memory tree.** +[[reference_corviduo_dev_emergency_ops]] carries that exact string as a **Bifrost "affect" plane** +entry in the personal Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` — an unrelated service that +happens to share the port. Incidentally, moving the post office off nh3-dev:8390 also cleared a +latent collision with it. + +## Evidence the outage semantics work under a real outage + +During the gap the nh3-dev herald logged, verbatim: *"push outage on nh3-dev: the post office did +not answer, so push is DOWN on this box. **This is an outage, not an empty poke list.**"* Then +after the repoint: `10:00:27 INFO poked infra-ops on nh3-dev via fifo (rung 0)` — poke path +re-verified end to end. + +## Open follow-up + +**The image has no registry push.** It moves by `docker save | ssh | docker load`, so a rebuild +means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull; +this should join it. The old nh3-dev volume is left in place untouched — not a rollback path +(the operator ruled that out), just not deleting the only other copy on the day of the move. diff --git a/persistent-memory.md b/persistent-memory.md index 07036aa..73467cd 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -27,7 +27,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) | | `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) | | `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) | -| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.10.50:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | +| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) | | `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) | | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) | diff --git a/stacks/althing-post-office/compose.yaml b/stacks/althing-post-office/compose.yaml new file mode 100644 index 0000000..93b36c0 --- /dev/null +++ b/stacks/althing-post-office/compose.yaml @@ -0,0 +1,121 @@ +# althing v3 — the post office. The fleet's message bus. +# +# MOVED nh3-dev -> nh3-docker on 2026-08-28 (operator: "I want it on the docker +# machine — that was always the goal"). It was deployed to nh3-dev at the U9b flag +# day because the herald lives there; but the herald is the piece that MUST be +# host-local (it reads route files and pokes FIFOs), and the post office is +# explicitly the piece that is not. +# +# Three reasons the dev box was wrong: +# - our own server table calls nh3-dev "not a Docker-stack host", and NH-site +# non-GPU services belong here. +# - nh3-dev had three OOM events in fourteen days with the interval HALVING +# (2026-08-14, -26, -28), and the confirmed hog is Claude Code sessions at +# 5-18 GB each, which is that box's actual job. +# - `mem_limit` protects the fleet FROM the post office. It does nothing to +# protect the post office from the box: oom_score_adj is 0, so it was an +# ordinary kill candidate, and the 08-28 sweep took althing-herald and +# uvicorn. A sweep that takes the post office takes mail for all 73 handles. +# +# ⚠ MOVING THE DATA: `docker stop` does NOT checkpoint the WAL. Measured on the +# 08-28 move: post_office.db was 155 KB / mtime 15:09 while post_office.db-wal +# was 4.1 MB / mtime 16:56 — every recent message lived in the WAL. Copying the +# .db alone yields a database that opens cleanly, passes a smoke test, and is +# missing the day's mail. Stop the container, then +# `PRAGMA wal_checkpoint(TRUNCATE)` explicitly, then verify row counts on BOTH +# sides before deleting anything. +# +# Deploy: scripts/deploy-stack.sh nh3-docker althing-post-office +# Image: built from the althing repo's Dockerfile (vh/althing @ v3.0.0) and +# transferred with `docker save | ssh | docker load`. There is no +# registry push for this image yet — that is a follow-up, and until it +# exists a rebuild means repeating the save/load. + +services: + post-office: + image: althing-post-office:local + container_name: althing-post-office + + # ─── Host networking, so the bind guard keeps working ──────────── + # + # ⚠ DO NOT "fix" this into a bridged container with `-p`. api.py's + # resolve_bind_host refuses any address that resolves to a wildcard, and it + # tests the RESOLVED PROPERTY rather than matching strings, so there is no + # spelling of "everything" that gets past it. A bridged container cannot + # satisfy that guard honestly: inside its own netns the only reachable bind + # is a wildcard, and publishing the port would move access control from the + # address the application checks to a `-p` flag it cannot see. + # + # Reachability on the private network IS the authorisation story here — + # there is no login and none is wanted. + # + # Cost, stated plainly: no network namespace isolation, and port 8390 is + # claimed host-wide. For a single-service private-network deployment that is + # the right trade, but it IS a trade. + network_mode: host + + # The store is the only thing that must survive. Named volume rather than a + # bind mount: uid 1000 inside the container owns it, and docker creates it + # with the right ownership instead of inheriting the host path's. + volumes: + - post-office-data:/var/lib/althing + + # The private address of THIS box. Single place it is named; the image ships + # no default on purpose, so a deployment that omits it is refused at startup + # rather than binding wide. + environment: + ALTHING_BIND_HOST: "10.100.50.40" + ALTHING_PORT: "8390" + ALTHING_DB: "/var/lib/althing/post_office.db" + + # `unless-stopped` rather than `always` so an operator who deliberately stops + # it during a flag day does not find it running again after a reboot. + restart: unless-stopped + + # One Python interpreter holding one SQLite connection; it idles far below + # this. The cap is not a tuning parameter, it is a promise that the post + # office can never be its host's next OOM story. + # + # ⚠ nh3-docker runs Compose v5, which ignores `version:` and honours + # `mem_limit` directly. On a docker-compose 1.x host this needs schema 2.4 — + # under 3.x the key moves to `deploy:`, which is swarm-only and SILENTLY + # IGNORED. Verify with `docker inspect` (want 536870912), never by reading + # the yaml: a cap that does nothing reads as protection. + mem_limit: 512m + + # ⚠ The fleet's entire bus. Make the kernel shoot almost anything else first. + # This is the gap the nh3-dev deployment had: a 512m cap and oom_score_adj 0 + # means "cannot cause an OOM, is an ordinary victim of one". + oom_score_adj: -500 + + # Docker's default json-file driver has no size limit. v2's herald left a + # 60 MB log on nh3-dev; an uncapped container log is the same mistake with a + # different name. + logging: + driver: json-file + options: + max-size: "10m" + max-file: "5" + + # Inherited from the image, restated so it is visible at deploy time rather + # than only in `docker inspect`. + healthcheck: + test: + - CMD + - python + - -c + - "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://10.100.50.40:8390/',timeout=4).status==200 else 1)" + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + + # No homepage labels, deliberately. There is an operator page at :8390/, but + # no group in stacks/homepage/conf/settings.yaml's `layout:` fits it, and a + # group the layout has never heard of gets no `tab:` and renders on ALL FOUR + # tabs (the Scriberr "AI Systems" bug, 2026-08-23). Adding a group is a + # separate, deliberate change — not a side effect of moving a container. + +volumes: + post-office-data: + name: althing-post-office-data