feat(althing): move the post office to nh3-docker
Operator directive, and a standing goal: the bus belongs on the docker host. The flag-day deployment put it on nh3-dev because the herald lives there -- but the herald is the piece that must be host-local, and the post office is explicitly the piece that is not. nh3-dev was wrong on three counts. Our own server table calls it "not a Docker-stack host". It has had three OOM events in fourteen days with the interval halving, and the confirmed hog is Claude Code sessions at 5-18 GB, which is that box's actual job. And mem_limit protects the fleet from the post office while doing nothing in the other direction: oom_score_adj was 0, an ordinary kill candidate, on a box whose last sweep took althing-herald and uvicorn. The new deployment sets oom_score_adj=-500. The compose is now version-controlled here as a normal stack rather than living only in the althing repo's deploy dir. ## docker stop does not checkpoint the WAL The database was 155 KB with a 4.1 MB write-ahead log, and every recent message was in the log. A clean container stop left it untouched -- an explicit PRAGMA wal_checkpoint(TRUNCATE) was required. A docker cp of the .db alone would have produced a database that opens cleanly, passes integrity_check, serves the full 73-handle roster, and is missing the day's mail, with nothing raising an error. Row counts were verified at source, in the staged copy, and after seeding, because the count is the only thing that separates those two outcomes. The old volume is left in place. Not a rollback path, which the operator ruled out -- just not deleting the only other copy on the day of a move. ## Follow-up left open The image has no registry push and moves by save/ssh/load, so a rebuild means repeating that by hand. It should join the gitea registry pattern the other stacks use.
This commit is contained in:
@@ -3,7 +3,7 @@
|
|||||||
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
|
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
|
||||||
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
|
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
|
||||||
|
|
||||||
post office ONE container on nh3-dev, http://10.100.10.50:8390
|
post office ONE container on nh3-dev, http://10.100.50.40:8390
|
||||||
the only stateful component. SQLite + FTS + the typed API + the operator page.
|
the only stateful component. SQLite + FTS + the typed API + the operator page.
|
||||||
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
|
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
|
||||||
holds no state. Refuses to start if another herald holds the node.
|
holds no state. Refuses to start if another herald holds the node.
|
||||||
@@ -14,7 +14,7 @@ deleted, not deprecated.** No rollback was designed or tested; failures are fixe
|
|||||||
althing-light-monitor -> GONE althing-receiver -> GONE
|
althing-light-monitor -> GONE althing-receiver -> GONE
|
||||||
althing-herald -> althing-po-herald
|
althing-herald -> althing-po-herald
|
||||||
|
|
||||||
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and
|
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.50.40:8390` and
|
||||||
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
|
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
|
||||||
address** — a bare `postbox status` errors out rather than guessing.
|
address** — a bare `postbox status` errors out rather than guessing.
|
||||||
|
|
||||||
@@ -86,3 +86,65 @@ never-broadcast-unsolicited directive.
|
|||||||
|
|
||||||
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
|
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
|
||||||
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
|
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## `[2026-08-28, same day]` MOVED to nh3-docker — and the WAL nearly ate the mail
|
||||||
|
|
||||||
|
Operator: *"I want it on the docker machine — that was always the goal."* The flag-day
|
||||||
|
deployment put the post office on **nh3-dev**, which was wrong on three counts:
|
||||||
|
|
||||||
|
- our own server table calls nh3-dev **"not a Docker-stack host"**; NH-site non-GPU → nh3-docker
|
||||||
|
- nh3-dev had **three OOM events in fourteen days, interval halving**, and the confirmed hog is
|
||||||
|
CC sessions at 5-18 GB — the box's actual job. See [[2026-08-28-nh3-dev-oom-attribution]].
|
||||||
|
- `mem_limit: 512m` protects the fleet **FROM** the post office. It does nothing to protect the
|
||||||
|
post office **from the box**: `oom_score_adj` was 0, an ordinary kill candidate, and the 08-28
|
||||||
|
sweep took althing-herald and uvicorn. A sweep taking the post office takes all 73 handles.
|
||||||
|
|
||||||
|
NOW http://10.100.50.40:8390 nh3-docker, oom_score_adj=-500, mem 512m verified honoured
|
||||||
|
WAS http://10.100.10.50:8390 nh3-dev (address now refuses)
|
||||||
|
canon stacks/althing-post-office/compose.yaml -> /opt/docker/compose/ on nh3-docker
|
||||||
|
|
||||||
|
## ⚠ THE DURABLE FINDING — `docker stop` does NOT checkpoint the SQLite WAL
|
||||||
|
|
||||||
|
post_office.db 155 KB mtime 15:09
|
||||||
|
post_office.db-wal 4.1 MB mtime 16:56 <- every recent message lived HERE
|
||||||
|
|
||||||
|
I expected a clean container stop to checkpoint. **It did not** — after `docker stop` the WAL was
|
||||||
|
still 4,124,152 bytes, unchanged. An explicit `PRAGMA wal_checkpoint(TRUNCATE)` was required,
|
||||||
|
after which the .db grew 155 KB → 163,840 B and the WAL/-shm vanished.
|
||||||
|
|
||||||
|
**A `docker cp` of `post_office.db` alone would have produced a database that opens cleanly,
|
||||||
|
passes `PRAGMA integrity_check`, serves the full 73-handle roster — and is missing the day's
|
||||||
|
mail. Nothing would have errored.** Only a row count distinguishes those two outcomes.
|
||||||
|
|
||||||
|
**Procedure for moving any WAL-mode SQLite service: stop → EXPLICIT checkpoint → verify counts →
|
||||||
|
copy → verify counts again on the far side, before deleting anything.** Not stop → copy. Verified
|
||||||
|
8 messages / 73 handles / 2 nodes / 8 recipients at source, in the staged copy, and after seeding.
|
||||||
|
|
||||||
|
## Repoint list (everything that names the address)
|
||||||
|
|
||||||
|
~/.config/systemd/user/althing-po-herald.service nh3-dev herald (user unit)
|
||||||
|
/etc/systemd/system/althing-po-herald.service nh3-extdev herald (system unit)
|
||||||
|
~/.claude/statusline-command.sh the hardcoded statusline fallback
|
||||||
|
every session's ALTHING_POST_OFFICE + re-arm althing-listen
|
||||||
|
|
||||||
|
⚠ **Do NOT blind-sed `10.100.10.50:8390` across the memory tree.**
|
||||||
|
[[reference_corviduo_dev_emergency_ops]] carries that exact string as a **Bifrost "affect" plane**
|
||||||
|
entry in the personal Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` — an unrelated service that
|
||||||
|
happens to share the port. Incidentally, moving the post office off nh3-dev:8390 also cleared a
|
||||||
|
latent collision with it.
|
||||||
|
|
||||||
|
## Evidence the outage semantics work under a real outage
|
||||||
|
|
||||||
|
During the gap the nh3-dev herald logged, verbatim: *"push outage on nh3-dev: the post office did
|
||||||
|
not answer, so push is DOWN on this box. **This is an outage, not an empty poke list.**"* Then
|
||||||
|
after the repoint: `10:00:27 INFO poked infra-ops on nh3-dev via fifo (rung 0)` — poke path
|
||||||
|
re-verified end to end.
|
||||||
|
|
||||||
|
## Open follow-up
|
||||||
|
|
||||||
|
**The image has no registry push.** It moves by `docker save | ssh | docker load`, so a rebuild
|
||||||
|
means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull;
|
||||||
|
this should join it. The old nh3-dev volume is left in place untouched — not a rollback path
|
||||||
|
(the operator ruled that out), just not deleting the only other copy on the day of the move.
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
|||||||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||||||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||||||
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
||||||
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.10.50:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
|
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
|
||||||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||||||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||||||
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
||||||
|
|||||||
@@ -0,0 +1,121 @@
|
|||||||
|
# althing v3 — the post office. The fleet's message bus.
|
||||||
|
#
|
||||||
|
# MOVED nh3-dev -> nh3-docker on 2026-08-28 (operator: "I want it on the docker
|
||||||
|
# machine — that was always the goal"). It was deployed to nh3-dev at the U9b flag
|
||||||
|
# day because the herald lives there; but the herald is the piece that MUST be
|
||||||
|
# host-local (it reads route files and pokes FIFOs), and the post office is
|
||||||
|
# explicitly the piece that is not.
|
||||||
|
#
|
||||||
|
# Three reasons the dev box was wrong:
|
||||||
|
# - our own server table calls nh3-dev "not a Docker-stack host", and NH-site
|
||||||
|
# non-GPU services belong here.
|
||||||
|
# - nh3-dev had three OOM events in fourteen days with the interval HALVING
|
||||||
|
# (2026-08-14, -26, -28), and the confirmed hog is Claude Code sessions at
|
||||||
|
# 5-18 GB each, which is that box's actual job.
|
||||||
|
# - `mem_limit` protects the fleet FROM the post office. It does nothing to
|
||||||
|
# protect the post office from the box: oom_score_adj is 0, so it was an
|
||||||
|
# ordinary kill candidate, and the 08-28 sweep took althing-herald and
|
||||||
|
# uvicorn. A sweep that takes the post office takes mail for all 73 handles.
|
||||||
|
#
|
||||||
|
# ⚠ MOVING THE DATA: `docker stop` does NOT checkpoint the WAL. Measured on the
|
||||||
|
# 08-28 move: post_office.db was 155 KB / mtime 15:09 while post_office.db-wal
|
||||||
|
# was 4.1 MB / mtime 16:56 — every recent message lived in the WAL. Copying the
|
||||||
|
# .db alone yields a database that opens cleanly, passes a smoke test, and is
|
||||||
|
# missing the day's mail. Stop the container, then
|
||||||
|
# `PRAGMA wal_checkpoint(TRUNCATE)` explicitly, then verify row counts on BOTH
|
||||||
|
# sides before deleting anything.
|
||||||
|
#
|
||||||
|
# Deploy: scripts/deploy-stack.sh nh3-docker althing-post-office
|
||||||
|
# Image: built from the althing repo's Dockerfile (vh/althing @ v3.0.0) and
|
||||||
|
# transferred with `docker save | ssh | docker load`. There is no
|
||||||
|
# registry push for this image yet — that is a follow-up, and until it
|
||||||
|
# exists a rebuild means repeating the save/load.
|
||||||
|
|
||||||
|
services:
|
||||||
|
post-office:
|
||||||
|
image: althing-post-office:local
|
||||||
|
container_name: althing-post-office
|
||||||
|
|
||||||
|
# ─── Host networking, so the bind guard keeps working ────────────
|
||||||
|
#
|
||||||
|
# ⚠ DO NOT "fix" this into a bridged container with `-p`. api.py's
|
||||||
|
# resolve_bind_host refuses any address that resolves to a wildcard, and it
|
||||||
|
# tests the RESOLVED PROPERTY rather than matching strings, so there is no
|
||||||
|
# spelling of "everything" that gets past it. A bridged container cannot
|
||||||
|
# satisfy that guard honestly: inside its own netns the only reachable bind
|
||||||
|
# is a wildcard, and publishing the port would move access control from the
|
||||||
|
# address the application checks to a `-p` flag it cannot see.
|
||||||
|
#
|
||||||
|
# Reachability on the private network IS the authorisation story here —
|
||||||
|
# there is no login and none is wanted.
|
||||||
|
#
|
||||||
|
# Cost, stated plainly: no network namespace isolation, and port 8390 is
|
||||||
|
# claimed host-wide. For a single-service private-network deployment that is
|
||||||
|
# the right trade, but it IS a trade.
|
||||||
|
network_mode: host
|
||||||
|
|
||||||
|
# The store is the only thing that must survive. Named volume rather than a
|
||||||
|
# bind mount: uid 1000 inside the container owns it, and docker creates it
|
||||||
|
# with the right ownership instead of inheriting the host path's.
|
||||||
|
volumes:
|
||||||
|
- post-office-data:/var/lib/althing
|
||||||
|
|
||||||
|
# The private address of THIS box. Single place it is named; the image ships
|
||||||
|
# no default on purpose, so a deployment that omits it is refused at startup
|
||||||
|
# rather than binding wide.
|
||||||
|
environment:
|
||||||
|
ALTHING_BIND_HOST: "10.100.50.40"
|
||||||
|
ALTHING_PORT: "8390"
|
||||||
|
ALTHING_DB: "/var/lib/althing/post_office.db"
|
||||||
|
|
||||||
|
# `unless-stopped` rather than `always` so an operator who deliberately stops
|
||||||
|
# it during a flag day does not find it running again after a reboot.
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
|
# One Python interpreter holding one SQLite connection; it idles far below
|
||||||
|
# this. The cap is not a tuning parameter, it is a promise that the post
|
||||||
|
# office can never be its host's next OOM story.
|
||||||
|
#
|
||||||
|
# ⚠ nh3-docker runs Compose v5, which ignores `version:` and honours
|
||||||
|
# `mem_limit` directly. On a docker-compose 1.x host this needs schema 2.4 —
|
||||||
|
# under 3.x the key moves to `deploy:`, which is swarm-only and SILENTLY
|
||||||
|
# IGNORED. Verify with `docker inspect` (want 536870912), never by reading
|
||||||
|
# the yaml: a cap that does nothing reads as protection.
|
||||||
|
mem_limit: 512m
|
||||||
|
|
||||||
|
# ⚠ The fleet's entire bus. Make the kernel shoot almost anything else first.
|
||||||
|
# This is the gap the nh3-dev deployment had: a 512m cap and oom_score_adj 0
|
||||||
|
# means "cannot cause an OOM, is an ordinary victim of one".
|
||||||
|
oom_score_adj: -500
|
||||||
|
|
||||||
|
# Docker's default json-file driver has no size limit. v2's herald left a
|
||||||
|
# 60 MB log on nh3-dev; an uncapped container log is the same mistake with a
|
||||||
|
# different name.
|
||||||
|
logging:
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "5"
|
||||||
|
|
||||||
|
# Inherited from the image, restated so it is visible at deploy time rather
|
||||||
|
# than only in `docker inspect`.
|
||||||
|
healthcheck:
|
||||||
|
test:
|
||||||
|
- CMD
|
||||||
|
- python
|
||||||
|
- -c
|
||||||
|
- "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://10.100.50.40:8390/',timeout=4).status==200 else 1)"
|
||||||
|
interval: 30s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 3
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
|
# No homepage labels, deliberately. There is an operator page at :8390/, but
|
||||||
|
# no group in stacks/homepage/conf/settings.yaml's `layout:` fits it, and a
|
||||||
|
# group the layout has never heard of gets no `tab:` and renders on ALL FOUR
|
||||||
|
# tabs (the Scriberr "AI Systems" bug, 2026-08-23). Adding a group is a
|
||||||
|
# separate, deliberate change — not a side effect of moving a container.
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
post-office-data:
|
||||||
|
name: althing-post-office-data
|
||||||
Reference in New Issue
Block a user