vh
9d4e7bd34a
Operator directive, and a standing goal: the bus belongs on the docker host. The flag-day deployment put it on nh3-dev because the herald lives there -- but the herald is the piece that must be host-local, and the post office is explicitly the piece that is not. nh3-dev was wrong on three counts. Our own server table calls it "not a Docker-stack host". It has had three OOM events in fourteen days with the interval halving, and the confirmed hog is Claude Code sessions at 5-18 GB, which is that box's actual job. And mem_limit protects the fleet from the post office while doing nothing in the other direction: oom_score_adj was 0, an ordinary kill candidate, on a box whose last sweep took althing-herald and uvicorn. The new deployment sets oom_score_adj=-500. The compose is now version-controlled here as a normal stack rather than living only in the althing repo's deploy dir. ## docker stop does not checkpoint the WAL The database was 155 KB with a 4.1 MB write-ahead log, and every recent message was in the log. A clean container stop left it untouched -- an explicit PRAGMA wal_checkpoint(TRUNCATE) was required. A docker cp of the .db alone would have produced a database that opens cleanly, passes integrity_check, serves the full 73-handle roster, and is missing the day's mail, with nothing raising an error. Row counts were verified at source, in the staged copy, and after seeding, because the count is the only thing that separates those two outcomes. The old volume is left in place. Not a rollback path, which the operator ruled out -- just not deleting the only other copy on the day of a move. ## Follow-up left open The image has no registry push and moves by save/ssh/load, so a rebuild means repeating that by hand. It should join the gitea registry pattern the other stacks use.
122 lines
5.7 KiB
YAML
122 lines
5.7 KiB
YAML
# althing v3 — the post office. The fleet's message bus.
|
|
#
|
|
# MOVED nh3-dev -> nh3-docker on 2026-08-28 (operator: "I want it on the docker
|
|
# machine — that was always the goal"). It was deployed to nh3-dev at the U9b flag
|
|
# day because the herald lives there; but the herald is the piece that MUST be
|
|
# host-local (it reads route files and pokes FIFOs), and the post office is
|
|
# explicitly the piece that is not.
|
|
#
|
|
# Three reasons the dev box was wrong:
|
|
# - our own server table calls nh3-dev "not a Docker-stack host", and NH-site
|
|
# non-GPU services belong here.
|
|
# - nh3-dev had three OOM events in fourteen days with the interval HALVING
|
|
# (2026-08-14, -26, -28), and the confirmed hog is Claude Code sessions at
|
|
# 5-18 GB each, which is that box's actual job.
|
|
# - `mem_limit` protects the fleet FROM the post office. It does nothing to
|
|
# protect the post office from the box: oom_score_adj is 0, so it was an
|
|
# ordinary kill candidate, and the 08-28 sweep took althing-herald and
|
|
# uvicorn. A sweep that takes the post office takes mail for all 73 handles.
|
|
#
|
|
# ⚠ MOVING THE DATA: `docker stop` does NOT checkpoint the WAL. Measured on the
|
|
# 08-28 move: post_office.db was 155 KB / mtime 15:09 while post_office.db-wal
|
|
# was 4.1 MB / mtime 16:56 — every recent message lived in the WAL. Copying the
|
|
# .db alone yields a database that opens cleanly, passes a smoke test, and is
|
|
# missing the day's mail. Stop the container, then
|
|
# `PRAGMA wal_checkpoint(TRUNCATE)` explicitly, then verify row counts on BOTH
|
|
# sides before deleting anything.
|
|
#
|
|
# Deploy: scripts/deploy-stack.sh nh3-docker althing-post-office
|
|
# Image: built from the althing repo's Dockerfile (vh/althing @ v3.0.0) and
|
|
# transferred with `docker save | ssh | docker load`. There is no
|
|
# registry push for this image yet — that is a follow-up, and until it
|
|
# exists a rebuild means repeating the save/load.
|
|
|
|
services:
|
|
post-office:
|
|
image: althing-post-office:local
|
|
container_name: althing-post-office
|
|
|
|
# ─── Host networking, so the bind guard keeps working ────────────
|
|
#
|
|
# ⚠ DO NOT "fix" this into a bridged container with `-p`. api.py's
|
|
# resolve_bind_host refuses any address that resolves to a wildcard, and it
|
|
# tests the RESOLVED PROPERTY rather than matching strings, so there is no
|
|
# spelling of "everything" that gets past it. A bridged container cannot
|
|
# satisfy that guard honestly: inside its own netns the only reachable bind
|
|
# is a wildcard, and publishing the port would move access control from the
|
|
# address the application checks to a `-p` flag it cannot see.
|
|
#
|
|
# Reachability on the private network IS the authorisation story here —
|
|
# there is no login and none is wanted.
|
|
#
|
|
# Cost, stated plainly: no network namespace isolation, and port 8390 is
|
|
# claimed host-wide. For a single-service private-network deployment that is
|
|
# the right trade, but it IS a trade.
|
|
network_mode: host
|
|
|
|
# The store is the only thing that must survive. Named volume rather than a
|
|
# bind mount: uid 1000 inside the container owns it, and docker creates it
|
|
# with the right ownership instead of inheriting the host path's.
|
|
volumes:
|
|
- post-office-data:/var/lib/althing
|
|
|
|
# The private address of THIS box. Single place it is named; the image ships
|
|
# no default on purpose, so a deployment that omits it is refused at startup
|
|
# rather than binding wide.
|
|
environment:
|
|
ALTHING_BIND_HOST: "10.100.50.40"
|
|
ALTHING_PORT: "8390"
|
|
ALTHING_DB: "/var/lib/althing/post_office.db"
|
|
|
|
# `unless-stopped` rather than `always` so an operator who deliberately stops
|
|
# it during a flag day does not find it running again after a reboot.
|
|
restart: unless-stopped
|
|
|
|
# One Python interpreter holding one SQLite connection; it idles far below
|
|
# this. The cap is not a tuning parameter, it is a promise that the post
|
|
# office can never be its host's next OOM story.
|
|
#
|
|
# ⚠ nh3-docker runs Compose v5, which ignores `version:` and honours
|
|
# `mem_limit` directly. On a docker-compose 1.x host this needs schema 2.4 —
|
|
# under 3.x the key moves to `deploy:`, which is swarm-only and SILENTLY
|
|
# IGNORED. Verify with `docker inspect` (want 536870912), never by reading
|
|
# the yaml: a cap that does nothing reads as protection.
|
|
mem_limit: 512m
|
|
|
|
# ⚠ The fleet's entire bus. Make the kernel shoot almost anything else first.
|
|
# This is the gap the nh3-dev deployment had: a 512m cap and oom_score_adj 0
|
|
# means "cannot cause an OOM, is an ordinary victim of one".
|
|
oom_score_adj: -500
|
|
|
|
# Docker's default json-file driver has no size limit. v2's herald left a
|
|
# 60 MB log on nh3-dev; an uncapped container log is the same mistake with a
|
|
# different name.
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "5"
|
|
|
|
# Inherited from the image, restated so it is visible at deploy time rather
|
|
# than only in `docker inspect`.
|
|
healthcheck:
|
|
test:
|
|
- CMD
|
|
- python
|
|
- -c
|
|
- "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://10.100.50.40:8390/',timeout=4).status==200 else 1)"
|
|
interval: 30s
|
|
timeout: 5s
|
|
retries: 3
|
|
start_period: 10s
|
|
|
|
# No homepage labels, deliberately. There is an operator page at :8390/, but
|
|
# no group in stacks/homepage/conf/settings.yaml's `layout:` fits it, and a
|
|
# group the layout has never heard of gets no `tab:` and renders on ALL FOUR
|
|
# tabs (the Scriberr "AI Systems" bug, 2026-08-23). Adding a group is a
|
|
# separate, deliberate change — not a side effect of moving a container.
|
|
|
|
volumes:
|
|
post-office-data:
|
|
name: althing-post-office-data
|