vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(althing): move the post office to nh3-docker

Browse Source 
Operator directive, and a standing goal: the bus belongs on the docker
host. The flag-day deployment put it on nh3-dev because the herald lives
there -- but the herald is the piece that must be host-local, and the
post office is explicitly the piece that is not.

nh3-dev was wrong on three counts. Our own server table calls it "not a
Docker-stack host". It has had three OOM events in fourteen days with
the interval halving, and the confirmed hog is Claude Code sessions at
5-18 GB, which is that box's actual job. And mem_limit protects the
fleet from the post office while doing nothing in the other direction:
oom_score_adj was 0, an ordinary kill candidate, on a box whose last
sweep took althing-herald and uvicorn. The new deployment sets
oom_score_adj=-500.

The compose is now version-controlled here as a normal stack rather than
living only in the althing repo's deploy dir.

## docker stop does not checkpoint the WAL

The database was 155 KB with a 4.1 MB write-ahead log, and every recent
message was in the log. A clean container stop left it untouched -- an
explicit PRAGMA wal_checkpoint(TRUNCATE) was required.

A docker cp of the .db alone would have produced a database that opens
cleanly, passes integrity_check, serves the full 73-handle roster, and
is missing the day's mail, with nothing raising an error. Row counts
were verified at source, in the staged copy, and after seeding, because
the count is the only thing that separates those two outcomes.

The old volume is left in place. Not a rollback path, which the operator
ruled out -- just not deleting the only other copy on the day of a move.

## Follow-up left open

The image has no registry push and moves by save/ssh/load, so a rebuild
means repeating that by hand. It should join the gitea registry pattern
the other stacks use.
This commit is contained in:
vh
2026-08-28 10:02:13 -07:00
parent e58360668e
commit 9d4e7bd34a
3 changed files with 186 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-08-28-althing-v3-cutover.md
+64 -2
View File
@@ -3,7 +3,7 @@
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
post office ONE container on nh3-dev, http://10.100.10.50:8390
post office ONE container on nh3-dev, http://10.100.50.40:8390
the only stateful component. SQLite + FTS + the typed API + the operator page.
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
holds no state. Refuses to start if another herald holds the node.
@@ -14,7 +14,7 @@ deleted, not deprecated.** No rollback was designed or tested; failures are fixe
althing-light-monitor -> GONE althing-receiver -> GONE
althing-herald -> althing-po-herald
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.50.40:8390` and
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
address** — a bare `postbox status` errors out rather than guessing.
@@ -86,3 +86,65 @@ never-broadcast-unsolicited directive.
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
---
## `[2026-08-28, same day]` MOVED to nh3-docker — and the WAL nearly ate the mail
Operator: *"I want it on the docker machine — that was always the goal."* The flag-day
deployment put the post office on **nh3-dev**, which was wrong on three counts:
- our own server table calls nh3-dev **"not a Docker-stack host"**; NH-site non-GPU → nh3-docker
- nh3-dev had **three OOM events in fourteen days, interval halving**, and the confirmed hog is
CC sessions at 5-18 GB — the box's actual job. See [[2026-08-28-nh3-dev-oom-attribution]].
- `mem_limit: 512m` protects the fleet **FROM** the post office. It does nothing to protect the
post office **from the box**: `oom_score_adj` was 0, an ordinary kill candidate, and the 08-28
sweep took althing-herald and uvicorn. A sweep taking the post office takes all 73 handles.
NOW http://10.100.50.40:8390 nh3-docker, oom_score_adj=-500, mem 512m verified honoured
WAS http://10.100.10.50:8390 nh3-dev (address now refuses)
canon stacks/althing-post-office/compose.yaml -> /opt/docker/compose/ on nh3-docker
## ⚠ THE DURABLE FINDING — `docker stop` does NOT checkpoint the SQLite WAL
post_office.db 155 KB mtime 15:09
post_office.db-wal 4.1 MB mtime 16:56 <- every recent message lived HERE
I expected a clean container stop to checkpoint. **It did not** — after `docker stop` the WAL was
still 4,124,152 bytes, unchanged. An explicit `PRAGMA wal_checkpoint(TRUNCATE)` was required,
after which the .db grew 155 KB → 163,840 B and the WAL/-shm vanished.
**A `docker cp` of `post_office.db` alone would have produced a database that opens cleanly,
passes `PRAGMA integrity_check`, serves the full 73-handle roster — and is missing the day's
mail. Nothing would have errored.** Only a row count distinguishes those two outcomes.
**Procedure for moving any WAL-mode SQLite service: stop → EXPLICIT checkpoint → verify counts →
copy → verify counts again on the far side, before deleting anything.** Not stop → copy. Verified
8 messages / 73 handles / 2 nodes / 8 recipients at source, in the staged copy, and after seeding.
## Repoint list (everything that names the address)
~/.config/systemd/user/althing-po-herald.service nh3-dev herald (user unit)
/etc/systemd/system/althing-po-herald.service nh3-extdev herald (system unit)
~/.claude/statusline-command.sh the hardcoded statusline fallback
every session's ALTHING_POST_OFFICE + re-arm althing-listen
⚠ **Do NOT blind-sed `10.100.10.50:8390` across the memory tree.**
[[reference_corviduo_dev_emergency_ops]] carries that exact string as a **Bifrost "affect" plane**
entry in the personal Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` — an unrelated service that
happens to share the port. Incidentally, moving the post office off nh3-dev:8390 also cleared a
latent collision with it.
## Evidence the outage semantics work under a real outage
During the gap the nh3-dev herald logged, verbatim: *"push outage on nh3-dev: the post office did
not answer, so push is DOWN on this box. **This is an outage, not an empty poke list.**"* Then
after the repoint: `10:00:27 INFO poked infra-ops on nh3-dev via fifo (rung 0)` — poke path
re-verified end to end.
## Open follow-up
**The image has no registry push.** It moves by `docker save | ssh | docker load`, so a rebuild
means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull;
this should join it. The old nh3-dev volume is left in place untouched — not a rollback path
(the operator ruled that out), just not deleting the only other copy on the day of the move.