esh-pfi-infrastructure/persistent-memory.d/2026-08-28-althing-v3-cutover.md

# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way

Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.

    post office   ONE container on nh3-dev, http://10.100.50.40:8390
                  the only stateful component. SQLite + FTS + the typed API + the operator page.
    herald        althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
                  holds no state. Refuses to start if another herald holds the node.
    waiter        althing-listen, one per session. Holds a FIFO, blocks, exits when poked.
    client        postbox  (+ althing-mcp for the stdio tool surface)

    althing-cli -> postbox                althing-wake-listener -> althing-listen
    althing-light-monitor -> GONE         althing-receiver      -> GONE
    althing-herald -> althing-po-herald

**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.50.40:8390` and
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
address** — a bare `postbox status` errors out rather than guessing.

## ⚠ An unreachable post office is an OUTAGE, never an empty inbox

v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if
postbox says it could not reach the post office, that is the fault. Do not read it as "no mail".

## Deployment facts worth not rediscovering

- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.**
  `api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the
  RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally
  impossible. Reachability on the private network IS the authorisation story; there is no login.
  Bridging would move access control to a `-p` flag the application cannot see.
- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no
  `docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it
  moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured:
  `docker inspect -> 536870912`.
- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at
  `/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were
  **SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at
  `/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp,
  `sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin <uv> tool install --force`.
  Playbook: `playbooks/nh3-extdev-althing-v3.yaml`.
- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service`
  (Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at
  `/etc/systemd/system/althing-po-herald.service` with `User=lkraven`.

## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT

5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five
thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and
never reaped them, so every pane that ever armed left one behind permanently. **v3 names them
per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup.

## The roster: 73, and the authoritative source is the CLI, not the DB

Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is
rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be.

⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy —
superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant,
and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes
@-qualification: identity no longer has a home, so an @-qualified handle is a category error.**
Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's
"72 rendered vs 73 declared" was a line-count artifact and the sets are identical.

## v2 history is inert, not migrated

`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path
exists and none should be improvised.** Plain SQLite if something must be recovered by hand.

## Two-agent flag day: the collision worth remembering

forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My
`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as
"already done at some point" rather than "someone is working in here right now." Cost was one
redundant backup. **Two agents worked the same checklist with no ownership marked per line.**
Next flag day: name an owner per item.

## Peers notified individually (operator-directed), NOT broadcast

`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md
copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev.
Operator's ruling on a general fleet announcement: **pointless in both directions — anyone
already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing
never-broadcast-unsolicited directive.

⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.

---

## `[2026-08-28, same day]` MOVED to nh3-docker — and the WAL nearly ate the mail

Operator: *"I want it on the docker machine — that was always the goal."* The flag-day
deployment put the post office on **nh3-dev**, which was wrong on three counts:

- our own server table calls nh3-dev **"not a Docker-stack host"**; NH-site non-GPU → nh3-docker
- nh3-dev had **three OOM events in fourteen days, interval halving**, and the confirmed hog is
  CC sessions at 5-18 GB — the box's actual job. See [[2026-08-28-nh3-dev-oom-attribution]].
- `mem_limit: 512m` protects the fleet **FROM** the post office. It does nothing to protect the
  post office **from the box**: `oom_score_adj` was 0, an ordinary kill candidate, and the 08-28
  sweep took althing-herald and uvicorn. A sweep taking the post office takes all 73 handles.

    NOW    http://10.100.50.40:8390   nh3-docker, oom_score_adj=-500, mem 512m verified honoured
    WAS    http://10.100.10.50:8390   nh3-dev  (address now refuses)
    canon  stacks/althing-post-office/compose.yaml -> /opt/docker/compose/ on nh3-docker

## ⚠ THE DURABLE FINDING — `docker stop` does NOT checkpoint the SQLite WAL

    post_office.db       155 KB  mtime 15:09
    post_office.db-wal   4.1 MB  mtime 16:56    <- every recent message lived HERE

I expected a clean container stop to checkpoint. **It did not** — after `docker stop` the WAL was
still 4,124,152 bytes, unchanged. An explicit `PRAGMA wal_checkpoint(TRUNCATE)` was required,
after which the .db grew 155 KB → 163,840 B and the WAL/-shm vanished.

**A `docker cp` of `post_office.db` alone would have produced a database that opens cleanly,
passes `PRAGMA integrity_check`, serves the full 73-handle roster — and is missing the day's
mail. Nothing would have errored.** Only a row count distinguishes those two outcomes.

**Procedure for moving any WAL-mode SQLite service: stop → EXPLICIT checkpoint → verify counts →
copy → verify counts again on the far side, before deleting anything.** Not stop → copy. Verified
8 messages / 73 handles / 2 nodes / 8 recipients at source, in the staged copy, and after seeding.

## Repoint list (everything that names the address)

    ~/.config/systemd/user/althing-po-herald.service        nh3-dev herald (user unit)
    /etc/systemd/system/althing-po-herald.service           nh3-extdev herald (system unit)
    ~/.claude/statusline-command.sh                         the hardcoded statusline fallback
    every session's ALTHING_POST_OFFICE                     + re-arm althing-listen

⚠ **Do NOT blind-sed `10.100.10.50:8390` across the memory tree.**
[[reference_corviduo_dev_emergency_ops]] carries that exact string as a **Bifrost "affect" plane**
entry in the personal Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` — an unrelated service that
happens to share the port. Incidentally, moving the post office off nh3-dev:8390 also cleared a
latent collision with it.

## Evidence the outage semantics work under a real outage

During the gap the nh3-dev herald logged, verbatim: *"push outage on nh3-dev: the post office did
not answer, so push is DOWN on this box. **This is an outage, not an empty poke list.**"* Then
after the repoint: `10:00:27 INFO poked infra-ops on nh3-dev via fifo (rung 0)` — poke path
re-verified end to end.

## Open follow-up

**The image has no registry push.** It moves by `docker save | ssh | docker load`, so a rebuild
means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull;
this should join it. The old nh3-dev volume is left in place untouched — not a rollback path
(the operator ruled that out), just not deleting the only other copy on the day of the move.

---

## `[2026-08-28, later]` The release train: 3.0.1 → 3.1.1 in one afternoon, and the registry

Six releases landed the same day as the cutover. **The container was touched exactly once** (the
nh3-docker move); every other release was herald- or client-side, established each time by
forseti's **import-graph argument** — asking what `althing/post_office/*` IMPORTS rather than
reading the diff. A diff tells you what moved; an import graph tells you what can be affected.

    3.0.1  pane routes (a pane agent can register its own route)
    3.0.2  the docs are now checked against the CLI, not against each other  — NO redeploy
    3.0.3  PANE_SETTLE_S=0.3, the write/submit race
    3.1.0  herald writes $ALTHING_ROOT/post-office; dev-launch reads it  — BOTH binaries
    3.1.1  warn when ALTHING_HANDLE disagrees with launch-history

**Deploy recipe per release:** `uv build --wheel` on nh3-dev → `uv tool install --force .`
locally → stage the wheel to nh3-extdev and install under `UV_TOOL_DIR=/opt/uv-tools` with
`/home/infra-ops/.local/bin/uv` → restart both `althing-po-herald` units.
`playbooks/nh3-extdev-althing-v3.yaml` does the extdev half.

## ⚠ THE REPEATED DEFECT — a verify that half-passes, four costumes in one day

Every one of these was written correctly for its first run and silently wrong on the next:

    1. install step gated on `postbox` not existing   -> right for the cutover, would SKIP
                                                          every release after and report success
    2. content check pinned to the PREVIOUS release's markers -> passes forever, asserts nothing
    3. ONE file:marker pair against a TWO-file release  -> asserts half a release
    4. a Go template `{{ }}` inside elway, whose own substitution ate it -> FAILED on a green deploy

**The check now matches on PRESENCE (`grep -q`), never a count**, and takes a LIST of
`file:marker` pairs bumped per release. 3.1.1's own release note said
`grep -c handles_launched_at dev_launch.py # 2+`; the real count there is 1 (the definition,
with 2 in postbox.py), so a count assertion would have reported FAILED on a byte-perfect install.

## The registry, and why the namespace is `claude-bot`

    gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0@sha256:410fed41...

Digest-pinned, not tag-floating — a tag is a mutable pointer on a registry anyone can re-push and
this container is the whole bus. ⚠ **claude-bot's token carries `write:package` and `docker
login` SUCCEEDS, but package namespaces are owned**: pushing to `vh/` returns
`unauthorized: authentication required` AFTER a successful login — an ownership refusal wearing a
credential error's clothes. Publishing under claude-bot's own namespace also satisfies the
standing directive to stop reusing the operator's personal credentials.

## The deployed CC plugin copies are a release step NOBODY owns

`sync_skill.sh` covers the SKILL, not the plugin. Nothing in the repo reaches
`~/.local/share/althing-plugin/` or `~/.claude/plugins/cache/althing/althing/0.0.1/`. Both must be
`rsync -a --delete`'d from the repo's `plugin/` **on every release**, by hand, or they carry the
previous release's bugs into the live surface — which happened: forseti's new docs-vs-CLI check
found `postbox reply --to` twice in `plugin/commands/inbox.md`, **the file a CC session reads
every time it drains its inbox**, and both my deployed copies had it.

⚠ **A running CC session keeps the plugin text it loaded at startup.** Files being right is
necessary and not sufficient — the session has to restart. I synced the copies at 09:30 and was
handed the v2 `/althing:monitor` text an hour later, calling three deleted binaries.

## Peer skill copies: I stopped hand-syncing, deliberately

I wrote into eitri/dvalin/bil's repos three times (operator-authorised, and right while they were
dark and could not pull). **Once they were awake and pulling, it became a race I was losing** —
canonical moved three times in an hour and I was chasing a one-line version-banner lag. It also
cost provenance: dvalin had to correct their own account of their file because my write looked
like a pre-existing partial. Two writers, no lock. `sync_skill.sh` deliberately does not write
into peer trees; althing owns canonical and peers pull. Respect that boundary once they can.