feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0

Two operator-authorised changes on the same afternoon.

## althing v3 (U9b flag day, one-way, no rollback)

The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.

73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.

Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.

nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.

The v2 database is left inert on disk. There is no import path and none
was improvised.

## sec onto GPU0

GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.

Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
This commit is contained in:
vh
2026-08-28 07:19:18 -07:00
parent f875f746b8
commit e58360668e
6 changed files with 290 additions and 7 deletions
@@ -0,0 +1,88 @@
# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
post office ONE container on nh3-dev, http://10.100.10.50:8390
the only stateful component. SQLite + FTS + the typed API + the operator page.
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
holds no state. Refuses to start if another herald holds the node.
waiter althing-listen, one per session. Holds a FIFO, blocks, exits when poked.
client postbox (+ althing-mcp for the stdio tool surface)
althing-cli -> postbox althing-wake-listener -> althing-listen
althing-light-monitor -> GONE althing-receiver -> GONE
althing-herald -> althing-po-herald
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
address** — a bare `postbox status` errors out rather than guessing.
## ⚠ An unreachable post office is an OUTAGE, never an empty inbox
v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if
postbox says it could not reach the post office, that is the fault. Do not read it as "no mail".
## Deployment facts worth not rediscovering
- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.**
`api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the
RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally
impossible. Reachability on the private network IS the authorisation story; there is no login.
Bridging would move access control to a `-p` flag the application cannot see.
- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no
`docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it
moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured:
`docker inspect -> 536870912`.
- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at
`/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were
**SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at
`/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp,
`sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin <uv> tool install --force`.
Playbook: `playbooks/nh3-extdev-althing-v3.yaml`.
- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service`
(Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at
`/etc/systemd/system/althing-po-herald.service` with `User=lkraven`.
## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT
5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five
thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and
never reaped them, so every pane that ever armed left one behind permanently. **v3 names them
per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup.
## The roster: 73, and the authoritative source is the CLI, not the DB
Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is
rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be.
⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy —
superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant,
and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes
@-qualification: identity no longer has a home, so an @-qualified handle is a category error.**
Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's
"72 rendered vs 73 declared" was a line-count artifact and the sets are identical.
## v2 history is inert, not migrated
`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path
exists and none should be improvised.** Plain SQLite if something must be recovered by hand.
## Two-agent flag day: the collision worth remembering
forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My
`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as
"already done at some point" rather than "someone is working in here right now." Cost was one
redundant backup. **Two agents worked the same checklist with no ownership marked per line.**
Next flag day: name an owner per item.
## Peers notified individually (operator-directed), NOT broadcast
`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md
copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev.
Operator's ruling on a general fleet announcement: **pointless in both directions — anyone
already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing
never-broadcast-unsolicited directive.
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.