feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
This commit is contained in:
@@ -0,0 +1,88 @@
|
||||
# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
|
||||
|
||||
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
|
||||
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
|
||||
|
||||
post office ONE container on nh3-dev, http://10.100.10.50:8390
|
||||
the only stateful component. SQLite + FTS + the typed API + the operator page.
|
||||
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
|
||||
holds no state. Refuses to start if another herald holds the node.
|
||||
waiter althing-listen, one per session. Holds a FIFO, blocks, exits when poked.
|
||||
client postbox (+ althing-mcp for the stdio tool surface)
|
||||
|
||||
althing-cli -> postbox althing-wake-listener -> althing-listen
|
||||
althing-light-monitor -> GONE althing-receiver -> GONE
|
||||
althing-herald -> althing-po-herald
|
||||
|
||||
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.10.50:8390` and
|
||||
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
|
||||
address** — a bare `postbox status` errors out rather than guessing.
|
||||
|
||||
## ⚠ An unreachable post office is an OUTAGE, never an empty inbox
|
||||
|
||||
v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if
|
||||
postbox says it could not reach the post office, that is the fault. Do not read it as "no mail".
|
||||
|
||||
## Deployment facts worth not rediscovering
|
||||
|
||||
- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.**
|
||||
`api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the
|
||||
RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally
|
||||
impossible. Reachability on the private network IS the authorisation story; there is no login.
|
||||
Bridging would move access control to a `-p` flag the application cannot see.
|
||||
- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no
|
||||
`docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it
|
||||
moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured:
|
||||
`docker inspect -> 536870912`.
|
||||
- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at
|
||||
`/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were
|
||||
**SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at
|
||||
`/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp,
|
||||
`sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin <uv> tool install --force`.
|
||||
Playbook: `playbooks/nh3-extdev-althing-v3.yaml`.
|
||||
- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service`
|
||||
(Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at
|
||||
`/etc/systemd/system/althing-po-herald.service` with `User=lkraven`.
|
||||
|
||||
## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT
|
||||
|
||||
5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five
|
||||
thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and
|
||||
never reaped them, so every pane that ever armed left one behind permanently. **v3 names them
|
||||
per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup.
|
||||
|
||||
## The roster: 73, and the authoritative source is the CLI, not the DB
|
||||
|
||||
Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is
|
||||
rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be.
|
||||
|
||||
⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy —
|
||||
superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant,
|
||||
and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes
|
||||
@-qualification: identity no longer has a home, so an @-qualified handle is a category error.**
|
||||
Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's
|
||||
"72 rendered vs 73 declared" was a line-count artifact and the sets are identical.
|
||||
|
||||
## v2 history is inert, not migrated
|
||||
|
||||
`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path
|
||||
exists and none should be improvised.** Plain SQLite if something must be recovered by hand.
|
||||
|
||||
## Two-agent flag day: the collision worth remembering
|
||||
|
||||
forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My
|
||||
`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as
|
||||
"already done at some point" rather than "someone is working in here right now." Cost was one
|
||||
redundant backup. **Two agents worked the same checklist with no ownership marked per line.**
|
||||
Next flag day: name an owner per item.
|
||||
|
||||
## Peers notified individually (operator-directed), NOT broadcast
|
||||
|
||||
`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md
|
||||
copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev.
|
||||
Operator's ruling on a general fleet announcement: **pointless in both directions — anyone
|
||||
already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing
|
||||
never-broadcast-unsolicited directive.
|
||||
|
||||
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
|
||||
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
|
||||
Reference in New Issue
Block a user