feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0

Two operator-authorised changes on the same afternoon.

## althing v3 (U9b flag day, one-way, no rollback)

The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.

73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.

Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.

nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.

The v2 database is left inert on disk. There is no import path and none
was improvised.

## sec onto GPU0

GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.

Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.

playbooks/mog-sec-move-to-gpu0.yaml

@@ -0,0 +1,80 @@
+# Move the `sec` pen-test seat (M.O.G.-SEC-27B) from ana-ml2 GPU 1 to GPU 0 and bring it up.
+#
+# Why: GPU 1 carries the five resident fleet seats (gen 46 GB + embed 9.8 + coder 8.4 +
+# rerank 3.5 + reward 2.1 = ~69.9 GB of 97.9), leaving ~28 GB. This seat reserves
+# MOG_GPU_MEM_UTIL=0.52 -> ~51 GB, so it could not start on GPU 1 at all. GPU 0 has been
+# idle since run 3c was stopped on 2026-08-26. Operator-directed 2026-08-28.
+#
+# ⚠ POWER. This re-arms the two-GPU load condition that tripped the Anaheim rack breaker
+# on 2026-08-26. One circuit feeds the whole rack including ana-gw and ana-wg, so a trip
+# costs the site AND the way back in. Idle draw is negligible; the risk materialises when
+# sec and gen are under concurrent load. Operator accepted this with the constraint stated.
+#
+# Labels only apply at container CREATION, so this uses `up -d`, never `restart` --
+# the homepage description carries the GPU number and would otherwise stay stale.
+#
+# Run:  scripts/elway infra-ops@ana-ml2 --playbook playbooks/mog-sec-move-to-gpu0.yaml
+# Model load is slow (22 GB + 262K ctx + MTP graft); the verify phase polls rather than
+# assuming readiness, and the compose healthcheck allows a 900s start_period.
+
+vars:
+  stack_dir: /opt/docker/compose/mog-sec
+  container: vllm-mog-sec
+  service: vllm-mog-sec
+  gpu_id: "0"
+  port: "8019"
+  staging: /tmp/mog-sec-compose.yaml
+
+steps:
+  - name: Stage the updated compose (GPU pin default + label now say GPU 0)
+    upload:
+      src: stacks/mog-sec/compose.yaml
+      dest: "{{ staging }}"
+      mode: "0644"
+
+  - name: Install it over the deployed copy
+    # /opt/docker/compose is root-owned, so the scp above lands in /tmp and this
+    # promotes it. Verified byte-identical against the deployed file beforehand:
+    # the only diff was these edits, so nothing on the host is being clobbered.
+    shell: sudo install -o root -g root -m 0644 {{ staging }} {{ stack_dir }}/compose.yaml
+    changed_when: "! sudo cmp -s {{ staging }} {{ stack_dir }}/compose.yaml"
+
+  - name: Pin the seat to GPU {{ gpu_id }} in the host .env
+    # The .env is the tunable surface and is NOT in git (secrets//tunables are
+    # excluded both directions). The compose default now matches, but the .env
+    # is what actually decides, so set it explicitly rather than relying on the
+    # default resolving.
+    shell: sudo sed -i 's/^MOG_GPU_ID=.*/MOG_GPU_ID={{ gpu_id }}/' {{ stack_dir }}/.env
+    when: "! sudo grep -qxF 'MOG_GPU_ID={{ gpu_id }}' {{ stack_dir }}/.env"
+
+  - name: Bring the seat up (up -d, not restart — labels apply at creation)
+    shell: cd {{ stack_dir }} && sudo docker compose up -d {{ service }}
+
+verify:
+  - name: Container exists and is running
+    # ⚠ No `docker inspect -f` here. Go templates use {{ }} and so does elway's own
+    # variable substitution, so an inspect format string gets eaten before it reaches
+    # the host -- these two checks reported FAILED on a deploy that had in fact
+    # succeeded. Filter-and-grep has no such collision.
+    shell: sudo docker ps --filter name={{ container }} --filter status=running --quiet | grep -q .
+    changed_when: "false"
+
+  - name: The container is actually pinned to GPU {{ gpu_id }}
+    # Assert the EFFECTIVE device reservation on the running container, not the
+    # .env string we wrote -- the .env is an input, this is the outcome.
+    shell: sudo docker inspect {{ container }} | tr -d ' \n' | grep -q '"DeviceIDs":\["{{ gpu_id }}"\]'
+    changed_when: "false"
+
+  - name: GPU 0 now holds a vLLM process (the seat really loaded onto that card)
+    shell: nvidia-smi --id={{ gpu_id }} --query-compute-apps=pid,used_memory --format=csv,noheader | grep -qE '[0-9]'
+    changed_when: "false"
+
+  - name: Health endpoint answers
+    shell: curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/health >/dev/null
+    changed_when: "false"
+
+  - name: Both served names are advertised (base + thinking)
+    shell: |
+      MODELS=$(curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/v1/models)
+      echo "$MODELS" | grep -q 'mog-sec-27b' && echo "$MODELS" | grep -q 'mog-sec-27b-thinking'
+    changed_when: "false"

playbooks/nh3-extdev-althing-v3.yaml

@@ -0,0 +1,104 @@
+# nh3-extdev: cut the system-wide althing install over from v2.1.0 to v3.0.0 (U9b flag day).
+#
+# nh3-extdev is the one box a `git pull` cannot move: althing lives there as a system WHEEL
+# under /opt/uv-tools/althing-core with entry points in /usr/local/bin, installed from a wheel
+# that was copied to /tmp -- not from a checkout. So it needs its own install or it goes dark
+# at the cutover.
+#
+# ⚠ Two things about this box that differ from nh3-dev:
+#   - the v2 daemons are SYSTEM units here (althing-herald, althing-receiver), not user units.
+#   - `uv` is not on lkraven's PATH; it lives at /home/infra-ops/.local/bin/uv. The original
+#     install used it under sudo with UV_TOOL_DIR=/opt/uv-tools, per the uv-receipt.toml.
+#
+# ⚠ There is a live agent session here (ldp-dev) holding a v2 light-monitor. Retiring the v2
+# herald does not kill it, but it will never fire again -- that session has to re-arm on
+# althing-listen after this. Its handle survives: bare `ldp-dev` is in the authoritative 73;
+# only the machine-qualified `ldp-dev@nh3-extdev` was on the legacy exclusion list.
+#
+# Run:  scripts/elway lkraven@10.100.50.42 --playbook playbooks/nh3-extdev-althing-v3.yaml
+# Rerunnable: a second run shows the install and unit steps skipped.
+
+vars:
+  wheel_src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl
+  wheel_dest: /tmp/althing_core-3.0.0-py3-none-any.whl
+  uv: /home/infra-ops/.local/bin/uv
+  tool_dir: /opt/uv-tools
+  bin_dir: /usr/local/bin
+  post_office: http://10.100.10.50:8390
+
+steps:
+  - name: Stage the v3.0.0 wheel
+    upload:
+      src: /home/lkraven/development/althing/dist/althing_core-3.0.0-py3-none-any.whl
+      dest: "{{ wheel_dest }}"
+      mode: "0644"
+
+  - name: Retire the v2 system daemons BEFORE swapping the package
+    # Order matters: these run out of /opt/uv-tools/althing-core/bin/python, which the
+    # install is about to replace. Stopping first means they never see a half-swapped tree.
+    # v3 has no counterpart to either -- the post office replaced the herald and deleted the
+    # reason for the receiver, since there is no longer a mailbox per machine to deliver between.
+    shell: sudo systemctl disable --now althing-herald.service althing-receiver.service
+    when: "systemctl is-active --quiet althing-herald.service || systemctl is-active --quiet althing-receiver.service"
+
+  - name: Install althing-core 3.0.0 over the system wheel install
+    shell: sudo env UV_TOOL_DIR={{ tool_dir }} UV_TOOL_BIN_DIR={{ bin_dir }} {{ uv }} tool install --force {{ wheel_dest }}
+    when: "! {{ bin_dir }}/postbox --help >/dev/null 2>&1"
+
+  - name: Install the post-office herald as a system unit
+    # A system unit rather than a user unit because that is how v2 was supervised here and
+    # because this box has no lingering user session to hang a --user unit from.
+    shell: |
+      printf '%s\n' \
+        '[Unit]' \
+        'Description=Althing post-office herald — per-machine relay (v3)' \
+        'Documentation=https://gitea.phasefinal.com/vh/althing' \
+        'After=network-online.target' \
+        'Wants=network-online.target' \
+        '' \
+        '[Service]' \
+        'Type=simple' \
+        'User=lkraven' \
+        'Environment=ALTHING_POST_OFFICE={{ post_office }}' \
+        'ExecStart={{ bin_dir }}/althing-po-herald' \
+        'Restart=always' \
+        'RestartSec=5' \
+        '' \
+        '# Dials out, opens no port, holds no state. Refuses to start if another herald' \
+        '# already holds this node — two would double every poke and both write liveness.' \
+        '# Replaces althing-herald.service + althing-receiver.service, retired 2026-08-28.' \
+        '' \
+        '[Install]' \
+        'WantedBy=multi-user.target' \
+      | sudo tee /etc/systemd/system/althing-po-herald.service >/dev/null
+      sudo systemctl daemon-reload
+    when: "! test -f /etc/systemd/system/althing-po-herald.service"
+
+  - name: Enable and start the herald
+    shell: sudo systemctl enable --now althing-po-herald.service
+
+verify:
+  - name: postbox is installed and is v3
+    shell: "{{ bin_dir }}/postbox --help | grep -q 'send,reply,read,peek,thread,search,status,handles,register,sign-off'"
+    changed_when: "false"
+
+  - name: the v2 entry points are GONE, not merely shadowed
+    # Assert absence of the binaries themselves. A `which` that still resolves would mean the
+    # old wheel's entry points survived the --force and agents could keep calling a dead CLI.
+    shell: "! test -e {{ bin_dir }}/althing-cli && ! test -e {{ bin_dir }}/althing-receiver && ! test -e {{ bin_dir }}/althing-herald"
+    changed_when: "false"
+
+  - name: v2 daemons are stopped and disabled
+    shell: "! systemctl is-active --quiet althing-herald.service && ! systemctl is-active --quiet althing-receiver.service"
+    changed_when: "false"
+
+  - name: the po-herald is running
+    shell: systemctl is-active --quiet althing-po-herald.service
+    changed_when: "false"
+
+  - name: this box can reach the post office and the roster is populated
+    # ⚠ --handle is required. postbox resolves its identity from ALTHING_HANDLE, which
+    # dev-launch sets per pane and which a playbook shell does not have -- without it this
+    # check fails on identity, not on reachability, and reads as a deployment fault.
+    shell: ALTHING_POST_OFFICE={{ post_office }} {{ bin_dir }}/postbox --handle operator handles | wc -l | awk '{ if ($1 >= 70) exit 0; else exit 1 }'
+    changed_when: "false"