Deploys althing-core 3.1.1 to nh3-extdev. Both markers present on both
boxes; the warning verified behaviourally in four conditions rather than
by grep alone -- mismatched inherited handle warns, matching handle
silent, explicit --handle silent, unlaunched directory silent, and the
warning precedes the output it is about.
The release's own verification line says
`grep -c handles_launched_at dev_launch.py # 2+`. The real count there
is 1, the definition; the other two occurrences are in postbox.py. The
installed tree is byte-identical to the repo at the pushed tag, so the
instruction is wrong rather than the install.
This playbook matches on presence via grep -q, so it passed. Had it
asserted the stated count it would have reported FAILED on a perfect
deploy -- a verification instruction that fails on correct input, which
is the same false-negative this file has now produced three times in
different costumes. Recorded above the variable so the next bump does
not reintroduce a count.
Deploys althing-core 3.1.0 to nh3-extdev and restarts the herald.
Verified by content on both boxes: POST_OFFICE_HINT 0 -> 4 in
post_office_herald.py and resolve_post_office 0 -> 3 in dev_launch.py,
dist-info 3.0.3 -> 3.1.0.
The check took one file:marker pair. 3.1.0 changed two files, so a
single pair would have asserted half a release and passed -- the same
half-passing-silently shape as the version-string check it replaced two
releases ago, one level up. It now takes a space-separated list, reports
each pair individually, and fails if any is missing. Every release's
markers so far are recorded above the variable so the next bump is a
lookup rather than an archaeology exercise.
Also verified the behaviour the release exists for rather than just its
markers. The herald writes its address to $ALTHING_ROOT/post-office and
dev_launch.resolve_post_office reads it when the variable is unset:
env unset -> http://10.100.50.40:8390
env set -> the env value, which wins
env set to blank -> the file, because blank counts as unset
My first attempt tested this through postbox, which still requires the
variable and reported "no post office address is configured" -- correct
behaviour that looked like a failed deploy. dev-launch is the reader,
not postbox.
Deploys althing-core 3.0.3 to nh3-extdev and restarts the herald.
Verified by content on both boxes: PANE_SETTLE_S 0 -> 2 occurrences,
value 0.3, dist-info 3.0.1 -> 3.0.3.
The content check was hardcoded to the 3.0.1 markers, so from the next
release onward it would have kept passing while asserting nothing about
what had just been installed -- a check that verifies the previous
release is indistinguishable from one that works. It now takes the
marker and file as variables, bumped per release, with both releases'
markers recorded so the pattern is obvious rather than folklore.
That is the same defect class as the install step gated on `postbox` not
existing, which this playbook carried until last round: a guard written
correctly for the first run and never re-read on the second.
The post office container was not touched. forseti established by
import graph that althing/post_office/* imports neither changed module
-- the fix is in reach_pane, which is herald code -- and the container
has been up two hours across both herald restarts.
Reinstalls althing-core on nh3-extdev for 3.0.1 (the pane-route fix) and
restarts the herald. Both boxes verified BY CONTENT rather than by
version string -- forseti's own checks, grep for _PANE_ID and _live_pid,
because a dist-info directory records what was installed, not what the
files contain. Both went 0 -> 3 and 0 -> 2.
Two bugs in the playbook this run exposed, both of which only appear on
the second use:
The install step was gated on `postbox` not existing. That guard was
correct for the cutover, when postbox genuinely was absent, and wrong
for every release after it -- postbox exists now, so a version bump
would have silently skipped the install and the playbook would have
reported success having done nothing. `--force` already makes the
reinstall idempotent, so the guard bought nothing and cost correctness.
The post_office variable still pointed at nh3-dev, three hours after the
post office moved to nh3-docker. It failed in the verify rather than at
install time, which reads as a broken deploy rather than as a stale
constant. Worth noting the failure message was the outage semantics
working exactly as designed: "This is an outage, not an answer: do not
treat it as 'no mail'."
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
vh