esh-pfi-infrastructure/docs/runbooks/althing-deploy.md

althing deploy — six surfaces

scripts/deploy-althing.sh does all four. --check reports drift without touching anything. This page is the why.

Deploying althing means updating four independent surfaces on nh3-dev. Three were known; the fourth had no step in any runbook and drifted for five days before anyone noticed, on 2026-09-01.

#	surface	what it is	how it drifts
1	binaries	uv tool install --force . — the 7 console scripts	silently, see below
2	herald	systemctl --user restart althing-po-herald	new guards live here; nothing takes effect until it restarts
3	skill	scripts/sync_skill.sh → ~/.agents/skills/althing/SKILL.md	covered by its own --check
4	plugin	repo plugin/ → marketplace dir → Claude Code cache	two hops, neither was automated
5	per-seat route	althing-route declare on each seat	a channel change needs a re-declare; the plugin's SessionStart hook does it for CC seats
6	~/.claude/settings.json	crossSessionInbound: "accept"	outside the althing repo entirely — no version can carry it

⚠ uv tool install . without --force is a silent no-op

$ uv tool install .
`althing-core @ file:///home/lkraven/development/althing` is already installed
$ echo $?
0

uv matches on the source spec, not its contents. On a box that already installed from that path it declines and reports success. An operator following a runbook literally would restart the herald, see everything green, and wonder why the new binary was missing — with every command exiting 0.

⚠ Surface 4 is the one that bites, and it ate a hook

The chain is:

repo plugin/  →  ~/.local/share/althing-plugin/  →  ~/.claude/plugins/cache/althing/althing/<ver>/
              ^^^ nothing synced this hop

On 2026-09-01 the marketplace directory was a frozen copy from Aug 28:

deployed 0.0.1   hooks: ['UserPromptSubmit']
repo     0.1.1   hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
                 + scripts/pane-route.sh   (absent from the deployment entirely)

So "CC seats re-declare their pane route automatically at next SessionStart" was never true on this box. The hook existed and worked upstream; it was never deployed. That is why every seat — including infra-ops' own Claude Code seat — had to be hand-declared with a pid someone measured by hand, and why the idle_cursor pin from 3.2.1 would not have self-applied either.

⚠ Compare the HOOK LIST, not the version string

A version number cannot tell you what a stale plugin actually cost you. 0.0.1 and 0.1.1 differ by two hooks and a script:

for d in ~/.claude/plugins/cache/althing/althing/*/; do
  printf "  %-8s hooks: %s  pane-route.sh: %s\n" "$(basename "$d")" \
    "$(python3 -c "import json;print(list(json.load(open('$d/.claude-plugin/plugin.json'))['hooks'].keys()))")" \
    "$(test -f "$d/scripts/pane-route.sh" && echo yes || echo NO)"
done

Expect ['UserPromptSubmit', 'SessionStart', 'SessionEnd'] and pane-route.sh: yes. Anything less and pane routes are not being declared or released automatically.

⚠ WHY THIS HID FOR FIVE DAYS — the shape to recognise

A missing deploy surface does not present as an error. It presents as "the migration needs manual work" — and we had a ready explanation for that, because four of five seats were non-Claude and genuinely did need hand-holding.

The seat that falsified the story was infra-ops' own: a Claude Code seat that should have self-declared and didn't, and it looked exactly like the other four. Neither infra-ops nor forseti asked why the automatic path had not fired for the one seat it was supposed to work on.

⚠ When a migration needs manual intervention, check whether the automatic path was ever deployed before concluding it does not apply to your case.

Use the supported CLI for the second hop. claude plugin update althing (also install / uninstall / list / details / validate / marketplace). ⚠ Do not hand-edit ~/.claude/plugins/installed_plugins.json or fabricate a cache directory — that is Claude Code's own bookkeeping, and a subtle mistake there breaks the plugin in a way that looks like an upstream bug.

⚠ A Claude Code restart is required for new plugin hooks. They are read at session start; a running session keeps the old ones.

Verifying a seat is actually reachable

postbox status reports mode: push, but until 3.2.4 a failed push declaration was silent — _declare_push devnulled both streams with check=False, so a missing binary, a non-zero exit and a server-side rejection rendered identically as nothing. A seat could have been silently pull-only since 3.1.2.

Audit for "looks armed but is not", cross-referencing local waiter locks against what the post office believes:

for f in ~/.althing/wake-listener-*.lock; do
  h=$(basename "$f" .lock); h=${h#wake-listener-}
  pid=$(cat "$f" 2>/dev/null)
  [ -n "$pid" ] && [ -r "/proc/$pid/cmdline" ] || continue
  # IDENTITY, not liveness: a lock left by a reaped listener names a pid the
  # kernel is free to hand to anything, so `kill -0` alone reports a stranger
  # as a live waiter. Match the same cmdline segment `althing-listen --stop`
  # requires before it will signal anything.
  ours=0
  while IFS= read -r -d '' seg; do [ "$seg" = "--_route=$h" ] && ours=1; done \
    < "/proc/$pid/cmdline"
  [ "$ours" = 1 ] || { printf "  %-24s pid %-8s STRANGER (recycled pid)\n" "$h" "$pid"; continue; }
  printf "  %-24s waiter %-8s mode: %s\n" "$h" "$pid" \
    "$(postbox status --handle "$h" 2>/dev/null | grep -oP 'mode: \K\w+')"
done

A live waiter reporting mode: pull is a seat that will never be poked. From 3.2.4 onward $ALTHING_ROOT/listen.log records failed declarations directly.

⚠ Liveness is not identity, and this loop is the place that gets it wrong. The earlier kill -0 form would print a phantom waiter for any handle whose stale lock happens to name a recycled pid — and a phantom waiter is exactly what sends a false "you are unreachable" notice to a seat that is fine. session_listener.sh refuses to SIGTERM on liveness alone for precisely this reason; an audit that only reads has no excuse for a weaker standard than the one that kills. (Adopted 2026-09-02 after the regin-smithy-dev cross-reference, where the pid happened to be genuine and the weaker check happened to be right.)

⚠ A stale wake-listener-*.lock is NOT a fault. The gate is flock -n on an open fd, which the kernel releases when the holder dies, so a lock file left by a reaped listener is inert and exit 3 only fires against a genuinely live holder. The pid in the file is read by --stop alone. (Recorded because infra-ops claimed the opposite, untested, on 2026-09-01; forseti measured it.)

⚠ The plugin cache does not take a content-only change

claude plugin update althing matches on the version in plugin.json and declines when it is unchanged. A release that edits hook or script CONTENT without bumping the plugin version therefore leaves the Claude Code cache stale while every version check reports success — the marketplace dir gets the new bytes, the cache does not, and update says "already at the latest version".

Measured on the 3.3.0 deploy (2026-09-02): marketplace and live cache both read 0.1.1, and pane-route.sh + README.md differed. That delta was documentation-only and harmless. The next one need not be.

scripts/deploy-althing.sh now diffs the marketplace tree against the live cache dir on every run and on --check, so the drift is reported rather than inferred. The fix is a plugin version bump upstream, not a hand-edit — ~/.claude/plugins/installed_plugins.json and the cache directory are Claude Code's own bookkeeping.

The 2026-09-01 lesson was compare the hook list, not the version string. This is the same lesson one turn deeper: here the hook list was identical too, and only the file contents moved.

A seat already running when the plugin lands never auto-declares — leave it

The SessionStart hook fires at session start and nowhere else, so any session that was already running when a plugin deploy landed keeps its old channel until it restarts. After the 3.3.1 deploy that was eight of seventeen live seats, all still on fifo.

Operator ruling 2026-09-02: leave them. They are reachable now, the post office holds mail regardless, and each self-corrects at its next session start. The residual exposure is that a fifo waiter is what Claude Code 2.1.257 reaps — one seat was reaped three times in a morning — and a reap converts a healthy seat into a silent one until the next restart. That costs timeliness, never mail.

⚠ Do not mass-message the affected seats. Standing operator directive: no unsolicited fleet notification, ever, unless asked. Direct correspondence with a counterparty about work in hand is fine; sweeping the handle list is not. Audit it, report the count, and let him decide — as he did here.

The one-liner, for a seat that asks or an operator who wants one moved:

althing-route declare --discover-pid

⚠ The cc channel is an undocumented interface, taken deliberately

From 3.3.0 a Claude Code seat is poked over its own message socket ($XDG_RUNTIME_DIR/cc-socks/<pid>.sock) rather than by typing into its pane. No process to reap, nothing near the input line, and delivery lands at the receiver's next turn boundary.

It is not a published interface and is expected to break on some future Claude Code release. Accepted on the operator's reasoning: the FIFO poker it replaces was also an unsanctioned hack of the background-watcher system, so this is a better instance of a dependency class we already had, not a new one.

When it breaks the failure mode is a seat going pull-only with a logged reason — not lost mail; the post office holds it either way. Recovery is one command:

althing-route declare --pid <pid>     # prefers cc, falls back to pane

Why cc is worth an undocumented dependency, in one measurement. A pane poke is typed into the session and submitted, so Claude Code records it in the operator's prompt history exactly as if he had written it — 8 herald-written entries accumulated across three projects, permanently in up-arrow recall. A cc poke enters none: 2 delivered, 0 entries. Verified from the herald log against the history on 2026-09-02. Channel sequence on one seat that day, all three in order:

09-01 21:44  poked via fifo            the old way, later reaped
09-02 08:34  could NOT reach via pane  guard 4 refusing: he was typing
09-02 09:07  poked via pane            typed into his input line
09-02 09:14  poked via cc              arrived as a peer message
09-02 09:31  poked via cc              after crossSessionInbound accept

⚠ Count herald pokes by the whole display string, not by grepping ALTHING-HERALD. The substring matches the operator's own messages when he pastes a hold notice into one, so the count runs high and the excess reads as new activity. See the identity-not-resemblance note in the audit section.

⚠ Ordering constraint at any upgrade that changes the channel set: a pre-3.3.0 herald refuses channel=cc at parse, so a seat that declares the new channel before the herald restarts goes silently pull-only. uv tool install --force then the herald restart, and only then let anything re-declare. deploy-althing.sh already runs them in that order and updates the plugin (whose SessionStart hook declares the route) last, which is what keeps the window to a couple of seconds.

⚠ Surface 6 — crossSessionInbound, and it is not in the althing repo

A cc poke to a default-configured seat is HELD, not delivered. Claude Code auto-delivers an inbound cross-session message only when the sender's permission-mode class matches the receiver's, and a sender that asserts no class is held. The herald is a daemon and asserts none, deliberately. What the operator sees instead of a delivery:

Held peer message — from an unidentified session [verified pid <herald>];
preview: «ALTHING-HERALD althing: you have mail ...» — not delivered to
Claude (1 held). The sender did not attest its permission mode and this
session bypasses prompts.

Claude Code verified the herald through SO_PEERCRED and then correctly declined to let it speak, because it would not say what it was.

The fix is one key in ~/.claude/settings.json:

"crossSessionInbound": "accept"

Set on nh3-dev 2026-09-02. The operator's reasoning is the part to keep: the herald can only reach local seats, so bypass is the correct authorization type; when the guard was not there, that was our default posture, including the pane poke. A pane poke types into a session and presses Enter — bypass-level access by any measure, and what we had been doing all along. The socket channel is strictly narrower than what it replaces, so accepting here states the existing trust boundary rather than widening it.

⚠ There is no attestation the herald could send instead. Four probes established that Claude Code identifies a sender by verified pid against the session registry and reads that session's live runtime permission mode; a daemon has none and the registry has no field for one. A from_mode field on a type:"user" frame is not consulted at all — it belongs to the control actions. Adding it would be shipping a field nothing reads.

Cost without the setting is smaller than it looks: the correspondent record is in-memory session state, so the hold is first-contact, not per-message — one approval per seat per session lifetime, and a long-lived pane pays it once.

⚠ Why this surface is the dangerous one. A seat without it is declared, reachable and green, and its pokes go to a human instead of to the session. That is the same shape as the SessionStart hook that was never deployed: everything reports success, nothing arrives. deploy-althing.sh reads the key and reports it on every run and on --check — and deliberately never sets it. That file is the operator's permission configuration; a deploy script that edits it is a deploy script granting itself trust.

The status-line segment belongs to althing, not to this repo

From althing 3.4.0 the segment is a console script, althing-statusline, installed on PATH by the same uv tool install --force as the other seven executables. scripts/claude-statusline-command.sh calls it and does not reimplement it:

althing=$(printf '%s' "$input" | timeout 5 althing-statusline 2>/dev/null)

⚠ Keep that outer timeout LOOSER than the program's own 2 s budget. If the outer one fires first the segment comes back empty, which renders as "not an althing directory" — the outage conflation, reintroduced by the guard meant to prevent a hang.

Why a console script rather than the sync_skill.sh canonical-file pattern: a script, unlike a document, has somewhere to be installed. Installing it makes drift impossible instead of merely visible.

Why it needed solving at all. On 2026-09-02 the same twelve lines existed in three places — althing's plugin/scripts/statusline.sh, the operator's wired ~/.claude/statusline-command.sh, and this repo's copy — and one audit found a different defect in each. All three were fixed separately, by hand, the same day. Two of the defects were in this repo's copy and are worth knowing because they are both silent:

• the handle was resolved as the most recent launch in the directory, so a directory hosting both a claude and a codex seat (~/development/Worldtree) reports the codex handle's unread count and reachability to a Claude session the moment the codex pane relaunches last;
• the post-office address was hardcoded, so it survives exactly until the post office moves and then reports an outage that is really a stale constant.

Deploying it

scripts/claude-statusline-command.sh is the tracked copy of what Claude Code actually runs at ~/.claude/statusline-command.sh (settings.json → statusLine.command). Copies, not symlinks — same rule as stacks/.

diff -u scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh
cp scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh

⚠ Diff before you copy, in both directions. On 2026-09-02 the live path was edited directly and was ahead of the tracked copy; a reflexive cp from the repo would have destroyed the better version. The tracked copy is intent, the live path is reality, and reality is sometimes right.

Smoke test — strip the althing env vars, or you will test your own session's identity instead of the resolution logic. A hand-run inherits the exported ALTHING_HANDLE from the session you run it in, so the seat's own directory and a directory with no althing identity render identically and both look right. That false pass caught two agents within one hour on 2026-09-02.

⚠ Open question — whether Claude Code's own invocation inherits the environment is NOT established, and nothing here depends on it. An earlier version of this page asserted a "bare shell"; that claim was never verified and the evidence points the other way. Measured 2026-09-02: this box's Claude Code process carries ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ (inherited from the dev-launch shell), and a spawned child would normally inherit both — observed independently on two seats. Whether Claude Code scrubs the environment before spawning the status-line command was not observed either way. Write code that does not depend on the environment being present, which althing-statusline already does: environment first, launch history second, and a seat started outside dev-launch has no handle in either place. Test with the variables stripped regardless, because that is the harder case:

probe() {  # $1 = cwd, $2 = what you expect
  got=$(printf '{"workspace":{"current_dir":"%s"}}' "$1" \
        | env -u ALTHING_HANDLE -u ALTHING_POST_OFFICE \
          bash scripts/claude-statusline-command.sh)
  printf 'got [%s]  wanted [%s]\n' "$got" "$2"
}
probe /tmp                                    ''      # no althing identity
probe /home/lkraven/development/eshpfi-management '🔔 …'

⚠ Print what you got beside what you wanted, on the same line. Expected values in trailing comments are what produced the false pass that caught two agents inside an hour on 2026-09-02: /tmp rendered 🔔 and both of us read it as "fine" rather than "that directory has no identity and should be empty." A plausible value in the right shape is the hardest kind of wrong to see, and the only thing that reliably catches it is the wanted value sitting on the same line as the got value.

⚠ The outage probe: a refused port does NOT test the timeout

📵 has two causes and they take different paths. Measured 2026-09-02 on nh3-dev:

address	elapsed	what it proves
http://10.100.50.40:9999 — refused port on a live host	0.079 s	the glyph, and nothing about the deadline
http://10.100.50.199:8390 — unrouted host, black-holes	2.065 s	the 2 s budget actually fires

A refused port answers instantly with an RST, so the fast probe returns 📵 without ever approaching the timeout. Use both, and know which is which — letting the fast case imply the slow one is how a status line that hangs the prompt ships with a green test beside it.

Rollback

uv tool install althing-core==3.1.2

Routes written by later versions stay parseable — the old reader ignores unknown keys — so nothing is stranded.