84349d7a0e
A version number cannot tell you what a stale plugin cost. 0.0.1 and 0.1.1 differ by two hooks and a script, so the runbook now carries a check that compares hook lists across cached versions and looks for pane-route.sh directly. Also records why this hid for five days, which is the more transferable half. A missing deploy surface does not present as an error -- it presents as "the migration needs manual work", and there was a ready explanation for that, because four of five seats were non-Claude and genuinely did need hand-holding. The seat that falsified the story was our own: a Claude Code seat that should have self-declared and did not, and it looked exactly like the other four. Nobody asked why the automatic path had not fired on the one seat it was built for. So: when a migration needs manual intervention, verify the automatic path was actually deployed before concluding it does not apply to your case.
537 lines
31 KiB
Markdown
537 lines
31 KiB
Markdown
# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies
|
|
|
|
Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops
|
|
deployed. Host work, nh3-dev only.
|
|
|
|
## What 3.2.0 changes
|
|
|
|
A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a
|
|
detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat
|
|
sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter
|
|
helps. A pane route is a file; there is no process to reap.
|
|
|
|
⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops'
|
|
own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat
|
|
silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness
|
|
reaper" rather than guessed at. 2.1.257 is the mechanism.
|
|
|
|
## The deploy — three steps, order matters
|
|
|
|
22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables
|
|
22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064
|
|
22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean
|
|
|
|
`althing-route` is the **seventh** console script and the whole point — the plugin's new
|
|
SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades
|
|
to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it
|
|
lands. Skill sync must come **after** the install or it documents a binary that is not there.
|
|
|
|
⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on
|
|
`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy
|
|
step in between.
|
|
|
|
## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL
|
|
|
|
$ uv tool install .
|
|
`althing-core @ file:///home/lkraven/development/althing` is already installed
|
|
$ echo $?
|
|
0
|
|
|
|
**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was
|
|
already installed from that same path, it declines and reports success. Nothing says "I did
|
|
not do the thing you asked."
|
|
|
|
**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the
|
|
nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every
|
|
local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the
|
|
command exited 0 and the state was unchanged.
|
|
|
|
## Fleet-visible: 4 pane routes revoked, and the count to trust
|
|
|
|
Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot
|
|
inherit a route) and `command` (so a pane that has dropped to a shell stops being poked).
|
|
Routes without them are **observed and excluded with a reason in the herald log**, not
|
|
silently ignored — this closes a real hazard where the herald would write a line into whatever
|
|
shell inherited an exited agent's pane and press Enter, running whatever was half-typed there.
|
|
|
|
Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve
|
|
route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only
|
|
**pane** routes are affected:
|
|
|
|
pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev
|
|
fifo: 8 (+ 9 fifos in ~/.althing/wake/)
|
|
|
|
forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are
|
|
untouched" — **twelve is the total route count; fifo is 8.** Untouched either way.
|
|
|
|
Re-declare is one command per agent: `althing-route declare --pid <long-lived agent pid>`.
|
|
|
|
**Not notified.** Four targeted messages about a change none of them asked about reads as a
|
|
broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the
|
|
operator instead; the herald logs the exclusion every tick so it is discoverable.
|
|
|
|
⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every
|
|
tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared —
|
|
enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a
|
|
back-off after N consecutive exclusions. Their call.
|
|
|
|
## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat
|
|
|
|
althing-route declare --discover-pid
|
|
-> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume)
|
|
|
|
pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2
|
|
command=claude --allow-dangerously-skip-permissions --resume
|
|
postbox status -> mode: push reachable: true
|
|
|
|
`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the
|
|
ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with
|
|
the tool call and the route dies with it.
|
|
|
|
⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving.
|
|
|
|
## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU
|
|
|
|
infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in
|
|
one session** (2026-09-01), the second time **within minutes of being armed**. Both times:
|
|
`unread: 0`, no mail, seat silently back to `mode: pull`.
|
|
|
|
**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER
|
|
confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So:
|
|
|
|
- **A green `postbox status` is not durable evidence of monitoring.** True when read, false
|
|
minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]]
|
|
- **Anything still on `althing-listen` is monitored only until the reaper notices.** The
|
|
waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones.
|
|
- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised
|
|
both consequences with forseti on the deploy thread.
|
|
|
|
## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min
|
|
|
|
Operator directed notification; all four told individually with their own measured pid, pane
|
|
and start time. Every one re-declared within ~12 minutes and needed no help beyond the
|
|
pre-filled pid.
|
|
|
|
handle pid started_at command status
|
|
bil-smithy-dev 98493 1713267 kimi push, poked via pane
|
|
dvalin-smithy-dev 96050 1691681 grok push
|
|
eitri-smithy-dev 103662 1752089 node/codex push
|
|
regin-smithy-dev 96864 1700952 pi push, poked via pane
|
|
infra-ops 2175612 38608652 claude push, poked via pane
|
|
|
|
Zero stale routes; last exclusion line 22:25:01, everything after is `poked <handle> via pane`.
|
|
|
|
## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case
|
|
|
|
The herald logs *"the process that declared this route is gone, or its pid was reused by
|
|
something that started at a different time."* **Neither disjunct was true for any of the
|
|
four.** Measured before notifying: each declaring process started 1-3 min *before* its route
|
|
was written and was still running four days later; `pid_max` 4,194,304 against a current
|
|
~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against
|
|
pid 98493.
|
|
|
|
Real cause is a third branch the message never offers: **the route predates the guard fields,
|
|
so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation —
|
|
it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log
|
|
line at face value.** Raised with forseti with a suggested third branch.
|
|
|
|
## ⚠ MY MISS — I called an open risk on something the release author had already measured
|
|
|
|
I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the
|
|
release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane
|
|
pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes
|
|
split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin
|
|
had also smoke-tested 3.0.3 on the `pi` seat.
|
|
|
|
**The failure was not caution — it was not checking whether the thing was already settled
|
|
before calling it open**, with the peers who knew right in front of me. This is exactly the
|
|
ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a
|
|
"you missed X" claim from someone who had not checked what the other party already read.
|
|
⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the
|
|
first was honest; asserting the second was not warranted. Retracted with forseti and with all
|
|
three seats. → [[feedback_canonical_sourced_values]]
|
|
|
|
## Verified: pane delivery works on Claude AND non-Claude seats
|
|
|
|
- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns.
|
|
- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat:
|
|
own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference.
|
|
- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover
|
|
unsubmitted text.
|
|
|
|
## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE
|
|
|
|
`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the
|
|
herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a
|
|
failure, in case anyone else reports it as one."* Asked forseti for a line in the skill.
|
|
|
|
## Open with forseti (nothing blocking)
|
|
|
|
1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person.
|
|
2. Exclusion-message third branch.
|
|
3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the
|
|
explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree.
|
|
4. One-tick-latency note in the skill.
|
|
5. `postbox status` push/reachable is not durable evidence.
|
|
|
|
**Withdrawn:** non-Claude delivery risk (was forseti's own prior work).
|
|
**Moot:** log back-off — zero stale routes, the log went quiet on its own.
|
|
|
|
## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN
|
|
|
|
**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its
|
|
poke into a pane's input line and presses Enter. If someone is **mid-sentence there**,
|
|
`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their
|
|
half-written message with the herald's line stapled on.
|
|
|
|
forseti's post-mortem, worth keeping verbatim because it generalises well past althing:
|
|
|
|
> The contract named this hazard on one road only — a submit after a FAILED write — and I
|
|
> built three guards that all answer the same question, "is this the right pane". **None of
|
|
> them asked whether it was a good MOMENT.**
|
|
|
|
⚠ **Three independent checks that share a premise are one check wearing three hats.**
|
|
|
|
**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with
|
|
its input line empty, captured at declare time. Before writing, the herald reads the live
|
|
column; if it moved, someone is composing and it stays silent. The signal was already in the
|
|
`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost
|
|
nothing at the wire.
|
|
|
|
22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables
|
|
22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820
|
|
22:40:40 scripts/sync_skill.sh --check clean
|
|
|
|
## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER
|
|
|
|
forseti measured the idle columns; applied to the live route table this is not a footnote:
|
|
|
|
handle TUI guard 4?
|
|
infra-ops claude YES idle_cursor=3 (pinned 22:41)
|
|
eitri-smithy-dev codex YES expects 3
|
|
dvalin-smithy-dev grok YES expects 7
|
|
bil-smithy-dev kimi NO -- reports no cursor at all
|
|
regin-smithy-dev pi NO -- reports no cursor at all
|
|
|
|
**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes
|
|
it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible —
|
|
⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is
|
|
logged, not silently assumed safe) but logged-and-poked is still poked.
|
|
|
|
**Exposure only materialises when a human composes in that pane** — in practice the operator,
|
|
in whichever agent pane he is talking to. Mitigation for an unguardable seat is
|
|
`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with
|
|
`althing-route declare --discover-pid`.
|
|
|
|
⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes
|
|
needlessly quiet.
|
|
|
|
**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in
|
|
5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms
|
|
collision window remains.
|
|
|
|
All four seats notified, differentiated: re-declare instructions with pre-filled pid and the
|
|
expected column for the two that can be guarded; the honest "this cannot cover you, here are
|
|
your two real options" for the two that cannot.
|
|
|
|
**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats
|
|
re-pin at next SessionStart; TUI peers when they next re-declare.
|
|
|
|
## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families
|
|
|
|
handle TUI idle_cursor outcome
|
|
infra-ops claude 3 push, positive
|
|
eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex")
|
|
dvalin-smithy-dev grok 7 push, positive (exact submitted text verified)
|
|
bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk
|
|
regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice**
|
|
|
|
forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess.
|
|
|
|
## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle
|
|
|
|
> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like
|
|
> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than
|
|
> **cursor column** (e.g. input-line length via terminal query), those seats become coverable.
|
|
|
|
Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong
|
|
question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual
|
|
*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the
|
|
lead item, credited. Difference between a workaround and a fix.
|
|
|
|
## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite
|
|
|
|
Both were given the same two options and neither was chosen for them. Opposite calls, both right:
|
|
|
|
- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every
|
|
operator turn this session, including multi-sentence research directives, was typed here."*
|
|
Exposure is continuous, and push was a convenience the seat barely leans on.
|
|
- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's
|
|
composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already**
|
|
cost them: their 3.2.0 revocation notice **sat unread for days**.
|
|
|
|
⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should
|
|
release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to
|
|
frame the choice by pane usage in the skill.
|
|
⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the
|
|
sharpest argument on record for that.
|
|
|
|
## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case
|
|
|
|
bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane,
|
|
so the stapling failure mode is inferred, not observed."*
|
|
|
|
Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They
|
|
never exercised **collision**, because nobody was typing. **Two different questions, one body
|
|
of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the
|
|
"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and
|
|
silent about stapling. The operator's report remains the single observed instance.
|
|
|
|
Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti
|
|
on the first observed stapling.
|
|
|
|
⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check.
|
|
Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the
|
|
other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."*
|
|
|
|
## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed
|
|
|
|
22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables
|
|
22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372
|
|
22:51:21 scripts/sync_skill.sh --check clean
|
|
|
|
1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched
|
|
`comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer
|
|
and refused — four of five pane seats could never have used it.** Now matches the **pane's
|
|
own command** (`pane_command`), which zellij already reports and guard 1 already compares
|
|
against, so discovery and the guard read one string and a route declared that way cannot
|
|
fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches
|
|
`node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the
|
|
obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an
|
|
identical `claude ...`, so a global /proc search is ambiguous by construction.
|
|
2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts,
|
|
all four were rendered as "gone, or pid reused". Migration case now says what is true and
|
|
names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one
|
|
vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a
|
|
wrong error message until it is true in all cases makes it true and worthless.**
|
|
3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the
|
|
editable-install warning it rhymes with.
|
|
4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a
|
|
receipt for the declare, never proof you are still being watched.*
|
|
|
|
Count correction accepted: 8 fifo, not 12.
|
|
|
|
## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT
|
|
|
|
forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug
|
|
"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then
|
|
reported the suspicion as an open question**, with four live non-Claude seats in front of me and
|
|
the answer one ancestry-walk away. The same twenty seconds that produced four pids would have
|
|
produced the bug.
|
|
|
|
⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had
|
|
already measured). **Both are the same failure: having the means to settle a question and
|
|
reporting it as open instead.** Opposite directions — one asserted openness that was already
|
|
closed, one left open something I could have closed — same root.
|
|
|
|
## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover
|
|
|
|
forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery
|
|
against another's safety on seats that are not mine."* Their dead-ends, worth keeping as
|
|
negative results:
|
|
|
|
- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI
|
|
parsing** — exactly the hardcoding the pinned baseline exists to avoid.
|
|
- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token
|
|
counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke
|
|
forever.
|
|
|
|
Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs
|
|
the wake) / leave pull-only.
|
|
|
|
**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this
|
|
call on identical facts, reached opposite conclusions, and both were right — the deciding
|
|
variable (who composes in that pane and why) is local knowledge a global policy cannot encode.
|
|
Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is
|
|
the exact thing they accepted risk to keep; for regin it removes only the submission, leaving
|
|
the operator to delete herald text from his in-flight sentences routinely.
|
|
|
|
## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE
|
|
|
|
The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and
|
|
`scripts/sync_skill.sh` does NOT cover it:
|
|
|
|
canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced
|
|
plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md
|
|
-> pre-3.2.0 text, ZERO occurrences of "pane", never synced
|
|
|
|
⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical
|
|
skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running,
|
|
that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter
|
|
**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps —
|
|
the path that died **twice in one session** here. The command does not warn; it predates the
|
|
problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.**
|
|
|
|
**On this seat: declined to arm, route left intact.** Reported to forseti with three options,
|
|
recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route
|
|
exists for the handle**, so the stale doc becomes harmless because the binary enforces the
|
|
documented precedence. The doc-only fixes rely on a reader noticing.
|
|
|
|
## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE
|
|
|
|
infra-ops claimed a `wake-listener-<handle>.lock` holding a dead pid would make the next
|
|
`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a
|
|
reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does
|
|
not happen:**
|
|
|
|
lock file holds pid 999999 (dead) -> althing-listen ARMED FINE
|
|
lock held by a live process -> rc=3, correctly refused
|
|
|
|
⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an
|
|
**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is
|
|
**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file
|
|
is read by `--stop` alone — which is why the file is opened non-truncating.
|
|
|
|
⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the
|
|
LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim
|
|
that would have justified real guard code for a condition that cannot occur. forseti tested it
|
|
before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug:
|
|
check `flock -n` on the fd before suspecting the pid.
|
|
[[feedback_assert_effective_value_not_substring]]
|
|
|
|
## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4)
|
|
|
|
$ althing-listen # on a channel=pane seat
|
|
'infra-ops' already has a PANE route, and arming a waiter would silently take
|
|
precedence over it ... To switch back on purpose: althing-route release
|
|
rc=4
|
|
|
|
Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that
|
|
never arrived), and **not 3** ("already running" says leave the incumbent alone, a different
|
|
instruction from "you do not need this at all"). Scoped to the handle's own route, checked
|
|
before the lock, and `--stop` deliberately ungated.
|
|
|
|
## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX
|
|
|
|
Not a stale doc. The deploy chain is:
|
|
|
|
repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/
|
|
^^^ NOTHING SYNCS THIS HOP
|
|
|
|
The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed
|
|
against the repo:
|
|
|
|
deployed 0.0.1 hooks: ['UserPromptSubmit']
|
|
repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
|
|
+ scripts/pane-route.sh (absent from the deployment entirely)
|
|
|
|
⚠ **So forseti's migration note — "CC seats re-declare automatically at their next
|
|
SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was
|
|
simply never deployed. That explains what we both treated as normal: all four smithy seats
|
|
needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code
|
|
seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin.
|
|
|
|
**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first,
|
|
`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware
|
|
`monitor.md` now present).
|
|
|
|
🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin`
|
|
update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache
|
|
directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the
|
|
plugin in a way that looks like an upstream bug.
|
|
|
|
⚠ **There is still no deploy script for the first hop**, which is why it drifted five days
|
|
unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo
|
|
vs eshpfi) is forseti's call.
|
|
|
|
## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1)
|
|
|
|
1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e)
|
|
2. systemctl --user restart althing-po-herald
|
|
3. scripts/sync_skill.sh (already current at 3.2.4)
|
|
4. claude plugin update althing <- THE NEW, FOURTH SURFACE
|
|
|
|
⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync,
|
|
**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1.
|
|
|
|
⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.**
|
|
`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist.
|
|
`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart
|
|
to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I
|
|
would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the
|
|
conclusion that no supported path existed was an untested assumption.**
|
|
|
|
⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions.
|
|
|
|
## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES
|
|
|
|
Four times in one session I reported a **proxy or an assumption as the fact itself**:
|
|
|
|
sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15
|
|
command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin
|
|
a lock file's pid for a flock -> flock is fd-held; stale files are inert
|
|
"no CLI path exists" for a CLI I never ran -> claude plugin update did it
|
|
|
|
⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any
|
|
one of them — it is reaching for the nearest observable and reporting it as the thing.
|
|
[[feedback_assert_effective_value_not_substring]]
|
|
|
|
## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be
|
|
|
|
`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit
|
|
and a server-side rejection were **three facts with one rendering, and that rendering was
|
|
silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to
|
|
`$ALTHING_ROOT/listen.log`. Success stays silent by design.
|
|
|
|
**Audit run on nh3-dev — live waiter locks vs what the post office believes:**
|
|
|
|
9 of 10 seats live waiter -> mode: push (correct)
|
|
regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked
|
|
|
|
⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their
|
|
pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a
|
|
`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be
|
|
distinguished from outside** — asked regin, who has the local history. Handing forseti an
|
|
ambiguous instance beats a confidently mislabelled one.
|
|
|
|
**Standing audit command** for "looks armed but isn't" — worth re-running after any incident:
|
|
cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against
|
|
`postbox status --handle <h>`. From the next arm onward `listen.log` answers it directly.
|
|
|
|
## Rollback (unused)
|
|
|
|
`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns
|
|
to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader
|
|
ignores unknown keys), so nothing is stranded.
|
|
|
|
Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`.
|
|
|
|
## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT
|
|
|
|
> **A missing deploy surface does not present as an error. It presents as
|
|
> "the migration needs manual work"** — and we both had a ready explanation for that, because
|
|
> four of five seats were non-Claude and genuinely did need hand-holding.
|
|
|
|
⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have
|
|
self-declared and did not, and it looked exactly like the other four. Neither of us asked why
|
|
the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats
|
|
re-declare automatically at their next SessionStart" in **three** deploy notes; it was never
|
|
true on the only box running it.
|
|
|
|
⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a
|
|
script; a version number cannot tell you what a stale plugin cost. Check in
|
|
`docs/runbooks/althing-deploy.md`.
|
|
|
|
⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before
|
|
concluding it does not apply to your case.**
|
|
|
|
forseti also corrected an error of their own that I did not catch: the contract's Migration
|
|
section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook.
|
|
Wrong since 3.2.0.
|
|
|
|
**On the four retractions**, forseti's framing, kept because it is fair and I would not have
|
|
written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive
|
|
and quiet, so your errors are all on the record and mine mostly are not. You catching four of
|
|
your own in one night is the system working."* Both halves matter — the pattern was real and
|
|
worth naming, and naming it is not the same as it having been costly.
|