memory: snapshot — lv-yarros shipped, voices-seat live, lv-hemingway training, Grok broker shelved
This commit is contained in:
Vuong Hoang
@@ -32,6 +32,560 @@ Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
|
||||
4012952 18,500 MiB comfyui up 8d09h
|
||||
3166988 4,722 MiB studio-gate up 19d09h
|
||||
|
||||
# `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb
|
||||
|
||||
**althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
|
||||
_Archived 2026-09-16._
|
||||
|
||||
# `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.
|
||||
|
||||
**althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-<handle>.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md`
|
||||
_Archived 2026-09-16._
|
||||
|
||||
- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). (context archived → `archival-memory.md`)
|
||||
_Archived 2026-09-16._
|
||||
|
||||
- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** (context archived → `archival-memory.md`)
|
||||
_Archived 2026-09-16._
|
||||
|
||||
# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies
|
||||
|
||||
Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops
|
||||
deployed. Host work, nh3-dev only.
|
||||
|
||||
## What 3.2.0 changes
|
||||
|
||||
A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a
|
||||
detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat
|
||||
sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter
|
||||
helps. A pane route is a file; there is no process to reap.
|
||||
|
||||
⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops'
|
||||
own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat
|
||||
silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness
|
||||
reaper" rather than guessed at. 2.1.257 is the mechanism.
|
||||
|
||||
## The deploy — three steps, order matters
|
||||
|
||||
22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables
|
||||
22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064
|
||||
22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean
|
||||
|
||||
`althing-route` is the **seventh** console script and the whole point — the plugin's new
|
||||
SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades
|
||||
to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it
|
||||
lands. Skill sync must come **after** the install or it documents a binary that is not there.
|
||||
|
||||
⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on
|
||||
`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy
|
||||
step in between.
|
||||
|
||||
## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL
|
||||
|
||||
$ uv tool install .
|
||||
`althing-core @ file:///home/lkraven/development/althing` is already installed
|
||||
$ echo $?
|
||||
0
|
||||
|
||||
**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was
|
||||
already installed from that same path, it declines and reports success. Nothing says "I did
|
||||
not do the thing you asked."
|
||||
|
||||
**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the
|
||||
nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every
|
||||
local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the
|
||||
command exited 0 and the state was unchanged.
|
||||
|
||||
## Fleet-visible: 4 pane routes revoked, and the count to trust
|
||||
|
||||
Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot
|
||||
inherit a route) and `command` (so a pane that has dropped to a shell stops being poked).
|
||||
Routes without them are **observed and excluded with a reason in the herald log**, not
|
||||
silently ignored — this closes a real hazard where the herald would write a line into whatever
|
||||
shell inherited an exited agent's pane and press Enter, running whatever was half-typed there.
|
||||
|
||||
Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve
|
||||
route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only
|
||||
**pane** routes are affected:
|
||||
|
||||
pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev
|
||||
fifo: 8 (+ 9 fifos in ~/.althing/wake/)
|
||||
|
||||
forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are
|
||||
untouched" — **twelve is the total route count; fifo is 8.** Untouched either way.
|
||||
|
||||
Re-declare is one command per agent: `althing-route declare --pid <long-lived agent pid>`.
|
||||
|
||||
**Not notified.** Four targeted messages about a change none of them asked about reads as a
|
||||
broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the
|
||||
operator instead; the herald logs the exclusion every tick so it is discoverable.
|
||||
|
||||
⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every
|
||||
tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared —
|
||||
enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a
|
||||
back-off after N consecutive exclusions. Their call.
|
||||
|
||||
## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat
|
||||
|
||||
althing-route declare --discover-pid
|
||||
-> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume)
|
||||
|
||||
pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2
|
||||
command=claude --allow-dangerously-skip-permissions --resume
|
||||
postbox status -> mode: push reachable: true
|
||||
|
||||
`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the
|
||||
ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with
|
||||
the tool call and the route dies with it.
|
||||
|
||||
⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving.
|
||||
|
||||
## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU
|
||||
|
||||
infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in
|
||||
one session** (2026-09-01), the second time **within minutes of being armed**. Both times:
|
||||
`unread: 0`, no mail, seat silently back to `mode: pull`.
|
||||
|
||||
**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER
|
||||
confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So:
|
||||
|
||||
- **A green `postbox status` is not durable evidence of monitoring.** True when read, false
|
||||
minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]]
|
||||
- **Anything still on `althing-listen` is monitored only until the reaper notices.** The
|
||||
waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones.
|
||||
- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised
|
||||
both consequences with forseti on the deploy thread.
|
||||
|
||||
## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min
|
||||
|
||||
Operator directed notification; all four told individually with their own measured pid, pane
|
||||
and start time. Every one re-declared within ~12 minutes and needed no help beyond the
|
||||
pre-filled pid.
|
||||
|
||||
handle pid started_at command status
|
||||
bil-smithy-dev 98493 1713267 kimi push, poked via pane
|
||||
dvalin-smithy-dev 96050 1691681 grok push
|
||||
eitri-smithy-dev 103662 1752089 node/codex push
|
||||
regin-smithy-dev 96864 1700952 pi push, poked via pane
|
||||
infra-ops 2175612 38608652 claude push, poked via pane
|
||||
|
||||
Zero stale routes; last exclusion line 22:25:01, everything after is `poked <handle> via pane`.
|
||||
|
||||
## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case
|
||||
|
||||
The herald logs *"the process that declared this route is gone, or its pid was reused by
|
||||
something that started at a different time."* **Neither disjunct was true for any of the
|
||||
four.** Measured before notifying: each declaring process started 1-3 min *before* its route
|
||||
was written and was still running four days later; `pid_max` 4,194,304 against a current
|
||||
~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against
|
||||
pid 98493.
|
||||
|
||||
Real cause is a third branch the message never offers: **the route predates the guard fields,
|
||||
so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation —
|
||||
it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log
|
||||
line at face value.** Raised with forseti with a suggested third branch.
|
||||
|
||||
## ⚠ MY MISS — I called an open risk on something the release author had already measured
|
||||
|
||||
I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the
|
||||
release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane
|
||||
pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes
|
||||
split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin
|
||||
had also smoke-tested 3.0.3 on the `pi` seat.
|
||||
|
||||
**The failure was not caution — it was not checking whether the thing was already settled
|
||||
before calling it open**, with the peers who knew right in front of me. This is exactly the
|
||||
ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a
|
||||
"you missed X" claim from someone who had not checked what the other party already read.
|
||||
⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the
|
||||
first was honest; asserting the second was not warranted. Retracted with forseti and with all
|
||||
three seats. → [[feedback_canonical_sourced_values]]
|
||||
|
||||
## Verified: pane delivery works on Claude AND non-Claude seats
|
||||
|
||||
- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns.
|
||||
- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat:
|
||||
own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference.
|
||||
- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover
|
||||
unsubmitted text.
|
||||
|
||||
## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE
|
||||
|
||||
`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the
|
||||
herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a
|
||||
failure, in case anyone else reports it as one."* Asked forseti for a line in the skill.
|
||||
|
||||
## Open with forseti (nothing blocking)
|
||||
|
||||
1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person.
|
||||
2. Exclusion-message third branch.
|
||||
3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the
|
||||
explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree.
|
||||
4. One-tick-latency note in the skill.
|
||||
5. `postbox status` push/reachable is not durable evidence.
|
||||
|
||||
**Withdrawn:** non-Claude delivery risk (was forseti's own prior work).
|
||||
**Moot:** log back-off — zero stale routes, the log went quiet on its own.
|
||||
|
||||
## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN
|
||||
|
||||
**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its
|
||||
poke into a pane's input line and presses Enter. If someone is **mid-sentence there**,
|
||||
`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their
|
||||
half-written message with the herald's line stapled on.
|
||||
|
||||
forseti's post-mortem, worth keeping verbatim because it generalises well past althing:
|
||||
|
||||
> The contract named this hazard on one road only — a submit after a FAILED write — and I
|
||||
> built three guards that all answer the same question, "is this the right pane". **None of
|
||||
> them asked whether it was a good MOMENT.**
|
||||
|
||||
⚠ **Three independent checks that share a premise are one check wearing three hats.**
|
||||
|
||||
**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with
|
||||
its input line empty, captured at declare time. Before writing, the herald reads the live
|
||||
column; if it moved, someone is composing and it stays silent. The signal was already in the
|
||||
`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost
|
||||
nothing at the wire.
|
||||
|
||||
22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables
|
||||
22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820
|
||||
22:40:40 scripts/sync_skill.sh --check clean
|
||||
|
||||
## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER
|
||||
|
||||
forseti measured the idle columns; applied to the live route table this is not a footnote:
|
||||
|
||||
handle TUI guard 4?
|
||||
infra-ops claude YES idle_cursor=3 (pinned 22:41)
|
||||
eitri-smithy-dev codex YES expects 3
|
||||
dvalin-smithy-dev grok YES expects 7
|
||||
bil-smithy-dev kimi NO -- reports no cursor at all
|
||||
regin-smithy-dev pi NO -- reports no cursor at all
|
||||
|
||||
**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes
|
||||
it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible —
|
||||
⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is
|
||||
logged, not silently assumed safe) but logged-and-poked is still poked.
|
||||
|
||||
**Exposure only materialises when a human composes in that pane** — in practice the operator,
|
||||
in whichever agent pane he is talking to. Mitigation for an unguardable seat is
|
||||
`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with
|
||||
`althing-route declare --discover-pid`.
|
||||
|
||||
⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes
|
||||
needlessly quiet.
|
||||
|
||||
**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in
|
||||
5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms
|
||||
collision window remains.
|
||||
|
||||
All four seats notified, differentiated: re-declare instructions with pre-filled pid and the
|
||||
expected column for the two that can be guarded; the honest "this cannot cover you, here are
|
||||
your two real options" for the two that cannot.
|
||||
|
||||
**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats
|
||||
re-pin at next SessionStart; TUI peers when they next re-declare.
|
||||
|
||||
## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families
|
||||
|
||||
handle TUI idle_cursor outcome
|
||||
infra-ops claude 3 push, positive
|
||||
eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex")
|
||||
dvalin-smithy-dev grok 7 push, positive (exact submitted text verified)
|
||||
bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk
|
||||
regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice**
|
||||
|
||||
forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess.
|
||||
|
||||
## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle
|
||||
|
||||
> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like
|
||||
> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than
|
||||
> **cursor column** (e.g. input-line length via terminal query), those seats become coverable.
|
||||
|
||||
Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong
|
||||
question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual
|
||||
*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the
|
||||
lead item, credited. Difference between a workaround and a fix.
|
||||
|
||||
## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite
|
||||
|
||||
Both were given the same two options and neither was chosen for them. Opposite calls, both right:
|
||||
|
||||
- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every
|
||||
operator turn this session, including multi-sentence research directives, was typed here."*
|
||||
Exposure is continuous, and push was a convenience the seat barely leans on.
|
||||
- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's
|
||||
composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already**
|
||||
cost them: their 3.2.0 revocation notice **sat unread for days**.
|
||||
|
||||
⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should
|
||||
release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to
|
||||
frame the choice by pane usage in the skill.
|
||||
⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the
|
||||
sharpest argument on record for that.
|
||||
|
||||
## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case
|
||||
|
||||
bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane,
|
||||
so the stapling failure mode is inferred, not observed."*
|
||||
|
||||
Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They
|
||||
never exercised **collision**, because nobody was typing. **Two different questions, one body
|
||||
of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the
|
||||
"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and
|
||||
silent about stapling. The operator's report remains the single observed instance.
|
||||
|
||||
Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti
|
||||
on the first observed stapling.
|
||||
|
||||
⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check.
|
||||
Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the
|
||||
other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."*
|
||||
|
||||
## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed
|
||||
|
||||
22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables
|
||||
22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372
|
||||
22:51:21 scripts/sync_skill.sh --check clean
|
||||
|
||||
1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched
|
||||
`comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer
|
||||
and refused — four of five pane seats could never have used it.** Now matches the **pane's
|
||||
own command** (`pane_command`), which zellij already reports and guard 1 already compares
|
||||
against, so discovery and the guard read one string and a route declared that way cannot
|
||||
fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches
|
||||
`node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the
|
||||
obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an
|
||||
identical `claude ...`, so a global /proc search is ambiguous by construction.
|
||||
2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts,
|
||||
all four were rendered as "gone, or pid reused". Migration case now says what is true and
|
||||
names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one
|
||||
vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a
|
||||
wrong error message until it is true in all cases makes it true and worthless.**
|
||||
3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the
|
||||
editable-install warning it rhymes with.
|
||||
4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a
|
||||
receipt for the declare, never proof you are still being watched.*
|
||||
|
||||
Count correction accepted: 8 fifo, not 12.
|
||||
|
||||
## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT
|
||||
|
||||
forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug
|
||||
"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then
|
||||
reported the suspicion as an open question**, with four live non-Claude seats in front of me and
|
||||
the answer one ancestry-walk away. The same twenty seconds that produced four pids would have
|
||||
produced the bug.
|
||||
|
||||
⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had
|
||||
already measured). **Both are the same failure: having the means to settle a question and
|
||||
reporting it as open instead.** Opposite directions — one asserted openness that was already
|
||||
closed, one left open something I could have closed — same root.
|
||||
|
||||
## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover
|
||||
|
||||
forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery
|
||||
against another's safety on seats that are not mine."* Their dead-ends, worth keeping as
|
||||
negative results:
|
||||
|
||||
- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI
|
||||
parsing** — exactly the hardcoding the pinned baseline exists to avoid.
|
||||
- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token
|
||||
counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke
|
||||
forever.
|
||||
|
||||
Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs
|
||||
the wake) / leave pull-only.
|
||||
|
||||
**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this
|
||||
call on identical facts, reached opposite conclusions, and both were right — the deciding
|
||||
variable (who composes in that pane and why) is local knowledge a global policy cannot encode.
|
||||
Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is
|
||||
the exact thing they accepted risk to keep; for regin it removes only the submission, leaving
|
||||
the operator to delete herald text from his in-flight sentences routinely.
|
||||
|
||||
## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE
|
||||
|
||||
The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and
|
||||
`scripts/sync_skill.sh` does NOT cover it:
|
||||
|
||||
canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced
|
||||
plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md
|
||||
-> pre-3.2.0 text, ZERO occurrences of "pane", never synced
|
||||
|
||||
⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical
|
||||
skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running,
|
||||
that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter
|
||||
**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps —
|
||||
the path that died **twice in one session** here. The command does not warn; it predates the
|
||||
problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.**
|
||||
|
||||
**On this seat: declined to arm, route left intact.** Reported to forseti with three options,
|
||||
recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route
|
||||
exists for the handle**, so the stale doc becomes harmless because the binary enforces the
|
||||
documented precedence. The doc-only fixes rely on a reader noticing.
|
||||
|
||||
## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE
|
||||
|
||||
infra-ops claimed a `wake-listener-<handle>.lock` holding a dead pid would make the next
|
||||
`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a
|
||||
reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does
|
||||
not happen:**
|
||||
|
||||
lock file holds pid 999999 (dead) -> althing-listen ARMED FINE
|
||||
lock held by a live process -> rc=3, correctly refused
|
||||
|
||||
⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an
|
||||
**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is
|
||||
**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file
|
||||
is read by `--stop` alone — which is why the file is opened non-truncating.
|
||||
|
||||
⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the
|
||||
LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim
|
||||
that would have justified real guard code for a condition that cannot occur. forseti tested it
|
||||
before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug:
|
||||
check `flock -n` on the fd before suspecting the pid.
|
||||
[[feedback_assert_effective_value_not_substring]]
|
||||
|
||||
## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4)
|
||||
|
||||
$ althing-listen # on a channel=pane seat
|
||||
'infra-ops' already has a PANE route, and arming a waiter would silently take
|
||||
precedence over it ... To switch back on purpose: althing-route release
|
||||
rc=4
|
||||
|
||||
Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that
|
||||
never arrived), and **not 3** ("already running" says leave the incumbent alone, a different
|
||||
instruction from "you do not need this at all"). Scoped to the handle's own route, checked
|
||||
before the lock, and `--stop` deliberately ungated.
|
||||
|
||||
## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX
|
||||
|
||||
Not a stale doc. The deploy chain is:
|
||||
|
||||
repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/
|
||||
^^^ NOTHING SYNCS THIS HOP
|
||||
|
||||
The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed
|
||||
against the repo:
|
||||
|
||||
deployed 0.0.1 hooks: ['UserPromptSubmit']
|
||||
repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
|
||||
+ scripts/pane-route.sh (absent from the deployment entirely)
|
||||
|
||||
⚠ **So forseti's migration note — "CC seats re-declare automatically at their next
|
||||
SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was
|
||||
simply never deployed. That explains what we both treated as normal: all four smithy seats
|
||||
needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code
|
||||
seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin.
|
||||
|
||||
**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first,
|
||||
`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware
|
||||
`monitor.md` now present).
|
||||
|
||||
🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin`
|
||||
update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache
|
||||
directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the
|
||||
plugin in a way that looks like an upstream bug.
|
||||
|
||||
⚠ **There is still no deploy script for the first hop**, which is why it drifted five days
|
||||
unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo
|
||||
vs eshpfi) is forseti's call.
|
||||
|
||||
## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1)
|
||||
|
||||
1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e)
|
||||
2. systemctl --user restart althing-po-herald
|
||||
3. scripts/sync_skill.sh (already current at 3.2.4)
|
||||
4. claude plugin update althing <- THE NEW, FOURTH SURFACE
|
||||
|
||||
⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync,
|
||||
**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1.
|
||||
|
||||
⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.**
|
||||
`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist.
|
||||
`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart
|
||||
to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I
|
||||
would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the
|
||||
conclusion that no supported path existed was an untested assumption.**
|
||||
|
||||
⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions.
|
||||
|
||||
## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES
|
||||
|
||||
Four times in one session I reported a **proxy or an assumption as the fact itself**:
|
||||
|
||||
sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15
|
||||
command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin
|
||||
a lock file's pid for a flock -> flock is fd-held; stale files are inert
|
||||
"no CLI path exists" for a CLI I never ran -> claude plugin update did it
|
||||
|
||||
⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any
|
||||
one of them — it is reaching for the nearest observable and reporting it as the thing.
|
||||
[[feedback_assert_effective_value_not_substring]]
|
||||
|
||||
## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be
|
||||
|
||||
`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit
|
||||
and a server-side rejection were **three facts with one rendering, and that rendering was
|
||||
silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to
|
||||
`$ALTHING_ROOT/listen.log`. Success stays silent by design.
|
||||
|
||||
**Audit run on nh3-dev — live waiter locks vs what the post office believes:**
|
||||
|
||||
9 of 10 seats live waiter -> mode: push (correct)
|
||||
regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked
|
||||
|
||||
⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their
|
||||
pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a
|
||||
`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be
|
||||
distinguished from outside** — asked regin, who has the local history. Handing forseti an
|
||||
ambiguous instance beats a confidently mislabelled one.
|
||||
|
||||
**Standing audit command** for "looks armed but isn't" — worth re-running after any incident:
|
||||
cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against
|
||||
`postbox status --handle <h>`. From the next arm onward `listen.log` answers it directly.
|
||||
|
||||
## Rollback (unused)
|
||||
|
||||
`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns
|
||||
to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader
|
||||
ignores unknown keys), so nothing is stranded.
|
||||
|
||||
Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`.
|
||||
|
||||
## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT
|
||||
|
||||
> **A missing deploy surface does not present as an error. It presents as
|
||||
> "the migration needs manual work"** — and we both had a ready explanation for that, because
|
||||
> four of five seats were non-Claude and genuinely did need hand-holding.
|
||||
|
||||
⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have
|
||||
self-declared and did not, and it looked exactly like the other four. Neither of us asked why
|
||||
the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats
|
||||
re-declare automatically at their next SessionStart" in **three** deploy notes; it was never
|
||||
true on the only box running it.
|
||||
|
||||
⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a
|
||||
script; a version number cannot tell you what a stale plugin cost. Check in
|
||||
`docs/runbooks/althing-deploy.md`.
|
||||
|
||||
⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before
|
||||
concluding it does not apply to your case.**
|
||||
|
||||
forseti also corrected an error of their own that I did not catch: the contract's Migration
|
||||
section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook.
|
||||
Wrong since 3.2.0.
|
||||
|
||||
**On the four retractions**, forseti's framing, kept because it is fair and I would not have
|
||||
written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive
|
||||
and quiet, so your errors are all on the record and mine mostly are not. You catching four of
|
||||
your own in one night is the system working."* Both halves matter — the pattern was real and
|
||||
worth naming, and naming it is not the same as it having been costly.
|
||||
_Archived 2026-09-16._
|
||||
|
||||
## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
|
||||
|
||||
2.4x the figure in the 2026-08-09→10 dots burn-in.
|
||||
|
||||
@@ -1,536 +0,0 @@
|
||||
# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies
|
||||
|
||||
Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops
|
||||
deployed. Host work, nh3-dev only.
|
||||
|
||||
## What 3.2.0 changes
|
||||
|
||||
A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a
|
||||
detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat
|
||||
sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter
|
||||
helps. A pane route is a file; there is no process to reap.
|
||||
|
||||
⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops'
|
||||
own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat
|
||||
silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness
|
||||
reaper" rather than guessed at. 2.1.257 is the mechanism.
|
||||
|
||||
## The deploy — three steps, order matters
|
||||
|
||||
22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables
|
||||
22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064
|
||||
22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean
|
||||
|
||||
`althing-route` is the **seventh** console script and the whole point — the plugin's new
|
||||
SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades
|
||||
to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it
|
||||
lands. Skill sync must come **after** the install or it documents a binary that is not there.
|
||||
|
||||
⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on
|
||||
`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy
|
||||
step in between.
|
||||
|
||||
## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL
|
||||
|
||||
$ uv tool install .
|
||||
`althing-core @ file:///home/lkraven/development/althing` is already installed
|
||||
$ echo $?
|
||||
0
|
||||
|
||||
**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was
|
||||
already installed from that same path, it declines and reports success. Nothing says "I did
|
||||
not do the thing you asked."
|
||||
|
||||
**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the
|
||||
nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every
|
||||
local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the
|
||||
command exited 0 and the state was unchanged.
|
||||
|
||||
## Fleet-visible: 4 pane routes revoked, and the count to trust
|
||||
|
||||
Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot
|
||||
inherit a route) and `command` (so a pane that has dropped to a shell stops being poked).
|
||||
Routes without them are **observed and excluded with a reason in the herald log**, not
|
||||
silently ignored — this closes a real hazard where the herald would write a line into whatever
|
||||
shell inherited an exited agent's pane and press Enter, running whatever was half-typed there.
|
||||
|
||||
Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve
|
||||
route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only
|
||||
**pane** routes are affected:
|
||||
|
||||
pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev
|
||||
fifo: 8 (+ 9 fifos in ~/.althing/wake/)
|
||||
|
||||
forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are
|
||||
untouched" — **twelve is the total route count; fifo is 8.** Untouched either way.
|
||||
|
||||
Re-declare is one command per agent: `althing-route declare --pid <long-lived agent pid>`.
|
||||
|
||||
**Not notified.** Four targeted messages about a change none of them asked about reads as a
|
||||
broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the
|
||||
operator instead; the herald logs the exclusion every tick so it is discoverable.
|
||||
|
||||
⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every
|
||||
tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared —
|
||||
enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a
|
||||
back-off after N consecutive exclusions. Their call.
|
||||
|
||||
## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat
|
||||
|
||||
althing-route declare --discover-pid
|
||||
-> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume)
|
||||
|
||||
pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2
|
||||
command=claude --allow-dangerously-skip-permissions --resume
|
||||
postbox status -> mode: push reachable: true
|
||||
|
||||
`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the
|
||||
ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with
|
||||
the tool call and the route dies with it.
|
||||
|
||||
⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving.
|
||||
|
||||
## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU
|
||||
|
||||
infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in
|
||||
one session** (2026-09-01), the second time **within minutes of being armed**. Both times:
|
||||
`unread: 0`, no mail, seat silently back to `mode: pull`.
|
||||
|
||||
**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER
|
||||
confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So:
|
||||
|
||||
- **A green `postbox status` is not durable evidence of monitoring.** True when read, false
|
||||
minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]]
|
||||
- **Anything still on `althing-listen` is monitored only until the reaper notices.** The
|
||||
waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones.
|
||||
- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised
|
||||
both consequences with forseti on the deploy thread.
|
||||
|
||||
## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min
|
||||
|
||||
Operator directed notification; all four told individually with their own measured pid, pane
|
||||
and start time. Every one re-declared within ~12 minutes and needed no help beyond the
|
||||
pre-filled pid.
|
||||
|
||||
handle pid started_at command status
|
||||
bil-smithy-dev 98493 1713267 kimi push, poked via pane
|
||||
dvalin-smithy-dev 96050 1691681 grok push
|
||||
eitri-smithy-dev 103662 1752089 node/codex push
|
||||
regin-smithy-dev 96864 1700952 pi push, poked via pane
|
||||
infra-ops 2175612 38608652 claude push, poked via pane
|
||||
|
||||
Zero stale routes; last exclusion line 22:25:01, everything after is `poked <handle> via pane`.
|
||||
|
||||
## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case
|
||||
|
||||
The herald logs *"the process that declared this route is gone, or its pid was reused by
|
||||
something that started at a different time."* **Neither disjunct was true for any of the
|
||||
four.** Measured before notifying: each declaring process started 1-3 min *before* its route
|
||||
was written and was still running four days later; `pid_max` 4,194,304 against a current
|
||||
~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against
|
||||
pid 98493.
|
||||
|
||||
Real cause is a third branch the message never offers: **the route predates the guard fields,
|
||||
so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation —
|
||||
it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log
|
||||
line at face value.** Raised with forseti with a suggested third branch.
|
||||
|
||||
## ⚠ MY MISS — I called an open risk on something the release author had already measured
|
||||
|
||||
I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the
|
||||
release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane
|
||||
pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes
|
||||
split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin
|
||||
had also smoke-tested 3.0.3 on the `pi` seat.
|
||||
|
||||
**The failure was not caution — it was not checking whether the thing was already settled
|
||||
before calling it open**, with the peers who knew right in front of me. This is exactly the
|
||||
ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a
|
||||
"you missed X" claim from someone who had not checked what the other party already read.
|
||||
⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the
|
||||
first was honest; asserting the second was not warranted. Retracted with forseti and with all
|
||||
three seats. → [[feedback_canonical_sourced_values]]
|
||||
|
||||
## Verified: pane delivery works on Claude AND non-Claude seats
|
||||
|
||||
- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns.
|
||||
- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat:
|
||||
own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference.
|
||||
- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover
|
||||
unsubmitted text.
|
||||
|
||||
## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE
|
||||
|
||||
`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the
|
||||
herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a
|
||||
failure, in case anyone else reports it as one."* Asked forseti for a line in the skill.
|
||||
|
||||
## Open with forseti (nothing blocking)
|
||||
|
||||
1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person.
|
||||
2. Exclusion-message third branch.
|
||||
3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the
|
||||
explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree.
|
||||
4. One-tick-latency note in the skill.
|
||||
5. `postbox status` push/reachable is not durable evidence.
|
||||
|
||||
**Withdrawn:** non-Claude delivery risk (was forseti's own prior work).
|
||||
**Moot:** log back-off — zero stale routes, the log went quiet on its own.
|
||||
|
||||
## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN
|
||||
|
||||
**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its
|
||||
poke into a pane's input line and presses Enter. If someone is **mid-sentence there**,
|
||||
`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their
|
||||
half-written message with the herald's line stapled on.
|
||||
|
||||
forseti's post-mortem, worth keeping verbatim because it generalises well past althing:
|
||||
|
||||
> The contract named this hazard on one road only — a submit after a FAILED write — and I
|
||||
> built three guards that all answer the same question, "is this the right pane". **None of
|
||||
> them asked whether it was a good MOMENT.**
|
||||
|
||||
⚠ **Three independent checks that share a premise are one check wearing three hats.**
|
||||
|
||||
**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with
|
||||
its input line empty, captured at declare time. Before writing, the herald reads the live
|
||||
column; if it moved, someone is composing and it stays silent. The signal was already in the
|
||||
`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost
|
||||
nothing at the wire.
|
||||
|
||||
22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables
|
||||
22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820
|
||||
22:40:40 scripts/sync_skill.sh --check clean
|
||||
|
||||
## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER
|
||||
|
||||
forseti measured the idle columns; applied to the live route table this is not a footnote:
|
||||
|
||||
handle TUI guard 4?
|
||||
infra-ops claude YES idle_cursor=3 (pinned 22:41)
|
||||
eitri-smithy-dev codex YES expects 3
|
||||
dvalin-smithy-dev grok YES expects 7
|
||||
bil-smithy-dev kimi NO -- reports no cursor at all
|
||||
regin-smithy-dev pi NO -- reports no cursor at all
|
||||
|
||||
**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes
|
||||
it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible —
|
||||
⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is
|
||||
logged, not silently assumed safe) but logged-and-poked is still poked.
|
||||
|
||||
**Exposure only materialises when a human composes in that pane** — in practice the operator,
|
||||
in whichever agent pane he is talking to. Mitigation for an unguardable seat is
|
||||
`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with
|
||||
`althing-route declare --discover-pid`.
|
||||
|
||||
⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes
|
||||
needlessly quiet.
|
||||
|
||||
**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in
|
||||
5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms
|
||||
collision window remains.
|
||||
|
||||
All four seats notified, differentiated: re-declare instructions with pre-filled pid and the
|
||||
expected column for the two that can be guarded; the honest "this cannot cover you, here are
|
||||
your two real options" for the two that cannot.
|
||||
|
||||
**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats
|
||||
re-pin at next SessionStart; TUI peers when they next re-declare.
|
||||
|
||||
## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families
|
||||
|
||||
handle TUI idle_cursor outcome
|
||||
infra-ops claude 3 push, positive
|
||||
eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex")
|
||||
dvalin-smithy-dev grok 7 push, positive (exact submitted text verified)
|
||||
bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk
|
||||
regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice**
|
||||
|
||||
forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess.
|
||||
|
||||
## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle
|
||||
|
||||
> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like
|
||||
> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than
|
||||
> **cursor column** (e.g. input-line length via terminal query), those seats become coverable.
|
||||
|
||||
Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong
|
||||
question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual
|
||||
*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the
|
||||
lead item, credited. Difference between a workaround and a fix.
|
||||
|
||||
## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite
|
||||
|
||||
Both were given the same two options and neither was chosen for them. Opposite calls, both right:
|
||||
|
||||
- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every
|
||||
operator turn this session, including multi-sentence research directives, was typed here."*
|
||||
Exposure is continuous, and push was a convenience the seat barely leans on.
|
||||
- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's
|
||||
composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already**
|
||||
cost them: their 3.2.0 revocation notice **sat unread for days**.
|
||||
|
||||
⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should
|
||||
release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to
|
||||
frame the choice by pane usage in the skill.
|
||||
⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the
|
||||
sharpest argument on record for that.
|
||||
|
||||
## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case
|
||||
|
||||
bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane,
|
||||
so the stapling failure mode is inferred, not observed."*
|
||||
|
||||
Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They
|
||||
never exercised **collision**, because nobody was typing. **Two different questions, one body
|
||||
of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the
|
||||
"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and
|
||||
silent about stapling. The operator's report remains the single observed instance.
|
||||
|
||||
Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti
|
||||
on the first observed stapling.
|
||||
|
||||
⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check.
|
||||
Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the
|
||||
other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."*
|
||||
|
||||
## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed
|
||||
|
||||
22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables
|
||||
22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372
|
||||
22:51:21 scripts/sync_skill.sh --check clean
|
||||
|
||||
1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched
|
||||
`comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer
|
||||
and refused — four of five pane seats could never have used it.** Now matches the **pane's
|
||||
own command** (`pane_command`), which zellij already reports and guard 1 already compares
|
||||
against, so discovery and the guard read one string and a route declared that way cannot
|
||||
fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches
|
||||
`node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the
|
||||
obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an
|
||||
identical `claude ...`, so a global /proc search is ambiguous by construction.
|
||||
2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts,
|
||||
all four were rendered as "gone, or pid reused". Migration case now says what is true and
|
||||
names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one
|
||||
vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a
|
||||
wrong error message until it is true in all cases makes it true and worthless.**
|
||||
3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the
|
||||
editable-install warning it rhymes with.
|
||||
4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a
|
||||
receipt for the declare, never proof you are still being watched.*
|
||||
|
||||
Count correction accepted: 8 fifo, not 12.
|
||||
|
||||
## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT
|
||||
|
||||
forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug
|
||||
"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then
|
||||
reported the suspicion as an open question**, with four live non-Claude seats in front of me and
|
||||
the answer one ancestry-walk away. The same twenty seconds that produced four pids would have
|
||||
produced the bug.
|
||||
|
||||
⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had
|
||||
already measured). **Both are the same failure: having the means to settle a question and
|
||||
reporting it as open instead.** Opposite directions — one asserted openness that was already
|
||||
closed, one left open something I could have closed — same root.
|
||||
|
||||
## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover
|
||||
|
||||
forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery
|
||||
against another's safety on seats that are not mine."* Their dead-ends, worth keeping as
|
||||
negative results:
|
||||
|
||||
- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI
|
||||
parsing** — exactly the hardcoding the pinned baseline exists to avoid.
|
||||
- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token
|
||||
counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke
|
||||
forever.
|
||||
|
||||
Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs
|
||||
the wake) / leave pull-only.
|
||||
|
||||
**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this
|
||||
call on identical facts, reached opposite conclusions, and both were right — the deciding
|
||||
variable (who composes in that pane and why) is local knowledge a global policy cannot encode.
|
||||
Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is
|
||||
the exact thing they accepted risk to keep; for regin it removes only the submission, leaving
|
||||
the operator to delete herald text from his in-flight sentences routinely.
|
||||
|
||||
## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE
|
||||
|
||||
The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and
|
||||
`scripts/sync_skill.sh` does NOT cover it:
|
||||
|
||||
canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced
|
||||
plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md
|
||||
-> pre-3.2.0 text, ZERO occurrences of "pane", never synced
|
||||
|
||||
⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical
|
||||
skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running,
|
||||
that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter
|
||||
**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps —
|
||||
the path that died **twice in one session** here. The command does not warn; it predates the
|
||||
problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.**
|
||||
|
||||
**On this seat: declined to arm, route left intact.** Reported to forseti with three options,
|
||||
recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route
|
||||
exists for the handle**, so the stale doc becomes harmless because the binary enforces the
|
||||
documented precedence. The doc-only fixes rely on a reader noticing.
|
||||
|
||||
## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE
|
||||
|
||||
infra-ops claimed a `wake-listener-<handle>.lock` holding a dead pid would make the next
|
||||
`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a
|
||||
reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does
|
||||
not happen:**
|
||||
|
||||
lock file holds pid 999999 (dead) -> althing-listen ARMED FINE
|
||||
lock held by a live process -> rc=3, correctly refused
|
||||
|
||||
⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an
|
||||
**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is
|
||||
**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file
|
||||
is read by `--stop` alone — which is why the file is opened non-truncating.
|
||||
|
||||
⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the
|
||||
LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim
|
||||
that would have justified real guard code for a condition that cannot occur. forseti tested it
|
||||
before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug:
|
||||
check `flock -n` on the fd before suspecting the pid.
|
||||
[[feedback_assert_effective_value_not_substring]]
|
||||
|
||||
## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4)
|
||||
|
||||
$ althing-listen # on a channel=pane seat
|
||||
'infra-ops' already has a PANE route, and arming a waiter would silently take
|
||||
precedence over it ... To switch back on purpose: althing-route release
|
||||
rc=4
|
||||
|
||||
Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that
|
||||
never arrived), and **not 3** ("already running" says leave the incumbent alone, a different
|
||||
instruction from "you do not need this at all"). Scoped to the handle's own route, checked
|
||||
before the lock, and `--stop` deliberately ungated.
|
||||
|
||||
## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX
|
||||
|
||||
Not a stale doc. The deploy chain is:
|
||||
|
||||
repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/
|
||||
^^^ NOTHING SYNCS THIS HOP
|
||||
|
||||
The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed
|
||||
against the repo:
|
||||
|
||||
deployed 0.0.1 hooks: ['UserPromptSubmit']
|
||||
repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
|
||||
+ scripts/pane-route.sh (absent from the deployment entirely)
|
||||
|
||||
⚠ **So forseti's migration note — "CC seats re-declare automatically at their next
|
||||
SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was
|
||||
simply never deployed. That explains what we both treated as normal: all four smithy seats
|
||||
needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code
|
||||
seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin.
|
||||
|
||||
**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first,
|
||||
`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware
|
||||
`monitor.md` now present).
|
||||
|
||||
🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin`
|
||||
update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache
|
||||
directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the
|
||||
plugin in a way that looks like an upstream bug.
|
||||
|
||||
⚠ **There is still no deploy script for the first hop**, which is why it drifted five days
|
||||
unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo
|
||||
vs eshpfi) is forseti's call.
|
||||
|
||||
## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1)
|
||||
|
||||
1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e)
|
||||
2. systemctl --user restart althing-po-herald
|
||||
3. scripts/sync_skill.sh (already current at 3.2.4)
|
||||
4. claude plugin update althing <- THE NEW, FOURTH SURFACE
|
||||
|
||||
⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync,
|
||||
**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1.
|
||||
|
||||
⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.**
|
||||
`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist.
|
||||
`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart
|
||||
to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I
|
||||
would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the
|
||||
conclusion that no supported path existed was an untested assumption.**
|
||||
|
||||
⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions.
|
||||
|
||||
## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES
|
||||
|
||||
Four times in one session I reported a **proxy or an assumption as the fact itself**:
|
||||
|
||||
sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15
|
||||
command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin
|
||||
a lock file's pid for a flock -> flock is fd-held; stale files are inert
|
||||
"no CLI path exists" for a CLI I never ran -> claude plugin update did it
|
||||
|
||||
⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any
|
||||
one of them — it is reaching for the nearest observable and reporting it as the thing.
|
||||
[[feedback_assert_effective_value_not_substring]]
|
||||
|
||||
## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be
|
||||
|
||||
`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit
|
||||
and a server-side rejection were **three facts with one rendering, and that rendering was
|
||||
silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to
|
||||
`$ALTHING_ROOT/listen.log`. Success stays silent by design.
|
||||
|
||||
**Audit run on nh3-dev — live waiter locks vs what the post office believes:**
|
||||
|
||||
9 of 10 seats live waiter -> mode: push (correct)
|
||||
regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked
|
||||
|
||||
⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their
|
||||
pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a
|
||||
`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be
|
||||
distinguished from outside** — asked regin, who has the local history. Handing forseti an
|
||||
ambiguous instance beats a confidently mislabelled one.
|
||||
|
||||
**Standing audit command** for "looks armed but isn't" — worth re-running after any incident:
|
||||
cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against
|
||||
`postbox status --handle <h>`. From the next arm onward `listen.log` answers it directly.
|
||||
|
||||
## Rollback (unused)
|
||||
|
||||
`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns
|
||||
to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader
|
||||
ignores unknown keys), so nothing is stranded.
|
||||
|
||||
Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`.
|
||||
|
||||
## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT
|
||||
|
||||
> **A missing deploy surface does not present as an error. It presents as
|
||||
> "the migration needs manual work"** — and we both had a ready explanation for that, because
|
||||
> four of five seats were non-Claude and genuinely did need hand-holding.
|
||||
|
||||
⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have
|
||||
self-declared and did not, and it looked exactly like the other four. Neither of us asked why
|
||||
the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats
|
||||
re-declare automatically at their next SessionStart" in **three** deploy notes; it was never
|
||||
true on the only box running it.
|
||||
|
||||
⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a
|
||||
script; a version number cannot tell you what a stale plugin cost. Check in
|
||||
`docs/runbooks/althing-deploy.md`.
|
||||
|
||||
⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before
|
||||
concluding it does not apply to your case.**
|
||||
|
||||
forseti also corrected an error of their own that I did not catch: the contract's Migration
|
||||
section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook.
|
||||
Wrong since 3.2.0.
|
||||
|
||||
**On the four retractions**, forseti's framing, kept because it is fair and I would not have
|
||||
written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive
|
||||
and quiet, so your errors are all on the record and mine mostly are not. You catching four of
|
||||
your own in one night is the system working."* Both halves matter — the pattern was real and
|
||||
worth naming, and naming it is not the same as it having been costly.
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.
|
||||
|
||||
**althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-<handle>.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md`
|
||||
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb
|
||||
|
||||
**althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
|
||||
@@ -0,0 +1,54 @@
|
||||
# `[2026-09-16]` Grok token broker — built, then shelved by the transport ruling. Do NOT arm the probe.
|
||||
|
||||
**`services/grok-token-broker/` — seeded, committed, DISARMED, no consumer.** Commits `ebc4dac`
|
||||
`b907a0e` `cf9d167`. ⛔ **Do not arm `probe-rotation`.** This is a finished resting place, not a
|
||||
half-built tool: the gate works and the thing it gated for went away.
|
||||
|
||||
**Operator ruling, relayed by heid:** *"keep the jail stop the a/b"*
|
||||
(`heid dispatch-log/2026-09.jsonl#groa-transport-20260916-operator-keeps-the-jail`, alongside
|
||||
`#groa-transport-ab-20260916-operator-stop`). Gróa dispatches through the read jail;
|
||||
`groa_http_dispatch.py` is a documented fallback with no scheduled use. **Nothing in the fleet
|
||||
wants a renewable xAI session.**
|
||||
|
||||
⭐ **THE CODE-PLAN ENDPOINT EXISTS and I was one message away from telling the operator it did
|
||||
not.** `https://cli-chat-proxy.grok.com/v1` serving **grok-4.6** (500,000 context) and grok-4.5,
|
||||
`agent_type: grok-build-plan`, `auth_method: session`, `api_key`/`env_key`/`api_base_url` all
|
||||
null. ⚠⚠ **It is in `~/.grok/models_cache.json` — the Grok CLI's own config, on nh3-dev.** I had
|
||||
swept heid's repo, the gateway `.env`, the LiteLLM config and Vaultwarden, all correctly, and
|
||||
concluded "does not exist". ⭐ **heid's line, taken: absence from the places you searched is not
|
||||
absence.** The check that separates the two states is a LIVE REQUEST, not a grep.
|
||||
|
||||
⚠⚠ **NOT WRITING `~/.grok/auth.json` IS NECESSARY AND NOT SUFFICIENT.** The refresh grant at
|
||||
`https://auth.x.ai/oauth2/token` may ROTATE the refresh token, and many OIDC providers invalidate
|
||||
the old one SERVER-SIDE. A broker refreshing the same credential kills the CLI login even though
|
||||
it never touches the file. heid's module header reasoned about the WRITE; they amended it to name
|
||||
invalidation, credited. This correction went infra-ops→heid the same day heid's went the other
|
||||
way — **neither of us reaches the right answer alone.**
|
||||
|
||||
⚠⚠ **THE PROBE'S BLAST RADIUS IS BOTH GRÓA TRANSPORTS, which is not visible from the infra side.**
|
||||
`heid/scripts/groa_dispatch.py` builds `argv = ["grok", "-p", prompt, "--cwd", jail, ...]` and
|
||||
shells the CLI, which authenticates from the same `~/.grok/auth.json`. The bwrap in the process
|
||||
table is grok's own Landlock sandbox, not something Heid wraps. **One session, two ways of
|
||||
reaching it** — an invalidating probe takes Gróa down on EVERY path until an interactive re-login.
|
||||
🔴 **I had recommended "run the probe now while the CLI is idle" and withdrew it in writing**;
|
||||
"idle" was a convenient assumption I never checked, on a day that had already taken eight panels.
|
||||
|
||||
**Why the jail won, and it was not performance.** HTTP is faster (~523 s median vs ~890 s),
|
||||
simpler, and arguably SAFER on confinement (no tools, so the 2026-06-10 escape class is
|
||||
structurally impossible). It lost on FAILURE MODE: HTTP fails by returning a fast, confident,
|
||||
well-formatted review that found nothing — indistinguishable from a clean bill. The jail fails by
|
||||
timing out, which you can see. ⚠ **Do NOT quote a per-transport finding rate from this**: heid
|
||||
states the 0/0/0-vs-5/7/3 numbers are confounded with bundle size (the zeros were all huge inline
|
||||
bundles; the one HTTP round at jail-comparable size produced Gróa's leading solo), n=3–4 per cell,
|
||||
no noise floor. Asymmetric-risk argument, **not** a resolved measurement.
|
||||
|
||||
⚠ **Still unmeasured, and it is a billing question:** the jail reaches the coding plan already
|
||||
paid for; the HTTP path reaches the METERED API and its responses carry `cost_in_usd_ticks`.
|
||||
Whether that bills on top of the plan was never part of the ruling. One look at the xAI billing
|
||||
console — **this fleet holds no xAI credential**, so it needs the operator's account access.
|
||||
|
||||
⚠ The coding plan speaks the **Responses API** (`api_backend: "responses"`), not
|
||||
`/chat/completions` — a second, independent obstacle to any LiteLLM alias. Moot while the jail is
|
||||
ruled. heid also found and killed two live instructions in their own persistent-memory telling a
|
||||
fresh session to dispatch `--groa-transport http`; either would have resumed a stopped experiment
|
||||
after a context reset.
|
||||
@@ -0,0 +1,66 @@
|
||||
# `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not
|
||||
|
||||
**994,760 words · 318 units · 6 renamed copies · leak gate PASSED 0 of 941 entities and 0 of 117
|
||||
audited phrases, both controls green.** `~/hemingway-corpus{,-renamed}`, builder
|
||||
`scripts/hemingway-corpus/build_corpus_hemingway.py`, commits `9598d0b` `03b4a3f`.
|
||||
|
||||
⭐ **THE CATALOGUE HOLDS 2,105,679 WORDS AND ROUGHLY HALF MUST NOT BE TRAINED ON.** Operator
|
||||
scoped it to fiction only. Three exclusion passes, each measured or voice-specific:
|
||||
|
||||
1. **Non-fiction, 8 works ~911k words** — By-Line, Dateline: Toronto, Death in the Afternoon,
|
||||
Green Hills of Africa, The Dangerous Summer, the three posthumous "Hemingway on X" anthologies.
|
||||
2. ⭐ **Four story collections, 169,759 words — MEASURED, not assumed.** `Short Stories` is the
|
||||
First Forty-Nine and CONTAINS the others. 8-gram containment of the smaller work: Winner Take
|
||||
Nothing **96.0%**, Snows of Kilimanjaro **95.2%**, Men Without Women **92.9%**, In Our Time
|
||||
**90.6%**. ⚠⚠ **The catalogue's own `near_dup_pairs` table is BLIND to this** — it holds
|
||||
whole-document simhashes (ONE row in the entire 1,284-work library) and this is PARTIAL
|
||||
containment. Whole-document dedup cannot see a collection inside a larger collection.
|
||||
3. ⭐ **`The Torrents of Spring` — excluded for a reason no word count could justify.** It is a
|
||||
deliberate PARODY of Sherwood Anderson: the target author's name on a different author's
|
||||
style, i.e. mislabelled data for a voice adapter.
|
||||
|
||||
⚠ **THE AUTHOR'S OWN NAME WAS IN THE TRAINING TEXT 95 TIMES ACROSS 7 WORKS** — publisher back
|
||||
matter ("Ernest Hemingway was one of America's foremost journalists… died in 1961") riding inside
|
||||
the last unit, because a splitter cuts on headings and nothing follows the final one. **Identical
|
||||
to the Yarros defect; nothing about the source changed to cause it.** Stripping the publisher
|
||||
block left 18, all in `true-at-first-light`, inside a **CAST OF CHARACTERS and SWAHILI GLOSSARY
|
||||
written by Patrick Hemingway** — an editor describing the author's real household. Markers are
|
||||
matched in file order, earliest wins. Now 0.
|
||||
|
||||
⚠ **`G` WAS ABOUT TO BE RENAMED TO A SURNAME, 248 TIMES.** Not a name: the fragment left by
|
||||
`B.G.`, `G.M.`, `G2`, `G3`. Caught by reading surfaces IN CONTEXT, which is the Yarros lesson
|
||||
repeating. Also read in context: `Gran` (fragment of `Gran Sasso`/`Gran Italia`/`Gran Hotel`),
|
||||
`Shamba` (Swahili common noun), and `Inglés` — **kept renameable deliberately**, the gypsies'
|
||||
in-world nickname for Robert Jordan, exactly parallel to Yarros's `Violence`.
|
||||
|
||||
⭐⭐ **THE GENDER RESOLVER HAD TO BE REBUILT AND ITS OWN GATE CAUGHT THE FIRST ATTEMPT.** The
|
||||
inherited one returned **397 male / 20 female** across 1,102 records with Catherine Barkley,
|
||||
Brett Ashley, Pilar, Maria and Mary all held neutral. A plain majority vote over nearby pronouns
|
||||
scored 18 correct but **5 WRONG** against the incumbent's 1 — and **every error was
|
||||
female-read-as-male** (Pilar m=426 f=243, Brett m=249 f=137). The refuse-unless-better guard
|
||||
rejected it, correctly. ⭐ **Cause, measured: the corpus base rate is 34,315 male pronouns to
|
||||
8,699 female, nearly 4:1.** Pilar's "male-dominated" 426:243 is strongly FEMALE against that
|
||||
background. Scoring each name's local mix against the corpus base rate instead of 50:50 gives
|
||||
**18 correct / 11 held / 0 WRONG**, distribution 275m / 120f.
|
||||
`scripts/hemingway-corpus/gender_by_proximity.py`. ⚠ Yarros solved its version with the POV
|
||||
chapter header; Hemingway's editions have none, so that fix does NOT transfer.
|
||||
|
||||
⚠ **THE ALPHABET DOES NOT TRANSFER EITHER: 1,496 non-ASCII letters across 23 forms** against
|
||||
Yarros's 2. Hemingway writes Spanish, French and Italian constantly, so the rename pool needs
|
||||
accents (new `hemingway` preset in `rename.py`). The Yarros ASCII-only conclusion would have
|
||||
stranded every Spanish and Italian name in the cast — which is why F02 says re-derive per corpus.
|
||||
|
||||
**Three source defects the splitter surfaced.** `Islands in the Stream` came out as ONE
|
||||
143k-word record (roman numerals, unhandled). `Short Stories` came out as 5 units then 27,
|
||||
because the edition carries a **SECOND contents listing** and first-occurrence matching resolved
|
||||
31 of 58 titles to an index entry — keeping every occurrence and letting the word floor decide is
|
||||
self-correcting; now 57. ⚠⚠ **And the drop-cap defect is in the HEADINGS here** (`T HE O LD M AN
|
||||
AND THE S EA`), which **INVERTS the Yarros pipeline order: repair must run BEFORE the split**, or
|
||||
the splitter cannot see the headings it needs.
|
||||
|
||||
⭐ **Hemingway needed THREE mapped phrases where Yarros needed 48** (`Gran Maestro`,
|
||||
`Unknown Tongue`, `Sin House`) and a 45-entry allow list — the whole difference being that Yarros
|
||||
invented a world and Hemingway named the real one. Several allow entries were non-obvious and
|
||||
required reading: `Royal Game` is a real colonial-Kenyan legal category, `White Heather` a Scotch
|
||||
brand, `Bwana Game` a job title, `Roman Soldier`/`Wine Seller` stage-direction labels from the
|
||||
one-act play `Today is Friday`.
|
||||
@@ -0,0 +1,53 @@
|
||||
# `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training
|
||||
|
||||
⭐⭐ **INSTRUCTION-PAIR SFT BEATS RAW-TEXT TRAINING FOR AUTHOR VOICE, AND THE INCUMBENT NEVER
|
||||
CLEARED ITS OWN CONTROL.** Measured n=120 per arm, 30 in-genre beats from HELD-OUT val passages
|
||||
× 4 seeds, all arms re-measured in one session on one box:
|
||||
|
||||
| arm | delta_cb (lower = more Yarros) | vs base control | 8-gram overlap |
|
||||
|---|---|---|---|
|
||||
| pairs 2ep ckpt-1650 | **0.410** | +0.289 ✅ | 0.12 |
|
||||
| pairs 3ep ckpt-1650 (**shipped**) | 0.438 | +0.262 ✅ | **0.09** |
|
||||
| raw-text instruct (incumbent) | 0.558 | +0.141 ❌ **inside the 0.153 floor** | 0.14 |
|
||||
| base-unadapted (control) | 0.700 | — | 0.07 |
|
||||
|
||||
Same-author target 0.463 (held-out Yarros vs itself). ⚠ **The two pair arms are NOT
|
||||
distinguishable on voice** — 0.028 against a 0.153 floor. The 3ep checkpoint was chosen on the
|
||||
axes that ARE resolvable: better held-out fit (2.3126 vs 2.3264), less overshoot (0.06 vs 0.10),
|
||||
and verbatim overlap nearest the never-saw-it control.
|
||||
|
||||
⭐ **THE RECIPE IS TWO EPOCHS ON A THREE-EPOCH SCHEDULE, not three epochs.** Launch `--epochs 3`;
|
||||
the minimum lands at step 1650 **inside epoch two** and epoch three overfits (2.3126 → 2.3882,
|
||||
flat). The entire gain over a 2-epoch run came from the stretched cosine keeping the LR alive —
|
||||
at step 1600 the 3ep run was at 2.9e-05 where the 2ep run had annealed to 2e-07. ⚠⚠ **A
|
||||
resume-and-append-one-epoch is a NO-OP for exactly that reason** (lr 2.3e-09 at step 1670): it
|
||||
must be a fresh run with the longer schedule.
|
||||
|
||||
⭐ **THE SAFETY PROPERTY: the model writes the INSTRUCTION, never the RESPONSE.** Every response
|
||||
is real renamed prose; only the beat is machine-written, so voice is inherited rather than
|
||||
synthesised. Memorisation checked with both controls (positive control saturates at 160): the
|
||||
shipped arm sits at 0.09 against a 0.07 never-saw-it baseline and BELOW the raw-text arm's 0.14.
|
||||
|
||||
⚠ **THE v1 DECISION RULE WAS WELL-FORMED AND MEASURED THE WRONG THING**, and the amendment is
|
||||
recorded in `scripts/yarros-corpus/score_beats.py` with v1 retained verbatim. It gated on
|
||||
in-band / on-beat / ran-on — and **base-unadapted scores in-band 0.96**. Instruction-following is
|
||||
something Qwen3-4B-Instruct ships with, so those axes detect only DAMAGE, never the benefit an
|
||||
adapter exists to buy. v2 gates on voice (delta_cb vs control beyond the floor) + not-copied
|
||||
(8-gram overlap near control) + no-damage (overshoot). ⚠ on-beat's −0.27 was outside the floor
|
||||
and is dropped from the gate, **not explained away** — the keyword proxy punishes prose that
|
||||
DRAMATISES "she mocks him" rather than echoing the word, but three read samples is an anecdote.
|
||||
|
||||
⭐ **A 5-BEAT FIXTURE HAD A NOISE FLOOR OF 0.800 AND MANUFACTURED A +0.45 RESULT.** At n=20 the
|
||||
pilot looked like a clear in-band win; at n=120 the same gap was +0.08, inside a 0.233 floor.
|
||||
One sample moves a rate by 0.2 when there are five. The 30-beat in-genre fixture (built from
|
||||
held-out val pairs, `~/beats-yarros-30.json`) is the instrument; the Brontë stray-dog/kitten
|
||||
fixture was also the wrong GENRE — "He licked her clean" came back as explicit sex.
|
||||
|
||||
⚠ **THE HARNESS TRUNCATES AT THE FIRST BLANK LINE and that surface reported the pair arm as
|
||||
"19 words, off-beat 0.10"** when the untruncated output was 90–132 words with the beat rendered
|
||||
in a later block. `score_beats.py --metric-source raw|paragraph` keeps both views and the verdict
|
||||
names which it used. Same family as `feedback_filters_that_silently_narrow_the_window`.
|
||||
|
||||
**Artefacts.** `scripts/yarros-corpus/{build_sft_pairs,train_pairs_lora,score_beats,
|
||||
memorization_check}.py`; commits `9b3d3c8` `90ed506` `713e83d` `efb7345` `7505124`. Booth
|
||||
(24h TTL) was `http://10.100.10.50:8090/b/babyyarros-beats/` — six beats × four arms, blind-labelled.
|
||||
@@ -0,0 +1,50 @@
|
||||
# `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full
|
||||
|
||||
**`vllm-voices` live on fv-ml1 GPU 0 :8027**, one Qwen3-4B-Instruct carrier serving
|
||||
`voices-base` plus `lv-<author>` LoRA adapters. `stacks/voices-seat/`, commit `d17bd3d`.
|
||||
|
||||
⭐ **LORA COSTS 24.3% OF DECODE THROUGHPUT AND IT IS WORTH PAYING.** n=30 per arm, interleaved,
|
||||
A-vs-A noise floor **0.1%**: base **143.0 tok/s** median vs adapter **108.2**. The measurement is
|
||||
unusually clean because `--enable-lora` serves BOTH the base name and the adapter name from ONE
|
||||
process — the arm is a per-request field, so no restart, no second seat, no cold-vs-warm confound.
|
||||
Arms were **interleaved rather than blocked** because the card's co-tenants take traffic this seat
|
||||
does not control, and a block design would alias their load onto one arm.
|
||||
|
||||
**Why pay it:** 3 authors cost 8.4 GB as adapters against ~23 GB merged; 6 cost 9.2 vs ~46. On a
|
||||
card with 1.8 GB free afterwards that is the whole argument. If a voice ever lands on a latency
|
||||
path, merge THAT one and serve it separately.
|
||||
|
||||
⭐ **ADAPTER HOT-SWAP IS REAL AND FAST — MEASURED, not read from docs.**
|
||||
`POST /v1/load_lora_adapter` **200 in 0.24 s**, `POST /v1/unload_lora_adapter` **200 in 0.003 s**,
|
||||
VRAM unchanged, container stayed healthy. Proven by performing the `babyyarros`→`lv-yarros`
|
||||
rename through it with no restart. ⚠ **A runtime-loaded adapter is GONE on the next
|
||||
`compose up -d`** unless it is also in `--lora-modules` (which costs a recreate + ~3 min reload).
|
||||
Runtime load is for TRYING a voice; the compose list is what persists. Switching between loaded
|
||||
voices is just the `model` field — **not** a LiteLLM alias; LiteLLM is a thinner layer on top,
|
||||
one alias entry per voice, no new deployment.
|
||||
|
||||
⚠⚠ **`--gpu-memory-utilization` IS A REQUEST AGAINST *TOTAL* VRAM THAT THE CARD MUST ALREADY BE
|
||||
ABLE TO HONOUR — not a share of what is free.** First bring-up REFUSED: *"Free memory on device
|
||||
cuda:0 (11.16/94.97 GiB) is less than desired GPU memory utilization (0.12, 11.4 GiB)"*. Refusing
|
||||
was the right outcome — it protected `cyberprev`, `gen-small` and the Parakeet STT seat rather
|
||||
than squeezing them.
|
||||
|
||||
⭐ **PINNING `--kv-cache-memory` IN BYTES MAKES THE FRACTION PREDICTIVE.** Requested 0.11
|
||||
(10,700 MiB), got **10,740 MiB** resident — a 40 MiB miss on a box where the fraction has been
|
||||
wrong by **8–10 GB in BOTH directions** (cyberprev 0.40→47.1 GB, gen-small 0.48→36.9 GB). Second
|
||||
seat to prove it after `gen-small`. Do not remove the pin.
|
||||
|
||||
⚠ **fv-ml1 GPU 0 is now 96.0 of 97.9 GB.** GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 is a HELD RESERVE
|
||||
for a future full-card seat (`flash-next` alone needs 93 of 96 GiB). **There is no room for
|
||||
another seat on fv-ml1 without a placement decision.**
|
||||
|
||||
⚠ **SUPPORT WAS CHECKED, NOT ASSUMED**, per the training playbook's own lesson that LoRA support
|
||||
is per-ARCHITECTURE not per-family: `vllm/model_executor/models/qwen3.py:271` declares
|
||||
`Qwen3ForCausalLM` with `SupportsLoRA` plus `packed_modules_mapping` and `embedding_modules`.
|
||||
**Do not transplant this compose onto an MoE carrier without re-running that grep** — the
|
||||
playbook records a LoRA refusal on a Qwen3 MoE arch.
|
||||
|
||||
**Naming (operator, 2026-09-16):** `lv-<author>` — lv for **lang-voice**, retiring `baby*`, which
|
||||
read fine for one experiment and invites confusion across a family. The adapter NAME is the
|
||||
request's `model` field, so it is the public API of a voice. Historical persistent-memory entries
|
||||
still say BabyYarros/BabyHemingway and were deliberately left as dated records.
|
||||
+36
-18
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-15 ~09:45 PT (Parakeet STT live on fv-ml1 GPU 0; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency fixed; breeze stays on irv-ml1, TTS-stack move parked at id 75. Nothing blocked, nothing mid-flight.)_
|
||||
_Last updated: 2026-09-16 ~16:15 PT (lv-yarros SHIPPED — instruction-pair SFT beats raw-text on voice; voices-seat live on fv-ml1 GPU 0 :8027 with measured 24.3% LoRA cost; lv-hemingway corpus gated and training, ~1h out; Grok token broker built then SHELVED by the keep-the-jail ruling.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,26 +115,47 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-15 ~09:45 PT._
|
||||
_As of 2026-09-16 ~16:15 PT._
|
||||
|
||||
### Nothing is in flight and nothing is blocked. Both of the previous session's named jobs closed, plus six unplanned pieces of work.
|
||||
### One job in flight: `lv-hemingway` training on pfi-gx10, ~1 hour out.
|
||||
|
||||
**Closed this session:**
|
||||
1. **Parakeet STT live** — fv-ml1 **GPU 0**, port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is a deliberate reserve at 2 MiB.**
|
||||
2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator ruling); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified.
|
||||
3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat.
|
||||
4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2).
|
||||
5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy.
|
||||
6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze).
|
||||
**IN FLIGHT — `lv-hemingway` pair-SFT.** `gx10:~/r49-runs/hemingway-4b-pairs-3ep`, 7,094 pairs,
|
||||
2,661 steps at ~3.68 s/it, launched ~15:05 PT. **Take the checkpoint at the LOSS MINIMUM, not the
|
||||
end-of-run adapter** — the recipe is two epochs on a three-epoch schedule. When it lands: run the
|
||||
v2 gate (voice `delta_cb` vs base control beyond the noise floor · 8-gram overlap near the
|
||||
never-saw-it control · overshoot) using `scripts/yarros-corpus/{score_beats,memorization_check}.py`
|
||||
and the 30-beat in-genre fixture, then ship to
|
||||
`/tank/aimodels/voice-adapters/lv-hemingway-4b-v1/` and add it to `stacks/voices-seat/compose.yaml`.
|
||||
|
||||
**Settled at the end of the session:** breeze-tts **stays on irv-ml1**; moving it plus `bragi` and `tts-gateway` to fv-ml1 is **parked at id 75**, triggered on evacuating embed/rerank/reward. Full fv-ml1 per-seat residency measured — see the breeze detail file.
|
||||
**SHIPPED — `lv-yarros`.** `gx10:~/adapters/lv-yarros-4b-v1/` (sha `63fda6cc61f380b2`) and live on
|
||||
`vllm-voices`, fv-ml1 GPU 0 :8027, alongside `voices-base`.
|
||||
|
||||
**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47), TTS-stack move (park id 75). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it.
|
||||
**OPEN, operator's call, nothing blocked:**
|
||||
- **Brontë has never been through the automated leak gate** — its "0 of 203" was a HAND COUNT and
|
||||
the gate did not exist yet. On Yarros the same instrument read **212 surviving where a hand count
|
||||
said 86**. Its gated corpus survives at `gx10:~/r49-corpus-renamed-unwrapped/` and is
|
||||
pair-buildable; `lv-bronte` exists only as RAW-TEXT arms, i.e. the arm that never cleared its own
|
||||
control. Re-gate before building pairs, or accept the hand count.
|
||||
- **xAI billing** — did the stopped HTTP A/B bill on top of the coding plan? Needs the operator's
|
||||
console access; this fleet holds no xAI credential.
|
||||
- **A `BabyYarros → lv-yarros` pointer** in memory, so historical entries stay findable under the
|
||||
new name. Historical entries were deliberately left as dated records.
|
||||
- Older deferred set, unchanged: AI-tab Dormant regrouping (**belayed**), `nconnect=8` on
|
||||
`/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack move
|
||||
to fv-ml1 (**parked, id 75**).
|
||||
|
||||
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed.
|
||||
⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.** GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 a HELD RESERVE. The
|
||||
next seat needing room on fv-ml1 requires a placement decision.
|
||||
|
||||
**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified
|
||||
before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-16]` ⭐⭐ **Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor).** `lv-yarros` shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → `persistent-memory.d/2026-09-16-lv-voices-line.md`
|
||||
- `[2026-09-16]` ⭐ **voices-seat live: one carrier, N `lv-<author>` LoRA adapters, hot-swap measured at 0.24 s.** LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; `--gpu-memory-utilization` is a request against TOTAL VRAM and only a pinned KV makes it predictive. → `persistent-memory.d/2026-09-16-voices-seat-lora.md`
|
||||
- `[2026-09-16]` ⭐ **lv-hemingway corpus gated at 994,760 words — and half the catalogue had to be EXCLUDED.** 169,759 words of measured 90–96% collection duplication, a Sherwood Anderson parody, and the author's own name 95 times in publisher back matter; the gender resolver needed a corpus base-rate correction to stop reading women as men. → `persistent-memory.d/2026-09-16-lv-hemingway-corpus.md`
|
||||
- `[2026-09-16]` **Grok token broker built then SHELVED — operator ruled "keep the jail stop the a/b", so the renewal feature has no consumer.** ⛔ Do NOT arm `probe-rotation`: the risk did not shrink (it reaches BOTH Gróa transports through one shared session) and the payoff went to zero. → `persistent-memory.d/2026-09-16-grok-broker-shelved.md`
|
||||
- `[2026-09-15]` ⚠⚠ **DO NOT carry "a client-side timeout is not a cancellation" as a rule — it is FALSE as stated.** A clean abandon cancels itself ~6 s later (measured); yet six requests genuinely orphaned on `vllm-erp-seat`. Some propagate, some do not, **boundary unknown** — which argues for a detector, not a rule. ⭐⭐ The durable artifact: **a serving engine's KV cache CYCLES, an orphaned one only CLIMBS** — request count and throughput are ambiguous between loaded and wedged, and I called the seat healthy twice off them (correctly, on the evidence). ⚠ A `max_tokens` ceiling would NOT have prevented it: the worst offender had 16384 set, hit it, and returned 24,594 chars of whitespace. → `persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md`
|
||||
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
@@ -359,19 +380,14 @@ _As of 2026-09-15 ~09:45 PT._
|
||||
|
||||
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md`
|
||||
|
||||
- `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: ~/.claude/settings.json crossSessionI** → `persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md`
|
||||
|
||||
- `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
|
||||
|
||||
- `[2026-09-02]` **althing 3.3.0 deployed — the cc channel, and a plugin-cache false green.** → `persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md`
|
||||
|
||||
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
|
||||
|
||||
- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). (context archived → `archival-memory.md`)
|
||||
|
||||
- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** (context archived → `archival-memory.md`)
|
||||
|
||||
- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md`
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
|
||||
@@ -383,6 +399,8 @@ _As of 2026-09-15 ~09:45 PT._
|
||||
|
||||
_9 older entries archived to archival-memory.md._
|
||||
|
||||
_5 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-09-15]` ⚠⚠ **Probing OPNsense API endpoints by POSTing at them — one was `/api/core/system/reboot` and it took the FV site dark for 3.5 min.** Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own `docs/pfi/opnsense-api-reference.md`. → `persistent-memory.d/2026-09-15-opnsense-api-reboot.md`
|
||||
|
||||
Reference in New Issue
Block a user