From e8086941e288a9521a124a78987ff6ca83c598c5 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 16 Sep 2026 16:19:58 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20lv-yarros=20sh?= =?UTF-8?q?ipped,=20voices-seat=20live,=20lv-hemingway=20training,=20Grok?= =?UTF-8?q?=20broker=20shelved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 554 ++++++++++++++++++ .../2026-09-01-althing-320-deploy.md | 536 ----------------- .../2026-09-02-althing-3-3-0-deployed-the.md | 3 - ...9-02-althing-deploy-is-six-surfaces-and.md | 3 - .../2026-09-16-grok-broker-shelved.md | 54 ++ .../2026-09-16-lv-hemingway-corpus.md | 66 +++ .../2026-09-16-lv-voices-line.md | 53 ++ .../2026-09-16-voices-seat-lora.md | 50 ++ persistent-memory.md | 54 +- 9 files changed, 813 insertions(+), 560 deletions(-) delete mode 100644 persistent-memory.d/2026-09-01-althing-320-deploy.md delete mode 100644 persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md delete mode 100644 persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md create mode 100644 persistent-memory.d/2026-09-16-grok-broker-shelved.md create mode 100644 persistent-memory.d/2026-09-16-lv-hemingway-corpus.md create mode 100644 persistent-memory.d/2026-09-16-lv-voices-line.md create mode 100644 persistent-memory.d/2026-09-16-voices-seat-lora.md diff --git a/archival-memory.md b/archival-memory.md index d8c2e1b..8a8268e 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -32,6 +32,560 @@ Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via 4012952 18,500 MiB comfyui up 8d09h 3166988 4,722 MiB studio-gate up 19d09h +# `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb + +**althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` +_Archived 2026-09-16._ + +# `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green. + +**althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md` +_Archived 2026-09-16._ + +- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). (context archived → `archival-memory.md`) +_Archived 2026-09-16._ + +- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** (context archived → `archival-memory.md`) +_Archived 2026-09-16._ + +# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies + +Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops +deployed. Host work, nh3-dev only. + +## What 3.2.0 changes + +A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a +detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat +sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter +helps. A pane route is a file; there is no process to reap. + +⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops' +own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat +silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness +reaper" rather than guessed at. 2.1.257 is the mechanism. + +## The deploy — three steps, order matters + + 22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables + 22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064 + 22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean + +`althing-route` is the **seventh** console script and the whole point — the plugin's new +SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades +to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it +lands. Skill sync must come **after** the install or it documents a binary that is not there. + +⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on +`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy +step in between. + +## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL + + $ uv tool install . + `althing-core @ file:///home/lkraven/development/althing` is already installed + $ echo $? + 0 + +**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was +already installed from that same path, it declines and reports success. Nothing says "I did +not do the thing you asked." + +**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the +nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every +local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the +command exited 0 and the state was unchanged. + +## Fleet-visible: 4 pane routes revoked, and the count to trust + +Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot +inherit a route) and `command` (so a pane that has dropped to a shell stops being poked). +Routes without them are **observed and excluded with a reason in the herald log**, not +silently ignored — this closes a real hazard where the herald would write a line into whatever +shell inherited an exited agent's pane and press Enter, running whatever was half-typed there. + +Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve +route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only +**pane** routes are affected: + + pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev + fifo: 8 (+ 9 fifos in ~/.althing/wake/) + +forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are +untouched" — **twelve is the total route count; fifo is 8.** Untouched either way. + +Re-declare is one command per agent: `althing-route declare --pid `. + +**Not notified.** Four targeted messages about a change none of them asked about reads as a +broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the +operator instead; the herald logs the exclusion every tick so it is discoverable. + +⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every +tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared — +enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a +back-off after N consecutive exclusions. Their call. + +## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat + + althing-route declare --discover-pid + -> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume) + + pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2 + command=claude --allow-dangerously-skip-permissions --resume + postbox status -> mode: push reachable: true + +`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the +ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with +the tool call and the route dies with it. + +⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving. + +## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU + +infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in +one session** (2026-09-01), the second time **within minutes of being armed**. Both times: +`unread: 0`, no mail, seat silently back to `mode: pull`. + +**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER +confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So: + +- **A green `postbox status` is not durable evidence of monitoring.** True when read, false + minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]] +- **Anything still on `althing-listen` is monitored only until the reaper notices.** The + waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones. +- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised + both consequences with forseti on the deploy thread. + +## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min + +Operator directed notification; all four told individually with their own measured pid, pane +and start time. Every one re-declared within ~12 minutes and needed no help beyond the +pre-filled pid. + + handle pid started_at command status + bil-smithy-dev 98493 1713267 kimi push, poked via pane + dvalin-smithy-dev 96050 1691681 grok push + eitri-smithy-dev 103662 1752089 node/codex push + regin-smithy-dev 96864 1700952 pi push, poked via pane + infra-ops 2175612 38608652 claude push, poked via pane + +Zero stale routes; last exclusion line 22:25:01, everything after is `poked via pane`. + +## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case + +The herald logs *"the process that declared this route is gone, or its pid was reused by +something that started at a different time."* **Neither disjunct was true for any of the +four.** Measured before notifying: each declaring process started 1-3 min *before* its route +was written and was still running four days later; `pid_max` 4,194,304 against a current +~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against +pid 98493. + +Real cause is a third branch the message never offers: **the route predates the guard fields, +so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation — +it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log +line at face value.** Raised with forseti with a suggested third branch. + +## ⚠ MY MISS — I called an open risk on something the release author had already measured + +I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the +release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane +pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes +split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin +had also smoke-tested 3.0.3 on the `pi` seat. + +**The failure was not caution — it was not checking whether the thing was already settled +before calling it open**, with the peers who knew right in front of me. This is exactly the +ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a +"you missed X" claim from someone who had not checked what the other party already read. +⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the +first was honest; asserting the second was not warranted. Retracted with forseti and with all +three seats. → [[feedback_canonical_sourced_values]] + +## Verified: pane delivery works on Claude AND non-Claude seats + +- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns. +- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat: + own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference. +- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover + unsubmitted text. + +## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE + +`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the +herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a +failure, in case anyone else reports it as one."* Asked forseti for a line in the skill. + +## Open with forseti (nothing blocking) + +1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person. +2. Exclusion-message third branch. +3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the + explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree. +4. One-tick-latency note in the skill. +5. `postbox status` push/reachable is not durable evidence. + +**Withdrawn:** non-Claude delivery risk (was forseti's own prior work). +**Moot:** log back-off — zero stale routes, the log went quiet on its own. + +## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN + +**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its +poke into a pane's input line and presses Enter. If someone is **mid-sentence there**, +`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their +half-written message with the herald's line stapled on. + +forseti's post-mortem, worth keeping verbatim because it generalises well past althing: + +> The contract named this hazard on one road only — a submit after a FAILED write — and I +> built three guards that all answer the same question, "is this the right pane". **None of +> them asked whether it was a good MOMENT.** + +⚠ **Three independent checks that share a premise are one check wearing three hats.** + +**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with +its input line empty, captured at declare time. Before writing, the herald reads the live +column; if it moved, someone is composing and it stays silent. The signal was already in the +`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost +nothing at the wire. + + 22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables + 22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820 + 22:40:40 scripts/sync_skill.sh --check clean + +## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER + +forseti measured the idle columns; applied to the live route table this is not a footnote: + + handle TUI guard 4? + infra-ops claude YES idle_cursor=3 (pinned 22:41) + eitri-smithy-dev codex YES expects 3 + dvalin-smithy-dev grok YES expects 7 + bil-smithy-dev kimi NO -- reports no cursor at all + regin-smithy-dev pi NO -- reports no cursor at all + +**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes +it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible — +⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is +logged, not silently assumed safe) but logged-and-poked is still poked. + +**Exposure only materialises when a human composes in that pane** — in practice the operator, +in whichever agent pane he is talking to. Mitigation for an unguardable seat is +`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with +`althing-route declare --discover-pid`. + +⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes +needlessly quiet. + +**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in +5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms +collision window remains. + +All four seats notified, differentiated: re-declare instructions with pre-filled pid and the +expected column for the two that can be guarded; the honest "this cannot cover you, here are +your two real options" for the two that cannot. + +**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats +re-pin at next SessionStart; TUI peers when they next re-declare. + +## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families + + handle TUI idle_cursor outcome + infra-ops claude 3 push, positive + eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex") + dvalin-smithy-dev grok 7 push, positive (exact submitted text verified) + bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk + regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice** + +forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess. + +## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle + +> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like +> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than +> **cursor column** (e.g. input-line length via terminal query), those seats become coverable. + +Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong +question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual +*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the +lead item, credited. Difference between a workaround and a fix. + +## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite + +Both were given the same two options and neither was chosen for them. Opposite calls, both right: + +- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every + operator turn this session, including multi-sentence research directives, was typed here."* + Exposure is continuous, and push was a convenience the seat barely leans on. +- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's + composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already** + cost them: their 3.2.0 revocation notice **sat unread for days**. + +⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should +release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to +frame the choice by pane usage in the skill. +⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the +sharpest argument on record for that. + +## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case + +bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane, +so the stapling failure mode is inferred, not observed."* + +Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They +never exercised **collision**, because nobody was typing. **Two different questions, one body +of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the +"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and +silent about stapling. The operator's report remains the single observed instance. + +Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti +on the first observed stapling. + +⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check. +Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the +other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."* + +## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed + + 22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables + 22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372 + 22:51:21 scripts/sync_skill.sh --check clean + +1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched + `comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer + and refused — four of five pane seats could never have used it.** Now matches the **pane's + own command** (`pane_command`), which zellij already reports and guard 1 already compares + against, so discovery and the guard read one string and a route declared that way cannot + fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches + `node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the + obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an + identical `claude ...`, so a global /proc search is ambiguous by construction. +2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts, + all four were rendered as "gone, or pid reused". Migration case now says what is true and + names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one + vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a + wrong error message until it is true in all cases makes it true and worthless.** +3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the + editable-install warning it rhymes with. +4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a + receipt for the declare, never proof you are still being watched.* + +Count correction accepted: 8 fifo, not 12. + +## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT + +forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug +"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then +reported the suspicion as an open question**, with four live non-Claude seats in front of me and +the answer one ancestry-walk away. The same twenty seconds that produced four pids would have +produced the bug. + +⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had +already measured). **Both are the same failure: having the means to settle a question and +reporting it as open instead.** Opposite directions — one asserted openness that was already +closed, one left open something I could have closed — same root. + +## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover + +forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery +against another's safety on seats that are not mine."* Their dead-ends, worth keeping as +negative results: + +- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI + parsing** — exactly the hardcoding the pinned baseline exists to avoid. +- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token + counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke + forever. + +Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs +the wake) / leave pull-only. + +**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this +call on identical facts, reached opposite conclusions, and both were right — the deciding +variable (who composes in that pane and why) is local knowledge a global policy cannot encode. +Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is +the exact thing they accepted risk to keep; for regin it removes only the submission, leaving +the operator to delete herald text from his in-flight sentences routinely. + +## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE + +The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and +`scripts/sync_skill.sh` does NOT cover it: + + canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced + plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md + -> pre-3.2.0 text, ZERO occurrences of "pane", never synced + +⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical +skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running, +that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter +**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps — +the path that died **twice in one session** here. The command does not warn; it predates the +problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.** + +**On this seat: declined to arm, route left intact.** Reported to forseti with three options, +recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route +exists for the handle**, so the stale doc becomes harmless because the binary enforces the +documented precedence. The doc-only fixes rely on a reader noticing. + +## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE + +infra-ops claimed a `wake-listener-.lock` holding a dead pid would make the next +`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a +reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does +not happen:** + + lock file holds pid 999999 (dead) -> althing-listen ARMED FINE + lock held by a live process -> rc=3, correctly refused + +⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an +**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is +**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file +is read by `--stop` alone — which is why the file is opened non-truncating. + +⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the +LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim +that would have justified real guard code for a condition that cannot occur. forseti tested it +before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug: +check `flock -n` on the fd before suspecting the pid. +[[feedback_assert_effective_value_not_substring]] + +## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4) + + $ althing-listen # on a channel=pane seat + 'infra-ops' already has a PANE route, and arming a waiter would silently take + precedence over it ... To switch back on purpose: althing-route release + rc=4 + +Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that +never arrived), and **not 3** ("already running" says leave the incumbent alone, a different +instruction from "you do not need this at all"). Scoped to the handle's own route, checked +before the lock, and `--stop` deliberately ungated. + +## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX + +Not a stale doc. The deploy chain is: + + repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/ + ^^^ NOTHING SYNCS THIS HOP + +The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed +against the repo: + + deployed 0.0.1 hooks: ['UserPromptSubmit'] + repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd'] + + scripts/pane-route.sh (absent from the deployment entirely) + +⚠ **So forseti's migration note — "CC seats re-declare automatically at their next +SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was +simply never deployed. That explains what we both treated as normal: all four smithy seats +needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code +seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin. + +**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first, +`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware +`monitor.md` now present). + +🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin` +update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache +directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the +plugin in a way that looks like an upstream bug. + +⚠ **There is still no deploy script for the first hop**, which is why it drifted five days +unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo +vs eshpfi) is forseti's call. + +## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1) + + 1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e) + 2. systemctl --user restart althing-po-herald + 3. scripts/sync_skill.sh (already current at 3.2.4) + 4. claude plugin update althing <- THE NEW, FOURTH SURFACE + +⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync, +**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1. + +⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.** +`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist. +`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart +to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I +would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the +conclusion that no supported path existed was an untested assumption.** + +⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions. + +## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES + +Four times in one session I reported a **proxy or an assumption as the fact itself**: + + sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15 + command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin + a lock file's pid for a flock -> flock is fd-held; stale files are inert + "no CLI path exists" for a CLI I never ran -> claude plugin update did it + +⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any +one of them — it is reaching for the nearest observable and reporting it as the thing. +[[feedback_assert_effective_value_not_substring]] + +## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be + +`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit +and a server-side rejection were **three facts with one rendering, and that rendering was +silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to +`$ALTHING_ROOT/listen.log`. Success stays silent by design. + +**Audit run on nh3-dev — live waiter locks vs what the post office believes:** + + 9 of 10 seats live waiter -> mode: push (correct) + regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked + +⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their +pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a +`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be +distinguished from outside** — asked regin, who has the local history. Handing forseti an +ambiguous instance beats a confidently mislabelled one. + +**Standing audit command** for "looks armed but isn't" — worth re-running after any incident: +cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against +`postbox status --handle `. From the next arm onward `listen.log` answers it directly. + +## Rollback (unused) + +`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns +to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader +ignores unknown keys), so nothing is stranded. + +Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`. + +## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT + +> **A missing deploy surface does not present as an error. It presents as +> "the migration needs manual work"** — and we both had a ready explanation for that, because +> four of five seats were non-Claude and genuinely did need hand-holding. + +⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have +self-declared and did not, and it looked exactly like the other four. Neither of us asked why +the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats +re-declare automatically at their next SessionStart" in **three** deploy notes; it was never +true on the only box running it. + +⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a +script; a version number cannot tell you what a stale plugin cost. Check in +`docs/runbooks/althing-deploy.md`. + +⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before +concluding it does not apply to your case.** + +forseti also corrected an error of their own that I did not catch: the contract's Migration +section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook. +Wrong since 3.2.0. + +**On the four retractions**, forseti's framing, kept because it is fair and I would not have +written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive +and quiet, so your errors are all on the record and mine mostly are not. You catching four of +your own in one night is the system working."* Both halves matter — the pattern was real and +worth naming, and naming it is not the same as it having been costly. +_Archived 2026-09-16._ + ## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev 2.4x the figure in the 2026-08-09→10 dots burn-in. diff --git a/persistent-memory.d/2026-09-01-althing-320-deploy.md b/persistent-memory.d/2026-09-01-althing-320-deploy.md deleted file mode 100644 index 4372026..0000000 --- a/persistent-memory.d/2026-09-01-althing-320-deploy.md +++ /dev/null @@ -1,536 +0,0 @@ -# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies - -Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops -deployed. Host work, nh3-dev only. - -## What 3.2.0 changes - -A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a -detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat -sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter -helps. A pane route is a file; there is no process to reap. - -⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops' -own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat -silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness -reaper" rather than guessed at. 2.1.257 is the mechanism. - -## The deploy — three steps, order matters - - 22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables - 22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064 - 22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean - -`althing-route` is the **seventh** console script and the whole point — the plugin's new -SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades -to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it -lands. Skill sync must come **after** the install or it documents a binary that is not there. - -⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on -`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy -step in between. - -## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL - - $ uv tool install . - `althing-core @ file:///home/lkraven/development/althing` is already installed - $ echo $? - 0 - -**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was -already installed from that same path, it declines and reports success. Nothing says "I did -not do the thing you asked." - -**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the -nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every -local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the -command exited 0 and the state was unchanged. - -## Fleet-visible: 4 pane routes revoked, and the count to trust - -Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot -inherit a route) and `command` (so a pane that has dropped to a shell stops being poked). -Routes without them are **observed and excluded with a reason in the herald log**, not -silently ignored — this closes a real hazard where the herald would write a line into whatever -shell inherited an exited agent's pane and press Enter, running whatever was half-typed there. - -Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve -route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only -**pane** routes are affected: - - pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev - fifo: 8 (+ 9 fifos in ~/.althing/wake/) - -forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are -untouched" — **twelve is the total route count; fifo is 8.** Untouched either way. - -Re-declare is one command per agent: `althing-route declare --pid `. - -**Not notified.** Four targeted messages about a change none of them asked about reads as a -broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the -operator instead; the herald logs the exclusion every tick so it is discoverable. - -⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every -tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared — -enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a -back-off after N consecutive exclusions. Their call. - -## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat - - althing-route declare --discover-pid - -> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume) - - pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2 - command=claude --allow-dangerously-skip-permissions --resume - postbox status -> mode: push reachable: true - -`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the -ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with -the tool call and the route dies with it. - -⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving. - -## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU - -infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in -one session** (2026-09-01), the second time **within minutes of being armed**. Both times: -`unread: 0`, no mail, seat silently back to `mode: pull`. - -**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER -confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So: - -- **A green `postbox status` is not durable evidence of monitoring.** True when read, false - minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]] -- **Anything still on `althing-listen` is monitored only until the reaper notices.** The - waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones. -- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised - both consequences with forseti on the deploy thread. - -## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min - -Operator directed notification; all four told individually with their own measured pid, pane -and start time. Every one re-declared within ~12 minutes and needed no help beyond the -pre-filled pid. - - handle pid started_at command status - bil-smithy-dev 98493 1713267 kimi push, poked via pane - dvalin-smithy-dev 96050 1691681 grok push - eitri-smithy-dev 103662 1752089 node/codex push - regin-smithy-dev 96864 1700952 pi push, poked via pane - infra-ops 2175612 38608652 claude push, poked via pane - -Zero stale routes; last exclusion line 22:25:01, everything after is `poked via pane`. - -## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case - -The herald logs *"the process that declared this route is gone, or its pid was reused by -something that started at a different time."* **Neither disjunct was true for any of the -four.** Measured before notifying: each declaring process started 1-3 min *before* its route -was written and was still running four days later; `pid_max` 4,194,304 against a current -~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against -pid 98493. - -Real cause is a third branch the message never offers: **the route predates the guard fields, -so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation — -it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log -line at face value.** Raised with forseti with a suggested third branch. - -## ⚠ MY MISS — I called an open risk on something the release author had already measured - -I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the -release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane -pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes -split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin -had also smoke-tested 3.0.3 on the `pi` seat. - -**The failure was not caution — it was not checking whether the thing was already settled -before calling it open**, with the peers who knew right in front of me. This is exactly the -ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a -"you missed X" claim from someone who had not checked what the other party already read. -⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the -first was honest; asserting the second was not warranted. Retracted with forseti and with all -three seats. → [[feedback_canonical_sourced_values]] - -## Verified: pane delivery works on Claude AND non-Claude seats - -- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns. -- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat: - own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference. -- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover - unsubmitted text. - -## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE - -`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the -herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a -failure, in case anyone else reports it as one."* Asked forseti for a line in the skill. - -## Open with forseti (nothing blocking) - -1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person. -2. Exclusion-message third branch. -3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the - explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree. -4. One-tick-latency note in the skill. -5. `postbox status` push/reachable is not durable evidence. - -**Withdrawn:** non-Claude delivery risk (was forseti's own prior work). -**Moot:** log back-off — zero stale routes, the log went quiet on its own. - -## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN - -**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its -poke into a pane's input line and presses Enter. If someone is **mid-sentence there**, -`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their -half-written message with the herald's line stapled on. - -forseti's post-mortem, worth keeping verbatim because it generalises well past althing: - -> The contract named this hazard on one road only — a submit after a FAILED write — and I -> built three guards that all answer the same question, "is this the right pane". **None of -> them asked whether it was a good MOMENT.** - -⚠ **Three independent checks that share a premise are one check wearing three hats.** - -**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with -its input line empty, captured at declare time. Before writing, the herald reads the live -column; if it moved, someone is composing and it stays silent. The signal was already in the -`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost -nothing at the wire. - - 22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables - 22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820 - 22:40:40 scripts/sync_skill.sh --check clean - -## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER - -forseti measured the idle columns; applied to the live route table this is not a footnote: - - handle TUI guard 4? - infra-ops claude YES idle_cursor=3 (pinned 22:41) - eitri-smithy-dev codex YES expects 3 - dvalin-smithy-dev grok YES expects 7 - bil-smithy-dev kimi NO -- reports no cursor at all - regin-smithy-dev pi NO -- reports no cursor at all - -**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes -it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible — -⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is -logged, not silently assumed safe) but logged-and-poked is still poked. - -**Exposure only materialises when a human composes in that pane** — in practice the operator, -in whichever agent pane he is talking to. Mitigation for an unguardable seat is -`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with -`althing-route declare --discover-pid`. - -⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes -needlessly quiet. - -**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in -5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms -collision window remains. - -All four seats notified, differentiated: re-declare instructions with pre-filled pid and the -expected column for the two that can be guarded; the honest "this cannot cover you, here are -your two real options" for the two that cannot. - -**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats -re-pin at next SessionStart; TUI peers when they next re-declare. - -## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families - - handle TUI idle_cursor outcome - infra-ops claude 3 push, positive - eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex") - dvalin-smithy-dev grok 7 push, positive (exact submitted text verified) - bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk - regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice** - -forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess. - -## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle - -> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like -> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than -> **cursor column** (e.g. input-line length via terminal query), those seats become coverable. - -Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong -question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual -*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the -lead item, credited. Difference between a workaround and a fix. - -## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite - -Both were given the same two options and neither was chosen for them. Opposite calls, both right: - -- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every - operator turn this session, including multi-sentence research directives, was typed here."* - Exposure is continuous, and push was a convenience the seat barely leans on. -- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's - composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already** - cost them: their 3.2.0 revocation notice **sat unread for days**. - -⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should -release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to -frame the choice by pane usage in the skill. -⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the -sharpest argument on record for that. - -## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case - -bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane, -so the stapling failure mode is inferred, not observed."* - -Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They -never exercised **collision**, because nobody was typing. **Two different questions, one body -of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the -"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and -silent about stapling. The operator's report remains the single observed instance. - -Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti -on the first observed stapling. - -⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check. -Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the -other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."* - -## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed - - 22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables - 22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372 - 22:51:21 scripts/sync_skill.sh --check clean - -1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched - `comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer - and refused — four of five pane seats could never have used it.** Now matches the **pane's - own command** (`pane_command`), which zellij already reports and guard 1 already compares - against, so discovery and the guard read one string and a route declared that way cannot - fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches - `node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the - obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an - identical `claude ...`, so a global /proc search is ambiguous by construction. -2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts, - all four were rendered as "gone, or pid reused". Migration case now says what is true and - names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one - vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a - wrong error message until it is true in all cases makes it true and worthless.** -3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the - editable-install warning it rhymes with. -4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a - receipt for the declare, never proof you are still being watched.* - -Count correction accepted: 8 fifo, not 12. - -## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT - -forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug -"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then -reported the suspicion as an open question**, with four live non-Claude seats in front of me and -the answer one ancestry-walk away. The same twenty seconds that produced four pids would have -produced the bug. - -⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had -already measured). **Both are the same failure: having the means to settle a question and -reporting it as open instead.** Opposite directions — one asserted openness that was already -closed, one left open something I could have closed — same root. - -## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover - -forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery -against another's safety on seats that are not mine."* Their dead-ends, worth keeping as -negative results: - -- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI - parsing** — exactly the hardcoding the pinned baseline exists to avoid. -- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token - counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke - forever. - -Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs -the wake) / leave pull-only. - -**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this -call on identical facts, reached opposite conclusions, and both were right — the deciding -variable (who composes in that pane and why) is local knowledge a global policy cannot encode. -Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is -the exact thing they accepted risk to keep; for regin it removes only the submission, leaving -the operator to delete herald text from his in-flight sentences routinely. - -## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE - -The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and -`scripts/sync_skill.sh` does NOT cover it: - - canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced - plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md - -> pre-3.2.0 text, ZERO occurrences of "pane", never synced - -⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical -skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running, -that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter -**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps — -the path that died **twice in one session** here. The command does not warn; it predates the -problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.** - -**On this seat: declined to arm, route left intact.** Reported to forseti with three options, -recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route -exists for the handle**, so the stale doc becomes harmless because the binary enforces the -documented precedence. The doc-only fixes rely on a reader noticing. - -## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE - -infra-ops claimed a `wake-listener-.lock` holding a dead pid would make the next -`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a -reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does -not happen:** - - lock file holds pid 999999 (dead) -> althing-listen ARMED FINE - lock held by a live process -> rc=3, correctly refused - -⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an -**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is -**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file -is read by `--stop` alone — which is why the file is opened non-truncating. - -⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the -LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim -that would have justified real guard code for a condition that cannot occur. forseti tested it -before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug: -check `flock -n` on the fd before suspecting the pid. -[[feedback_assert_effective_value_not_substring]] - -## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4) - - $ althing-listen # on a channel=pane seat - 'infra-ops' already has a PANE route, and arming a waiter would silently take - precedence over it ... To switch back on purpose: althing-route release - rc=4 - -Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that -never arrived), and **not 3** ("already running" says leave the incumbent alone, a different -instruction from "you do not need this at all"). Scoped to the handle's own route, checked -before the lock, and `--stop` deliberately ungated. - -## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX - -Not a stale doc. The deploy chain is: - - repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/ - ^^^ NOTHING SYNCS THIS HOP - -The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed -against the repo: - - deployed 0.0.1 hooks: ['UserPromptSubmit'] - repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd'] - + scripts/pane-route.sh (absent from the deployment entirely) - -⚠ **So forseti's migration note — "CC seats re-declare automatically at their next -SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was -simply never deployed. That explains what we both treated as normal: all four smithy seats -needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code -seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin. - -**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first, -`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware -`monitor.md` now present). - -🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin` -update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache -directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the -plugin in a way that looks like an upstream bug. - -⚠ **There is still no deploy script for the first hop**, which is why it drifted five days -unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo -vs eshpfi) is forseti's call. - -## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1) - - 1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e) - 2. systemctl --user restart althing-po-herald - 3. scripts/sync_skill.sh (already current at 3.2.4) - 4. claude plugin update althing <- THE NEW, FOURTH SURFACE - -⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync, -**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1. - -⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.** -`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist. -`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart -to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I -would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the -conclusion that no supported path existed was an untested assumption.** - -⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions. - -## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES - -Four times in one session I reported a **proxy or an assumption as the fact itself**: - - sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15 - command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin - a lock file's pid for a flock -> flock is fd-held; stale files are inert - "no CLI path exists" for a CLI I never ran -> claude plugin update did it - -⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any -one of them — it is reaching for the nearest observable and reporting it as the thing. -[[feedback_assert_effective_value_not_substring]] - -## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be - -`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit -and a server-side rejection were **three facts with one rendering, and that rendering was -silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to -`$ALTHING_ROOT/listen.log`. Success stays silent by design. - -**Audit run on nh3-dev — live waiter locks vs what the post office believes:** - - 9 of 10 seats live waiter -> mode: push (correct) - regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked - -⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their -pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a -`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be -distinguished from outside** — asked regin, who has the local history. Handing forseti an -ambiguous instance beats a confidently mislabelled one. - -**Standing audit command** for "looks armed but isn't" — worth re-running after any incident: -cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against -`postbox status --handle `. From the next arm onward `listen.log` answers it directly. - -## Rollback (unused) - -`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns -to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader -ignores unknown keys), so nothing is stranded. - -Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`. - -## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT - -> **A missing deploy surface does not present as an error. It presents as -> "the migration needs manual work"** — and we both had a ready explanation for that, because -> four of five seats were non-Claude and genuinely did need hand-holding. - -⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have -self-declared and did not, and it looked exactly like the other four. Neither of us asked why -the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats -re-declare automatically at their next SessionStart" in **three** deploy notes; it was never -true on the only box running it. - -⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a -script; a version number cannot tell you what a stale plugin cost. Check in -`docs/runbooks/althing-deploy.md`. - -⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before -concluding it does not apply to your case.** - -forseti also corrected an error of their own that I did not catch: the contract's Migration -section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook. -Wrong since 3.2.0. - -**On the four retractions**, forseti's framing, kept because it is fair and I would not have -written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive -and quiet, so your errors are all on the record and mine mostly are not. You catching four of -your own in one night is the system working."* Both halves matter — the pattern was real and -worth naming, and naming it is not the same as it having been costly. diff --git a/persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md b/persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md deleted file mode 100644 index ad9f3fa..0000000 --- a/persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-02]` althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green. - -**althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md` diff --git a/persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md b/persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md deleted file mode 100644 index f2d0e86..0000000 --- a/persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md +++ /dev/null @@ -1,3 +0,0 @@ -# `[2026-09-02]` althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInb - -**althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` diff --git a/persistent-memory.d/2026-09-16-grok-broker-shelved.md b/persistent-memory.d/2026-09-16-grok-broker-shelved.md new file mode 100644 index 0000000..7768c10 --- /dev/null +++ b/persistent-memory.d/2026-09-16-grok-broker-shelved.md @@ -0,0 +1,54 @@ +# `[2026-09-16]` Grok token broker — built, then shelved by the transport ruling. Do NOT arm the probe. + +**`services/grok-token-broker/` — seeded, committed, DISARMED, no consumer.** Commits `ebc4dac` +`b907a0e` `cf9d167`. ⛔ **Do not arm `probe-rotation`.** This is a finished resting place, not a +half-built tool: the gate works and the thing it gated for went away. + +**Operator ruling, relayed by heid:** *"keep the jail stop the a/b"* +(`heid dispatch-log/2026-09.jsonl#groa-transport-20260916-operator-keeps-the-jail`, alongside +`#groa-transport-ab-20260916-operator-stop`). Gróa dispatches through the read jail; +`groa_http_dispatch.py` is a documented fallback with no scheduled use. **Nothing in the fleet +wants a renewable xAI session.** + +⭐ **THE CODE-PLAN ENDPOINT EXISTS and I was one message away from telling the operator it did +not.** `https://cli-chat-proxy.grok.com/v1` serving **grok-4.6** (500,000 context) and grok-4.5, +`agent_type: grok-build-plan`, `auth_method: session`, `api_key`/`env_key`/`api_base_url` all +null. ⚠⚠ **It is in `~/.grok/models_cache.json` — the Grok CLI's own config, on nh3-dev.** I had +swept heid's repo, the gateway `.env`, the LiteLLM config and Vaultwarden, all correctly, and +concluded "does not exist". ⭐ **heid's line, taken: absence from the places you searched is not +absence.** The check that separates the two states is a LIVE REQUEST, not a grep. + +⚠⚠ **NOT WRITING `~/.grok/auth.json` IS NECESSARY AND NOT SUFFICIENT.** The refresh grant at +`https://auth.x.ai/oauth2/token` may ROTATE the refresh token, and many OIDC providers invalidate +the old one SERVER-SIDE. A broker refreshing the same credential kills the CLI login even though +it never touches the file. heid's module header reasoned about the WRITE; they amended it to name +invalidation, credited. This correction went infra-ops→heid the same day heid's went the other +way — **neither of us reaches the right answer alone.** + +⚠⚠ **THE PROBE'S BLAST RADIUS IS BOTH GRÓA TRANSPORTS, which is not visible from the infra side.** +`heid/scripts/groa_dispatch.py` builds `argv = ["grok", "-p", prompt, "--cwd", jail, ...]` and +shells the CLI, which authenticates from the same `~/.grok/auth.json`. The bwrap in the process +table is grok's own Landlock sandbox, not something Heid wraps. **One session, two ways of +reaching it** — an invalidating probe takes Gróa down on EVERY path until an interactive re-login. +🔴 **I had recommended "run the probe now while the CLI is idle" and withdrew it in writing**; +"idle" was a convenient assumption I never checked, on a day that had already taken eight panels. + +**Why the jail won, and it was not performance.** HTTP is faster (~523 s median vs ~890 s), +simpler, and arguably SAFER on confinement (no tools, so the 2026-06-10 escape class is +structurally impossible). It lost on FAILURE MODE: HTTP fails by returning a fast, confident, +well-formatted review that found nothing — indistinguishable from a clean bill. The jail fails by +timing out, which you can see. ⚠ **Do NOT quote a per-transport finding rate from this**: heid +states the 0/0/0-vs-5/7/3 numbers are confounded with bundle size (the zeros were all huge inline +bundles; the one HTTP round at jail-comparable size produced Gróa's leading solo), n=3–4 per cell, +no noise floor. Asymmetric-risk argument, **not** a resolved measurement. + +⚠ **Still unmeasured, and it is a billing question:** the jail reaches the coding plan already +paid for; the HTTP path reaches the METERED API and its responses carry `cost_in_usd_ticks`. +Whether that bills on top of the plan was never part of the ruling. One look at the xAI billing +console — **this fleet holds no xAI credential**, so it needs the operator's account access. + +⚠ The coding plan speaks the **Responses API** (`api_backend: "responses"`), not +`/chat/completions` — a second, independent obstacle to any LiteLLM alias. Moot while the jail is +ruled. heid also found and killed two live instructions in their own persistent-memory telling a +fresh session to dispatch `--groa-transport http`; either would have resumed a stopped experiment +after a context reset. diff --git a/persistent-memory.d/2026-09-16-lv-hemingway-corpus.md b/persistent-memory.d/2026-09-16-lv-hemingway-corpus.md new file mode 100644 index 0000000..232a7f5 --- /dev/null +++ b/persistent-memory.d/2026-09-16-lv-hemingway-corpus.md @@ -0,0 +1,66 @@ +# `[2026-09-16]` lv-hemingway corpus: half the work was EXCLUSION, and the gate found what a hand count would not + +**994,760 words · 318 units · 6 renamed copies · leak gate PASSED 0 of 941 entities and 0 of 117 +audited phrases, both controls green.** `~/hemingway-corpus{,-renamed}`, builder +`scripts/hemingway-corpus/build_corpus_hemingway.py`, commits `9598d0b` `03b4a3f`. + +⭐ **THE CATALOGUE HOLDS 2,105,679 WORDS AND ROUGHLY HALF MUST NOT BE TRAINED ON.** Operator +scoped it to fiction only. Three exclusion passes, each measured or voice-specific: + +1. **Non-fiction, 8 works ~911k words** — By-Line, Dateline: Toronto, Death in the Afternoon, + Green Hills of Africa, The Dangerous Summer, the three posthumous "Hemingway on X" anthologies. +2. ⭐ **Four story collections, 169,759 words — MEASURED, not assumed.** `Short Stories` is the + First Forty-Nine and CONTAINS the others. 8-gram containment of the smaller work: Winner Take + Nothing **96.0%**, Snows of Kilimanjaro **95.2%**, Men Without Women **92.9%**, In Our Time + **90.6%**. ⚠⚠ **The catalogue's own `near_dup_pairs` table is BLIND to this** — it holds + whole-document simhashes (ONE row in the entire 1,284-work library) and this is PARTIAL + containment. Whole-document dedup cannot see a collection inside a larger collection. +3. ⭐ **`The Torrents of Spring` — excluded for a reason no word count could justify.** It is a + deliberate PARODY of Sherwood Anderson: the target author's name on a different author's + style, i.e. mislabelled data for a voice adapter. + +⚠ **THE AUTHOR'S OWN NAME WAS IN THE TRAINING TEXT 95 TIMES ACROSS 7 WORKS** — publisher back +matter ("Ernest Hemingway was one of America's foremost journalists… died in 1961") riding inside +the last unit, because a splitter cuts on headings and nothing follows the final one. **Identical +to the Yarros defect; nothing about the source changed to cause it.** Stripping the publisher +block left 18, all in `true-at-first-light`, inside a **CAST OF CHARACTERS and SWAHILI GLOSSARY +written by Patrick Hemingway** — an editor describing the author's real household. Markers are +matched in file order, earliest wins. Now 0. + +⚠ **`G` WAS ABOUT TO BE RENAMED TO A SURNAME, 248 TIMES.** Not a name: the fragment left by +`B.G.`, `G.M.`, `G2`, `G3`. Caught by reading surfaces IN CONTEXT, which is the Yarros lesson +repeating. Also read in context: `Gran` (fragment of `Gran Sasso`/`Gran Italia`/`Gran Hotel`), +`Shamba` (Swahili common noun), and `Inglés` — **kept renameable deliberately**, the gypsies' +in-world nickname for Robert Jordan, exactly parallel to Yarros's `Violence`. + +⭐⭐ **THE GENDER RESOLVER HAD TO BE REBUILT AND ITS OWN GATE CAUGHT THE FIRST ATTEMPT.** The +inherited one returned **397 male / 20 female** across 1,102 records with Catherine Barkley, +Brett Ashley, Pilar, Maria and Mary all held neutral. A plain majority vote over nearby pronouns +scored 18 correct but **5 WRONG** against the incumbent's 1 — and **every error was +female-read-as-male** (Pilar m=426 f=243, Brett m=249 f=137). The refuse-unless-better guard +rejected it, correctly. ⭐ **Cause, measured: the corpus base rate is 34,315 male pronouns to +8,699 female, nearly 4:1.** Pilar's "male-dominated" 426:243 is strongly FEMALE against that +background. Scoring each name's local mix against the corpus base rate instead of 50:50 gives +**18 correct / 11 held / 0 WRONG**, distribution 275m / 120f. +`scripts/hemingway-corpus/gender_by_proximity.py`. ⚠ Yarros solved its version with the POV +chapter header; Hemingway's editions have none, so that fix does NOT transfer. + +⚠ **THE ALPHABET DOES NOT TRANSFER EITHER: 1,496 non-ASCII letters across 23 forms** against +Yarros's 2. Hemingway writes Spanish, French and Italian constantly, so the rename pool needs +accents (new `hemingway` preset in `rename.py`). The Yarros ASCII-only conclusion would have +stranded every Spanish and Italian name in the cast — which is why F02 says re-derive per corpus. + +**Three source defects the splitter surfaced.** `Islands in the Stream` came out as ONE +143k-word record (roman numerals, unhandled). `Short Stories` came out as 5 units then 27, +because the edition carries a **SECOND contents listing** and first-occurrence matching resolved +31 of 58 titles to an index entry — keeping every occurrence and letting the word floor decide is +self-correcting; now 57. ⚠⚠ **And the drop-cap defect is in the HEADINGS here** (`T HE O LD M AN +AND THE S EA`), which **INVERTS the Yarros pipeline order: repair must run BEFORE the split**, or +the splitter cannot see the headings it needs. + +⭐ **Hemingway needed THREE mapped phrases where Yarros needed 48** (`Gran Maestro`, +`Unknown Tongue`, `Sin House`) and a 45-entry allow list — the whole difference being that Yarros +invented a world and Hemingway named the real one. Several allow entries were non-obvious and +required reading: `Royal Game` is a real colonial-Kenyan legal category, `White Heather` a Scotch +brand, `Bwana Game` a job title, `Roman Soldier`/`Wine Seller` stage-direction labels from the +one-act play `Today is Friday`. diff --git a/persistent-memory.d/2026-09-16-lv-voices-line.md b/persistent-memory.d/2026-09-16-lv-voices-line.md new file mode 100644 index 0000000..1662d04 --- /dev/null +++ b/persistent-memory.d/2026-09-16-lv-voices-line.md @@ -0,0 +1,53 @@ +# `[2026-09-16]` The lv-* voice line: Option C proved, lv-yarros shipped, lv-hemingway training + +⭐⭐ **INSTRUCTION-PAIR SFT BEATS RAW-TEXT TRAINING FOR AUTHOR VOICE, AND THE INCUMBENT NEVER +CLEARED ITS OWN CONTROL.** Measured n=120 per arm, 30 in-genre beats from HELD-OUT val passages +× 4 seeds, all arms re-measured in one session on one box: + +| arm | delta_cb (lower = more Yarros) | vs base control | 8-gram overlap | +|---|---|---|---| +| pairs 2ep ckpt-1650 | **0.410** | +0.289 ✅ | 0.12 | +| pairs 3ep ckpt-1650 (**shipped**) | 0.438 | +0.262 ✅ | **0.09** | +| raw-text instruct (incumbent) | 0.558 | +0.141 ❌ **inside the 0.153 floor** | 0.14 | +| base-unadapted (control) | 0.700 | — | 0.07 | + +Same-author target 0.463 (held-out Yarros vs itself). ⚠ **The two pair arms are NOT +distinguishable on voice** — 0.028 against a 0.153 floor. The 3ep checkpoint was chosen on the +axes that ARE resolvable: better held-out fit (2.3126 vs 2.3264), less overshoot (0.06 vs 0.10), +and verbatim overlap nearest the never-saw-it control. + +⭐ **THE RECIPE IS TWO EPOCHS ON A THREE-EPOCH SCHEDULE, not three epochs.** Launch `--epochs 3`; +the minimum lands at step 1650 **inside epoch two** and epoch three overfits (2.3126 → 2.3882, +flat). The entire gain over a 2-epoch run came from the stretched cosine keeping the LR alive — +at step 1600 the 3ep run was at 2.9e-05 where the 2ep run had annealed to 2e-07. ⚠⚠ **A +resume-and-append-one-epoch is a NO-OP for exactly that reason** (lr 2.3e-09 at step 1670): it +must be a fresh run with the longer schedule. + +⭐ **THE SAFETY PROPERTY: the model writes the INSTRUCTION, never the RESPONSE.** Every response +is real renamed prose; only the beat is machine-written, so voice is inherited rather than +synthesised. Memorisation checked with both controls (positive control saturates at 160): the +shipped arm sits at 0.09 against a 0.07 never-saw-it baseline and BELOW the raw-text arm's 0.14. + +⚠ **THE v1 DECISION RULE WAS WELL-FORMED AND MEASURED THE WRONG THING**, and the amendment is +recorded in `scripts/yarros-corpus/score_beats.py` with v1 retained verbatim. It gated on +in-band / on-beat / ran-on — and **base-unadapted scores in-band 0.96**. Instruction-following is +something Qwen3-4B-Instruct ships with, so those axes detect only DAMAGE, never the benefit an +adapter exists to buy. v2 gates on voice (delta_cb vs control beyond the floor) + not-copied +(8-gram overlap near control) + no-damage (overshoot). ⚠ on-beat's −0.27 was outside the floor +and is dropped from the gate, **not explained away** — the keyword proxy punishes prose that +DRAMATISES "she mocks him" rather than echoing the word, but three read samples is an anecdote. + +⭐ **A 5-BEAT FIXTURE HAD A NOISE FLOOR OF 0.800 AND MANUFACTURED A +0.45 RESULT.** At n=20 the +pilot looked like a clear in-band win; at n=120 the same gap was +0.08, inside a 0.233 floor. +One sample moves a rate by 0.2 when there are five. The 30-beat in-genre fixture (built from +held-out val pairs, `~/beats-yarros-30.json`) is the instrument; the Brontë stray-dog/kitten +fixture was also the wrong GENRE — "He licked her clean" came back as explicit sex. + +⚠ **THE HARNESS TRUNCATES AT THE FIRST BLANK LINE and that surface reported the pair arm as +"19 words, off-beat 0.10"** when the untruncated output was 90–132 words with the beat rendered +in a later block. `score_beats.py --metric-source raw|paragraph` keeps both views and the verdict +names which it used. Same family as `feedback_filters_that_silently_narrow_the_window`. + +**Artefacts.** `scripts/yarros-corpus/{build_sft_pairs,train_pairs_lora,score_beats, +memorization_check}.py`; commits `9b3d3c8` `90ed506` `713e83d` `efb7345` `7505124`. Booth +(24h TTL) was `http://10.100.10.50:8090/b/babyyarros-beats/` — six beats × four arms, blind-labelled. diff --git a/persistent-memory.d/2026-09-16-voices-seat-lora.md b/persistent-memory.d/2026-09-16-voices-seat-lora.md new file mode 100644 index 0000000..f6a9d7a --- /dev/null +++ b/persistent-memory.d/2026-09-16-voices-seat-lora.md @@ -0,0 +1,50 @@ +# `[2026-09-16]` voices-seat: LoRA over merge, measured — and GPU 0 is now full + +**`vllm-voices` live on fv-ml1 GPU 0 :8027**, one Qwen3-4B-Instruct carrier serving +`voices-base` plus `lv-` LoRA adapters. `stacks/voices-seat/`, commit `d17bd3d`. + +⭐ **LORA COSTS 24.3% OF DECODE THROUGHPUT AND IT IS WORTH PAYING.** n=30 per arm, interleaved, +A-vs-A noise floor **0.1%**: base **143.0 tok/s** median vs adapter **108.2**. The measurement is +unusually clean because `--enable-lora` serves BOTH the base name and the adapter name from ONE +process — the arm is a per-request field, so no restart, no second seat, no cold-vs-warm confound. +Arms were **interleaved rather than blocked** because the card's co-tenants take traffic this seat +does not control, and a block design would alias their load onto one arm. + +**Why pay it:** 3 authors cost 8.4 GB as adapters against ~23 GB merged; 6 cost 9.2 vs ~46. On a +card with 1.8 GB free afterwards that is the whole argument. If a voice ever lands on a latency +path, merge THAT one and serve it separately. + +⭐ **ADAPTER HOT-SWAP IS REAL AND FAST — MEASURED, not read from docs.** +`POST /v1/load_lora_adapter` **200 in 0.24 s**, `POST /v1/unload_lora_adapter` **200 in 0.003 s**, +VRAM unchanged, container stayed healthy. Proven by performing the `babyyarros`→`lv-yarros` +rename through it with no restart. ⚠ **A runtime-loaded adapter is GONE on the next +`compose up -d`** unless it is also in `--lora-modules` (which costs a recreate + ~3 min reload). +Runtime load is for TRYING a voice; the compose list is what persists. Switching between loaded +voices is just the `model` field — **not** a LiteLLM alias; LiteLLM is a thinner layer on top, +one alias entry per voice, no new deployment. + +⚠⚠ **`--gpu-memory-utilization` IS A REQUEST AGAINST *TOTAL* VRAM THAT THE CARD MUST ALREADY BE +ABLE TO HONOUR — not a share of what is free.** First bring-up REFUSED: *"Free memory on device +cuda:0 (11.16/94.97 GiB) is less than desired GPU memory utilization (0.12, 11.4 GiB)"*. Refusing +was the right outcome — it protected `cyberprev`, `gen-small` and the Parakeet STT seat rather +than squeezing them. + +⭐ **PINNING `--kv-cache-memory` IN BYTES MAKES THE FRACTION PREDICTIVE.** Requested 0.11 +(10,700 MiB), got **10,740 MiB** resident — a 40 MiB miss on a box where the fraction has been +wrong by **8–10 GB in BOTH directions** (cyberprev 0.40→47.1 GB, gen-small 0.48→36.9 GB). Second +seat to prove it after `gen-small`. Do not remove the pin. + +⚠ **fv-ml1 GPU 0 is now 96.0 of 97.9 GB.** GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 is a HELD RESERVE +for a future full-card seat (`flash-next` alone needs 93 of 96 GiB). **There is no room for +another seat on fv-ml1 without a placement decision.** + +⚠ **SUPPORT WAS CHECKED, NOT ASSUMED**, per the training playbook's own lesson that LoRA support +is per-ARCHITECTURE not per-family: `vllm/model_executor/models/qwen3.py:271` declares +`Qwen3ForCausalLM` with `SupportsLoRA` plus `packed_modules_mapping` and `embedding_modules`. +**Do not transplant this compose onto an MoE carrier without re-running that grep** — the +playbook records a LoRA refusal on a Qwen3 MoE arch. + +**Naming (operator, 2026-09-16):** `lv-` — lv for **lang-voice**, retiring `baby*`, which +read fine for one experiment and invites confusion across a family. The adapter NAME is the +request's `model` field, so it is the public API of a voice. Historical persistent-memory entries +still say BabyYarros/BabyHemingway and were deliberately left as dated records. diff --git a/persistent-memory.md b/persistent-memory.md index 74b2a09..672b928 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-15 ~09:45 PT (Parakeet STT live on fv-ml1 GPU 0; `svos_miranda` LIVE in Hermes; talk v10 deployed; irv-ml1 dead-address sweep COMPLETE; secrets-broker concurrency fixed; breeze stays on irv-ml1, TTS-stack move parked at id 75. Nothing blocked, nothing mid-flight.)_ +_Last updated: 2026-09-16 ~16:15 PT (lv-yarros SHIPPED — instruction-pair SFT beats raw-text on voice; voices-seat live on fv-ml1 GPU 0 :8027 with measured 24.3% LoRA cost; lv-hemingway corpus gated and training, ~1h out; Grok token broker built then SHELVED by the keep-the-jail ruling.)_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under **8 hours** old, read it (it carries the in-flight @@ -115,26 +115,47 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-09-15 ~09:45 PT._ +_As of 2026-09-16 ~16:15 PT._ -### Nothing is in flight and nothing is blocked. Both of the previous session's named jobs closed, plus six unplanned pieces of work. +### One job in flight: `lv-hemingway` training on pfi-gx10, ~1 hour out. -**Closed this session:** -1. **Parakeet STT live** — fv-ml1 **GPU 0**, port 8300, v3 int8 25-language model, behind LiteLLM `ext-stt` / `whisper-1`. ⚠ Placed on GPU 3 first; operator corrected it — a ~800 MiB seat belongs on the card with the most uncommitted headroom, not on the one pristine 96 GB card, because vLLM sizes KV against TOTAL VRAM. **GPU 3 is a deliberate reserve at 2 MiB.** -2. **`svos_miranda` LIVE in Hermes** — gateway restarted, 29 toolsets, Miranda scoped to exactly 8, operator's own surface intact at 46. ⚠ `agent.disabled_toolsets` is DELETED and stays out (operator ruling); svos-dev fixed their roster check at `c9d2a96`. SVOS restarted itself; both roster lines verified. -3. **talk v10 deployed** on nh3-dev :8092 — push-to-talk STT through `ext-stt`, barge-in. First consumer of the Parakeet seat. -4. **irv-ml1 dead-address sweep DONE** — 0 of 112 Homepage cards on `10.100.79.3`, was 9. Found and fixed four live breakages on OTHER hosts (Open WebUI TTS, asset-engine, skaldsong x2). -5. **`secret` concurrency bug fixed** — parallel `secret get` returned empty with exit 0. Command-level lock + empty-value guard + `find()` no longer coercing empty stdout to `[]`. `~/.local/bin/secret` is now a symlink, was a stale copy. -6. **Retired:** irv-ml1 parakeet (lost tts-dev's bench) and voice-studio (dots obsoleted by Breeze). +**IN FLIGHT — `lv-hemingway` pair-SFT.** `gx10:~/r49-runs/hemingway-4b-pairs-3ep`, 7,094 pairs, +2,661 steps at ~3.68 s/it, launched ~15:05 PT. **Take the checkpoint at the LOSS MINIMUM, not the +end-of-run adapter** — the recipe is two epochs on a three-epoch schedule. When it lands: run the +v2 gate (voice `delta_cb` vs base control beyond the noise floor · 8-gram overlap near the +never-saw-it control · overshoot) using `scripts/yarros-corpus/{score_beats,memorization_check}.py` +and the 30-beat in-genre fixture, then ship to +`/tank/aimodels/voice-adapters/lv-hemingway-4b-v1/` and add it to `stacks/voices-seat/compose.yaml`. -**Settled at the end of the session:** breeze-tts **stays on irv-ml1**; moving it plus `bragi` and `tts-gateway` to fv-ml1 is **parked at id 75**, triggered on evacuating embed/rerank/reward. Full fv-ml1 per-seat residency measured — see the breeze detail file. +**SHIPPED — `lv-yarros`.** `gx10:~/adapters/lv-yarros-4b-v1/` (sha `63fda6cc61f380b2`) and live on +`vllm-voices`, fv-ml1 GPU 0 :8027, alongside `voices-base`. -**Open, all operator-deferred, none blocking:** the AI-tab Dormant regrouping (belayed), `nconnect=8` on /mnt/smithy (deferred), fused MoE kernel path (park id 47), TTS-stack move (park id 75). `speaches`'s label claims `:8204`, which is breeze-tts's live port — a latent conflict if anyone starts it. +**OPEN, operator's call, nothing blocked:** +- **Brontë has never been through the automated leak gate** — its "0 of 203" was a HAND COUNT and + the gate did not exist yet. On Yarros the same instrument read **212 surviving where a hand count + said 86**. Its gated corpus survives at `gx10:~/r49-corpus-renamed-unwrapped/` and is + pair-buildable; `lv-bronte` exists only as RAW-TEXT arms, i.e. the arm that never cleared its own + control. Re-gate before building pairs, or accept the hand count. +- **xAI billing** — did the stopped HTTP A/B bill on top of the coding plan? Needs the operator's + console access; this fleet holds no xAI credential. +- **A `BabyYarros → lv-yarros` pointer** in memory, so historical entries stay findable under the + new name. Historical entries were deliberately left as dated records. +- Older deferred set, unchanged: AI-tab Dormant regrouping (**belayed**), `nconnect=8` on + `/mnt/smithy` (**declined in scope**), fused MoE kernel path (**parked, id 47**), TTS-stack move + to fv-ml1 (**parked, id 75**). -**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified before this session began; untouched and deliberately not committed. +⚠ **fv-ml1 GPU 0 is at 96.0 of 97.9 GB.** GPU 1 ~5.7 free, GPU 2 ~2.4, GPU 3 a HELD RESERVE. The +next seat needing room on fv-ml1 requires a placement decision. + +**Uncommitted:** `graphify-out/GRAPH_REPORT.md` and `scripts/seat-inventory.py` were modified +before this session began. ⚠ **Do NOT commit them** — untouched and deliberately left alone. ## Recent decisions +- `[2026-09-16]` ⭐⭐ **Instruction-pair SFT BEATS raw-text for author voice, and the raw-text incumbent never cleared its own control (+0.141 against a 0.153 floor).** `lv-yarros` shipped; the v1 decision rule was amended by the operator after measurement showed it gated on axes the unadapted carrier already maxes. → `persistent-memory.d/2026-09-16-lv-voices-line.md` +- `[2026-09-16]` ⭐ **voices-seat live: one carrier, N `lv-` LoRA adapters, hot-swap measured at 0.24 s.** LoRA costs 24.3% of decode against a 0.1% A-vs-A floor and is worth paying; `--gpu-memory-utilization` is a request against TOTAL VRAM and only a pinned KV makes it predictive. → `persistent-memory.d/2026-09-16-voices-seat-lora.md` +- `[2026-09-16]` ⭐ **lv-hemingway corpus gated at 994,760 words — and half the catalogue had to be EXCLUDED.** 169,759 words of measured 90–96% collection duplication, a Sherwood Anderson parody, and the author's own name 95 times in publisher back matter; the gender resolver needed a corpus base-rate correction to stop reading women as men. → `persistent-memory.d/2026-09-16-lv-hemingway-corpus.md` +- `[2026-09-16]` **Grok token broker built then SHELVED — operator ruled "keep the jail stop the a/b", so the renewal feature has no consumer.** ⛔ Do NOT arm `probe-rotation`: the risk did not shrink (it reaches BOTH Gróa transports through one shared session) and the payoff went to zero. → `persistent-memory.d/2026-09-16-grok-broker-shelved.md` - `[2026-09-15]` ⚠⚠ **DO NOT carry "a client-side timeout is not a cancellation" as a rule — it is FALSE as stated.** A clean abandon cancels itself ~6 s later (measured); yet six requests genuinely orphaned on `vllm-erp-seat`. Some propagate, some do not, **boundary unknown** — which argues for a detector, not a rule. ⭐⭐ The durable artifact: **a serving engine's KV cache CYCLES, an orphaned one only CLIMBS** — request count and throughput are ambiguous between loaded and wedged, and I called the seat healthy twice off them (correctly, on the evidence). ⚠ A `max_tokens` ceiling would NOT have prevented it: the worst offender had 16384 set, hit it, and returned 24,594 chars of whitespace. → `persistent-memory.d/2026-09-15-client-abandon-cancellation-boundary.md` - `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md` @@ -359,19 +380,14 @@ _As of 2026-09-15 ~09:45 PT._ - `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding ever** → `persistent-memory.d/2026-09-03-nh3-dev-wedged-for-40-min.md` -- `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: ~/.claude/settings.json crossSessionI** → `persistent-memory.d/2026-09-02-althing-deploy-is-six-surfaces-and.md` - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md` -- `[2026-09-02]` **althing 3.3.0 deployed — the cc channel, and a plugin-cache false green.** → `persistent-memory.d/2026-09-02-althing-3-3-0-deployed-the.md` - `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md` -- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). (context archived → `archival-memory.md`) -- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** (context archived → `archival-memory.md`) -- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` - `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312. @@ -383,6 +399,8 @@ _As of 2026-09-15 ~09:45 PT._ _9 older entries archived to archival-memory.md._ +_5 older entries archived to archival-memory.md._ + ## Tried and abandoned - `[2026-09-15]` ⚠⚠ **Probing OPNsense API endpoints by POSTing at them — one was `/api/core/system/reboot` and it took the FV site dark for 3.5 min.** Endpoints are ACTIONS; a 200 means it ran. The call I wanted was documented in this repo's own `docs/pfi/opnsense-api-reference.md`. → `persistent-memory.d/2026-09-15-opnsense-api-reboot.md`