The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.
Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.
The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.
ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.
Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.
Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.
Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:
io pressure full avg10 96.5% -> 0.01%
in-flight writes 45, completing none -> 0 0
D-state processes 11 -> 0
32 MiB dsync write never completed -> 0.24s (133 MB/s)
The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.
Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.
It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.
Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.
Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".
Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.
conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.
Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.
Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.
Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.
The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.
The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.
Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.
Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.
The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.
deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.
Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer,
meaning a manual edit; it was read as scheduler behaviour, and
`runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A
label list requires ALL the labels, so that form matches no runner and
queues forever instead of failing — the worst outcome of the three,
because a queued job looks busy rather than broken.
Standard Actions semantics rather than a gitea quirk; the ambiguity was
in my prose. Documented where the label guidance already lives so the
next reader does not have to have been in the conversation.
The bell reported 🔔 iff wake-listener-<handle>.lock named a live pid — one
MECHANISM, not the property it stands for. Correct while the FIFO waiter
was the only channel; wrong the moment 3.3.0 added `cc`. This seat is
reachable over its Claude Code socket, has no waiter lock at all, and was
rendering 🔕 while the post office reported push/reachable. Pane-routed
seats were wrong the same way before that.
It now reads `reachable` from the status payload it was already fetching
and discarding, which means the segment knows nothing about althing's
internals — no lock paths, no channel names — so a fourth channel cannot
make it stale.
Adds the third state: an unreachable post office rendered identically to a
healthy seat with no mail. It is 📵 now. An outage is not an empty inbox,
including on the status line.
Also retires a `kill -0` liveness check, the third instance of
liveness-standing-in-for-identity found on this object tonight.
Verified in all three states: 🔔 on this seat, 📵 against a dead address,
📬 4 🔕 on a pull-only handle.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
The README's capability table is a measurement with a date on it, and a
runner upgrade or a gitea bump can invalidate it without anything saying
so. Committing the workflow that produced it means the next person can
re-derive the claims instead of trusting them, which is the same posture
the training probes take.
Carries the three job shapes plus the traps each one cost to find: job
images need a node binary or the first `uses:` fails, the Postgres service
needed ~6s and a pg_isready wait, and a run must be polled at
/actions/runs because /actions/tasks reports it as empty on 1.26.1.
Requested by vastblue-dev as a reference for their first smoke.
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:
- `container.valid_volumes: []` does NOT keep the docker daemon out of
jobs. act_runner mounts /var/run/docker.sock on its own, so every job
on the shared runner is uid 0 with `docker ps` over all 49 containers
on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
It is also load-bearing: four repos drive buildx through it, so the
fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
resolves from the local mirrors today. That is github-independence per
workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
act_runner's action-fetch auth since 2026-08-05.
Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.
Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
The cross-reference loop filtered live waiters on `kill -0 "$pid"`. A lock
file left by a reaped listener names a pid the kernel is free to reissue,
so the loop would report an unrelated process as a live waiter — and a
phantom waiter is how a false "your seat is unreachable" notice reaches a
seat that is fine, with nothing in the output to falsify it.
`session_listener.sh --stop` already refuses this standard: it walks
/proc/<pid>/cmdline for the `--_route=<handle>` segment before signalling,
on the grounds that a live pid proves existence and not identity. The audit
now asserts the same thing and prints STRANGER on a mismatch.
Verified against the live box: six waiters, all identity-confirmed, all
reporting push.
The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.
Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.
Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.
Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.
Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.
Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
The operator overruled my ana-ml2 recommendation and was right. I had step
times in hand, so I priced a breaker trip as eleven minutes of lost
training. That is the recompute cost and it is the cheapest component of
the loss.
A breaker trip at Anaheim is a forty-minute drive each way on the
operator's time, whenever he happens to notice, with thirteen hosts dark
until he arrives -- including the hypervisor most of them run on, the
fleet's primary backup server, and three SureFire client machines that are
a customer's production hosts under a hosting agreement.
So save_steps 100 to 50 caps the recompute, not the outage, and it was
never the mitigation I claimed. Thirteen hours unattended on a desk in NH3
beats two and a half hours that can put a client's hosts dark, especially
when nothing is waiting on this run.
Recorded the general form at length because it is the transferable part:
when recommending between options, check whether you priced the failure
mode in whatever units you happened to be measuring. A metric in hand will
volunteer itself as the unit of risk.
Also measured and dismissed the obvious third option: the RTX PRO 6000s
have a 250 W floor against a 300 W default, so capping both saves 100 W on
a box drawing about a kilowatt. Not enough to matter, and it costs
throughput to buy.
The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.
That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.
Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.
A version number cannot tell you what a stale plugin cost. 0.0.1 and
0.1.1 differ by two hooks and a script, so the runbook now carries a
check that compares hook lists across cached versions and looks for
pane-route.sh directly.
Also records why this hid for five days, which is the more transferable
half. A missing deploy surface does not present as an error -- it presents
as "the migration needs manual work", and there was a ready explanation
for that, because four of five seats were non-Claude and genuinely did
need hand-holding. The seat that falsified the story was our own: a Claude
Code seat that should have self-declared and did not, and it looked
exactly like the other four. Nobody asked why the automatic path had not
fired on the one seat it was built for.
So: when a migration needs manual intervention, verify the automatic path
was actually deployed before concluding it does not apply to your case.
Recorded the ownership call, which forseti left open. The plugin deployer
lives in eshpfi rather than the althing repo because it targets
per-machine paths, and althing's sync_skill.sh deliberately reaches into
no other tree. Putting a plugin installer upstream would break that
boundary for one consumer's convenience. Their repo stays the source;
this one does the installing.
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.
The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.
The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.
The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
Deployed 3.2.4 and closed the fourth deploy surface. The plugin cache
moved 0.0.1 to 0.1.1 via `claude plugin update althing`, which means the
step I had escalated to the operator was mine to do. Refusing to
hand-edit installed_plugins.json was right; concluding no supported path
existed was an untested assumption, and `claude plugin` has install,
update, uninstall, list, details, validate and marketplace subcommands.
Recording the pattern once rather than four separate lessons: four times
tonight I reported a proxy or an assumption as the fact itself. sudo -n -v
for NOPASSWD, command -v nvcc for the toolkit, a lock file's pid for a
flock, and "no CLI path exists" for a CLI I never ran. Each was cheap to
test and expensive to assert.
Also ran the audit forseti's 3.2.4 makes possible, since a seat could have
been silently pull-only since 3.1.2 with every failure rendering as
silence. Nine of ten seats with a live waiter report push. regin-smithy-dev
holds a live waiter and the post office has it pull-only -- but regin
deliberately released their pane route earlier tonight, so release-then-arm
ordering explains the same observation with no bug involved. Not
distinguishable from outside, so it went to regin as a question rather
than to forseti as a confirmed instance.
forseti measured my stale-lock claim and it is false. I said a lock file
holding a dead pid would make the next althing-listen exit 3 and turn a
reap into a permanent monitoring outage that reports healthy. The gate is
flock -n on an open fd, which the kernel releases when the holder dies, so
a lock left by a reaped listener is inert and exit 3 only fires against a
live holder. The pid in the file is read by --stop alone.
I reasoned from the artifact's contents when the behaviour is set by the
locking mechanism, and put the consequence in durable memory without
testing it. forseti tested before writing code. That is the third time
tonight I reported a proxy as the thing itself, after sudo -n -v for
NOPASSWD and command -v nvcc for the toolkit.
Deployed 3.2.3, which makes althing-listen refuse on a pane seat with its
own exit code rather than silently demoting it.
And found the real shape of the plugin gap, which is worse than the stale
document forseti and I were both discussing. Nothing syncs the repo's
plugin directory into the marketplace directory, so it was frozen at Aug
28 with only the UserPromptSubmit hook -- no SessionStart, no SessionEnd,
no pane-route.sh at all. That means "CC seats re-declare automatically at
their next SessionStart" has never been true on this box, which is why
every seat including our own needed a hand-fed declare. Source is fixed
and bumped to 0.1.1; refreshing the plugin cache needs a /plugin update
from the operator, and I deliberately did not hand-edit Claude Code's own
plugin bookkeeping to force it.
Triton builds its CUDA-utils shim with gcc the first time a kernel runs,
and needs Python.h to do it. Without python3-dev the box looks entirely
healthy: torch imports, the 49 GB base loads, LoRA attaches with the
right parameter count, and then the first training step dies with a bare
CalledProcessError naming a gcc invocation and an exit code.
The real message -- "fatal error: Python.h: No such file or directory" --
is discarded, because Triton sends the compiler's stdout to DEVNULL. It
cost a probe run and a model load to find something a one-line manual
compile answered immediately.
Same shape as the dots-tts container needing a C compiler at runtime: a
JIT dependency invisible at install time that only surfaces under load.
The verify step runs the compile rather than checking the package is
present, because `dpkg -l python3-dev` would pass while the compile still
failed on a missing library path or header directory.
The comparison number nobody had written down: ana-ml2's real run-3c step
times, pulled out of run-03c.log before the breaker killed it. About 10.8
to 15.8 s/it over the first 24 steps, so 604 steps lands at roughly 2.2
to 2.7 hours. Anything under about 45 s/it on the GX10 makes it an
overnight run.
Also records the exact geometry from run-03c.json and a real
adapter_config.json, so the probe measures the shape that actually ran
rather than an approximation of it.
Checked the backend-delta trap the playbook warns about before running
anything rather than after: flash-attn is installed on neither box, so
both fall back to sdpa. Library versions do differ -- torch 2.13.0 versus
2.14.0, transformers 5.15.1 versus 5.16.1 -- and that is recorded rather
than assumed harmless. The probe reads the resolved attention
implementation back off the loaded model instead of trusting the request.
The probe discards its first two steps as warmup, which is not optional on
this box: an unwarmed benchmark here already read 27 TFLOP/s when the true
figure was 93, because it was timing the PTX JIT.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
The userspace half is done and needed no root. This covers what does:
NOPASSWD sudo for infra-ops (ending the gx10 fleet exception), the docker
group, the CUDA toolkit for nvcc, and the NVIDIA Container Toolkit wired
into dockerd.
Unrun -- it needs one interactive invocation to supply the sudo password,
because no gx10 credential is vaulted and elway prompts via getpass. After
its first step lands, the box stops being an exception and later runs are
unattended.
Guards worth noting: the sudoers drop-in is validated with visudo -cf
before install, since a malformed one locks every sudo user out of a box
sitting on a desk with no out-of-band access; and cuda-toolkit is
installed rather than the cuda metapackage, so the working 580.173.02
driver is not replaced.
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.
Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.
CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.
Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.
The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.
The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.
Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
The /althing:monitor slash command is a separate artifact from the
canonical skill and sync_skill.sh does not touch it. It sits at plugin
version 0.0.1 with pre-3.2.0 text and zero mentions of pane routes, while
the canonical skill it should mirror is at 3.2.2.
That makes it active harm rather than stale documentation. The canonical
precedence rule is that a live waiter wins over a pane entry, so an agent
on a healthy pane route who follows the command verbatim demotes itself
back onto the FIFO path the harness reaps -- the path that died twice on
this seat tonight. The command cannot warn about it because it predates
the problem, and every Claude Code seat reaches for the slash command
first because it is the discoverable surface.
Declined to arm on this seat and left the route intact. Reported to
forseti with a recommendation that althing-listen refuse to arm when a
pane route exists for the handle, since a binary enforcing the documented
precedence beats a doc that relies on a reader noticing.
Also cleared a stale wake-listener lock holding a dead pid, and recorded
why it matters: the next althing-listen seeing it exits 3, whose
documented response is do not drain and do not re-arm. A stale lock turns
a reap into a permanent monitoring outage that reports itself as healthy.
Checked the disk rather than waiting on comfy-dev. Both builds are there,
pulled a minute apart on Aug 23: a 26 GB int8 and a 15 GB nvfp4-awq.
I had asked whether their nvfp4 pin was set under a Blackwell assumption,
since Ada has no native nvfp4, which would make the int8 file the right
one on the new box. It cannot be. They pinned it on irv-ml1's A6000,
which is Ampere sm_86 and has neither native nvfp4 nor native fp8. Ada
sm_89 supports a strict superset, so a pin that was correct on the weaker
card cannot be invalidated by moving to the stronger one. The migration is
incapable of breaking it.
The pin is about VRAM, not architecture. Eleven gigabytes on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs, and a
text encoder runs once per prompt rather than once per diffusion step, so
its throughput matters far less than the DiT's. That also explains why
this pin went the opposite way from their other one without either being
inconsistent. The RTX 6000 Ada is also 48 GB, so nothing relaxes.
Reclassified the 26 GB int8 from orphan to spare: with the extra drives
the destination lands near 14% full, so disk stops being the constraint
and the pin-rot argument says keep it. Question withdrawn to comfy-dev.
Four of the five items I raised are closed and tagged. The one that
matters most: --discover-pid matched comm == "claude", which means kimi,
grok, codex and pi would each have walked to the multiplexer and refused.
Four of five pane seats could never have used it. It now matches the
pane's own command, which zellij already reports and guard 1 already
compares against, so discovery and the guard read one string.
Recording a second miss of my own alongside it. I handed all four seats
explicit measured pids because I suspected the ancestry walk was broken,
then reported the suspicion as an open question rather than spending the
same twenty seconds to settle it, with four live non-Claude seats in
front of me. That pairs with the earlier miss in the opposite direction:
asserting an open risk on something the author had already measured. Same
root -- having the means to settle a question and reporting it as open
instead.
Also keeping forseti's two dead ends as negative results, because they
are the obvious things to propose next and both fail: locating a TUI's
input box in a screen dump needs per-TUI parsing, and diffing two dumps
to detect typing refuses every poke forever, because status bars carry
live token counts and clocks so consecutive dumps differ on an idle pane.
The kimi/pi coverage gap is deliberately unfixed and now sits with the
operator, with a recommendation to leave it per-seat.
All four seats answered. Delivery is now verified on four TUI families and
forseti's measured idle columns held exactly, so nobody had to guess.
The two seats guard 4 cannot cover made opposite calls on the same facts
and both are right. regin-smithy-dev released to pull-only because the
operator composes in that pane routinely, so the exposure is continuous.
bil-smithy-dev kept its route because that pane is poke-driven, making the
collision window narrow, and because pull-only had already cost them a
notice that sat unread for days. The deciding variable is who composes in
the pane and why, not risk appetite -- a flat rule either way would have
been wrong for one of them.
regin also supplied the only real path to closing the hole: pi and kimi
report no cursor because their input line is not an empty-prompt-at-idle,
so a moment-check reading pane content rather than cursor column would
cover them. Column is a proxy; an empty input line is the actual
predicate. Relayed to forseti as the lead item.
And bil corrected something upstream: the Aug 28 probes proved delivery
against non-Claude TUIs but never exercised the collision case, because
nobody was typing during them. Two different questions, one body of
evidence, only one of them answered by it -- which means the earlier
retraction of the non-Claude risk flag was right about delivery and
silent about stapling.
Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will
not be poked until it declares again.
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone
mid-sentence had their half-written message submitted with the herald's
line stapled on. It hit the operator within an hour of this evening's
deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column
and stays silent when the live column has moved.
Deployed all three steps at 22:40 and re-declared this seat, which pinned
idle_cursor=3 as expected for a claude TUI.
The finding worth keeping is the hole the fix leaves. forseti's own
measurements say kimi and pi report no cursor at all, which means
bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter
what they run. Two of the five pane seats on this box stay permanently
exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the
write-into-a-typing-pane bug" is only true where the cursor is legible.
All four seats were told, differentiated: a pre-filled re-declare and the
expected column for the two that can be guarded, and the honest version
plus the release-to-pull-only option for the two that cannot.
Also keeping forseti's post-mortem line, because it generalises: three
guards that all answer "is this the right pane" and none that asks
whether it is a good moment are one check wearing three hats.
All four notified seats re-declared within about twelve minutes and the
log went quiet. Five of five pane routes now carry the guard fields, and
delivery is confirmed on a claude seat, a pi seat and a grok seat.
Two findings survive the close-out. The herald's exclusion reason is
false for the migration case -- every declaring process was alive and
four days old with no pid wrap, and the real cause is simply that the
route predates the fields the guard needs. And the uv trap stands as the
thing most likely to bite the next person.
The third is mine. I raised non-Claude pane delivery with forseti as an
open risk on their release when forseti had personally measured it days
earlier, on the exact seats in question, and the resulting matrix is what
characterised the settle bug they fixed. The error was not caution, it
was calling something open without checking whether it was already
settled, with the peers who knew right in front of me. Recorded because
"I don't know" and "this is an open risk" are different claims and I made
the second when only the first was true.
Still open with forseti: the uv --force runbook fix, an exclusion-message
third branch, --discover-pid against a non-Claude process tree, the
one-tick latency note, and status not being durable evidence.
Operator directed that the four affected seats be told directly. Measured
their state before writing, which turned up two things worth more than
the notification itself.
The herald logs each exclusion as "the process that declared this route
is gone, or its pid was reused by something that started at a different
time." Neither is true for any of the four. Each declaring process
started minutes before its route was written and is still running four
days later, and pid_max is 4194304 against a current 2.86M so the counter
has not wrapped. The real cause is a third one the message never offers:
the route predates the guard fields, so identity cannot be verified. The
behaviour is right and the explanation is wrong, and it would send anyone
debugging it hunting a dead agent that is alive.
Separately, all four seats run non-Claude CLIs -- kimi, grok, codex and
pi -- while the pane poke is built around typing into a Claude Code pane.
Whether --discover-pid walks a non-Claude process tree, and what a poke
does to a non-Claude TUI, are both unverified. Each agent was given their
measured pid to sidestep the first, told plainly about the second, and
offered the choice between re-declaring as a test or staying pull-only.
Both findings raised with forseti.
The waiter died a second time, within minutes of being armed, so this
seat stopped re-arming and declared a pane route instead. That is exactly
what 3.2.0 shipped for.
- althing-route declare --discover-pid walks the ancestry to the
long-lived claude process rather than the ephemeral bash that invoked
it. Passing --pid $$ would bind the route to a shell that dies with the
tool call.
- Recorded the sharper form of the failure: the waiter does not just die,
it dies after confirming it is up. Both waiters reported push and
reachable immediately after arming. So a green postbox status is not
durable evidence of monitoring, and the reap is not confined to the
pane seats that were migrated -- this seat is a fifo waiter and was hit
twice.
Both consequences raised with forseti on the deploy thread, along with a
note that reachable is a report rather than a delivered poke.
forseti requested the deploy, operator-approved and tagged at c4ede0f.
All three steps landed on nh3-dev and verified: force reinstall to 3.2.0
with seven binaries, herald restart, skill sync.
The durable lesson is the trap in step 1. `uv tool install .` matches on
the source spec rather than its contents, so on a box that already had
the tool installed from that path it prints "already installed" and exits
0 having done nothing. Following the runbook literally would have left
the new binary absent with every command reporting success. Always use
--force when reinstalling from a local path.
Also recorded: the pane-route migration revoked exactly four routes,
verified by splitting on channel= before the restart rather than by
auditing fields (all twelve route files lack the new fields, so a field
audit over-counts). The four affected agents were deliberately not
notified, per the standing rule against unsolicited fleet broadcast, and
that is surfaced to the operator instead.
And this closes an open question from earlier today: the waiter that died
with status "killed" was CC 2.1.257 reaping detached tasks, which is the
premise 3.2.0 exists to address.
vh