I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer,
meaning a manual edit; it was read as scheduler behaviour, and
`runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A
label list requires ALL the labels, so that form matches no runner and
queues forever instead of failing — the worst outcome of the three,
because a queued job looks busy rather than broken.
Standard Actions semantics rather than a gitea quirk; the ambiguity was
in my prose. Documented where the label guidance already lives so the
next reader does not have to have been in the conversation.
The bell reported 🔔 iff wake-listener-<handle>.lock named a live pid — one
MECHANISM, not the property it stands for. Correct while the FIFO waiter
was the only channel; wrong the moment 3.3.0 added `cc`. This seat is
reachable over its Claude Code socket, has no waiter lock at all, and was
rendering 🔕 while the post office reported push/reachable. Pane-routed
seats were wrong the same way before that.
It now reads `reachable` from the status payload it was already fetching
and discarding, which means the segment knows nothing about althing's
internals — no lock paths, no channel names — so a fourth channel cannot
make it stale.
Adds the third state: an unreachable post office rendered identically to a
healthy seat with no mail. It is 📵 now. An outage is not an empty inbox,
including on the status line.
Also retires a `kill -0` liveness check, the third instance of
liveness-standing-in-for-identity found on this object tonight.
Verified in all three states: 🔔 on this seat, 📵 against a dead address,
📬 4 🔕 on a pull-only handle.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
The README's capability table is a measurement with a date on it, and a
runner upgrade or a gitea bump can invalidate it without anything saying
so. Committing the workflow that produced it means the next person can
re-derive the claims instead of trusting them, which is the same posture
the training probes take.
Carries the three job shapes plus the traps each one cost to find: job
images need a node binary or the first `uses:` fails, the Postgres service
needed ~6s and a pg_isready wait, and a run must be polled at
/actions/runs because /actions/tasks reports it as empty on 1.26.1.
Requested by vastblue-dev as a reference for their first smoke.
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:
- `container.valid_volumes: []` does NOT keep the docker daemon out of
jobs. act_runner mounts /var/run/docker.sock on its own, so every job
on the shared runner is uid 0 with `docker ps` over all 49 containers
on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
It is also load-bearing: four repos drive buildx through it, so the
fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
resolves from the local mirrors today. That is github-independence per
workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
act_runner's action-fetch auth since 2026-08-05.
Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.
Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
The cross-reference loop filtered live waiters on `kill -0 "$pid"`. A lock
file left by a reaped listener names a pid the kernel is free to reissue,
so the loop would report an unrelated process as a live waiter — and a
phantom waiter is how a false "your seat is unreachable" notice reaches a
seat that is fine, with nothing in the output to falsify it.
`session_listener.sh --stop` already refuses this standard: it walks
/proc/<pid>/cmdline for the `--_route=<handle>` segment before signalling,
on the grounds that a live pid proves existence and not identity. The audit
now asserts the same thing and prints STRANGER on a mismatch.
Verified against the live box: six waiters, all identity-confirmed, all
reporting push.
The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.
Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.
Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.
Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.
Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.
Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
The operator overruled my ana-ml2 recommendation and was right. I had step
times in hand, so I priced a breaker trip as eleven minutes of lost
training. That is the recompute cost and it is the cheapest component of
the loss.
A breaker trip at Anaheim is a forty-minute drive each way on the
operator's time, whenever he happens to notice, with thirteen hosts dark
until he arrives -- including the hypervisor most of them run on, the
fleet's primary backup server, and three SureFire client machines that are
a customer's production hosts under a hosting agreement.
So save_steps 100 to 50 caps the recompute, not the outage, and it was
never the mitigation I claimed. Thirteen hours unattended on a desk in NH3
beats two and a half hours that can put a client's hosts dark, especially
when nothing is waiting on this run.
Recorded the general form at length because it is the transferable part:
when recommending between options, check whether you priced the failure
mode in whatever units you happened to be measuring. A metric in hand will
volunteer itself as the unit of risk.
Also measured and dismissed the obvious third option: the RTX PRO 6000s
have a 250 W floor against a 300 W default, so capping both saves 100 W on
a box drawing about a kilowatt. Not enough to matter, and it costs
throughput to buy.
The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.
That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.
Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.
A version number cannot tell you what a stale plugin cost. 0.0.1 and
0.1.1 differ by two hooks and a script, so the runbook now carries a
check that compares hook lists across cached versions and looks for
pane-route.sh directly.
Also records why this hid for five days, which is the more transferable
half. A missing deploy surface does not present as an error -- it presents
as "the migration needs manual work", and there was a ready explanation
for that, because four of five seats were non-Claude and genuinely did
need hand-holding. The seat that falsified the story was our own: a Claude
Code seat that should have self-declared and did not, and it looked
exactly like the other four. Nobody asked why the automatic path had not
fired on the one seat it was built for.
So: when a migration needs manual intervention, verify the automatic path
was actually deployed before concluding it does not apply to your case.
Recorded the ownership call, which forseti left open. The plugin deployer
lives in eshpfi rather than the althing repo because it targets
per-machine paths, and althing's sync_skill.sh deliberately reaches into
no other tree. Putting a plugin installer upstream would break that
boundary for one consumer's convenience. Their repo stays the source;
this one does the installing.
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.
The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.
The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.
The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
Deployed 3.2.4 and closed the fourth deploy surface. The plugin cache
moved 0.0.1 to 0.1.1 via `claude plugin update althing`, which means the
step I had escalated to the operator was mine to do. Refusing to
hand-edit installed_plugins.json was right; concluding no supported path
existed was an untested assumption, and `claude plugin` has install,
update, uninstall, list, details, validate and marketplace subcommands.
Recording the pattern once rather than four separate lessons: four times
tonight I reported a proxy or an assumption as the fact itself. sudo -n -v
for NOPASSWD, command -v nvcc for the toolkit, a lock file's pid for a
flock, and "no CLI path exists" for a CLI I never ran. Each was cheap to
test and expensive to assert.
Also ran the audit forseti's 3.2.4 makes possible, since a seat could have
been silently pull-only since 3.1.2 with every failure rendering as
silence. Nine of ten seats with a live waiter report push. regin-smithy-dev
holds a live waiter and the post office has it pull-only -- but regin
deliberately released their pane route earlier tonight, so release-then-arm
ordering explains the same observation with no bug involved. Not
distinguishable from outside, so it went to regin as a question rather
than to forseti as a confirmed instance.
forseti measured my stale-lock claim and it is false. I said a lock file
holding a dead pid would make the next althing-listen exit 3 and turn a
reap into a permanent monitoring outage that reports healthy. The gate is
flock -n on an open fd, which the kernel releases when the holder dies, so
a lock left by a reaped listener is inert and exit 3 only fires against a
live holder. The pid in the file is read by --stop alone.
I reasoned from the artifact's contents when the behaviour is set by the
locking mechanism, and put the consequence in durable memory without
testing it. forseti tested before writing code. That is the third time
tonight I reported a proxy as the thing itself, after sudo -n -v for
NOPASSWD and command -v nvcc for the toolkit.
Deployed 3.2.3, which makes althing-listen refuse on a pane seat with its
own exit code rather than silently demoting it.
And found the real shape of the plugin gap, which is worse than the stale
document forseti and I were both discussing. Nothing syncs the repo's
plugin directory into the marketplace directory, so it was frozen at Aug
28 with only the UserPromptSubmit hook -- no SessionStart, no SessionEnd,
no pane-route.sh at all. That means "CC seats re-declare automatically at
their next SessionStart" has never been true on this box, which is why
every seat including our own needed a hand-fed declare. Source is fixed
and bumped to 0.1.1; refreshing the plugin cache needs a /plugin update
from the operator, and I deliberately did not hand-edit Claude Code's own
plugin bookkeeping to force it.
Triton builds its CUDA-utils shim with gcc the first time a kernel runs,
and needs Python.h to do it. Without python3-dev the box looks entirely
healthy: torch imports, the 49 GB base loads, LoRA attaches with the
right parameter count, and then the first training step dies with a bare
CalledProcessError naming a gcc invocation and an exit code.
The real message -- "fatal error: Python.h: No such file or directory" --
is discarded, because Triton sends the compiler's stdout to DEVNULL. It
cost a probe run and a model load to find something a one-line manual
compile answered immediately.
Same shape as the dots-tts container needing a C compiler at runtime: a
JIT dependency invisible at install time that only surfaces under load.
The verify step runs the compile rather than checking the package is
present, because `dpkg -l python3-dev` would pass while the compile still
failed on a missing library path or header directory.
The comparison number nobody had written down: ana-ml2's real run-3c step
times, pulled out of run-03c.log before the breaker killed it. About 10.8
to 15.8 s/it over the first 24 steps, so 604 steps lands at roughly 2.2
to 2.7 hours. Anything under about 45 s/it on the GX10 makes it an
overnight run.
Also records the exact geometry from run-03c.json and a real
adapter_config.json, so the probe measures the shape that actually ran
rather than an approximation of it.
Checked the backend-delta trap the playbook warns about before running
anything rather than after: flash-attn is installed on neither box, so
both fall back to sdpa. Library versions do differ -- torch 2.13.0 versus
2.14.0, transformers 5.15.1 versus 5.16.1 -- and that is recorded rather
than assumed harmless. The probe reads the resolved attention
implementation back off the loaded model instead of trusting the request.
The probe discards its first two steps as warmup, which is not optional on
this box: an unwarmed benchmark here already read 27 TFLOP/s when the true
figure was 93, because it was timing the PTX JIT.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
The userspace half is done and needed no root. This covers what does:
NOPASSWD sudo for infra-ops (ending the gx10 fleet exception), the docker
group, the CUDA toolkit for nvcc, and the NVIDIA Container Toolkit wired
into dockerd.
Unrun -- it needs one interactive invocation to supply the sudo password,
because no gx10 credential is vaulted and elway prompts via getpass. After
its first step lands, the box stops being an exception and later runs are
unattended.
Guards worth noting: the sudoers drop-in is validated with visudo -cf
before install, since a malformed one locks every sudo user out of a box
sitting on a desk with no out-of-band access; and cuda-toolkit is
installed rather than the cuda metapackage, so the working 580.173.02
driver is not replaced.
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.
Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.
CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.
Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.
The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.
The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.
Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
The /althing:monitor slash command is a separate artifact from the
canonical skill and sync_skill.sh does not touch it. It sits at plugin
version 0.0.1 with pre-3.2.0 text and zero mentions of pane routes, while
the canonical skill it should mirror is at 3.2.2.
That makes it active harm rather than stale documentation. The canonical
precedence rule is that a live waiter wins over a pane entry, so an agent
on a healthy pane route who follows the command verbatim demotes itself
back onto the FIFO path the harness reaps -- the path that died twice on
this seat tonight. The command cannot warn about it because it predates
the problem, and every Claude Code seat reaches for the slash command
first because it is the discoverable surface.
Declined to arm on this seat and left the route intact. Reported to
forseti with a recommendation that althing-listen refuse to arm when a
pane route exists for the handle, since a binary enforcing the documented
precedence beats a doc that relies on a reader noticing.
Also cleared a stale wake-listener lock holding a dead pid, and recorded
why it matters: the next althing-listen seeing it exits 3, whose
documented response is do not drain and do not re-arm. A stale lock turns
a reap into a permanent monitoring outage that reports itself as healthy.
Checked the disk rather than waiting on comfy-dev. Both builds are there,
pulled a minute apart on Aug 23: a 26 GB int8 and a 15 GB nvfp4-awq.
I had asked whether their nvfp4 pin was set under a Blackwell assumption,
since Ada has no native nvfp4, which would make the int8 file the right
one on the new box. It cannot be. They pinned it on irv-ml1's A6000,
which is Ampere sm_86 and has neither native nvfp4 nor native fp8. Ada
sm_89 supports a strict superset, so a pin that was correct on the weaker
card cannot be invalidated by moving to the stronger one. The migration is
incapable of breaking it.
The pin is about VRAM, not architecture. Eleven gigabytes on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs, and a
text encoder runs once per prompt rather than once per diffusion step, so
its throughput matters far less than the DiT's. That also explains why
this pin went the opposite way from their other one without either being
inconsistent. The RTX 6000 Ada is also 48 GB, so nothing relaxes.
Reclassified the 26 GB int8 from orphan to spare: with the extra drives
the destination lands near 14% full, so disk stops being the constraint
and the pin-rot argument says keep it. Question withdrawn to comfy-dev.
Four of the five items I raised are closed and tagged. The one that
matters most: --discover-pid matched comm == "claude", which means kimi,
grok, codex and pi would each have walked to the multiplexer and refused.
Four of five pane seats could never have used it. It now matches the
pane's own command, which zellij already reports and guard 1 already
compares against, so discovery and the guard read one string.
Recording a second miss of my own alongside it. I handed all four seats
explicit measured pids because I suspected the ancestry walk was broken,
then reported the suspicion as an open question rather than spending the
same twenty seconds to settle it, with four live non-Claude seats in
front of me. That pairs with the earlier miss in the opposite direction:
asserting an open risk on something the author had already measured. Same
root -- having the means to settle a question and reporting it as open
instead.
Also keeping forseti's two dead ends as negative results, because they
are the obvious things to propose next and both fail: locating a TUI's
input box in a screen dump needs per-TUI parsing, and diffing two dumps
to detect typing refuses every poke forever, because status bars carry
live token counts and clocks so consecutive dumps differ on an idle pane.
The kimi/pi coverage gap is deliberately unfixed and now sits with the
operator, with a recommendation to leave it per-seat.
All four seats answered. Delivery is now verified on four TUI families and
forseti's measured idle columns held exactly, so nobody had to guess.
The two seats guard 4 cannot cover made opposite calls on the same facts
and both are right. regin-smithy-dev released to pull-only because the
operator composes in that pane routinely, so the exposure is continuous.
bil-smithy-dev kept its route because that pane is poke-driven, making the
collision window narrow, and because pull-only had already cost them a
notice that sat unread for days. The deciding variable is who composes in
the pane and why, not risk appetite -- a flat rule either way would have
been wrong for one of them.
regin also supplied the only real path to closing the hole: pi and kimi
report no cursor because their input line is not an empty-prompt-at-idle,
so a moment-check reading pane content rather than cursor column would
cover them. Column is a proxy; an empty input line is the actual
predicate. Relayed to forseti as the lead item.
And bil corrected something upstream: the Aug 28 probes proved delivery
against non-Claude TUIs but never exercised the collision case, because
nobody was typing during them. Two different questions, one body of
evidence, only one of them answered by it -- which means the earlier
retraction of the non-Claude risk flag was right about delivery and
silent about stapling.
Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will
not be poked until it declares again.
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone
mid-sentence had their half-written message submitted with the herald's
line stapled on. It hit the operator within an hour of this evening's
deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column
and stays silent when the live column has moved.
Deployed all three steps at 22:40 and re-declared this seat, which pinned
idle_cursor=3 as expected for a claude TUI.
The finding worth keeping is the hole the fix leaves. forseti's own
measurements say kimi and pi report no cursor at all, which means
bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter
what they run. Two of the five pane seats on this box stay permanently
exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the
write-into-a-typing-pane bug" is only true where the cursor is legible.
All four seats were told, differentiated: a pre-filled re-declare and the
expected column for the two that can be guarded, and the honest version
plus the release-to-pull-only option for the two that cannot.
Also keeping forseti's post-mortem line, because it generalises: three
guards that all answer "is this the right pane" and none that asks
whether it is a good moment are one check wearing three hats.
All four notified seats re-declared within about twelve minutes and the
log went quiet. Five of five pane routes now carry the guard fields, and
delivery is confirmed on a claude seat, a pi seat and a grok seat.
Two findings survive the close-out. The herald's exclusion reason is
false for the migration case -- every declaring process was alive and
four days old with no pid wrap, and the real cause is simply that the
route predates the fields the guard needs. And the uv trap stands as the
thing most likely to bite the next person.
The third is mine. I raised non-Claude pane delivery with forseti as an
open risk on their release when forseti had personally measured it days
earlier, on the exact seats in question, and the resulting matrix is what
characterised the settle bug they fixed. The error was not caution, it
was calling something open without checking whether it was already
settled, with the peers who knew right in front of me. Recorded because
"I don't know" and "this is an open risk" are different claims and I made
the second when only the first was true.
Still open with forseti: the uv --force runbook fix, an exclusion-message
third branch, --discover-pid against a non-Claude process tree, the
one-tick latency note, and status not being durable evidence.
Operator directed that the four affected seats be told directly. Measured
their state before writing, which turned up two things worth more than
the notification itself.
The herald logs each exclusion as "the process that declared this route
is gone, or its pid was reused by something that started at a different
time." Neither is true for any of the four. Each declaring process
started minutes before its route was written and is still running four
days later, and pid_max is 4194304 against a current 2.86M so the counter
has not wrapped. The real cause is a third one the message never offers:
the route predates the guard fields, so identity cannot be verified. The
behaviour is right and the explanation is wrong, and it would send anyone
debugging it hunting a dead agent that is alive.
Separately, all four seats run non-Claude CLIs -- kimi, grok, codex and
pi -- while the pane poke is built around typing into a Claude Code pane.
Whether --discover-pid walks a non-Claude process tree, and what a poke
does to a non-Claude TUI, are both unverified. Each agent was given their
measured pid to sidestep the first, told plainly about the second, and
offered the choice between re-declaring as a test or staying pull-only.
Both findings raised with forseti.
The waiter died a second time, within minutes of being armed, so this
seat stopped re-arming and declared a pane route instead. That is exactly
what 3.2.0 shipped for.
- althing-route declare --discover-pid walks the ancestry to the
long-lived claude process rather than the ephemeral bash that invoked
it. Passing --pid $$ would bind the route to a shell that dies with the
tool call.
- Recorded the sharper form of the failure: the waiter does not just die,
it dies after confirming it is up. Both waiters reported push and
reachable immediately after arming. So a green postbox status is not
durable evidence of monitoring, and the reap is not confined to the
pane seats that were migrated -- this seat is a fifo waiter and was hit
twice.
Both consequences raised with forseti on the deploy thread, along with a
note that reachable is a report rather than a delivered poke.
forseti requested the deploy, operator-approved and tagged at c4ede0f.
All three steps landed on nh3-dev and verified: force reinstall to 3.2.0
with seven binaries, herald restart, skill sync.
The durable lesson is the trap in step 1. `uv tool install .` matches on
the source spec rather than its contents, so on a box that already had
the tool installed from that path it prints "already installed" and exits
0 having done nothing. Following the runbook literally would have left
the new binary absent with every command reporting success. Always use
--force when reinstalling from a local path.
Also recorded: the pane-route migration revoked exactly four routes,
verified by splitting on channel= before the restart rather than by
auditing fields (all twelve route files lack the new fields, so a field
audit over-counts). The four affected agents were deliberately not
notified, per the standing rule against unsolicited fleet broadcast, and
that is surfaced to the operator instead.
And this closes an open question from earlier today: the waiter that died
with status "killed" was CC 2.1.257 reaping detached tasks, which is the
premise 3.2.0 exists to address.
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.
- Verified it was a model unload, not a stop: comfyui still up 8 days,
same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
and reloads ~18.5 GB on the next render, which puts the real floor at
~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
shrank from "stop ComfyUI" to "don't render during the bench" rather
than disappearing. Flagged to both; not volunteered on comfy-dev's
behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
identified it as their prompt-feature cache, capped at 32 entries on
2026-08-14 after two production incidents. A named mechanism with an
incident history beats a plausible story, and the useful finding is
that 14.43 GB sits inside a cap they deliberately chose.
Read-only probes; nothing on the box was changed.
Memory-only; no version bump per the SemVer SKIP list.
tts-dev asked for an A6000 window for an approved TTS bench and flagged a
3090 VRAM delta. Probed the box and mapped PID to container rather than
taking the reported figures.
- The 18.5 GB process they attributed to the 3090 is comfyui, on the
A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so
stopping it gives ~44.4 GB free, not the tight margin they expected.
- Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a
third tenant the doc figure never counted, and dots-tts alone is
holding 14,430 MiB against a burn-in figure of ~6 GB. The second is
the larger finding and it is theirs to act on; handed over with a
caching-allocator hypothesis and a one-restart discriminating test.
- Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a
container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090.
Window not granted unilaterally — comfyui is comfy-dev's and they are
mid-migration, so the request went to them directly and infra-ops relays.
Ruled that the bench runs as a plain container under lkraven rather than
under /opt/docker/compose/, which is for deployed stacks and would leave
a canonical entry reporting as drift until deleted.
Read-only probes; nothing on the box was changed.
Memory-only; no version bump per the SemVer SKIP list.
Operator asked why not move all 768 GB across. It does not fit the
board's shape: the R640 has 24 slots at 6 channels/socket x 2 DPC, the
R750xa has 16 at 8 channels/socket x 1 DPC.
768 GB is either 24x 32 GB (more DIMMs than slots) or 12x 64 GB (fits,
but populates 6 of 8 channels per socket and gives up ~25% of memory
bandwidth). The board wants 16 identical DIMMs. So the targets are
512 GB if they are 32s, or 1 TB if they are 64s — taking 12 from one
spare and 4 from the other. In the 64 GB case the answer beats the
question.
Also recorded: beyond ~512 GB the return is marginal for this workload,
so take 1 TB because it is free rather than because it is needed; 64 GB
LRDIMMs run ~50 W hotter in a chassis whose high-performance fans are
unaccounted for on the invoice; and DIMM slot count now needs to be on
the iDRAC pull, since the 16-slot figure is inferred from the factory
CSV and is load-bearing for a 512-vs-1024 decision.
Memory-only; no version bump per the SemVer SKIP list.
Operator asked whether 512 GB justifies a ramfs for hot models, then
self-corrected toward ARC with a 128 GB cap.
- No ramfs/tmpfs tier. ARC is the same cache done adaptively: no curated
hot-list to rot, no boot-time copy-in, and the memory comes back under
pressure. Neither tier beats the host-to-VRAM PCIe hop anyway, so the
warm-load time is identical.
- 128 GB is below the OpenZFS Linux default of 50% of RAM, so doing
nothing already gives 256 GB. Recommend ~320 GB, not past ~75%.
- Recorded the one honest argument for tmpfs: safetensors mmap gets
double-buffered on ZFS-on-Linux, so a checkpoint can cost ~2x. Sized
for, not architected around.
- Flagged that this is not the idle-VRAM-is-reserved case: arc_max is a
ceiling on an elastic cache, not a preallocation.
- recordsize=1M cannot be won through zfs send, since recv reproduces the
source's block structure. Not worth losing incremental send over; the
128K cost on flash is metadata overhead, not throughput.
Memory-only; no version bump per the SemVer SKIP list.
Operator has 2x Dell R640 at 768 GB each and asked whether the memory is
interchangeable with the R750xa. Both are DDR4 RDIMM platforms and Dell
does not vendor-lock DIMMs, so the answer is very likely yes.
- 16 slots in the R750xa x 32 GB = 512 GB, double the factory spec, and
it deletes the 8x M04W6 purchase.
- Gating question is RDIMM vs LRDIMM. 24x 32 GB 2Rx4 RDIMM is the safe
and most likely case; 12x 64 GB LRDIMM needs Ice Lake support checked.
- Cleanest harvest is to strip one R640 entirely and leave the other
whole, rather than half-emptying both into unbalanced populations.
Separately, reading the factory CSV to answer this surfaced a gap in the
diff table: the reseller also swapped 2x Xeon Platinum 8362 (32C/64T,
265 W, DDR4-3200) for 2x Xeon Silver 4314 (16C/32T, 135 W, DDR4-2666).
The Silvers were recorded under "As bought" but never diffed, so the swap
went unremarked. Two consequences: the box cannot use the 3200 rating the
buy list was paying for, which makes 2666 R640 DIMMs a free lunch; and
the CPUs draw 260 W less, which the existing ~1,020 W power figure
already assumed correctly.
Memory-only; no version bump per the SemVer SKIP list.
Six drives fills all eight bays, which turns a capacity top-up into a
one-shot build decision. Recorded because the reasoning survives whatever
he picks:
- raidz2 over mirrors. Workload is large sequential reads of safetensors,
ARC fronts it, SSD resilver has no seek penalty, and "expand two at a
time" is meaningless once every bay is full. raidz2 survives any two
failures; 4x mirrors dies to an unlucky pair.
- Buy 8, not 6. A raidz vdev caps every member at the smallest, so the
two as-bought 1.92 TB drives would cap all eight and put two used
drives of unknown endurance inside the parity set.
- Drive size is now the permanent ceiling. SAS/SATA backplane, all bays
full, and the free-PCIe-slot inventory is still unpulled.
Runway table included with an explicit caveat that the growth rate is
projected off one acquisition batch, not measured.
Memory-only; no version bump per the SemVer SKIP list.
comfy-dev's disk-vs-catalog diff and a recount of my own figures both
landed on this thread. Two numbers were wrong and both were headed for
the operator's sizing conversation.
- My ~90% was a double-count. I read ALLOC 1.45T while their pull was
running and then added the full ~112 GB on top; most of it was already
in that reading. "Onboarded" is not "landed". Settled payload is
~1.47 TiB and the as-bought mirror lands at 84%, not 90%.
- comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the
as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights
still lands at 72%, with ~140 GiB of runway on a store that took on
~100 GiB in one day. The constraint is vdev layout, not payload — a
1.75 TiB pool stays 1.75 TiB whatever goes in it.
- So the prune audit and the drive purchase are independent decisions
and neither gates the cutover. Presenting them to the operator that
way rather than as a trade.
Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev
smallest-member gotcha if the new drives get paired one-each with the
1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question
back to them on whether the H3 encoder's nvfp4 pin was set under a
Blackwell assumption that sm_89 does not satisfy.
Memory-only; no version bump per the SemVer SKIP list.
Measured irv-ml1's storetank against the R750xa's as-bought drives while
answering comfy-dev's "does the Ada box have ZFS?" question.
- storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression
off at compressratio 1.00x (safetensors incompressible — no win at
recv). comfy-dev's ~112 GB batch is landing into it now.
- The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB,
smaller than the pool it receives from. Migration would arrive at
~90% full with no growth room.
- Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) ->
two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact.
- Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN
recommended, surfaced to the operator.
Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all
twelve running containers: only comfyui mounts /storetank, so (b) was
unavailable during the transition, not structurally. Right conclusion,
wrong reason — infra-ops reasoned about the box when the question was
about the mount.
Memory-only; no version bump per the SemVer SKIP list.
Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing
Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the
2400 W PSUs, and the GPU risers, cables and high-performance fans all absent.
Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU
8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the
card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe
question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it.
Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line
target was not reached because the guards hold nearly everything else back as under 14 days
or carrying open deferred work.
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.
Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.
Synapse mounts its admin API on the same vhost as the client API, so publishing
matrix.phasefinal.com published the admin surface too -- it answered 200 from
the open internet. HMAC-protected, so not an open door, but Synapse's own
guidance is to keep it off the public listener.
A higher-priority router (explicit priority 100, not relying on Traefik's
rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist.
Verified from a genuinely external vantage rather than from a fleet host, since
nh3-dev sits inside the allowed range and would have proved nothing: via the NH3
residential egress proxy the admin path returns 403 while the client API returns
200 and Element is unaffected.
The 10.0.0.0/8 entry matches nothing today and the comment says so rather than
implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet
hosts hairpin out their own WAN -- a request from nh3-dev arrived as
70.230.226.88. The rule is effectively deny-all through Traefik, which is the
intended posture: admin work goes through docker exec to localhost:8008 and
never traverses Traefik. Allow-listing the sites' WAN addresses was considered
and rejected as a maintenance trap on dynamic addresses.
Also brings the stack under stacks/ with the Postgres password replaced by a
required .env variable. The tracked copy and the live file have therefore
DIVERGED and deploy-stack.sh must not be used until the live file reads from a
.env; the README says so.
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.
The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.
Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.
Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.
playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.
Two things learned the hard way and recorded in the playbook:
- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
'enabled'. A guard of always skips, and a verify
written the same way passes while the desktop is still running. Both now test
is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
the connecting user and must carry their own sudo; connect as infra-ops.
The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
Synapse v1.120.0 -> v1.159.0 and Element-web v1.11.80 -> v1.12.27 (2026-09-01).
The existing build steps date from the AIPA era and are now marked as provenance
rather than as instructions.
Records what only existed in a session transcript:
- Schema migrations are one-way; rollback is restore-from-dump. Pre-upgrade
pg_dump procedure, with a pg_restore --list verification step.
- Why the appservice user namespace is now exclusive: false. exclusive governs
who ELSE may act, not what the appservice may do, so on a closed single-admin
server it locked out all other account creation to prevent squatting that
cannot occur. Includes the two things not to do: narrow the regex (orphans 13
accounts) or rename the id (Synapse keys ownership on it).
- The shared-secret registration HMAC takes no trailing null after notadmin.
- Room alias convention #<agent>-<purpose>, operator-ratified, with its cost
accepted deliberately and its rationale stated as room-identity-carries-tier
rather than push-payload-carries-room-name.
- Push reality: the pusher is event_id_only, so the notification is assembled
on-device by Element X's service extension. Records the resulting
server-invisible failure mode when the phone cannot reach the homeserver.
- QR sign-in requires Matrix Authentication Service and why it is deferred.
Ops ownership recorded: worldtree-dev writes the bridge, infra-ops operates
this instance.
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin,
so cache policy stays declared once in nginx.conf rather than split between the
repo and the dashboard. Always Online enabled, which is what actually survives
an origin outage; a 300s document TTL alone would only mask five minutes.
Verified: document and assets both reach cf-cache-status HIT, apex 301s to www,
edge email obfuscation active.
vh