Step 8 executed on the operator's direct authorization with preconditions
checked before firing and the post-state read back rather than inferred. The
rollback window is closed and the arc is finished except the handle rename.
Records the remote-ssh-mcp fork and, more usefully, why it was chosen over the
far more popular alternative: it never handles key material, delegating to the
system OpenSSH client, and its dependency surface is two packages. Also records
that its denylist is four bypassable regexes and must not be counted as
security — the host allowlist is the real boundary — plus the two things to
settle before the tool is used in anger.
The delete was verified from outside rather than taken on report: an admin key
cannot see consumer agents, but the key roster shows the old key authenticating
once after 47 minutes of silence with the new key following 35 seconds later,
which is the signature of delete-then-confirm.
Step 8 is deliberately not done. ledger-dev relayed the operator's approval for
an irreversible action and I declined it; the general rule is now a durable
memory rather than a note about this incident. Both keys stay live.
Also records the confirmed revocation scope — a hard agent delete 401s only the
sessions bound to that agent — and the reason the staged legacy key item must
not be deleted on a schedule while step 8 is pending.
Cutover landed clean — a 201 define rather than 409/PATCH, which is what proves
the new identity is actually in use. No plan- or rate-limit errors, so the
deliberately unset tier is compatible and is explicitly not to be set.
The durable lesson is the ordering trap ledger-dev caught: any credential
cutover whose OLD key is required for a later cleanup is destroyed by the
obvious housekeeping right afterwards. Re-vaulting the post-cutover env.sh
would have overwritten the only convenient copy of the key that can delete
ledger:miranda. Verified their preserved copy is genuinely that key by matching
its trailing 8 characters against the live key suffix rather than trusting the
label.
Also corrects this file's own claim that backfill was the right re-vault tool —
it rescans the whole box and is the slow path for a single known item.
They pulled it from the vault and verified the sha independently, so delivery is
established rather than assumed. The remaining step — swapping the live
credential in env.sh, flipping worldtree.user_id, registering svos:miranda and
restarting the service — is the operator's, not mine and not theirs off a peer
message.
Also records that the global CLAUDE.md Heimdall routing line was corrected in
place on operator instruction, so the ruling is not carried by this file alone.
user_id=svos, key_id=eab3cdbe, wt_live_ format, on worldtree-personal
(10.250.50.152:8081). Deployment established by locating the ledger key there
rather than assumed: created 2026-07-13, last used today, matching ledger-dev's
account exactly.
Value vaulted at nh3-dev/development/svos/worldtree-api-key and verified against
the mint response before the scratch copy was removed. Delivered by vault path
rather than over althing — ledger-dev runs on the same box under the same uid,
so the bus never carried the credential.
The ledger key was read back from /admin/keys AFTER the mint and is untouched,
which is what keeps their deletion ordering possible.
Plan tier left unset deliberately and disclosed: the API offers no way to read a
user's current tier, so matching it would have been a guess.
Also records the operator ruling that worldtree-dev owns code only and no ops,
which supersedes the global CLAUDE.md routing of Heimdall auth to worldtree-dev.
Rename executed on the operator's direct authorization. Repo id 70 unchanged.
Redirect verified by measurement rather than by citing upstream #807: web and
API paths both 301, and `git ls-remote` against the old URL warns, follows, and
returns HEAD b48a11ca5183. Existing clones keep working.
Recorded as a standing hazard that the old path is now burned — a repo created
at corviduo/ledger kills the redirect silently.
Also opens the Heimdall item ledger-dev had been holding as "awaiting infra-ops"
without ever having sent it, including the deletion-ordering trap that makes
retiring the old key first unrecoverable.
The `secret` CLI has no rename, so this was re-put + rm. Stored
nh3-dev/development/svos/env.sh from the live file (sha 7253633d4155, verified
on read-back) and retired nh3-dev/development/ledger/env.sh (sha feb418634e10,
id 3a2af37c-c5aa-4f46-9178-f4fb6008a753, soft-deleted to trash and recoverable).
The two shas differ: the vaulted copy was a 2026-08-11 snapshot and the live
file had drifted since without ever being re-vaulted. Recorded because the
lesson generalises — the vault goes stale unless `secret backfill` is re-run.
Gitea corviduo/ledger deliberately NOT renamed. It is ledger-dev's repo; the
answer they needed is that 1.26.1 writes a repo_redirect on a same-org repo
rename, org/user renames do not redirect, the redirect dies if anything
re-creates the old path, and both the repo and the org carry zero webhooks.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.
It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.
Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.
Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".
Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:
- `container.valid_volumes: []` does NOT keep the docker daemon out of
jobs. act_runner mounts /var/run/docker.sock on its own, so every job
on the shared runner is uid 0 with `docker ps` over all 49 containers
on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
It is also load-bearing: four repos drive buildx through it, so the
fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
resolves from the local mirrors today. That is github-independence per
workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
act_runner's action-fetch auth since 2026-08-05.
Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.
Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.
Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.
Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.
Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.
Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.
Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
Recorded the ownership call, which forseti left open. The plugin deployer
lives in eshpfi rather than the althing repo because it targets
per-machine paths, and althing's sync_skill.sh deliberately reaches into
no other tree. Putting a plugin installer upstream would break that
boundary for one consumer's convenience. Their repo stays the source;
this one does the installing.
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.
The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.
Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.
Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.
CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.
Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.
The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.
The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.
Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
Checked the disk rather than waiting on comfy-dev. Both builds are there,
pulled a minute apart on Aug 23: a 26 GB int8 and a 15 GB nvfp4-awq.
I had asked whether their nvfp4 pin was set under a Blackwell assumption,
since Ada has no native nvfp4, which would make the int8 file the right
one on the new box. It cannot be. They pinned it on irv-ml1's A6000,
which is Ampere sm_86 and has neither native nvfp4 nor native fp8. Ada
sm_89 supports a strict superset, so a pin that was correct on the weaker
card cannot be invalidated by moving to the stronger one. The migration is
incapable of breaking it.
The pin is about VRAM, not architecture. Eleven gigabytes on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs, and a
text encoder runs once per prompt rather than once per diffusion step, so
its throughput matters far less than the DiT's. That also explains why
this pin went the opposite way from their other one without either being
inconsistent. The RTX 6000 Ada is also 48 GB, so nothing relaxes.
Reclassified the 26 GB int8 from orphan to spare: with the extra drives
the destination lands near 14% full, so disk stops being the constraint
and the pin-rot argument says keep it. Question withdrawn to comfy-dev.
Four of the five items I raised are closed and tagged. The one that
matters most: --discover-pid matched comm == "claude", which means kimi,
grok, codex and pi would each have walked to the multiplexer and refused.
Four of five pane seats could never have used it. It now matches the
pane's own command, which zellij already reports and guard 1 already
compares against, so discovery and the guard read one string.
Recording a second miss of my own alongside it. I handed all four seats
explicit measured pids because I suspected the ancestry walk was broken,
then reported the suspicion as an open question rather than spending the
same twenty seconds to settle it, with four live non-Claude seats in
front of me. That pairs with the earlier miss in the opposite direction:
asserting an open risk on something the author had already measured. Same
root -- having the means to settle a question and reporting it as open
instead.
Also keeping forseti's two dead ends as negative results, because they
are the obvious things to propose next and both fail: locating a TUI's
input box in a screen dump needs per-TUI parsing, and diffing two dumps
to detect typing refuses every poke forever, because status bars carry
live token counts and clocks so consecutive dumps differ on an idle pane.
The kimi/pi coverage gap is deliberately unfixed and now sits with the
operator, with a recommendation to leave it per-seat.
All four seats answered. Delivery is now verified on four TUI families and
forseti's measured idle columns held exactly, so nobody had to guess.
The two seats guard 4 cannot cover made opposite calls on the same facts
and both are right. regin-smithy-dev released to pull-only because the
operator composes in that pane routinely, so the exposure is continuous.
bil-smithy-dev kept its route because that pane is poke-driven, making the
collision window narrow, and because pull-only had already cost them a
notice that sat unread for days. The deciding variable is who composes in
the pane and why, not risk appetite -- a flat rule either way would have
been wrong for one of them.
regin also supplied the only real path to closing the hole: pi and kimi
report no cursor because their input line is not an empty-prompt-at-idle,
so a moment-check reading pane content rather than cursor column would
cover them. Column is a proxy; an empty input line is the actual
predicate. Relayed to forseti as the lead item.
And bil corrected something upstream: the Aug 28 probes proved delivery
against non-Claude TUIs but never exercised the collision case, because
nobody was typing during them. Two different questions, one body of
evidence, only one of them answered by it -- which means the earlier
retraction of the non-Claude risk flag was right about delivery and
silent about stapling.
Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will
not be poked until it declares again.
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone
mid-sentence had their half-written message submitted with the herald's
line stapled on. It hit the operator within an hour of this evening's
deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column
and stays silent when the live column has moved.
Deployed all three steps at 22:40 and re-declared this seat, which pinned
idle_cursor=3 as expected for a claude TUI.
The finding worth keeping is the hole the fix leaves. forseti's own
measurements say kimi and pi report no cursor at all, which means
bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter
what they run. Two of the five pane seats on this box stay permanently
exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the
write-into-a-typing-pane bug" is only true where the cursor is legible.
All four seats were told, differentiated: a pre-filled re-declare and the
expected column for the two that can be guarded, and the honest version
plus the release-to-pull-only option for the two that cannot.
Also keeping forseti's post-mortem line, because it generalises: three
guards that all answer "is this the right pane" and none that asks
whether it is a good moment are one check wearing three hats.
All four notified seats re-declared within about twelve minutes and the
log went quiet. Five of five pane routes now carry the guard fields, and
delivery is confirmed on a claude seat, a pi seat and a grok seat.
Two findings survive the close-out. The herald's exclusion reason is
false for the migration case -- every declaring process was alive and
four days old with no pid wrap, and the real cause is simply that the
route predates the fields the guard needs. And the uv trap stands as the
thing most likely to bite the next person.
The third is mine. I raised non-Claude pane delivery with forseti as an
open risk on their release when forseti had personally measured it days
earlier, on the exact seats in question, and the resulting matrix is what
characterised the settle bug they fixed. The error was not caution, it
was calling something open without checking whether it was already
settled, with the peers who knew right in front of me. Recorded because
"I don't know" and "this is an open risk" are different claims and I made
the second when only the first was true.
Still open with forseti: the uv --force runbook fix, an exclusion-message
third branch, --discover-pid against a non-Claude process tree, the
one-tick latency note, and status not being durable evidence.
Operator directed that the four affected seats be told directly. Measured
their state before writing, which turned up two things worth more than
the notification itself.
The herald logs each exclusion as "the process that declared this route
is gone, or its pid was reused by something that started at a different
time." Neither is true for any of the four. Each declaring process
started minutes before its route was written and is still running four
days later, and pid_max is 4194304 against a current 2.86M so the counter
has not wrapped. The real cause is a third one the message never offers:
the route predates the guard fields, so identity cannot be verified. The
behaviour is right and the explanation is wrong, and it would send anyone
debugging it hunting a dead agent that is alive.
Separately, all four seats run non-Claude CLIs -- kimi, grok, codex and
pi -- while the pane poke is built around typing into a Claude Code pane.
Whether --discover-pid walks a non-Claude process tree, and what a poke
does to a non-Claude TUI, are both unverified. Each agent was given their
measured pid to sidestep the first, told plainly about the second, and
offered the choice between re-declaring as a test or staying pull-only.
Both findings raised with forseti.
The waiter died a second time, within minutes of being armed, so this
seat stopped re-arming and declared a pane route instead. That is exactly
what 3.2.0 shipped for.
- althing-route declare --discover-pid walks the ancestry to the
long-lived claude process rather than the ephemeral bash that invoked
it. Passing --pid $$ would bind the route to a shell that dies with the
tool call.
- Recorded the sharper form of the failure: the waiter does not just die,
it dies after confirming it is up. Both waiters reported push and
reachable immediately after arming. So a green postbox status is not
durable evidence of monitoring, and the reap is not confined to the
pane seats that were migrated -- this seat is a fifo waiter and was hit
twice.
Both consequences raised with forseti on the deploy thread, along with a
note that reachable is a report rather than a delivered poke.
forseti requested the deploy, operator-approved and tagged at c4ede0f.
All three steps landed on nh3-dev and verified: force reinstall to 3.2.0
with seven binaries, herald restart, skill sync.
The durable lesson is the trap in step 1. `uv tool install .` matches on
the source spec rather than its contents, so on a box that already had
the tool installed from that path it prints "already installed" and exits
0 having done nothing. Following the runbook literally would have left
the new binary absent with every command reporting success. Always use
--force when reinstalling from a local path.
Also recorded: the pane-route migration revoked exactly four routes,
verified by splitting on channel= before the restart rather than by
auditing fields (all twelve route files lack the new fields, so a field
audit over-counts). The four affected agents were deliberately not
notified, per the standing rule against unsolicited fleet broadcast, and
that is surfaced to the operator instead.
And this closes an open question from earlier today: the waiter that died
with status "killed" was CC 2.1.257 reaping detached tasks, which is the
premise 3.2.0 exists to address.
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.
- Verified it was a model unload, not a stop: comfyui still up 8 days,
same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
and reloads ~18.5 GB on the next render, which puts the real floor at
~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
shrank from "stop ComfyUI" to "don't render during the bench" rather
than disappearing. Flagged to both; not volunteered on comfy-dev's
behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
identified it as their prompt-feature cache, capped at 32 entries on
2026-08-14 after two production incidents. A named mechanism with an
incident history beats a plausible story, and the useful finding is
that 14.43 GB sits inside a cap they deliberately chose.
Read-only probes; nothing on the box was changed.
Memory-only; no version bump per the SemVer SKIP list.
tts-dev asked for an A6000 window for an approved TTS bench and flagged a
3090 VRAM delta. Probed the box and mapped PID to container rather than
taking the reported figures.
- The 18.5 GB process they attributed to the 3090 is comfyui, on the
A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so
stopping it gives ~44.4 GB free, not the tight margin they expected.
- Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a
third tenant the doc figure never counted, and dots-tts alone is
holding 14,430 MiB against a burn-in figure of ~6 GB. The second is
the larger finding and it is theirs to act on; handed over with a
caching-allocator hypothesis and a one-restart discriminating test.
- Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a
container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090.
Window not granted unilaterally — comfyui is comfy-dev's and they are
mid-migration, so the request went to them directly and infra-ops relays.
Ruled that the bench runs as a plain container under lkraven rather than
under /opt/docker/compose/, which is for deployed stacks and would leave
a canonical entry reporting as drift until deleted.
Read-only probes; nothing on the box was changed.
Memory-only; no version bump per the SemVer SKIP list.
comfy-dev's disk-vs-catalog diff and a recount of my own figures both
landed on this thread. Two numbers were wrong and both were headed for
the operator's sizing conversation.
- My ~90% was a double-count. I read ALLOC 1.45T while their pull was
running and then added the full ~112 GB on top; most of it was already
in that reading. "Onboarded" is not "landed". Settled payload is
~1.47 TiB and the as-bought mirror lands at 84%, not 90%.
- comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the
as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights
still lands at 72%, with ~140 GiB of runway on a store that took on
~100 GiB in one day. The constraint is vdev layout, not payload — a
1.75 TiB pool stays 1.75 TiB whatever goes in it.
- So the prune audit and the drive purchase are independent decisions
and neither gates the cutover. Presenting them to the operator that
way rather than as a trade.
Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev
smallest-member gotcha if the new drives get paired one-each with the
1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question
back to them on whether the H3 encoder's nvfp4 pin was set under a
Blackwell assumption that sm_89 does not satisfy.
Memory-only; no version bump per the SemVer SKIP list.
Measured irv-ml1's storetank against the R750xa's as-bought drives while
answering comfy-dev's "does the Ada box have ZFS?" question.
- storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression
off at compressratio 1.00x (safetensors incompressible — no win at
recv). comfy-dev's ~112 GB batch is landing into it now.
- The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB,
smaller than the pool it receives from. Migration would arrive at
~90% full with no growth room.
- Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) ->
two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact.
- Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN
recommended, surfaced to the operator.
Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all
twelve running containers: only comfyui mounts /storetank, so (b) was
unavailable during the transition, not structurally. Right conclusion,
wrong reason — infra-ops reasoned about the box when the question was
about the mount.
Memory-only; no version bump per the SemVer SKIP list.
Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing
Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the
2400 W PSUs, and the GPU risers, cables and high-performance fans all absent.
Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU
8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the
card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe
question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it.
Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line
target was not reached because the guards hold nearly everything else back as under 14 days
or carrying open deferred work.
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.
Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.
The in-flight section was a day stale: it still described run 3c as the
live subject on a box where nothing had moved. Rewritten around what is
actually true now -- v3 deployed fleet-wide, the post office relocated
to nh3-docker, sec serving on GPU0, run 3c still held on power.
Six new decision entries, three of which carry findings that outlive
their incident: the inbound half of the handle-resolution bug (a stale
ALTHING_HANDLE reads another agent's mailbox and reports it empty,
which is a second route into the failure v3 exists to prevent), the OOM
attribution to Claude Code sessions, and the operator's two explicit
belays recorded so a later session does not re-raise them as new.
Auto-archival fired at 301 lines and moved exactly one entry. Three
others were old enough and every one carries a still-open deferred
pointer -- the parked CI flip, muninn-gate's submit path, and the triton
backend deferred to the Ada refresh. Held back per the guards; an
over-cap file that keeps live decisions beats a scannable one that lost
them. The entry that did move had its deferred item closed today:
nh3-extdev's staged v2.1.2 wheel is moot now that the box runs 3.1.1.
Operator directive, and a standing goal: the bus belongs on the docker
host. The flag-day deployment put it on nh3-dev because the herald lives
there -- but the herald is the piece that must be host-local, and the
post office is explicitly the piece that is not.
nh3-dev was wrong on three counts. Our own server table calls it "not a
Docker-stack host". It has had three OOM events in fourteen days with
the interval halving, and the confirmed hog is Claude Code sessions at
5-18 GB, which is that box's actual job. And mem_limit protects the
fleet from the post office while doing nothing in the other direction:
oom_score_adj was 0, an ordinary kill candidate, on a box whose last
sweep took althing-herald and uvicorn. The new deployment sets
oom_score_adj=-500.
The compose is now version-controlled here as a normal stack rather than
living only in the althing repo's deploy dir.
## docker stop does not checkpoint the WAL
The database was 155 KB with a 4.1 MB write-ahead log, and every recent
message was in the log. A clean container stop left it untouched -- an
explicit PRAGMA wal_checkpoint(TRUNCATE) was required.
A docker cp of the .db alone would have produced a database that opens
cleanly, passes integrity_check, serves the full 73-handle roster, and
is missing the day's mail, with nothing raising an error. Row counts
were verified at source, in the staged copy, and after seeding, because
the count is the only thing that separates those two outcomes.
The old volume is left in place. Not a rollback path, which the operator
ruled out -- just not deleting the only other copy on the day of a move.
## Follow-up left open
The image has no registry push and moves by save/ssh/load, so a rebuild
means repeating that by hand. It should join the gitea registry pattern
the other stacks use.
Two operator-authorised changes on the same afternoon.
## althing v3 (U9b flag day, one-way, no rollback)
The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.
73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.
Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.
nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.
The v2 database is left inert on disk. There is no import path and none
was improvised.
## sec onto GPU0
GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.
Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
forseti asked for journald kernel persistence plus sysstat, on the
premise that nh3-dev's three OOM events in 14 days left no evidence.
The premise was wrong. journald has been persistent all along: 15,068
kernel entries in the 82-day previous boot and 351 OOM records across
retained boots, full task tables included. `journalctl -b -1 -k`
returned one entry because it ran as a user in neither adm nor
systemd-journal, and journalctl shows only your own messages in that
case. The same artifact produced the "journal stops at 05:36:08 with no
shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM
kills logged at 05:38, 05:40 and 05:42.
So the fix for "no evidence" is a group membership, not a logging
change: usermod -aG adm lkraven, which is the group Debian's journald
ACL names explicitly.
With the journal readable the attribution is already in it. The
versioned Claude Code binary lives at .local/share/claude/versions/,
so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as
are those named claude. Every one of the twelve largest resident
processes ever recorded on this box is a CC session, topping out at
18.4 GB. Everything else killed is 30-55 MB collateral, which clears
the althing daemons by measurement rather than by their own sampling.
sysstat and atop are added because the journal records the moment of
the kill, not the ramp, and names the victim rather than the winner.
atop was not requested and is the one that matters: with a dozen panes
open, only a per-process timeseries says which session was growing.
Not done: a cgroup cap on CC sessions. It is the real mitigation and it
would kill long-running sessions mid-work, so it goes to the operator.
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."
That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.
Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.
Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.
Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.
#1 17:53:33 PDT killed by the power loss at step 80/604
#2 20:58:41 PDT stopped deliberately at 21:07:40, healthy
The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.
Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.
Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.
Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm
guardrail regression that its own preregistered rule passed -- a pooled preserve-list
test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched,
killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the
operator at step 22 pending a weekend power triage.
Also captured: the corpus mix was specified in a unit the optimiser never sees
(45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage
are one direction in weight space, so the merge-back measures the problem rather than
fixing it; four guests including the storage SPOF had onboot unset and never came back
from the outage, now fixed with dependency ordering; and a transport failure that
enters a measurement as a value looks like whatever you hoped to find -- which found a
live defect in another agent's instrument an hour after it was reported.
Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and
abandoned: 0); 4 held back on open deferred-work pointers.
Run 2 is finished, gated FAIL, and serving on the gateway at operator request.
Run 3's corpus was built to brokkr's first recipe and held before any GPU spend:
creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA,
LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the
two roots that overlap, at median jaccard 0.873. Containment, not overlap.
Dedup direction reversed so the primary source survives rather than the copy
inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the
largest contributor. Wholly-human share up, megamix share down, total context
unchanged at 12.49M so the operator's settled mix arithmetic survived.
Two structural findings recorded because they outlive this recipe: F1 'excise
PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that
work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP
and stheno remain unchecked against any other root.
Also records the correction I published wrong twice: run 2 was never unstable.
All 46 flags were too_short, the collapse guards fired zero times, and it is the
left tail of a length distribution -- not new to run 2 either, so it is a
property of the recipe and a further base swap will not fix it.
Operator asked for it so he can evaluate the failed tune by hand, overriding my
not-in-the-gateway recommendation. His call.
erp-tune-v1 was DELETED from the config in the same reload rather than
repointed, so the name now 400s cleanly instead of 500ing against a stopped
backend. Deleting rather than repointing is the point: repointing would resolve
a name a consumer already knows to different weights, silently.
The config entry carries the failed-gate table, the long-form truncation (9.9%)
and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone
reading the gateway config learns what they are calling without having to find
the althing thread. Fleet verified healthy after the restart.
Records the verdict as a FAIL without rounding it off, and the three findings
worth more than the verdict:
- T6 spatial +15.0 where run 1 failed the same axis at -3.5, with the base
swap as the only intended variable. Neither run ships; together they price
what the abliteration was costing, which neither could answer alone.
- an output-stability regression visible ONLY on long-form (truncated 0->38,
degenerate 0->19 per 384) that the reasoning battery could not see across
four passes because its answers are short
- PIPPA's 123-word product clip sitting in the length signal at 70.3% of bot
TURNS against 37.5% of bot WORDS, with the counter-evidence recorded too
(the tune landed near the median, not the cap)
Also records why keeping the tune out of the LiteLLM gateway now reads as
clearly right rather than merely cautious: a FAILED tune must not be one alias
resolution away from a consumer who has not read the thread.
Records the floors the tuned deltas have to clear, since they are the whole
point of the base pass and are not recoverable from anywhere else: reasoning
core 0.5 pt, diversity overall 0.0125, story attractor 0.0000.
Two caveats that would otherwise be misread:
- the rp family froze ZERO markers, so its attractor hit rate is structurally
0.0 on both arms. That reads as a clean result and means the instrument
cannot discriminate on that family; rp is measured on the distance axis
only.
- 'Elias' in 92/96 base stories is an independent replication of a published
102/144 on the same family, at a higher rate -- not a novel finding.
Image digest sha256:4091d5593f77 verified identical across both arms, which was
brokkr's stated void condition.
brokkr withdrew the both-arms-concurrent requirement himself: his diversity
battery emits the frozen marker list to a FILE, so the arms were never a live
dependency. The real constraint is narrower -- all of one arm's passes on one
served instance before the swap -- and sequential satisfies it. No fleet seats
displaced, operator not woken.
Records the two parity guards, both of which came out of failures rather than
foresight: the image is pinned by DIGEST (a vLLM version change between arms six
hours apart is a base swap that appears in no config diff), and /tank/aimodels
is mounted for BOTH arms even though only the base needs it, because a mount
that differs between arms is a difference between arms.
vh