Quantifying exposure to a gate-failed tune: counting by the gateway alias gave 363
rows, counting by the artifact gave 77, because the alias had carried three
different models that day. Wrong in the direction that looks careful.
From a near-miss brokkr-smithy-dev caught and declined to send: a post-fix
verification read raced a gateway restart and reported the alias still present.
A disagreement between two observers is not a finding until the boring
explanation is ruled out, and during an incident the pressure runs the other way.
- gx10:8098 erp-tune-v7 stopped 17:42; ana-ml2:8021 trial NVFP4 seat stopped 17:43
(infra-ops' call — the same adapter was on the SHARED-KEY gateway ~15:30-17:43
under the operator's pre-gate instruction; surfaced to him as reversible)
- probe NOT re-run, flagged generations NOT opened, length verdict left UNSET
- every artifact preserved; disposition is the operator's
- non-safety results recorded but explicitly NOT certified (primary flat +2,
diversity reduced, coherence 1.0 -> 0.875, control valid at 1.0)
The empty-snapshot promote and brokkr-smithy-dev's -s-on-a-touched-sentinel are
the same defect from opposite sides. Records the direction split (false
reassurance / false refusal / false alarm) because the mitigations differ and a
post-mortem that lumps them loses the actionable half.
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.
The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.
Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
Operator is keeping the override rather than reverting: he has a replacement
cable and will run it at the weekend. Recorded as a held position so a later
session does not revert it as an oversight or re-run the diagnosis.
Also corrects a claim this runbook made an hour earlier. I wrote that the
zero-error 1G link ruled out a marginal cable. It did not — it only proved the
cable was clean at 1G. The two-hour soak shows errors climbing monotonically at
10G (200 to 221 rx, ~1 per 23 MB) plus one self-recovering link drop, so the
autoneg fallback had a real basis and the fix is the cable, not the config.
The OEM SFP-H10GB-CU3M between ESH-Media SFP+1 and UDM SFP+2 linked cleanly
at 1000 with zero errors on both ends, which rules out a damaged cable — a
marginal one links at 10G and throws CRCs rather than dropping a rate. The
switch reported sfp_compliance "Unknown": it reads the EEPROM but cannot
parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate.
A TP-Link TL-SM5220 DAC on the adjacent UDM port runs at 10000 with the same
speed_caps and autoneg on, which exonerated the port, the firmware and
autoneg before anything was changed.
Forced autoneg off / 10000 / full duplex on the UDM end only. ESH-Media's
end followed to 10000 on its own, which is the proof the cable was always
capable and only the negotiation was wrong. Both ends now 10000.
The UDM end was chosen because it is the recoverable one: ESH-Media reaches
the controller through this link, so a failed force there would strand the
switch. The path to the UDM was traceroute-verified not to cross the link,
and the revert payload was written before the forward one was sent.
Cost ~60 s of link renegotiation. ESH-Media, the E7-ESH-Media AP and the
USW Flex XG ESH-Office-Vuong all returned at uplink_speed 10000 with uptimes
unbroken; the Zigbee coordinator and HA's path to it were re-verified.
⚠ The port is now pinned: a non-10G module will not link there. Documented,
along with the ~200 link-up errors that then went flat and the fact that
steadily climbing errors would mean the cable genuinely cannot do 10G.
I described ea:f6:0a:ca:f5:b4 as "locally administered, no OUI" and treated
it as the class of MAC that may regenerate at boot. ha-dev corrected it and
the correction verifies: the device advertises e8:f6:0a:ca:f5:b4 over mDNS,
which differs in exactly the locally-administered bit, and E8:F6:0A is
registered to Espressif Inc. in the IEEE registry while EA:F6:0A resolves to
nothing. That is the standard ESP32 pattern — one factory base MAC in eFuse,
sibling interface MACs derived deterministically — so the reservation is
keyed correctly and cannot drift on its own.
The PoE-cycle test still stands and is now corroborating evidence rather than
the only evidence. The residual risk narrows to a firmware change to the
derivation scheme.
Also from ha-dev: tcp/7638 is open alongside 6638; mDNS crosses the VLAN
boundary so HA rediscovers without help; and HA's pending smlight config flow
is keyed on the mDNS service name, which did not change, so a stale flow may
still hold the dead 10.0.10.58 and should be dismissed rather than confirmed.
ha-dev declined the dns: resolver fix on their stack — configuring by IP
costs them nothing and the entry would couple HA name resolution to AdGuard
uptime for no present benefit.
The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.
Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.
The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.
ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.
Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.
Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.
Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:
io pressure full avg10 96.5% -> 0.01%
in-flight writes 45, completing none -> 0 0
D-state processes 11 -> 0
32 MiB dsync write never completed -> 0.24s (133 MB/s)
The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.
Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.
It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.
Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.
Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".
Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.
Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.
The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.
The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.
Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.
Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.
The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.
deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.
Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.
That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).
The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.
Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
The cross-reference loop filtered live waiters on `kill -0 "$pid"`. A lock
file left by a reaped listener names a pid the kernel is free to reissue,
so the loop would report an unrelated process as a live waiter — and a
phantom waiter is how a false "your seat is unreachable" notice reaches a
seat that is fine, with nothing in the output to falsify it.
`session_listener.sh --stop` already refuses this standard: it walks
/proc/<pid>/cmdline for the `--_route=<handle>` segment before signalling,
on the grounds that a live pid proves existence and not identity. The audit
now asserts the same thing and prints STRANGER on a mismatch.
Verified against the live box: six waiters, all identity-confirmed, all
reporting push.
vh