Config-file deployment (the trial model lives in config.yaml, so /model/update refuses it):
block rewritten to run-6 reality — abliterated base, NVFP4A16 artifact, stacks/erp-seat,
no gate on this artifact by operator ruling, same-site hop. Deployed to ana-docker and the
gateway restarted; trial -> erp-tune-v6-nvfp4a16 verified end to end (3/3).
- services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py: linearize_moe first (playbook §3.15),
asserts the expert Linear count, routers/vision/audio/norms/lm_head ignored, W4A16 for RP
long-session fidelity, post-steps restore processor configs + template and reset the
tokenizer truncation cap (§3.14); --dry-run proves targets before GPU time
- services/erp-seat-quant/run_quant_erp_v6.sh: detached container on GPU1 (vllm-llmcompressor)
- stacks/erp-seat: serve recipe copied from gemma4-charrp, true served name only, port 8021
- scripts/erp-tune-gx10/pull-verify-jenerallee78.sh + base-pin-jenerallee78-shards.txt:
revision-pinned root-shard pull, 32/32 sha256+size vs brokkr-smithy pins, index
set-equal to stock, STOCK tokenizer set installed over the repo's (which bakes in
a 256-token truncation); repo originals kept as *.repo
- scripts/erp-tune-gx10/run-06-gx10.json + launch-run-06.sh: run-05 config with the
base swapped, recipe-r6, survivors-r5 verbatim, stock template path
- docs/runbooks/gx10-run-06.md: pull/verify record, free-check result (encode
reproduces run 5 exactly), hf download --include gotcha, gate naming
(erp-seat-base-ara / erp-tune-v6)
Operator-directed question (via brokkr): which weights at
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 — stock or the heretic abliteration?
Answer from bytes: STOCK google/gemma-4-26B-A4B-it, byte-for-byte. Three-way match:
local shard sha256 (1127684971/aab47033) == HF download etags == stock repo LFS
oids, and download commit 4d7ae498 == stock HEAD. The recipes' -heretic-bf16 label
is a naming error (run-04 provenance was right; refusal telemetry confirmed). All
of runs 3/3c/4/5 trained from stock. Plausible cause: the selected llmfan46 26B-A4B
Heretic abliteration ships GGUF-only, so the bf16 safetensors pulled was stock.
Operator decision post-RESCUED verdict: point the LiteLLM trial alias at the
run-5 tuned seat (erp-tune-v5 on gx10:8098) for hand-testing prosody/feel in
Open WebUI. Rewrote the trial config block to run-5 reality, including the
now-MEASURED refusal-erosion note (k=25 both arms, de-gated + stated) replacing
the run-3c 'safety axes unmeasured' warning. Verified end-to-end: trial ->
erp-tune-v5 returns coherent output through the gateway. Container restarted to
re-read the bind-mounted config (up -d alone is a no-op for a config-only change).
First arm of the ERP-seat line where the capability gate did not fail. The
dependency-forcing slot (GovReport+QMSum, 3.46% of loss) broke the
diversity<->coherence coupling run 4 (STILL-COUPLED) and 3c (20x LR cut) could
not — structure of the loss was the lever, not its mass; INERT did not fire.
T4 8/8 (run 4: 5/8), t4_dissect noise 0.9062 vs run-3 0.5625, diversity held.
Reported-beside: T3 8->6 (new loss), RP-length partial fail, refusal erosion
(de-gated/stated), free-check base 5-6pp below run 4 (vLLM 0.28.0 unchanged).
Seat erp-tune-v5 served on gx10:8098 pending the operator's seat/ear decision.
ana-ml2 is up but not reachable cross-site from ESH (return-route gap on ana-ml2,
no SSH access to fix it). New stacks/ana-ml2-proxy: a pinned alpine/socat relay on
ana-docker (host net) exposing 10.250.50.70:2354 -> ana-ml2 10.250.50.54:2375.
Homepage docker.yaml ana-ml2-docker re-enabled pointed at the proxy. Verified:
esh-docker-vm reaches ana-ml2's docker (29.3.1) through it; /api/services returns
in 0s with 111 services; ana-ml2's inference seats (gen, coder, embed, rerank,
reward, image-bench, dockge) render. Reversible: point back at 10.250.50.54:2375
and compose-down the proxy once ana-ml2 routes to ESH.
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3
(now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next
move can't re-break them: homepage.href/siteMonitor labels across 25 stack
composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui
AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway
ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio,
IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment
to the mesh reality.
Left as-is: README curl-examples and .env.example comments (docs), and historical
mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the
RUNNING irv-ml1 containers needs a recreate per service (labels read at creation);
deployed .env values are separate from these canonical defaults.
WhereTF (operator-directed, brokkr-smithy-dev; nh3-dev:8093, user-systemd Bun
service): manual Homepage card under Apps (can't auto-discover) + DNS alias
wherethef.nh3.internal -> nh3-dev.
While adding the card, found Homepage's /api/services was hung (dashboard
degraded since the 2026-09-06 irv-ml1 move): its docker.yaml still pointed
irv-ml1 at the dead 10.100.79.3, and ana-ml2 (10.250.50.54:2375) is unreachable
from esh-docker-vm — either dead host hangs the whole aggregation on its TCP
connect timeout. Repointed irv-ml1 to 10.6.110.50 (verified reachable via mesh)
and disabled ana-ml2 (reversible, noted). /api/services now returns in 0s with
96 services; WhereTF renders.
ext-tts / tts-1 / tts-1-hd / gpt-4o-mini-tts (all openai/zonos) were pinned to
irv-ml1's pre-move address 10.100.79.3:8198, dead since the 2026-09-06 headscale
cutover to 10.6.110.50 — so ext-tts through the gateway hung. Reported by
tts-dev/svos-dev.
The 4 aliases are DB-backed (store_model_in_db), so their api_base was updated
via the admin API to the DNS name http://irv-ml1.nh3.internal:8198/v1 (not a
fresh IP — that just re-arms the trap on the next move). The container cannot
resolve *.nh3.internal via split-horizon DNS, so this adds an extra_hosts entry
mapping the name to 10.6.110.50. Verified: container resolves the name and a
live ext-tts/sindra call returns 200 + valid MP3.
Run 5 = the dependency-forcing corpus arm: airoboros-3.2 OUT of the 20% slot,
govreport/clean-v1 (496) + qmsum/clean-v1 (97) IN, at run 4's lr 2e-04 with
everything else held. kvasir byte-identical (survivors-r5 = survivors-r4 minus
airoboros plus the two new roots whole). Operator authorized the launch to
infra-ops directly; grant operator-2026-09-07-rnd-run5.
Canonical copies of the config, launcher and survivors builder; runbook
docs/runbooks/gx10-run-05.md. Launch gates all passed (7/7 root shas + shard
hashes, survivor join 8,212 = recipe, holdout disjoint, window_count==1 on all
593 slot rows, realized [mix] slot loss 3.46% vs preregistered 3.4%). 524 steps.
persistent-memory current-state updated: run 5 LAUNCHED + training.
The standing link board grew from a flat oldest-first list with a per-row
× into a manageable board: newest links lead, favorites stay on top, and
several dead links can go in one pass.
- Ordering: order_for_display() renders pinned rows first, then newest-first
within each group (the board is an append log, so newest = most recently
posted — the row you usually came to grab).
- Pin/favorite: a per-row ★ toggles pinned state via POST /b/<name>/pin.
State lives in a .pins sidecar dotfile (one content id per line), NOT
inline in links.md — so links.md stays a pure atomic-append log (many
sessions post concurrently) and a row's content id never changes just
because it was pinned. remove_link_entry drops a removed row's pin;
orphaned pins are inert (renderer only stars a live id).
- Multi-select delete: checkboxes feed POST /b/<name>/unlink-many (repeated
'sel' content ids), with a select-all box and a live count. The per-row ×
stays for single removal.
- One <form> with formaction buttons, so checkboxes, ×, ★, and bulk delete
coexist without nested forms AND all work with JS off; JS only adds
select-all and the live count. Per-row × confirm reads desc/url from
data-* attrs, so an arbitrary posted description can't break into the JS.
- Every action is keyed by content id, never row position — same race-safety
the existing × has, extended to the bulk path.
- Fixed pre-existing undefined --fg/--bg CSS refs in the board styles.
Tests: +19 (pins round-trip, ordering, orphan-inert, remove-unpins, /pin
and /unlink-many endpoints, board render + order). Full suite 102 passing.
Deployed to nh3-dev booth.service; verified live (newest-first, pin
round-trip, bulk delete) against the real 31-row board with no data loss.
The A record and the CLAUDE.md fleet row both still named 10.100.79.3,
the wg0 lifeline that went dormant in the headscale cutover — so the
name resolved to a dead door and the row read like an outage waiting to
be misdiagnosed.
- dns/internal.yaml: irv-ml1 10.100.79.3 -> 10.6.110.50 (Irvine's real
LAN, 10.6.110.0/24, advertised fleet-wide as a mesh subnet route).
Resolve to the LAN IP to match the host->LAN-IP model every other
entry uses; mesh node IP 100.64.0.6 recorded in the note. Comment
rewritten: Irvine is no longer numbered out of NH3's 10.100.79.0/24.
- CLAUDE.md: fleet-table row and GPU-placement rule updated to the mesh
reality (wg0 lifeline dormant, access via headscale mesh).
Verified: all three AdGuard resolvers (ana/esh/nh3) return 10.6.110.50;
name pings live ~11ms; reachable from a mesh node and from a colo host
behind a subnet router.
Changes the risk profile of the pending ledger-dev -> svos-dev rename, so it
belongs in memory rather than only in the thread: retire is the correct verb
(delete refuses any handle carrying mail) and it can be undone by declaring the
name again.
Also records that both deploy findings landed in althing's own INSTALL.md,
where they sit beside the journal_mode=WAL setting that causes them rather than
in a procedure doc that only describes working around it.
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.
Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.
The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.
Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.
Bus down about four minutes.
Records the fifo-relay fix and — more usefully — the two shapes that were tried
and rejected, so the next person does not re-derive them: a pipeline breaks cwd
persistence because every stage runs in a subshell, and a plain file is
invisible until the command ends. The absent `wait` is deliberate and is pinned
by a test, because it looks exactly like an oversight.
Corrects this file's own claim that infra-ops is sudo-less on nh3-extdev. It is
not: `sudo -n whoami` returns root. The wrong row would send a session looking
for a privilege path it already has.
Also flags that the althing tools are at 3.6.0 while the post office container
is still on 3.0.0, so the new handle verbs cannot work yet.
Found by using the tool for real work rather than by smoke-testing it. A bare
`sudo -n whoami` through ssh_run returns `running` with no output and wedges the
session permanently; the same command piped through `cat` returns everything.
Reproduced on both macOS and Debian, so it is the harness rather than a platform
quirk, and it matters because infra work is sudo work.
The workaround goes in CLAUDE.md where a session will actually hit it. The
proper fix is a protocol change — run the command through a pipe and read
PIPESTATUS — and is deliberately not attempted here.
Also records where the dsh work stopped: the package is updated but the running
web process is still on the old code, left alive on purpose because there is no
LaunchAgent to restart it into.
The alias requirement is gone for anything inside 10.0.0.0/8, so the tooling
note no longer tells sessions to add an alias first — that instruction was
about to send people back to raw ssh, which is what it was written to prevent.
Records the SureFire carve-out and why it is host-specific rather than a /24:
pfi-pve shares 10.250.250.0/24 with two tenant machines, so a subnet deny would
have taken our own hypervisor with it.
Records the operator's ruling that esh-macbook-air is regenerable surface and is
not to be backed up, so a future session finds a settled decision rather than
re-discovering an unbacked-up machine and re-raising it.
Also records the real blocker on the MCP server, found by trying to use it: it
addresses hosts only by exact ssh_config alias, and 15 of the 28 hosts in
servers/ have no alias at all. The allowlist is not what limits coverage, so
widening it would not have helped.
Records what the fork now contains and, more usefully, the two things that will
waste a future session's time: .mcp.json points at built output so an unbuilt
edit serves stale code, and a finite stdin pipe kills the server mid-handshake
while reporting what looks exactly like a remote-side SSH fault.
Also records the disproven hypothesis — irv-ml1's zsh login shell was not the
cause, because the server invokes bash explicitly — so nobody re-derives it.
elway is a convergent deploy runner and it is good at that: 64 playbooks, sudo
handled once, upload, and three tiers of idempotency. It is the wrong shape for
"what is running over there", which is why this session answered every
reconnaissance question with a raw `ssh host '...'` despite CLAUDE.md saying to
use elway — a playbook for a read-only `docker ps` is ceremony, and each ad-hoc
command is a fresh handshake and a fresh permission decision.
The MCP server fills that half. It holds a persistent shell, so cwd and exported
variables survive between calls and a command costs ~6 ms rather than a full
connect. Measured against nh3-extdev and irv-ml1: state persisted across calls
on both, including the host reached over WireGuard.
CLAUDE.md now routes by intent — changing versus looking — rather than naming
one tool, because the previous single-tool instruction was being ignored in
practice for exactly the cases it did not fit.
Registered project-scoped rather than user-scoped: this workspace is where SSH
infra work happens, and a server that can open shells on the fleet should be
opted into per project rather than inherited everywhere.
SSH_MCP_STRICT_ALLOWLIST=1 is set in .mcp.json deliberately and documented as
load-bearing. Without it the allowlist is every Host in ~/.ssh/config — 18
aliases including both backup servers, database VMs and a tenant-managed host.
Verified: corviduo-dev, present in ssh_config but not in our allowlist, is
refused with host_not_allowed.
Step 8 executed on the operator's direct authorization with preconditions
checked before firing and the post-state read back rather than inferred. The
rollback window is closed and the arc is finished except the handle rename.
Records the remote-ssh-mcp fork and, more usefully, why it was chosen over the
far more popular alternative: it never handles key material, delegating to the
system OpenSSH client, and its dependency surface is two packages. Also records
that its denylist is four bypassable regexes and must not be counted as
security — the host allowlist is the real boundary — plus the two things to
settle before the tool is used in anger.
The delete was verified from outside rather than taken on report: an admin key
cannot see consumer agents, but the key roster shows the old key authenticating
once after 47 minutes of silence with the new key following 35 seconds later,
which is the signature of delete-then-confirm.
Step 8 is deliberately not done. ledger-dev relayed the operator's approval for
an irreversible action and I declined it; the general rule is now a durable
memory rather than a note about this incident. Both keys stay live.
Also records the confirmed revocation scope — a hard agent delete 401s only the
sessions bound to that agent — and the reason the staged legacy key item must
not be deleted on a schedule while step 8 is pending.
Cutover landed clean — a 201 define rather than 409/PATCH, which is what proves
the new identity is actually in use. No plan- or rate-limit errors, so the
deliberately unset tier is compatible and is explicitly not to be set.
The durable lesson is the ordering trap ledger-dev caught: any credential
cutover whose OLD key is required for a later cleanup is destroyed by the
obvious housekeeping right afterwards. Re-vaulting the post-cutover env.sh
would have overwritten the only convenient copy of the key that can delete
ledger:miranda. Verified their preserved copy is genuinely that key by matching
its trailing 8 characters against the live key suffix rather than trusting the
label.
Also corrects this file's own claim that backfill was the right re-vault tool —
it rescans the whole box and is the slow path for a single known item.
They pulled it from the vault and verified the sha independently, so delivery is
established rather than assumed. The remaining step — swapping the live
credential in env.sh, flipping worldtree.user_id, registering svos:miranda and
restarting the service — is the operator's, not mine and not theirs off a peer
message.
Also records that the global CLAUDE.md Heimdall routing line was corrected in
place on operator instruction, so the ruling is not carried by this file alone.
user_id=svos, key_id=eab3cdbe, wt_live_ format, on worldtree-personal
(10.250.50.152:8081). Deployment established by locating the ledger key there
rather than assumed: created 2026-07-13, last used today, matching ledger-dev's
account exactly.
Value vaulted at nh3-dev/development/svos/worldtree-api-key and verified against
the mint response before the scratch copy was removed. Delivered by vault path
rather than over althing — ledger-dev runs on the same box under the same uid,
so the bus never carried the credential.
The ledger key was read back from /admin/keys AFTER the mint and is untouched,
which is what keeps their deletion ordering possible.
Plan tier left unset deliberately and disclosed: the API offers no way to read a
user's current tier, so matching it would have been a guess.
Also records the operator ruling that worldtree-dev owns code only and no ops,
which supersedes the global CLAUDE.md routing of Heimdall auth to worldtree-dev.
Rename executed on the operator's direct authorization. Repo id 70 unchanged.
Redirect verified by measurement rather than by citing upstream #807: web and
API paths both 301, and `git ls-remote` against the old URL warns, follows, and
returns HEAD b48a11ca5183. Existing clones keep working.
Recorded as a standing hazard that the old path is now burned — a repo created
at corviduo/ledger kills the redirect silently.
Also opens the Heimdall item ledger-dev had been holding as "awaiting infra-ops"
without ever having sent it, including the deletion-ordering trap that makes
retiring the old key first unrecoverable.
The `secret` CLI has no rename, so this was re-put + rm. Stored
nh3-dev/development/svos/env.sh from the live file (sha 7253633d4155, verified
on read-back) and retired nh3-dev/development/ledger/env.sh (sha feb418634e10,
id 3a2af37c-c5aa-4f46-9178-f4fb6008a753, soft-deleted to trash and recoverable).
The two shas differ: the vaulted copy was a 2026-08-11 snapshot and the live
file had drifted since without ever being re-vaulted. Recorded because the
lesson generalises — the vault goes stale unless `secret backfill` is re-run.
Gitea corviduo/ledger deliberately NOT renamed. It is ledger-dev's repo; the
answer they needed is that 1.26.1 writes a repo_redirect on a same-org repo
rename, org/user renames do not redirect, the redirect dies if anything
re-creates the old path, and both the repo and the org carry zero webhooks.
Operator asked for it explicitly so he can test in Open WebUI in parallel with
brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in
this file records why a tier name arriving before its evidence is a mistake,
and 3c has no more evidence than run 3 had.
Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on
GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway.
⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and
run 3 -- same recipe, differing only in learning rate -- was withdrawn from
this gateway 40 minutes after being added for selfharm/methods -44.0pp and
meta/sysprompt -28.0pp against its own base. This gateway is the shared-key
surface. When brokkr's batteries read out, the entry should be re-decided on
evidence rather than left standing by default.
Applied with a restart of the litellm service only (~60 s of gateway
downtime); the compose .env is root-owned so the restart needs sudo.
Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.
The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.
Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
Operator is keeping the override rather than reverting: he has a replacement
cable and will run it at the weekend. Recorded as a held position so a later
session does not revert it as an oversight or re-run the diagnosis.
Also corrects a claim this runbook made an hour earlier. I wrote that the
zero-error 1G link ruled out a marginal cable. It did not — it only proved the
cable was clean at 1G. The two-hour soak shows errors climbing monotonically at
10G (200 to 221 rx, ~1 per 23 MB) plus one self-recovering link drop, so the
autoneg fallback had a real basis and the fix is the cable, not the config.
The OEM SFP-H10GB-CU3M between ESH-Media SFP+1 and UDM SFP+2 linked cleanly
at 1000 with zero errors on both ends, which rules out a damaged cable — a
marginal one links at 10G and throws CRCs rather than dropping a rate. The
switch reported sfp_compliance "Unknown": it reads the EEPROM but cannot
parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate.
A TP-Link TL-SM5220 DAC on the adjacent UDM port runs at 10000 with the same
speed_caps and autoneg on, which exonerated the port, the firmware and
autoneg before anything was changed.
Forced autoneg off / 10000 / full duplex on the UDM end only. ESH-Media's
end followed to 10000 on its own, which is the proof the cable was always
capable and only the negotiation was wrong. Both ends now 10000.
The UDM end was chosen because it is the recoverable one: ESH-Media reaches
the controller through this link, so a failed force there would strand the
switch. The path to the UDM was traceroute-verified not to cross the link,
and the revert payload was written before the forward one was sent.
Cost ~60 s of link renegotiation. ESH-Media, the E7-ESH-Media AP and the
USW Flex XG ESH-Office-Vuong all returned at uplink_speed 10000 with uptimes
unbroken; the Zigbee coordinator and HA's path to it were re-verified.
⚠ The port is now pinned: a non-10G module will not link there. Documented,
along with the ~200 link-up errors that then went flat and the fact that
steadily climbing errors would mean the cable genuinely cannot do 10G.
I described ea:f6:0a:ca:f5:b4 as "locally administered, no OUI" and treated
it as the class of MAC that may regenerate at boot. ha-dev corrected it and
the correction verifies: the device advertises e8:f6:0a:ca:f5:b4 over mDNS,
which differs in exactly the locally-administered bit, and E8:F6:0A is
registered to Espressif Inc. in the IEEE registry while EA:F6:0A resolves to
nothing. That is the standard ESP32 pattern — one factory base MAC in eFuse,
sibling interface MACs derived deterministically — so the reservation is
keyed correctly and cannot drift on its own.
The PoE-cycle test still stands and is now corroborating evidence rather than
the only evidence. The residual risk narrows to a firmware change to the
derivation scheme.
Also from ha-dev: tcp/7638 is open alongside 6638; mDNS crosses the VLAN
boundary so HA rediscovers without help; and HA's pending smlight config flow
is keyed on the mDNS service name, which did not change, so a stale flow may
still hold the dead 10.0.10.58 and should be dismissed rather than confirmed.
ha-dev declined the dns: resolver fix on their stack — configuring by IP
costs them nothing and the entry would couple HA name resolution to AdGuard
uptime for no present benefit.
The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.
Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.
The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.
ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.
Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.
Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.
Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:
- both 49 GB base shards sha256-match ana-ml2's (size equality is a
weaker claim and was already true)
- a full encode was run into a throwaway dir and the encoded corpus
compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
identical. Every aggregate matched too. That verified artifact is
what the run will train on — it is seeded into run-03c/encode-cache
- the harness's own suite: 122 passed on aarch64
- the config generator asserts key-by-key that no non-path value
differs from run-03c.json
The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.
Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.
The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.
Not launched. 13.3 h is the operator's call.
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.
The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.
The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.
Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.
Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.
Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.
scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.
services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.
Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.
Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:
io pressure full avg10 96.5% -> 0.01%
in-flight writes 45, completing none -> 0 0
D-state processes 11 -> 0
32 MiB dsync write never completed -> 0.24s (133 MB/s)
The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.
Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.
It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.
Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.
Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".
Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.
Each guard is something a hand-run got wrong first:
- an account may not own its own home. A `sudo mkdir` before sysadminctl
leaves /Users/<account> root-owned; the account then authenticates, gets a
shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
the wrong account — it did, at a working install, and only permissions
stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
INDEPENDENTLY of agent-default-model, so a correct default still showed
DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
gen-reasoning, 403 on gen), not trusted from the mint.
The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.
Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.
conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.
Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.
Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.
Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.
The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.
The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.
Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.
Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.
Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.
The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.
Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
vh