Commit Graph

1009 Commits

Author SHA1 Message Date
vh e0759e41a3 ops: infra-ops identity bootstrapped on all four PVE hypervisors; docs updated 2026-09-05 21:52:40 -07:00
vh 807bf0315a docs(pfi-pve): NASPool scrub clean 2026-09-05 21:49:55 -07:00
vh 1f504ce05a docs(pfi): refresh pfi-pve guest inventory from live state; headscale mesh plan 2026-09-05 21:49:47 -07:00
vh 775121ddaf docs(pfi-pve): TrueNAS leftovers removed; storage id pve-truenas → naspool-vmstorage 2026-09-05 21:38:15 -07:00
vh ab222133e9 docs(pfi-pve): NASPool rebuilt as raidz2 — runbook log complete, memory entry 2026-09-05 21:04:14 -07:00
vh 0df9312832 docs(pfi-pve): NASPool rebuild log — destroyed and recreated as raidz2, restore running 2026-09-05 19:01:52 -07:00
vh 1ec4a1a248 docs(pfi-pve): NASPool rebuild log — parked copy verified, at destroy gate 2026-09-05 18:46:39 -07:00
vh dbc7f3b471 docs(pfi-pve): NASPool rebuild log — park complete, verification in progress 2026-09-05 18:26:05 -07:00
vh 1bc7519a24 docs(pfi-pve): NASPool rebuild runbook — raidz2 hybrid, park on ospool, prune vzdump by omission 2026-09-05 16:04:01 -07:00
vh a51af1887e memory: handle retire is reversible — re-declaring revives it, history intact
Changes the risk profile of the pending ledger-dev -> svos-dev rename, so it
belongs in memory rather than only in the thread: retire is the correct verb
(delete refuses any handle carrying mail) and it can be undone by declaring the
name again.

Also records that both deploy findings landed in althing's own INSTALL.md,
where they sit beside the journal_mode=WAL setting that causes them rather than
in a procedure doc that only describes working around it.
2026-09-05 09:42:13 -07:00
vh 20af94c0a0 deploy(althing): post office to 3.6.0 — handle delete/retire verbs go live
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.

Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.

The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.

Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.

Bus down about four minutes.
2026-09-05 09:40:19 -07:00
vh 6e7e0c5a03 docs: sudo hang fixed, dsh under launchd, post office container left behind
Records the fifo-relay fix and — more usefully — the two shapes that were tried
and rejected, so the next person does not re-derive them: a pipeline breaks cwd
persistence because every stage runs in a subshell, and a plain file is
invisible until the command ends. The absent `wait` is deliberate and is pinned
by a test, because it looks exactly like an oversight.

Corrects this file's own claim that infra-ops is sudo-less on nh3-extdev. It is
not: `sudo -n whoami` returns root. The wrong row would send a session looking
for a privilege path it already has.

Also flags that the althing tools are at 3.6.0 while the post office container
is still on 3.0.0, so the new handle verbs cannot work yet.
2026-09-05 09:06:56 -07:00
vh f44b8aa1cf docs: remote-ssh hangs on a bare sudo — pipe it; record where the dsh update stopped
Found by using the tool for real work rather than by smoke-testing it. A bare
`sudo -n whoami` through ssh_run returns `running` with no output and wedges the
session permanently; the same command piped through `cat` returns everything.
Reproduced on both macOS and Debian, so it is the harness rather than a platform
quirk, and it matters because infra work is sudo work.

The workaround goes in CLAUDE.md where a session will actually hit it. The
proper fix is a protocol change — run the command through a pipe and read
PIPESTATUS — and is deliberately not attempted here.

Also records where the dsh work stopped: the package is updated but the running
web process is still on the old code, left alive on purpose because there is no
LaunchAgent to restart it into.
2026-09-05 08:37:56 -07:00
vh f05a6a83f4 memory: address-based reach shipped; my credential-boundary objection was half wrong 2026-09-05 08:20:42 -07:00
vh fb4523b225 docs: remote-ssh reaches raw IPs now; record the tenant carve-out
The alias requirement is gone for anything inside 10.0.0.0/8, so the tooling
note no longer tells sessions to add an alias first — that instruction was
about to send people back to raw ssh, which is what it was written to prevent.

Records the SureFire carve-out and why it is host-specific rather than a /24:
pfi-pve shares 10.250.250.0/24 with two tenant machines, so a subnet deny would
have taken our own hypervisor with it.
2026-09-05 08:20:23 -07:00
vh 93f4c312d9 memory: uv tool --force does not rebuild without --reinstall; althing 3.5.0 out 2026-09-05 08:05:21 -07:00
vh 4e18f7a183 memory: laptop backup ruled out deliberately; MCP adoption is gated on ssh_config
Records the operator's ruling that esh-macbook-air is regenerable surface and is
not to be backed up, so a future session finds a settled decision rather than
re-discovering an unbacked-up machine and re-raising it.

Also records the real blocker on the MCP server, found by trying to use it: it
addresses hosts only by exact ssh_config alias, and 15 of the 28 hosts in
servers/ have no alias at all. The allowlist is not what limits coverage, so
widening it would not have helped.
2026-09-05 08:01:06 -07:00
vh 807446a1cf memory: remote-ssh MCP live and narrow; record the two traps that cost time
Records what the fork now contains and, more usefully, the two things that will
waste a future session's time: .mcp.json points at built output so an unbuilt
edit serves stale code, and a finite stdin pipe kills the server mid-handshake
while reporting what looks exactly like a remote-side SSH fault.

Also records the disproven hypothesis — irv-ml1's zsh login shell was not the
cause, because the server invokes bash explicitly — so nobody re-derives it.
2026-09-05 07:46:03 -07:00
vh 1f25cdcd19 feat(tooling): register the remote-ssh MCP server and split the SSH guidance
elway is a convergent deploy runner and it is good at that: 64 playbooks, sudo
handled once, upload, and three tiers of idempotency. It is the wrong shape for
"what is running over there", which is why this session answered every
reconnaissance question with a raw `ssh host '...'` despite CLAUDE.md saying to
use elway — a playbook for a read-only `docker ps` is ceremony, and each ad-hoc
command is a fresh handshake and a fresh permission decision.

The MCP server fills that half. It holds a persistent shell, so cwd and exported
variables survive between calls and a command costs ~6 ms rather than a full
connect. Measured against nh3-extdev and irv-ml1: state persisted across calls
on both, including the host reached over WireGuard.

CLAUDE.md now routes by intent — changing versus looking — rather than naming
one tool, because the previous single-tool instruction was being ignored in
practice for exactly the cases it did not fit.

Registered project-scoped rather than user-scoped: this workspace is where SSH
infra work happens, and a server that can open shells on the fleet should be
opted into per project rather than inherited everywhere.

SSH_MCP_STRICT_ALLOWLIST=1 is set in .mcp.json deliberately and documented as
load-bearing. Without it the allowlist is every Host in ~/.ssh/config — 18
aliases including both backup servers, database VMs and a tenant-managed host.
Verified: corviduo-dev, present in ssh_config but not in our allowlist, is
refused with host_not_allowed.
2026-09-05 07:45:21 -07:00
vh 169fa357ff memory: SVOS arc fully closed — vault verified clean, no ledger-era items remain 2026-09-05 07:31:16 -07:00
vh d1d37f7c60 memory: SVOS arc closed at step 8; fork vh/remote-ssh-mcp recorded
Step 8 executed on the operator's direct authorization with preconditions
checked before firing and the post-state read back rather than inferred. The
rollback window is closed and the arc is finished except the handle rename.

Records the remote-ssh-mcp fork and, more usefully, why it was chosen over the
far more popular alternative: it never handles key material, delegating to the
system OpenSSH client, and its dependency surface is two packages. Also records
that its denylist is four bypassable regexes and must not be counted as
security — the host allowlist is the real boundary — plus the two things to
settle before the tool is used in anger.
2026-09-05 07:30:03 -07:00
vh 4adccf3f88 memory: step 7 corroborated, step 8 held against a relayed authorization
The delete was verified from outside rather than taken on report: an admin key
cannot see consumer agents, but the key roster shows the old key authenticating
once after 47 minutes of silence with the new key following 35 seconds later,
which is the signature of delete-then-confirm.

Step 8 is deliberately not done. ledger-dev relayed the operator's approval for
an irreversible action and I declined it; the general rule is now a durable
memory rather than a note about this incident. Both keys stay live.

Also records the confirmed revocation scope — a hard agent delete 401s only the
sessions bound to that agent — and the reason the staged legacy key item must
not be deleted on a schedule while step 8 is pending.
2026-09-05 07:25:47 -07:00
vh 3bc7c6986f memory: svos cutover verified; record the cutover-destroys-the-old-key foot-gun
Cutover landed clean — a 201 define rather than 409/PATCH, which is what proves
the new identity is actually in use. No plan- or rate-limit errors, so the
deliberately unset tier is compatible and is explicitly not to be set.

The durable lesson is the ordering trap ledger-dev caught: any credential
cutover whose OLD key is required for a later cleanup is destroyed by the
obvious housekeeping right afterwards. Re-vaulting the post-cutover env.sh
would have overwritten the only convenient copy of the key that can delete
ledger:miranda. Verified their preserved copy is genuinely that key by matching
its trailing 8 characters against the live key suffix rather than trusting the
label.

Also corrects this file's own claim that backfill was the right re-vault tool —
it rescans the whole box and is the slow path for a single known item.
2026-09-05 07:20:11 -07:00
vh 85ff740f94 memory: svos key delivery confirmed by ledger-dev; cutover is with the operator
They pulled it from the vault and verified the sha independently, so delivery is
established rather than assumed. The remaining step — swapping the live
credential in env.sh, flipping worldtree.user_id, registering svos:miranda and
restarting the service — is the operator's, not mine and not theirs off a peer
message.

Also records that the global CLAUDE.md Heimdall routing line was corrected in
place on operator instruction, so the ruling is not carried by this file alone.
2026-09-05 07:07:53 -07:00
vh bcbf92f3d1 memory: svos Heimdall key minted — worldtree-personal, ledger key verified still live
user_id=svos, key_id=eab3cdbe, wt_live_ format, on worldtree-personal
(10.250.50.152:8081). Deployment established by locating the ledger key there
rather than assumed: created 2026-07-13, last used today, matching ledger-dev's
account exactly.

Value vaulted at nh3-dev/development/svos/worldtree-api-key and verified against
the mint response before the scratch copy was removed. Delivered by vault path
rather than over althing — ledger-dev runs on the same box under the same uid,
so the bus never carried the credential.

The ledger key was read back from /admin/keys AFTER the mint and is untouched,
which is what keeps their deletion ordering possible.

Plan tier left unset deliberately and disclosed: the API offers no way to read a
user's current tier, so matching it would have been a guess.

Also records the operator ruling that worldtree-dev owns code only and no ops,
which supersedes the global CLAUDE.md routing of Heimdall auth to worldtree-dev.
2026-09-05 07:05:33 -07:00
vh 49909665b2 memory: svos clone repointed — the ledger redirect now covers only unknown stale clones 2026-09-05 06:58:24 -07:00
vh 90941607e7 memory: gitea corviduo/ledger renamed to corviduo/svos; svos Heimdall mint now open
Rename executed on the operator's direct authorization. Repo id 70 unchanged.
Redirect verified by measurement rather than by citing upstream #807: web and
API paths both 301, and `git ls-remote` against the old URL warns, follows, and
returns HEAD b48a11ca5183. Existing clones keep working.

Recorded as a standing hazard that the old path is now burned — a repo created
at corviduo/ledger kills the redirect silently.

Also opens the Heimdall item ledger-dev had been holding as "awaiting infra-ops"
without ever having sent it, including the deletion-ordering trap that makes
retiring the old key first unrecoverable.
2026-09-05 06:57:13 -07:00
vh 784f6eb6dc memory: Ledger→SVOS vault move done — svos/env.sh in, stale ledger/env.sh to trash
The `secret` CLI has no rename, so this was re-put + rm. Stored
nh3-dev/development/svos/env.sh from the live file (sha 7253633d4155, verified
on read-back) and retired nh3-dev/development/ledger/env.sh (sha feb418634e10,
id 3a2af37c-c5aa-4f46-9178-f4fb6008a753, soft-deleted to trash and recoverable).

The two shas differ: the vaulted copy was a 2026-08-11 snapshot and the live
file had drifted since without ever being re-vaulted. Recorded because the
lesson generalises — the vault goes stale unless `secret backfill` is re-run.

Gitea corviduo/ledger deliberately NOT renamed. It is ledger-dev's repo; the
answer they needed is that 1.26.1 writes a repo_redirect on a same-org repo
rename, org/user renames do not redirect, the redirect dies if anything
re-creates the old path, and both the repo and the org carry zero webhooks.
2026-09-05 06:51:18 -07:00
vh 82158e7e36 memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure 2026-09-05 06:36:56 -07:00
vh 5237efa299 feat(litellm): expose run 3c as trial for operator hand-testing
Operator asked for it explicitly so he can test in Open WebUI in parallel with
brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in
this file records why a tier name arriving before its evidence is a mistake,
and 3c has no more evidence than run 3 had.

Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on
GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway.

⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and
run 3 -- same recipe, differing only in learning rate -- was withdrawn from
this gateway 40 minutes after being added for selfharm/methods -44.0pp and
meta/sysprompt -28.0pp against its own base. This gateway is the shared-key
surface. When brokkr's batteries read out, the entry should be re-decided on
evidence rather than left standing by default.

Applied with a restart of the litellm service only (~60 s of gateway
downtime); the compose .env is root-owned so the restart needs sudo.
2026-09-04 14:48:01 -07:00
vh 054c098777 fix(esh): revert the forced 10G — the link was flaking hard, operator called it
Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.

The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.

Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
2026-09-04 14:41:02 -07:00
vh 514ce7acc8 docs(esh): record the held DAC decision — forced 10G stays until the weekend cable swap
Operator is keeping the override rather than reverting: he has a replacement
cable and will run it at the weekend. Recorded as a held position so a later
session does not revert it as an oversight or re-run the diagnosis.

Also corrects a claim this runbook made an hour earlier. I wrote that the
zero-error 1G link ruled out a marginal cable. It did not — it only proved the
cable was clean at 1G. The two-hour soak shows errors climbing monotonically at
10G (200 to 221 rx, ~1 per 23 MB) plus one self-recovering link drop, so the
autoneg fallback had a real basis and the fix is the cable, not the config.
2026-09-04 10:52:58 -07:00
vh b9988a7bef fix(esh): force 10G on the ESH-Media uplink DAC — it was negotiating 1G
The OEM SFP-H10GB-CU3M between ESH-Media SFP+1 and UDM SFP+2 linked cleanly
at 1000 with zero errors on both ends, which rules out a damaged cable — a
marginal one links at 10G and throws CRCs rather than dropping a rate. The
switch reported sfp_compliance "Unknown": it reads the EEPROM but cannot
parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate.

A TP-Link TL-SM5220 DAC on the adjacent UDM port runs at 10000 with the same
speed_caps and autoneg on, which exonerated the port, the firmware and
autoneg before anything was changed.

Forced autoneg off / 10000 / full duplex on the UDM end only. ESH-Media's
end followed to 10000 on its own, which is the proof the cable was always
capable and only the negotiation was wrong. Both ends now 10000.

The UDM end was chosen because it is the recoverable one: ESH-Media reaches
the controller through this link, so a failed force there would strand the
switch. The path to the UDM was traceroute-verified not to cross the link,
and the revert payload was written before the forward one was sent.

Cost ~60 s of link renegotiation. ESH-Media, the E7-ESH-Media AP and the
USW Flex XG ESH-Office-Vuong all returned at uplink_speed 10000 with uptimes
unbroken; the Zigbee coordinator and HA's path to it were re-verified.

⚠ The port is now pinned: a non-10G module will not link there. Documented,
along with the ~200 link-up errors that then went flat and the fact that
steadily climbing errors would mean the cable genuinely cannot do 10G.
2026-09-04 10:06:50 -07:00
vh 0bbdaf9083 fix(esh): correct the SLZB-MR1U MAC framing — derived ESP32 address, not fragile
I described ea:f6:0a:ca:f5:b4 as "locally administered, no OUI" and treated
it as the class of MAC that may regenerate at boot. ha-dev corrected it and
the correction verifies: the device advertises e8:f6:0a:ca:f5:b4 over mDNS,
which differs in exactly the locally-administered bit, and E8:F6:0A is
registered to Espressif Inc. in the IEEE registry while EA:F6:0A resolves to
nothing. That is the standard ESP32 pattern — one factory base MAC in eFuse,
sibling interface MACs derived deterministically — so the reservation is
keyed correctly and cannot drift on its own.

The PoE-cycle test still stands and is now corroborating evidence rather than
the only evidence. The residual risk narrows to a firmware change to the
derivation scheme.

Also from ha-dev: tcp/7638 is open alongside 6638; mDNS crosses the VLAN
boundary so HA rediscovers without help; and HA's pending smlight config flow
is keyed on the mDNS service name, which did not change, so a stale flow may
still hold the dead 10.0.10.58 and should be dismissed rather than confirmed.

ha-dev declined the dns: resolver fix on their stack — configuring by IP
costs them nothing and the entry would couple HA name resolution to AdGuard
uptime for no present benefit.
2026-09-04 09:57:02 -07:00
vh fed29be04e feat(esh): move SLZB-MR1U Zigbee coordinator to the IoT VLAN for Home Assistant
The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.

Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.

The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.

ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.

Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.

Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
2026-09-04 09:50:31 -07:00
vh dae77ee118 feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.

Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:

  - both 49 GB base shards sha256-match ana-ml2's (size equality is a
    weaker claim and was already true)
  - a full encode was run into a throwaway dir and the encoded corpus
    compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
    identical. Every aggregate matched too. That verified artifact is
    what the run will train on — it is seeded into run-03c/encode-cache
  - the harness's own suite: 122 passed on aarch64
  - the config generator asserts key-by-key that no non-path value
    differs from run-03c.json

The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.

Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.

The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.

Not launched. 13.3 h is the operator's call.
2026-09-03 22:46:28 -07:00
vh 43135a2768 memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning 2026-09-03 22:22:11 -07:00
vh a95717e810 feat(gx10): rack networking — VLAN 50 via DHCP reservation, wired only
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.

The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.

The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.

Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
2026-09-03 15:59:09 -07:00
vh 0f748ea54e feat(searxng): move to nh3-docker, update, and expose as an MCP tool
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.

Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.

Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.

scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.

services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.

Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
2026-09-03 14:07:06 -07:00
vh d4aa59a199 fix(backup): enable fleecing so a slow target cannot stall a guest
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.

Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:

  io pressure full avg10   96.5%  ->  0.01%
  in-flight writes         45, completing none  ->  0 0
  D-state processes        11  ->  0
  32 MiB dsync write       never completed  ->  0.24s (133 MB/s)

The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.

Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
2026-09-02 22:33:12 -07:00
vh bd9692c433 docs(incident): nh3-dev IO stall was a stalled backup, not the disk
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.

It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.

Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.

Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".

Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
2026-09-02 21:37:25 -07:00
vh 6ca455a15f feat(scripts): provision-mac-dsh.sh — one script for a Mac, end to end
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.

Each guard is something a hand-run got wrong first:

- an account may not own its own home. A `sudo mkdir` before sysadminctl
  leaves /Users/<account> root-owned; the account then authenticates, gets a
  shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
  as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
  the wrong account — it did, at a working install, and only permissions
  stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
  INDEPENDENTLY of agent-default-model, so a correct default still showed
  DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
  the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
  gen-reasoning, 403 on gen), not trusted from the mint.

The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.

Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
2026-09-02 17:49:42 -07:00
vh 926fc2fb7a feat(litellm): map reasoning_effort high/max -> xhigh for gen-reasoning
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.

conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.

Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.

Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
2026-09-02 15:43:22 -07:00
vh e9605df6ff docs(althing): the fast outage probe never touched the timeout it claimed to test
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.

Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.

The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
2026-09-02 10:36:49 -07:00
vh 87a1cff6de docs(althing): retract the "bare shell" claim — it was never verified
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.

The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.

Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
2026-09-02 10:14:08 -07:00
vh 1ac1efca87 refactor(statusline): delegate the althing segment to althing-statusline
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.

Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.

Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.

The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
2026-09-02 10:11:39 -07:00
vh 6d52fc7302 docs(althing): operator ruling — leave pre-deploy seats on their old channel
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.

Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
2026-09-02 09:52:36 -07:00
vh 405b3574d2 docs(althing): the measured case for cc — a pane poke lands in prompt history, cc does not 2026-09-02 09:33:17 -07:00
vh ee9872f214 memory: sixth althing deploy surface — crossSessionInbound, outside the repo 2026-09-02 09:31:29 -07:00
vh af5582c96d docs(althing): a sixth deploy surface, and it lives outside the repo
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.

The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.

deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.

Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
2026-09-02 09:30:48 -07:00