vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: vh/esh-pfi-infrastructure:0ad332bb4a9bd00600816e424f18b907eb3c39d4
Branches Tags
vh/esh-pfi-infrastructure:main vh/esh-pfi-infrastructure:graphify-tooling
vh/esh-pfi-infrastructure:booth-v0.1.8 vh/esh-pfi-infrastructure:booth-v0.1.7 vh/esh-pfi-infrastructure:booth-v0.1.6 vh/esh-pfi-infrastructure:booth-v0.1.5 vh/esh-pfi-infrastructure:booth-v0.1.4
...
compare: vh/esh-pfi-infrastructure:booth-v0.1.8
Branches Tags
vh/esh-pfi-infrastructure:main vh/esh-pfi-infrastructure:graphify-tooling
vh/esh-pfi-infrastructure:booth-v0.1.8 vh/esh-pfi-infrastructure:booth-v0.1.7 vh/esh-pfi-infrastructure:booth-v0.1.6 vh/esh-pfi-infrastructure:booth-v0.1.5 vh/esh-pfi-infrastructure:booth-v0.1.4
210 Commits
0ad332bb4a ... booth-v0.1.8
Author SHA1 Message Date
vh 76fdf45925 feat(booth): pin/favorite, multi-select delete, newest-first link board 
The standing link board grew from a flat oldest-first list with a per-row
× into a manageable board: newest links lead, favorites stay on top, and
several dead links can go in one pass.

- Ordering: order_for_display() renders pinned rows first, then newest-first
  within each group (the board is an append log, so newest = most recently
  posted — the row you usually came to grab).
- Pin/favorite: a per-row ★ toggles pinned state via POST /b/<name>/pin.
  State lives in a .pins sidecar dotfile (one content id per line), NOT
  inline in links.md — so links.md stays a pure atomic-append log (many
  sessions post concurrently) and a row's content id never changes just
  because it was pinned. remove_link_entry drops a removed row's pin;
  orphaned pins are inert (renderer only stars a live id).
- Multi-select delete: checkboxes feed POST /b/<name>/unlink-many (repeated
  'sel' content ids), with a select-all box and a live count. The per-row ×
  stays for single removal.
- One <form> with formaction buttons, so checkboxes, ×, ★, and bulk delete
  coexist without nested forms AND all work with JS off; JS only adds
  select-all and the live count. Per-row × confirm reads desc/url from
  data-* attrs, so an arbitrary posted description can't break into the JS.
- Every action is keyed by content id, never row position — same race-safety
  the existing × has, extended to the bulk path.
- Fixed pre-existing undefined --fg/--bg CSS refs in the board styles.

Tests: +19 (pins round-trip, ordering, orphan-inert, remove-unpins, /pin
and /unlink-many endpoints, board render + order). Full suite 102 passing.
Deployed to nh3-dev booth.service; verified live (newest-first, pin
round-trip, bulk delete) against the real 31-row board with no data loss.
 2026-09-06 02:29:22 -07:00
vh 0336e033b8 fix(dns): repoint irv-ml1.nh3.internal to live Irvine LAN IP over mesh 
The A record and the CLAUDE.md fleet row both still named 10.100.79.3,
the wg0 lifeline that went dormant in the headscale cutover — so the
name resolved to a dead door and the row read like an outage waiting to
be misdiagnosed.

- dns/internal.yaml: irv-ml1 10.100.79.3 -> 10.6.110.50 (Irvine's real
  LAN, 10.6.110.0/24, advertised fleet-wide as a mesh subnet route).
  Resolve to the LAN IP to match the host->LAN-IP model every other
  entry uses; mesh node IP 100.64.0.6 recorded in the note. Comment
  rewritten: Irvine is no longer numbered out of NH3's 10.100.79.0/24.
- CLAUDE.md: fleet-table row and GPU-placement rule updated to the mesh
  reality (wg0 lifeline dormant, access via headscale mesh).

Verified: all three AdGuard resolvers (ana/esh/nh3) return 10.6.110.50;
name pings live ~11ms; reachable from a mesh node and from a colo host
behind a subnet router.
 2026-09-06 01:14:25 -07:00
vh ed3810aea2 memory: snapshot — irv-ml1 remote cutover complete + subnet discovery; session follow-ups 2026-09-06 01:05:14 -07:00
vh 02cf9d6cfb feat(mesh): irv-ml1 wg0 non-reboot-restorable + Irvine subnet discovery (bidirectional, verified) 2026-09-06 01:02:50 -07:00
vh b2c819ed51 docs(nh3-dev): mark SOCKS egress proxy retired (headscale exit nodes) 2026-09-06 00:58:36 -07:00
vh be4e7d68c9 feat(mesh): irv-ml1 cutover complete (remote, 5-day-safe) — 3 recovery paths, wg0 dormant, dante retired 2026-09-06 00:58:21 -07:00
vh f8eb3606eb feat(mesh): fold irv-ml1 in additively (node 100.64.0.6), wg0 lifeline preserved; wg0 retirement deferred to on-site 2026-09-06 00:43:48 -07:00
vh 13aaccf21e memory: snapshot — headscale cutover complete + NASPool rebuild; session open follow-ups 2026-09-06 00:29:14 -07:00
vh 1284c18fda docs(mesh): throughput baseline — 761/464 Mb/s vs old 250 IPsec; shared-fiber caveat 2026-09-06 00:24:41 -07:00
vh 7ce398f069 feat(mesh): selective masquerade — exit nodes AND site-to-site source preservation both work 2026-09-06 00:16:18 -07:00
vh b846a7c870 feat(mesh): advertise exit nodes at all three sites (SNAT re-enabled); usage + tradeoff documented 2026-09-06 00:09:43 -07:00
vh 815c283e44 feat(mesh): cutover COMPLETE — all three site-pairs on headscale, Site Magic + IPsec dormant 2026-09-06 00:05:18 -07:00
vh df8440e4b8 memory: headscale cutover — IPsec dormant, mesh primary, Site Magic pending operator 2026-09-06 00:02:15 -07:00
vh d25c096b6a feat(mesh): cut colo<->NH3 and colo<->ESH onto headscale (IPsec dormant); Site Magic pre-staged for operator UI disable 2026-09-06 00:01:43 -07:00
vh e7f8d1f227 docs(mesh): ESH egress whitelisted in crowdsec (temporary, pending static IP) 2026-09-05 23:36:11 -07:00
vh e88ec01726 docs(mesh): CrowdSec CGNAT false-ban incident — mesh bypasses it 2026-09-05 23:28:36 -07:00
vh 7382fdaa7b docs(mesh): first roaming client (vhlk-mba26) verified 2026-09-05 22:58:23 -07:00
vh a8a6f24b56 docs(mesh): return paths on all three gateways, verified; enrolment notes 2026-09-05 22:49:54 -07:00
vh 3ef87165df feat(mesh): headscale control plane + three subnet routers provisioned; DNS entries; plan status 2026-09-05 22:32:23 -07:00
vh e0759e41a3 ops: infra-ops identity bootstrapped on all four PVE hypervisors; docs updated 2026-09-05 21:52:40 -07:00
vh 807bf0315a docs(pfi-pve): NASPool scrub clean 2026-09-05 21:49:55 -07:00
vh 1f504ce05a docs(pfi): refresh pfi-pve guest inventory from live state; headscale mesh plan 2026-09-05 21:49:47 -07:00
vh 775121ddaf docs(pfi-pve): TrueNAS leftovers removed; storage id pve-truenas → naspool-vmstorage 2026-09-05 21:38:15 -07:00
vh ab222133e9 docs(pfi-pve): NASPool rebuilt as raidz2 — runbook log complete, memory entry 2026-09-05 21:04:14 -07:00
vh 0df9312832 docs(pfi-pve): NASPool rebuild log — destroyed and recreated as raidz2, restore running 2026-09-05 19:01:52 -07:00
vh 1ec4a1a248 docs(pfi-pve): NASPool rebuild log — parked copy verified, at destroy gate 2026-09-05 18:46:39 -07:00
vh dbc7f3b471 docs(pfi-pve): NASPool rebuild log — park complete, verification in progress 2026-09-05 18:26:05 -07:00
vh 1bc7519a24 docs(pfi-pve): NASPool rebuild runbook — raidz2 hybrid, park on ospool, prune vzdump by omission 2026-09-05 16:04:01 -07:00
vh a51af1887e memory: handle retire is reversible — re-declaring revives it, history intact 
Changes the risk profile of the pending ledger-dev -> svos-dev rename, so it
belongs in memory rather than only in the thread: retire is the correct verb
(delete refuses any handle carrying mail) and it can be undone by declaring the
name again.

Also records that both deploy findings landed in althing's own INSTALL.md,
where they sit beside the journal_mode=WAL setting that causes them rather than
in a procedure doc that only describes working around it.
 2026-09-05 09:42:13 -07:00
vh 20af94c0a0 deploy(althing): post office to 3.6.0 — handle delete/retire verbs go live 
The tools had been on 3.6.0 while the container sat on 3.0.0 for seven days, so
the operator verbs failed with "no tool named delete_handle" — they live in the
post office, not the client.

Image built on nh3-dev from a clean tree at 4d26226 and pushed under the
claude-bot namespace; the compose pin moves to the new digest rather than
floating on the tag, since this container is the fleet's whole message bus.

The backup procedure this file documents earned itself again: at stop time the
database was 23.8 MB with a 5.9 MB WAL beside it, so a plain copy would have
produced a database that opens cleanly, smokes green, and is missing the day's
mail. Stop, explicit checkpoint to a zero-byte WAL, copy, then verify counts on
both sides — 76 handles, 995 messages, 1022 recipients, integrity ok.

Post-deploy the same counts came back with handles.retired_at present, and the
memory cap and OOM guard were confirmed by `docker inspect` rather than by
reading the yaml, which is what that file asks for and the only check that can
tell a working cap from a decorative one.

Bus down about four minutes.
 2026-09-05 09:40:19 -07:00
vh 6e7e0c5a03 docs: sudo hang fixed, dsh under launchd, post office container left behind 
Records the fifo-relay fix and — more usefully — the two shapes that were tried
and rejected, so the next person does not re-derive them: a pipeline breaks cwd
persistence because every stage runs in a subshell, and a plain file is
invisible until the command ends. The absent `wait` is deliberate and is pinned
by a test, because it looks exactly like an oversight.

Corrects this file's own claim that infra-ops is sudo-less on nh3-extdev. It is
not: `sudo -n whoami` returns root. The wrong row would send a session looking
for a privilege path it already has.

Also flags that the althing tools are at 3.6.0 while the post office container
is still on 3.0.0, so the new handle verbs cannot work yet.
 2026-09-05 09:06:56 -07:00
vh f44b8aa1cf docs: remote-ssh hangs on a bare sudo — pipe it; record where the dsh update stopped 
Found by using the tool for real work rather than by smoke-testing it. A bare
`sudo -n whoami` through ssh_run returns `running` with no output and wedges the
session permanently; the same command piped through `cat` returns everything.
Reproduced on both macOS and Debian, so it is the harness rather than a platform
quirk, and it matters because infra work is sudo work.

The workaround goes in CLAUDE.md where a session will actually hit it. The
proper fix is a protocol change — run the command through a pipe and read
PIPESTATUS — and is deliberately not attempted here.

Also records where the dsh work stopped: the package is updated but the running
web process is still on the old code, left alive on purpose because there is no
LaunchAgent to restart it into.
 2026-09-05 08:37:56 -07:00
vh f05a6a83f4 memory: address-based reach shipped; my credential-boundary objection was half wrong 2026-09-05 08:20:42 -07:00
vh fb4523b225 docs: remote-ssh reaches raw IPs now; record the tenant carve-out 
The alias requirement is gone for anything inside 10.0.0.0/8, so the tooling
note no longer tells sessions to add an alias first — that instruction was
about to send people back to raw ssh, which is what it was written to prevent.

Records the SureFire carve-out and why it is host-specific rather than a /24:
pfi-pve shares 10.250.250.0/24 with two tenant machines, so a subnet deny would
have taken our own hypervisor with it.
 2026-09-05 08:20:23 -07:00
vh 93f4c312d9 memory: uv tool --force does not rebuild without --reinstall; althing 3.5.0 out 2026-09-05 08:05:21 -07:00
vh 4e18f7a183 memory: laptop backup ruled out deliberately; MCP adoption is gated on ssh_config 
Records the operator's ruling that esh-macbook-air is regenerable surface and is
not to be backed up, so a future session finds a settled decision rather than
re-discovering an unbacked-up machine and re-raising it.

Also records the real blocker on the MCP server, found by trying to use it: it
addresses hosts only by exact ssh_config alias, and 15 of the 28 hosts in
servers/ have no alias at all. The allowlist is not what limits coverage, so
widening it would not have helped.
 2026-09-05 08:01:06 -07:00
vh 807446a1cf memory: remote-ssh MCP live and narrow; record the two traps that cost time 
Records what the fork now contains and, more usefully, the two things that will
waste a future session's time: .mcp.json points at built output so an unbuilt
edit serves stale code, and a finite stdin pipe kills the server mid-handshake
while reporting what looks exactly like a remote-side SSH fault.

Also records the disproven hypothesis — irv-ml1's zsh login shell was not the
cause, because the server invokes bash explicitly — so nobody re-derives it.
 2026-09-05 07:46:03 -07:00
vh 1f25cdcd19 feat(tooling): register the remote-ssh MCP server and split the SSH guidance 
elway is a convergent deploy runner and it is good at that: 64 playbooks, sudo
handled once, upload, and three tiers of idempotency. It is the wrong shape for
"what is running over there", which is why this session answered every
reconnaissance question with a raw `ssh host '...'` despite CLAUDE.md saying to
use elway — a playbook for a read-only `docker ps` is ceremony, and each ad-hoc
command is a fresh handshake and a fresh permission decision.

The MCP server fills that half. It holds a persistent shell, so cwd and exported
variables survive between calls and a command costs ~6 ms rather than a full
connect. Measured against nh3-extdev and irv-ml1: state persisted across calls
on both, including the host reached over WireGuard.

CLAUDE.md now routes by intent — changing versus looking — rather than naming
one tool, because the previous single-tool instruction was being ignored in
practice for exactly the cases it did not fit.

Registered project-scoped rather than user-scoped: this workspace is where SSH
infra work happens, and a server that can open shells on the fleet should be
opted into per project rather than inherited everywhere.

SSH_MCP_STRICT_ALLOWLIST=1 is set in .mcp.json deliberately and documented as
load-bearing. Without it the allowlist is every Host in ~/.ssh/config — 18
aliases including both backup servers, database VMs and a tenant-managed host.
Verified: corviduo-dev, present in ssh_config but not in our allowlist, is
refused with host_not_allowed.
 2026-09-05 07:45:21 -07:00
vh 169fa357ff memory: SVOS arc fully closed — vault verified clean, no ledger-era items remain 2026-09-05 07:31:16 -07:00
vh d1d37f7c60 memory: SVOS arc closed at step 8; fork vh/remote-ssh-mcp recorded 
Step 8 executed on the operator's direct authorization with preconditions
checked before firing and the post-state read back rather than inferred. The
rollback window is closed and the arc is finished except the handle rename.

Records the remote-ssh-mcp fork and, more usefully, why it was chosen over the
far more popular alternative: it never handles key material, delegating to the
system OpenSSH client, and its dependency surface is two packages. Also records
that its denylist is four bypassable regexes and must not be counted as
security — the host allowlist is the real boundary — plus the two things to
settle before the tool is used in anger.
 2026-09-05 07:30:03 -07:00
vh 4adccf3f88 memory: step 7 corroborated, step 8 held against a relayed authorization 
The delete was verified from outside rather than taken on report: an admin key
cannot see consumer agents, but the key roster shows the old key authenticating
once after 47 minutes of silence with the new key following 35 seconds later,
which is the signature of delete-then-confirm.

Step 8 is deliberately not done. ledger-dev relayed the operator's approval for
an irreversible action and I declined it; the general rule is now a durable
memory rather than a note about this incident. Both keys stay live.

Also records the confirmed revocation scope — a hard agent delete 401s only the
sessions bound to that agent — and the reason the staged legacy key item must
not be deleted on a schedule while step 8 is pending.
 2026-09-05 07:25:47 -07:00
vh 3bc7c6986f memory: svos cutover verified; record the cutover-destroys-the-old-key foot-gun 
Cutover landed clean — a 201 define rather than 409/PATCH, which is what proves
the new identity is actually in use. No plan- or rate-limit errors, so the
deliberately unset tier is compatible and is explicitly not to be set.

The durable lesson is the ordering trap ledger-dev caught: any credential
cutover whose OLD key is required for a later cleanup is destroyed by the
obvious housekeeping right afterwards. Re-vaulting the post-cutover env.sh
would have overwritten the only convenient copy of the key that can delete
ledger:miranda. Verified their preserved copy is genuinely that key by matching
its trailing 8 characters against the live key suffix rather than trusting the
label.

Also corrects this file's own claim that backfill was the right re-vault tool —
it rescans the whole box and is the slow path for a single known item.
 2026-09-05 07:20:11 -07:00
vh 85ff740f94 memory: svos key delivery confirmed by ledger-dev; cutover is with the operator 
They pulled it from the vault and verified the sha independently, so delivery is
established rather than assumed. The remaining step — swapping the live
credential in env.sh, flipping worldtree.user_id, registering svos:miranda and
restarting the service — is the operator's, not mine and not theirs off a peer
message.

Also records that the global CLAUDE.md Heimdall routing line was corrected in
place on operator instruction, so the ruling is not carried by this file alone.
 2026-09-05 07:07:53 -07:00
vh bcbf92f3d1 memory: svos Heimdall key minted — worldtree-personal, ledger key verified still live 
user_id=svos, key_id=eab3cdbe, wt_live_ format, on worldtree-personal
(10.250.50.152:8081). Deployment established by locating the ledger key there
rather than assumed: created 2026-07-13, last used today, matching ledger-dev's
account exactly.

Value vaulted at nh3-dev/development/svos/worldtree-api-key and verified against
the mint response before the scratch copy was removed. Delivered by vault path
rather than over althing — ledger-dev runs on the same box under the same uid,
so the bus never carried the credential.

The ledger key was read back from /admin/keys AFTER the mint and is untouched,
which is what keeps their deletion ordering possible.

Plan tier left unset deliberately and disclosed: the API offers no way to read a
user's current tier, so matching it would have been a guess.

Also records the operator ruling that worldtree-dev owns code only and no ops,
which supersedes the global CLAUDE.md routing of Heimdall auth to worldtree-dev.
 2026-09-05 07:05:33 -07:00
vh 49909665b2 memory: svos clone repointed — the ledger redirect now covers only unknown stale clones 2026-09-05 06:58:24 -07:00
vh 90941607e7 memory: gitea corviduo/ledger renamed to corviduo/svos; svos Heimdall mint now open 
Rename executed on the operator's direct authorization. Repo id 70 unchanged.
Redirect verified by measurement rather than by citing upstream #807: web and
API paths both 301, and `git ls-remote` against the old URL warns, follows, and
returns HEAD b48a11ca5183. Existing clones keep working.

Recorded as a standing hazard that the old path is now burned — a repo created
at corviduo/ledger kills the redirect silently.

Also opens the Heimdall item ledger-dev had been holding as "awaiting infra-ops"
without ever having sent it, including the deletion-ordering trap that makes
retiring the old key first unrecoverable.
 2026-09-05 06:57:13 -07:00
vh 784f6eb6dc memory: Ledger→SVOS vault move done — svos/env.sh in, stale ledger/env.sh to trash 
The `secret` CLI has no rename, so this was re-put + rm. Stored
nh3-dev/development/svos/env.sh from the live file (sha 7253633d4155, verified
on read-back) and retired nh3-dev/development/ledger/env.sh (sha feb418634e10,
id 3a2af37c-c5aa-4f46-9178-f4fb6008a753, soft-deleted to trash and recoverable).

The two shas differ: the vaulted copy was a 2026-08-11 snapshot and the live
file had drifted since without ever being re-vaulted. Recorded because the
lesson generalises — the vault goes stale unless `secret backfill` is re-run.

Gitea corviduo/ledger deliberately NOT renamed. It is ledger-dev's repo; the
answer they needed is that 1.26.1 writes a repo_redirect on a same-org repo
rename, org/user renames do not redirect, the redirect dies if anything
re-creates the old path, and both the repo and the org carry zero webhooks.
 2026-09-05 06:51:18 -07:00
vh 82158e7e36 memory: snapshot — run 3c gated, run 4 training, DAC revert, GPU rebalance, NAS exposure 2026-09-05 06:36:56 -07:00
vh 5237efa299 feat(litellm): expose run 3c as trial for operator hand-testing 
Operator asked for it explicitly so he can test in Open WebUI in parallel with
brokkr's batteries. Named `trial` rather than erp-tune-v3c: the run-3 note in
this file records why a tier name arriving before its evidence is a mistake,
and 3c has no more evidence than run 3 had.

Points at the pfi-gx10 seat (http://10.100.50.60:8098/v1, vLLM 0.28.0 on
GB10/sm_121), so this is a cross-site route -- NH3 seat, Anaheim gateway.

⚠ Recorded in the entry, not just here: 3c's safety axes are unmeasured, and
run 3 -- same recipe, differing only in learning rate -- was withdrawn from
this gateway 40 minutes after being added for selfharm/methods -44.0pp and
meta/sysprompt -28.0pp against its own base. This gateway is the shared-key
surface. When brokkr's batteries read out, the entry should be re-decided on
evidence rather than left standing by default.

Applied with a restart of the litellm service only (~60 s of gateway
downtime); the compose .env is root-owned so the restart needs sudo.
 2026-09-04 14:48:01 -07:00
vh 054c098777 fix(esh): revert the forced 10G — the link was flaking hard, operator called it 
Both ends are back on autoneg at 1000 Mbps and stable; UDM SFP+2 restored to
autoneg True / setting_preference auto from the payload staged before the
original change. ~50 s of renegotiation, nothing rebooted.

The force failed progressively and I called it too early. Errors on the
ESH-Media side went 200 at link-up, 221 at 42 minutes, 416 by four hours, with
flapping the operator noticed before my soak had accumulated enough to raise
it. I read a two-minute flat window as a plateau and said so; a marginal link
declares itself over hours, not minutes.

Do not re-force this port. The cable is being replaced at the weekend and a
correctly-coded DAC negotiates 10G on its own.
 2026-09-04 14:41:02 -07:00
vh 514ce7acc8 docs(esh): record the held DAC decision — forced 10G stays until the weekend cable swap 
Operator is keeping the override rather than reverting: he has a replacement
cable and will run it at the weekend. Recorded as a held position so a later
session does not revert it as an oversight or re-run the diagnosis.

Also corrects a claim this runbook made an hour earlier. I wrote that the
zero-error 1G link ruled out a marginal cable. It did not — it only proved the
cable was clean at 1G. The two-hour soak shows errors climbing monotonically at
10G (200 to 221 rx, ~1 per 23 MB) plus one self-recovering link drop, so the
autoneg fallback had a real basis and the fix is the cable, not the config.
 2026-09-04 10:52:58 -07:00
vh b9988a7bef fix(esh): force 10G on the ESH-Media uplink DAC — it was negotiating 1G 
The OEM SFP-H10GB-CU3M between ESH-Media SFP+1 and UDM SFP+2 linked cleanly
at 1000 with zero errors on both ends, which rules out a damaged cable — a
marginal one links at 10G and throws CRCs rather than dropping a rate. The
switch reported sfp_compliance "Unknown": it reads the EEPROM but cannot
parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate.

A TP-Link TL-SM5220 DAC on the adjacent UDM port runs at 10000 with the same
speed_caps and autoneg on, which exonerated the port, the firmware and
autoneg before anything was changed.

Forced autoneg off / 10000 / full duplex on the UDM end only. ESH-Media's
end followed to 10000 on its own, which is the proof the cable was always
capable and only the negotiation was wrong. Both ends now 10000.

The UDM end was chosen because it is the recoverable one: ESH-Media reaches
the controller through this link, so a failed force there would strand the
switch. The path to the UDM was traceroute-verified not to cross the link,
and the revert payload was written before the forward one was sent.

Cost ~60 s of link renegotiation. ESH-Media, the E7-ESH-Media AP and the
USW Flex XG ESH-Office-Vuong all returned at uplink_speed 10000 with uptimes
unbroken; the Zigbee coordinator and HA's path to it were re-verified.

⚠ The port is now pinned: a non-10G module will not link there. Documented,
along with the ~200 link-up errors that then went flat and the fact that
steadily climbing errors would mean the cable genuinely cannot do 10G.
 2026-09-04 10:06:50 -07:00
vh 0bbdaf9083 fix(esh): correct the SLZB-MR1U MAC framing — derived ESP32 address, not fragile 
I described ea:f6:0a:ca:f5:b4 as "locally administered, no OUI" and treated
it as the class of MAC that may regenerate at boot. ha-dev corrected it and
the correction verifies: the device advertises e8:f6:0a:ca:f5:b4 over mDNS,
which differs in exactly the locally-administered bit, and E8:F6:0A is
registered to Espressif Inc. in the IEEE registry while EA:F6:0A resolves to
nothing. That is the standard ESP32 pattern — one factory base MAC in eFuse,
sibling interface MACs derived deterministically — so the reservation is
keyed correctly and cannot drift on its own.

The PoE-cycle test still stands and is now corroborating evidence rather than
the only evidence. The residual risk narrows to a firmware change to the
derivation scheme.

Also from ha-dev: tcp/7638 is open alongside 6638; mDNS crosses the VLAN
boundary so HA rediscovers without help; and HA's pending smlight config flow
is keyed on the mDNS service name, which did not change, so a stale flow may
still hold the dead 10.0.10.58 and should be dismissed rather than confirmed.

ha-dev declined the dns: resolver fix on their stack — configuring by IP
costs them nothing and the entry would couple HA name resolution to AdGuard
uptime for no present benefit.
 2026-09-04 09:57:02 -07:00
vh fed29be04e feat(esh): move SLZB-MR1U Zigbee coordinator to the IoT VLAN for Home Assistant 
The coordinator arrived on esh-userland (VLAN 10) at a DHCP-assigned
10.0.10.58. It now sits on esh-iot (VLAN 90) at 10.0.90.10, reserved on
the UDM and named slzb-mr1u.esh.internal.

Address on the network side, not the device — the pfi-gx10 ruling: a
reservation moves with the hardware, a device-side static goes stale.
10.0.90.10 is below the 10.0.90.40-250 pool so it cannot be handed out.

The MAC is locally administered (ea:f6:0a:ca:f5:b4, no OUI), which is
exactly the case where a reservation can silently stop matching. The
PoE power-cycle that makes the device re-DHCP is also the stability
test, so it cost nothing: it rebooted and came back on exactly
10.0.90.10, which only happens if the MAC held.

ESH-Media carried 8 port_overrides and that PUT is a whole-array
replace, so the array was diffed to prove exactly one field on one port
changed before sending, and read back after.

Inter-VLAN reachability needed no firewall work and was proven from
inside the HA container before the move, against existing IoT devices.
Testing from the Docker host would have proven the wrong thing — HA
runs on a macvlan address, 10.0.50.46, not the host's 10.0.50.45.

Documented but NOT fixed: the HA container cannot resolve any
.internal name (its resolver is Docker's 127.0.0.11, upstream excludes
the fleet AdGuard). Pre-existing, verified against names that predate
this change. HA should be configured with the IP; changing the DNS of
a live home-automation stack is ha-dev's call.
 2026-09-04 09:50:31 -07:00
vh dae77ee118 feat(pfi-gx10): stage ERP-seat SFT run 3c — verified, not launched 
Rehome run 3c from ana-ml2 to pfi-gx10 unchanged: same corpus, base,
recipe and hyperparameters, different host. Slower (~13.3 h vs ~2.5 h)
and correct — an Anaheim breaker trip costs a 40-minute drive each way
and 13 hosts down, three of them SureFire client machines, while the
GX10 is a ~240 W appliance at NH3 that can take nothing else down.

Verified rather than assumed, because ana-ml2 ran transformers 5.15.1
on x86-64 and this box runs 5.16.1 on aarch64 — the silent
backend-delta class that has already voided conclusions here:

  - both 49 GB base shards sha256-match ana-ml2's (size equality is a
    weaker claim and was already true)
  - a full encode was run into a throwaway dir and the encoded corpus
    compared byte-for-byte: 197,360,233 B, sha256 c08bb1fe2ecb0be3,
    identical. Every aggregate matched too. That verified artifact is
    what the run will train on — it is seeded into run-03c/encode-cache
  - the harness's own suite: 122 passed on aarch64
  - the config generator asserts key-by-key that no non-path value
    differs from run-03c.json

The encode-cache filename differs by design (base_model_path is part of
the key) — an input hash, not an output hash. Documented so it is not
misread as drift, or "fixed" by faking /tank on this box.

Corpus is copied to local NVMe; the box mounts no NFS. nh3-nas is now on
the same subnet, which makes mounting it tempting and still wrong under
a 13 h unattended run.

The launcher refuses on a live pidfile rather than a pgrep: `pgrep -f
erp_sft_harness` invoked over ssh matches the invoking shell's own argv.
That self-match cost a shell during staging.

Not launched. 13.3 h is the operator's call.
 2026-09-03 22:46:28 -07:00
vh 43135a2768 memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning 2026-09-03 22:22:11 -07:00
vh a95717e810 feat(gx10): rack networking — VLAN 50 via DHCP reservation, wired only 
pfi-gx10 moved from desk Wi-Fi (10.100.10.226, VLAN 10) to the rack on
10.100.50.60 (nh3-servers, VLAN 50), reachable as pfi-gx10.nh3.internal.

The address is a DHCP RESERVATION on the UDM, not a host static. Operator
ruling during the move, and the better design: a host-side static works
until the box moves and is then a stale netplan file on a machine whose
address you no longer know. The pre-written playbook wrote a host static;
it is kept for its safety ordering and annotated as retired.

The port arrived on the native VLAN, not the server VLAN, so switch port 22
was repointed first. port_overrides is a whole-array PUT — two unrelated
overrides on ports 21 and 23 were read, preserved and written back, with the
original array backed up to a file before the change.

Wi-Fi stayed up as the escape hatch until the wired path was proven from
outside, and was downed last as its own step. The step worth keeping: while
Wi-Fi was up, traffic to nh3-dev still preferred wlP9s9 because that
interface sits directly on the userland subnet, so reachability proved
nothing about the wired path. `ping -I enP7s7` across the VLAN boundary is
what actually settled it before the hatch came down.
 2026-09-03 15:59:09 -07:00
vh 0f748ea54e feat(searxng): move to nh3-docker, update, and expose as an MCP tool 
The ana-docker instance was returning zero results for every query while
reporting healthy — 4.5 months stale (2026.4.17 against a current 2026.9.3),
its engine scrapers rotted against sites that had changed. /healthz proves
the web app answers and says nothing about whether search works, so seven
days of green sat on top of a search box that found nothing.

Moved to nh3-docker rather than updated in place, because the colo egress is
the other half of the problem: 38.120.12.42 is a datacenter address that
DuckDuckGo and Startpage CAPTCHA, while nh3-docker egresses residentially at
70.230.226.88. Same reasoning as the fleet's residential proxy for yt-dlp,
applied at the source instead of around it.

Config corrected along the way: base_url said searxng.pfi.local, a name
retired on 2026-08-19, while the environment said something else — the env
won so nothing broke and the file quietly lied. The karmasearch.videos
removal key never matched, because the engine's real name has a space.

scripts/searxng-health.sh asserts results > 0 across three unrelated
queries. That is the check that would have caught this, and the only kind
that can: the mechanism was healthy throughout.

services/searxng-mcp exposes it as `web_search` at user scope, so every
Claude Code session has it. Zero results raise rather than returning an
empty list — an empty list is indistinguishable from a broken aggregator,
which is precisely how this hid.

Old instance stopped and removed; DNS alias repointed to searxng.nh3.internal.
 2026-09-03 14:07:06 -07:00
vh d4aa59a199 fix(backup): enable fleecing so a slow target cannot stall a guest 
Operator-approved after last night's stall. Proxmox's own man vzdump names
both the mechanism and the remedy: without fleecing, guest writes block on
the backup target's speed; with it, pre-write copies land in a local image
instead.

Verified by behaviour rather than by config read-back, on a live backup that
was still crawling at the same 1.4 MiB/s that caused the outage:

  io pressure full avg10   96.5%  ->  0.01%
  in-flight writes         45, completing none  ->  0 0
  D-state processes        11  ->  0
  32 MiB dsync write       never completed  ->  0.24s (133 MB/s)

The backup is exactly as slow as before and the guest no longer cares. That
is the point, and also the limit: this contains the blast radius and does
not explain why a cross-site backup that once ran at 941 MiB/s now runs at
1.4, with the link up and pbs-ana answering in 11 ms.

Job config backed up to /root/jobs.cfg.bak-* on nh3-pve before the change.
 2026-09-02 22:33:12 -07:00
vh bd9692c433 docs(incident): nh3-dev IO stall was a stalled backup, not the disk 
hamr-dev reported /dev/sda stalled on nh3-dev: 45 writes in flight
completing zero, jbd2 and flush kworkers in D state for 33 minutes, io
pressure full at 96%, load 26. Every symptom pointed at a dying disk.

It was a vzdump to pbs-ana that collapsed from 64 MiB/s to 1.4 MiB/s and
sat there. Proxmox interposes a copy-before-write filter during a backup,
so every guest write must copy the original block to the target first — the
guest was throttled to the speed of a stalled cross-site backup.

Three things distinguished it, all cheap: there were no device errors
anywhere (a failing disk says so); Dirty was 3.8 MB while 45 requests were
in flight, so the writes were submitted and not completing rather than
backed up in page cache; and the hypervisor was idle at 0.00 io pressure,
which means the writes were not reaching it at all.

Cancelling the task detached the filter: inflight 45 -> 0, D-states gone,
191 MB/s dsync restored. Tonight's incremental is lost and re-runs cheaply,
which is the standing trade in "prefer no backup over one that can crash
the service".

Recurs nightly at 21:00 until changed, and the job has fleecing disabled —
which is why a slow TARGET can stall a GUEST at all.
 2026-09-02 21:37:25 -07:00
vh 6ca455a15f feat(scripts): provision-mac-dsh.sh — one script for a Mac, end to end 
Three Macs and six accounts were done by hand, and the fourth would have
repeated every mistake the first three taught. This script carries them.

Each guard is something a hand-run got wrong first:

- an account may not own its own home. A `sudo mkdir` before sysadminctl
  leaves /Users/<account> root-owned; the account then authenticates, gets a
  shell, has a correct $HOME and cannot write to it. Surfaced on the Studio
  as a bare "Permission denied" hours after the account looked fine.
- `sudo -u` keeps the CALLER's $HOME. Without -H the install's rm -rf aims at
  the wrong account — it did, at a working install, and only permissions
  stopped it. The remote half refuses to run unless $HOME matches the target.
- the provider ships a hard-coded model catalog that the web GUI reads
  INDEPENDENTLY of agent-default-model, so a correct default still showed
  DeepSeek models in the picker. `models:` replaces it.
- reasoningEffort / maxTokens / defaultContextWindow are all measured against
  the seat; the harness defaults fail on every one.
- the key is scoped per machine and the scope is VERIFIED (200 on
  gen-reasoning, 403 on gen), not trusted from the mint.

The first run found two more: it named the vault item after the IP
(`mac-10-0-10-10/`, unreadable beside esh-mac-studio) and its config check
used grep -A3 where the block needs -A4, so it printed an empty model and
passed anyway. Both fixed, and verification now asserts the model rather
than only the answer token — a check that cannot fail is not a check.

Run twice against the same account to confirm idempotence, then against
vhpfi. docs/runbooks/mac-provisioning.md carries the operator-run stage and
the traps that are not the script's to solve.
 2026-09-02 17:49:42 -07:00
vh 926fc2fb7a feat(litellm): map reasoning_effort high/max -> xhigh for gen-reasoning 
The gen-reasoning seat accepts only xhigh/medium/low and 400s on anything
else — including `high`, which is the default of several clients, so the
seat presented as broken rather than as one enum value out of step. The
DeepSeek Harness failed every request on its default setting, and the only
working client value was `low`: the seat's WEAKEST reasoning tier, while
its own default is xhigh.

conf/reasoning_effort_map.py is a pre-call hook in the same shape as the
existing strip_empty_tools hook. It is scoped to one model group, measured
rather than assumed: gen-reasoning rejects `high`; gen, sec,
char-rp-reasoning and summarizer all accept it and are left alone. Paid
passthroughs were not probed, because probing them spends vendor credits,
and are not mapped.

Verified after deploy: high and max now succeed on gen-reasoning, low and
xhigh still work, a request with no effort param still works, `gen` with
`high` still passes through unmapped, and the harness completes a real
file-edit task at full reasoning.

Needed a compose change as well as a conf push — callbacks are bind-mounted
per file, so the volume only attaches on container create. Recreated the
litellm service by name so the DB was not bounced with it.
 2026-09-02 15:43:22 -07:00
vh e9605df6ff docs(althing): the fast outage probe never touched the timeout it claimed to test 
My smoke test used a refused port on a live host and presented it as the
check for the 2 s budget. Measured: it returns in 79 ms, because a refused
port answers instantly with an RST. It verifies the glyph and nothing about
the deadline. An address that black-holes — an unrouted LAN host — takes
2.065 s and is the one that exercises it. Both are now in the page with what
each actually proves, because letting the fast case imply the slow one is how
a status line that hangs the prompt ships with a green test beside it.

Found by forseti in their own copy of the same probe and confirmed here
rather than adopted.

The smoke test also moves from expected-values-in-trailing-comments to
printing `got [x] wanted [y]` on one line. The comment form is the shape that
produced the false pass that caught two agents inside an hour, and the
expected-value column is what caught it — so the page should use the thing it
recommends.
 2026-09-02 10:36:49 -07:00
vh 87a1cff6de docs(althing): retract the "bare shell" claim — it was never verified 
I wrote that Claude Code runs the status line in a bare shell. What I
actually measured was that a HAND-RUN from an interactive session inherits
that session's exported ALTHING_HANDLE; Claude Code's own invocation is a
different execution and was not observed. forseti caught it before the
sentence had been read by anyone else.

The evidence points the other way: this box's Claude Code process carries
ALTHING_HANDLE and ALTHING_POST_OFFICE in its own environ, inherited from
the dev-launch shell, and a spawned child would normally inherit both —
observed independently on two seats. Whether Claude Code scrubs before
spawning was not observed either.

Nothing depends on the answer, which is why the page now says "do not depend
on the environment being present" instead of asserting its absence, and
records it as an open question with both sides. The smoke test still strips
the variables, because that is the harder case and the one whose false pass
caught two agents inside an hour.
 2026-09-02 10:14:08 -07:00
vh 1ac1efca87 refactor(statusline): delegate the althing segment to althing-statusline 
althing 3.4.0 ships the segment as a console script on PATH, so the twelve
lines that existed in three hand-maintained copies now exist in one. A
script, unlike a document, has somewhere to be installed, and installing it
makes drift impossible rather than merely visible.

Two real defects left this file with the block, both silent: the handle was
resolved as the most recent launch in a directory, so a directory hosting a
claude and a codex seat reports the codex handle's unread count to a Claude
session as soon as the codex pane relaunches last; and the post-office
address was hardcoded, which survives until the post office moves and then
reports an outage that is really a stale constant.

Taken from the live path rather than pushed to it. ~/.claude/statusline-
command.sh had been migrated directly and was AHEAD of this tracked copy,
with a better `command -v` guard; a reflexive cp from the repo would have
destroyed it. Repo and live are now byte-identical.

The deploy instructions and the v3-cutover history move from the file header
into docs/runbooks/althing-deploy.md, where they cannot drift against the
script they describe, along with the diff-before-you-copy-in-both-directions
warning that this near-miss earned.
 2026-09-02 10:11:39 -07:00
vh 6d52fc7302 docs(althing): operator ruling — leave pre-deploy seats on their old channel 
Eight of seventeen live seats were still on fifo after the 3.3.1 deploy,
because a SessionStart hook cannot reach a session that was already running.
Operator's call: leave them. They are reachable, mail is held at the post
office regardless, and each self-corrects at its next session start; the
residual is that a reaped fifo waiter goes silent until restart, which costs
timeliness and never mail.

Recorded with the no-unsolicited-fleet-notification directive next to it,
since the tempting action here is precisely the prohibited one — audit,
report the count, let the operator decide.
 2026-09-02 09:52:36 -07:00
vh 405b3574d2 docs(althing): the measured case for cc — a pane poke lands in prompt history, cc does not 2026-09-02 09:33:17 -07:00
vh ee9872f214 memory: sixth althing deploy surface — crossSessionInbound, outside the repo 2026-09-02 09:31:29 -07:00
vh af5582c96d docs(althing): a sixth deploy surface, and it lives outside the repo 
Claude Code holds an inbound cross-session message unless the sender's
permission-mode class matches the receiver's, and holds any sender that
asserts no class. The herald is a daemon and asserts none, so on a
default-configured seat every cc poke is held at first contact and the
notice goes to whoever is watching the pane rather than to the session.

The seat still reports declared, reachable and healthy throughout — the
same failure shape as the SessionStart hook that was never deployed, where
every check was green and nothing arrived. That is what makes it worth a
surface of its own rather than a footnote.

deploy-althing.sh now reads crossSessionInbound from ~/.claude/settings.json
and reports it on every run and on --check. It deliberately does NOT set it:
that file is the operator's permission configuration, and a deploy script
that edits it is a deploy script granting itself trust.

Also records surface 5 (the per-seat route declare), which had been implicit,
and why no attestation exists for the herald to send instead — CC reads a
sender's live runtime permission mode from the session registry, which a
daemon does not appear in.
 2026-09-02 09:30:48 -07:00
vh 2575242e87 docs(gitea-runner): state that a runs-on label list is AND, not fallback 
I wrote "one-line fallback to pfi-fleet costs you nothing" to a peer,
meaning a manual edit; it was read as scheduler behaviour, and
`runs-on: [vastblue, pfi-fleet]` would have shipped into a contract. A
label list requires ALL the labels, so that form matches no runner and
queues forever instead of failing — the worst outcome of the three,
because a queued job looks busy rather than broken.

Standard Actions semantics rather than a gitea quirk; the ambiguity was
in my prose. Documented where the label guidance already lives so the
next reader does not have to have been in the conversation.
 2026-09-02 09:26:19 -07:00
vh e5cbb5485b memory: vastblue org created — org scope is the binding that was expensive to retrofit 2026-09-02 09:25:06 -07:00
vh 26706f58ca memory: statusline bell measured a mechanism; two copies now exist 2026-09-02 09:14:51 -07:00
vh 5e0c23b0b5 fix(statusline): ask the post office if this seat is reachable, not a lock file 
The bell reported 🔔 iff wake-listener-<handle>.lock named a live pid — one
MECHANISM, not the property it stands for. Correct while the FIFO waiter
was the only channel; wrong the moment 3.3.0 added `cc`. This seat is
reachable over its Claude Code socket, has no waiter lock at all, and was
rendering 🔕 while the post office reported push/reachable. Pane-routed
seats were wrong the same way before that.

It now reads `reachable` from the status payload it was already fetching
and discarding, which means the segment knows nothing about althing's
internals — no lock paths, no channel names — so a fourth channel cannot
make it stale.

Adds the third state: an unreachable post office rendered identically to a
healthy seat with no mail. It is 📵 now. An outage is not an empty inbox,
including on the status line.

Also retires a `kill -0` liveness check, the third instance of
liveness-standing-in-for-identity found on this object tonight.

Verified in all three states: 🔔 on this seat, 📵 against a dead address,
📬 4 🔕 on a pull-only handle.
 2026-09-02 09:14:06 -07:00
vh 1935e48456 memory: althing 3.3.0 deployed — cc channel, and the plugin-cache false green 2026-09-02 09:08:16 -07:00
vh 1147e14b79 fix(deploy-althing): diff plugin CONTENT, and document the cc channel 
The 3.3.0 deploy exposed a false green. `claude plugin update` matches on
the version in plugin.json and declines when it has not moved, so a release
that edits hook or script content without a version bump leaves the Claude
Code cache stale while every version check in this script reports success.
Marketplace and live cache both read 0.1.1, update said "already at the
latest version", and pane-route.sh + README differed.

That particular delta was documentation-only, so nothing was actually
broken — but the script had no way to say so, which is the defect. It now
diffs the marketplace tree against the live cache dir on every run and on
--check, ignoring orphaned version dirs, and says what to do about drift
(bump upstream; never hand-edit Claude Code's bookkeeping).

The 2026-09-01 lesson was "compare the hook list, not the version string".
This is that lesson one turn deeper: the hook list was identical too.

Also documents the cc channel as a deliberately-taken undocumented
interface — expected to break on some future Claude Code release, failing
to pull-only with a logged reason rather than losing mail — and the
herald-before-declare ordering constraint that the script already honours.
 2026-09-02 09:07:21 -07:00
vh 91e2b67074 docs(gitea-runner): commit the capability probe as a re-runnable instrument 
The README's capability table is a measurement with a date on it, and a
runner upgrade or a gitea bump can invalidate it without anything saying
so. Committing the workflow that produced it means the next person can
re-derive the claims instead of trusting them, which is the same posture
the training probes take.

Carries the three job shapes plus the traps each one cost to find: job
images need a node binary or the first `uses:` fails, the Postgres service
needed ~6s and a pg_isready wait, and a run must be polled at
/actions/runs because /actions/tasks reports it as empty on 1.26.1.

Requested by vastblue-dev as a reference for their first smoke.
 2026-09-02 08:58:45 -07:00
vh 89ffab69df docs(gitea-runner): record measured job capabilities, incl. root-equivalent docker access 
Answering a CI-posture question from vastblue-dev meant measuring three
things rather than recalling them. Two came back the opposite of the way
the config reads:

- `container.valid_volumes: []` does NOT keep the docker daemon out of
  jobs. act_runner mounts /var/run/docker.sock on its own, so every job
  on the shared runner is uid 0 with `docker ps` over all 49 containers
  on ana-docker — gitea, synapse, phasefinal-web, adguardhome included.
  It is also load-bearing: four repos drive buildx through it, so the
  fix is isolation onto a dedicated runner, not tightening this one.
- A full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4`
  resolves from the local mirrors today. That is github-independence per
  workflow without the DEFAULT_ACTIONS_URL flip that has been parked on
  act_runner's action-fetch auth since 2026-08-05.

Also recorded: `services:` containers work (Postgres 16 on the service
name), job images need a node binary for JS actions, and `/actions/runs`
lists runs that `/actions/tasks` reports as empty on gitea 1.26.1.

Measured on a throwaway repo under the claude-bot account, since deleted.
config.yaml change is comment-only and deliberately not deployed — it
would bounce the runner for no runtime effect.
 2026-09-02 08:57:25 -07:00
vh a4a529dd76 docs(runbook): assert waiter identity, not liveness, in the reachability audit 
The cross-reference loop filtered live waiters on `kill -0 "$pid"`. A lock
file left by a reaped listener names a pid the kernel is free to reissue,
so the loop would report an unrelated process as a live waiter — and a
phantom waiter is how a false "your seat is unreachable" notice reaches a
seat that is fine, with nothing in the output to falsify it.

`session_listener.sh --stop` already refuses this standard: it walks
/proc/<pid>/cmdline for the `--_route=<handle>` segment before signalling,
on the grounds that a live pid proves existence and not identity. The audit
now asserts the same thing and prints STRANGER on a mismatch.

Verified against the live box: six waiters, all identity-confirmed, all
reporting push.
 2026-09-02 08:34:43 -07:00
vh 7d0d991fcd memory: pin the GX10 baseline's final figures 
The probe finished. Median is 79.36 s/it across ten timed steps with a
min/max of 79.30 to 79.45, and peak memory is 75.1 of 121.6 GiB by
PyTorch's own max_memory_allocated -- 46 GiB spare rather than the 35 I
estimated from a live free reading, which was counting fragmentation and
the resident model rather than the allocation high-water mark.

Resolved attention backend recorded as flex_attention, read off the loaded
model rather than trusted from the request, which is the check I got wrong
the first time.
 2026-09-02 01:23:50 -07:00
vh e39106bd03 memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4 
Rewrote the in-flight section to reflect that nothing is running and the
operator stood the session down. The GX10 work is recorded as what it was
asked to be: a baseline for the box and a check that the tooling loads,
with the run-3c port scoped but explicitly declined.

Added index lines for four decisions that had detail files but no pointer:
the GX10 baseline at 79.35 s/it, the costing error the operator overruled,
the althing four-surface deploy finding, and the irv-ml1 GPU resident map.

Auto-archival fired at the soft cap and moved 17 entries dated on or
before 2026-08-18 to archival-memory.md, holding back 4 that carry open
deferred-work pointers. The index went from 410 lines to 296, mostly by
rewriting in-flight rather than by archiving -- the dated log was not what
made it long.
 2026-09-02 01:23:19 -07:00
vh b039aa19e8 memory: reframe the GX10 work as what it was — a baseline and a tooling check 
Operator clarified the purpose, so the record now leads with it: this was
a baseline for the box and a check that the tooling loads, not a decision
about where run 3c runs. The placement reasoning stays because it is
sound, but it is marked as a byproduct rather than the deliverable.

Two things were actually delivered. The box trains: aarch64 and sm_121
run torch 2.14.0+cu130 with transformers, accelerate, peft, trl, datasets,
safetensors and bitsandbytes, plus the harness's own flex_attention
backend and chunked-loss path, and nothing beyond python3-dev was needed.
And the baseline is 79.35 s/it median across seven timed steps with a
0.19% spread.

Also recorded the port scope without executing it, so nobody re-derives
it: about 2.5 GB of data, a venv rebuild on aarch64, no encode cache worth
moving since the encode runs in 14 seconds, and copy the corpus rather
than mounting NFS on a desk box that will be unattended for hours.
 2026-09-02 01:20:26 -07:00
vh dce261335e memory: run 3c goes to the GX10 — I priced the failure in the wrong units 
The operator overruled my ana-ml2 recommendation and was right. I had step
times in hand, so I priced a breaker trip as eleven minutes of lost
training. That is the recompute cost and it is the cheapest component of
the loss.

A breaker trip at Anaheim is a forty-minute drive each way on the
operator's time, whenever he happens to notice, with thirteen hosts dark
until he arrives -- including the hypervisor most of them run on, the
fleet's primary backup server, and three SureFire client machines that are
a customer's production hosts under a hosting agreement.

So save_steps 100 to 50 caps the recompute, not the outage, and it was
never the mitigation I claimed. Thirteen hours unattended on a desk in NH3
beats two and a half hours that can put a client's hosts dark, especially
when nothing is waiting on this run.

Recorded the general form at length because it is the transferable part:
when recommending between options, check whether you priced the failure
mode in whatever units you happened to be measuring. A metric in hand will
volunteer itself as the unit of risk.

Also measured and dismissed the obvious third option: the RTX PRO 6000s
have a 250 W floor against a 300 W default, so capping both saves 100 W on
a box drawing about a kilowatt. Not enough to matter, and it costs
throughput to buy.
 2026-09-02 01:17:49 -07:00
vh 5a24d77f12 memory: run-3c probe answered — 79.3 s/it, and it reverses the plan on file 
The GX10 runs run 3c at about 79.3 s/it with 35 GB of headroom, which puts
604 steps at 13.3 hours against ana-ml2's 2.2 to 2.7. Six times slower
where raw compute predicts 2.7, which points at memory bandwidth rather
than FLOPs -- recorded as a hypothesis, since confirming it needs a
bandwidth-bound microbenchmark nobody has run.

That reverses the standing plan. Moving run 3c here was framed as the
power answer, but the run did not die because ana-ml2 is unreliable. It
died because save_steps was 100 and the breaker tripped at step 80, so no
checkpoint existed. save_steps is now 50, which caps a power event at
about eleven minutes. Trading 2.5 hours for 13.3 buys insurance against a
risk already engineered out.

Also records the five launch failures and their causes, and the one that
matters most: the attention-backend trap was present and I first declared
it absent. I checked whether flash-attn was installed, which is the wrong
discriminator; the harness sets flex_attention explicitly in code. Absence
of an alternative is not evidence of the default. The probe now reads the
resolved backend back off the loaded model, and flex_attention does
compile and run on sm_121.
 2026-09-02 01:16:20 -07:00
vh 84349d7a0e docs(althing): check the hook list, not the version string 
A version number cannot tell you what a stale plugin cost. 0.0.1 and
0.1.1 differ by two hooks and a script, so the runbook now carries a
check that compares hook lists across cached versions and looks for
pane-route.sh directly.

Also records why this hid for five days, which is the more transferable
half. A missing deploy surface does not present as an error -- it presents
as "the migration needs manual work", and there was a ready explanation
for that, because four of five seats were non-Claude and genuinely did
need hand-holding. The seat that falsified the story was our own: a Claude
Code seat that should have self-declared and did not, and it looked
exactly like the other four. Nobody asked why the automatic path had not
fired on the one seat it was built for.

So: when a migration needs manual intervention, verify the automatic path
was actually deployed before concluding it does not apply to your case.
 2026-09-02 01:13:01 -07:00
vh 167a30a916 memory: althing deploy is one command now, and eshpfi owns the plugin hop 
Recorded the ownership call, which forseti left open. The plugin deployer
lives in eshpfi rather than the althing repo because it targets
per-machine paths, and althing's sync_skill.sh deliberately reaches into
no other tree. Putting a plugin installer upstream would break that
boundary for one consumer's convenience. Their repo stays the source;
this one does the installing.
 2026-09-02 01:05:48 -07:00
vh d0882fb830 feat(althing): four-surface deploy script + runbook 
Deploying althing touches four independent surfaces on nh3-dev. Three
were known. The fourth -- the plugin -- had no step in any runbook and
drifted for five days before anyone noticed.

The plugin chain is repo plugin/ to the marketplace directory to Claude
Code's cache, and neither hop was automated. The marketplace directory
was a frozen copy from Aug 28 carrying only the UserPromptSubmit hook,
with no SessionStart, no SessionEnd and no pane-route.sh at all. So the
claim that CC seats re-declare their pane route automatically at session
start was never true on this box, which is why every seat had to be
hand-declared with a pid measured by hand.

The script backs up the marketplace directory before syncing, re-stamps
its marketplace.json from the repo's plugin.json, and uses `claude plugin
update` for the cache rather than hand-editing installed_plugins.json --
that is Claude Code's own bookkeeping and a subtle mistake there breaks
the plugin in a way that looks like an upstream bug.

The runbook also carries the two things most likely to waste someone's
afternoon: `uv tool install .` without --force is a silent no-op that
exits 0 having done nothing, and a live waiter reporting mode:pull is a
seat that will never be poked, with the audit loop for finding them.
 2026-09-02 01:04:32 -07:00
vh 87f74f1fee memory: althing deploy is four surfaces, and the plugin CLI existed all along 
Deployed 3.2.4 and closed the fourth deploy surface. The plugin cache
moved 0.0.1 to 0.1.1 via `claude plugin update althing`, which means the
step I had escalated to the operator was mine to do. Refusing to
hand-edit installed_plugins.json was right; concluding no supported path
existed was an untested assumption, and `claude plugin` has install,
update, uninstall, list, details, validate and marketplace subcommands.

Recording the pattern once rather than four separate lessons: four times
tonight I reported a proxy or an assumption as the fact itself. sudo -n -v
for NOPASSWD, command -v nvcc for the toolkit, a lock file's pid for a
flock, and "no CLI path exists" for a CLI I never ran. Each was cheap to
test and expensive to assert.

Also ran the audit forseti's 3.2.4 makes possible, since a seat could have
been silently pull-only since 3.1.2 with every failure rendering as
silence. Nine of ten seats with a live waiter report push. regin-smithy-dev
holds a live waiter and the post office has it pull-only -- but regin
deliberately released their pane route earlier tonight, so release-then-arm
ordering explains the same observation with no bug involved. Not
distinguishable from outside, so it went to regin as a question rather
than to forseti as a confirmed instance.
 2026-09-02 00:48:18 -07:00
vh 83217553bb memory: retract the stale-lock claim; the plugin gap ate the SessionStart hook 
forseti measured my stale-lock claim and it is false. I said a lock file
holding a dead pid would make the next althing-listen exit 3 and turn a
reap into a permanent monitoring outage that reports healthy. The gate is
flock -n on an open fd, which the kernel releases when the holder dies, so
a lock left by a reaped listener is inert and exit 3 only fires against a
live holder. The pid in the file is read by --stop alone.

I reasoned from the artifact's contents when the behaviour is set by the
locking mechanism, and put the consequence in durable memory without
testing it. forseti tested before writing code. That is the third time
tonight I reported a proxy as the thing itself, after sudo -n -v for
NOPASSWD and command -v nvcc for the toolkit.

Deployed 3.2.3, which makes althing-listen refuse on a pane seat with its
own exit code rather than silently demoting it.

And found the real shape of the plugin gap, which is worse than the stale
document forseti and I were both discussing. Nothing syncs the repo's
plugin directory into the marketplace directory, so it was frozen at Aug
28 with only the UserPromptSubmit hook -- no SessionStart, no SessionEnd,
no pane-route.sh at all. That means "CC seats re-declare automatically at
their next SessionStart" has never been true on this box, which is why
every seat including our own needed a hand-fed declare. Source is fixed
and bumped to 0.1.1; refreshing the plugin cache needs a /plugin update
from the operator, and I deliberately did not hand-edit Claude Code's own
plugin bookkeeping to force it.
 2026-09-02 00:43:56 -07:00
vh 5563b77867 fix(pfi-gx10): install python3-dev — Triton JIT-compiles C at first use 
Triton builds its CUDA-utils shim with gcc the first time a kernel runs,
and needs Python.h to do it. Without python3-dev the box looks entirely
healthy: torch imports, the 49 GB base loads, LoRA attaches with the
right parameter count, and then the first training step dies with a bare
CalledProcessError naming a gcc invocation and an exit code.

The real message -- "fatal error: Python.h: No such file or directory" --
is discarded, because Triton sends the compiler's stdout to DEVNULL. It
cost a probe run and a model load to find something a one-line manual
compile answered immediately.

Same shape as the dots-tts container needing a C compiler at runtime: a
JIT dependency invisible at install time that only surfaces under load.

The verify step runs the compile rather than checking the package is
present, because `dpkg -l python3-dev` would pass while the compile still
failed on a missing library path or header directory.
 2026-09-02 00:36:16 -07:00
vh f870dbcbbb memory: snapshot — run-3c probe in flight; ana-ml2 baseline reduced from its log 
The comparison number nobody had written down: ana-ml2's real run-3c step
times, pulled out of run-03c.log before the breaker killed it. About 10.8
to 15.8 s/it over the first 24 steps, so 604 steps lands at roughly 2.2
to 2.7 hours. Anything under about 45 s/it on the GX10 makes it an
overnight run.

Also records the exact geometry from run-03c.json and a real
adapter_config.json, so the probe measures the shape that actually ran
rather than an approximation of it.

Checked the backend-delta trap the playbook warns about before running
anything rather than after: flash-attn is installed on neither box, so
both fall back to sdpa. Library versions do differ -- torch 2.13.0 versus
2.14.0, transformers 5.15.1 versus 5.16.1 -- and that is recorded rather
than assumed harmless. The probe reads the resolved attention
implementation back off the loaded model instead of trusting the request.

The probe discards its first two steps as warmup, which is not optional on
this box: an unwarmed benchmark here already read 27 TFLOP/s when the true
figure was 93, because it was timing the PTX JIT.
 2026-09-01 23:46:43 -07:00
vh c0e352a47b fix(elway): probe NOPASSWD with sudo -n true, never sudo -n -v 
`sudo -v` refreshes the auth timestamp, and a NOPASSWD-only rule creates
no timestamp to refresh, so on sudo >= 1.9.15 `sudo -n -v` returns
non-zero while every real command runs passwordless. Measured:

  pfi-gx10    sudo 1.9.15p5   sudo -n -v rc=1   sudo -n true rc=0
  nh3-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  ana-docker  sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0
  irv-ml1     sudo 1.9.13p3   sudo -n -v rc=0   sudo -n true rc=0

Only pfi-gx10 is new enough to hit it today, but every host does as it
moves past 1.9.13, and the failure mode is bad: elway prompts for a
password on a host with working NOPASSWD sudo, which in a
non-interactive run is an EOFError partway through a playbook.

The same probe in my own notes cost this session directly. gx10 looked
like a fleet exception with no NOPASSWD sudo when it had it from account
creation, and the operator was asked for a password that was never
needed. Corrected in auto-memory too.

Also lands the gx10 privileged outfit playbook, now green at 5/5:
NOPASSWD sudo, nvcc, docker group, a CUDA container seeing the GB10, and
the userspace torch stack still working afterward.
 2026-09-01 23:37:42 -07:00
vh 0b48517909 feat(pfi-gx10): privileged-half outfit playbook (sudo, CUDA toolkit, container toolkit) 
The userspace half is done and needed no root. This covers what does:
NOPASSWD sudo for infra-ops (ending the gx10 fleet exception), the docker
group, the CUDA toolkit for nvcc, and the NVIDIA Container Toolkit wired
into dockerd.

Unrun -- it needs one interactive invocation to supply the sudo password,
because no gx10 credential is vaulted and elway prompts via getpass. After
its first step lands, the box stops being an exception and later runs are
unattended.

Guards worth noting: the sudoers drop-in is validated with visudo -cf
before install, since a malformed one locks every sudo user out of a box
sitting on a desk with no out-of-band access; and cuda-toolkit is
installed rather than the cuda metapackage, so the working 580.173.02
driver is not replaced.
 2026-09-01 23:23:50 -07:00
vh fad1db96a0 memory: snapshot — GX10 outfitted userspace; bare metal ruled; CUDA works on sm_121 
Ruling on the architecture question: bare metal, not Proxmox. Proxmox VE
has no aarch64 build, and more fundamentally the GB10's GPU sits on an
on-package root complex cache-coherent with the CPU over NVLink-C2C,
sharing the same LPDDR5X. Passing it to a guest would mean partitioning
the unified memory that is the entire reason for the box. The fleet's
other GPU hosts are bare metal for the same class of reason.

Installed uv and a venv with torch 2.14.0+cu130 plus the full training
stack, and every one of transformers, accelerate, peft, trl, datasets,
safetensors, huggingface_hub and bitsandbytes imports clean on aarch64.
The per-arch unknowns warning did not materialise for any of them.

CUDA works: sm_121, 121.6 GiB addressable, about 93 TFLOP/s dense bf16
with tensor cores confirmed engaged by the bf16-to-fp32 ratio. That is
A6000-class throughput with two and a half times the memory, so capacity
rather than speed is what this box buys.

Two warnings worth keeping. sm_121 is not in torch's compiled arch list,
so everything runs by PTX JIT from sm_120: first use of every kernel pays
a compile, and any library shipping cubins without PTX will fail outright.
And an unwarmed benchmark read 27 TFLOP/s because it was timing that JIT,
which nearly became a phantom report that tensor cores were broken -- the
playbook's section 4 shape exactly, a run that completes and reports
plausible numbers and is wrong.

The privileged half is blocked: infra-ops has no NOPASSWD sudo on this
box, unlike the rest of the fleet, and no credential is vaulted. That
gates nvcc, the container toolkit and the docker group, but not the
run-3c throughput probe.
 2026-09-01 23:18:14 -07:00
vh 18fde5902e memory: snapshot — GX10 liveness confirmed; racking is not a prerequisite 
Probed the box read-only. Alive and idle at 11h48m uptime, 118 of 121 GB
memory free, 822 GB disk free, and completely unchanged since onboarding:
no torch, no nvcc, no uv, and infra-ops is not in the docker group.

The useful finding is a negative one. I assumed the temporary Wi-Fi would
gate getting a 49 GB base model onto the box and it does not. The link is
Wi-Fi 7 on 6 GHz at 2401.9 Mbit/s with a -48 dBm signal, and a measured
300 MB transfer ran at 67 MB/s over SSH, which puts the full base at
about twelve minutes. SSH's cipher is the limiter there, not the radio.
So the throughput probe can run from the desk today and racking is worth
doing for permanence rather than as a blocker.

Also recorded that nvidia-smi reporting FB Memory and BAR1 as N/A is
correct for GB10 rather than a driver fault, since the Grace Blackwell
superchip shares unified LPDDR5X between CPU and GPU and has no discrete
VRAM figure to report.
 2026-09-01 23:11:21 -07:00
vh 4c39ef08e2 memory: snapshot — never arm a waiter on a pane-routed seat 
The /althing:monitor slash command is a separate artifact from the
canonical skill and sync_skill.sh does not touch it. It sits at plugin
version 0.0.1 with pre-3.2.0 text and zero mentions of pane routes, while
the canonical skill it should mirror is at 3.2.2.

That makes it active harm rather than stale documentation. The canonical
precedence rule is that a live waiter wins over a pane entry, so an agent
on a healthy pane route who follows the command verbatim demotes itself
back onto the FIFO path the harness reaps -- the path that died twice on
this seat tonight. The command cannot warn about it because it predates
the problem, and every Claude Code seat reaches for the slash command
first because it is the discoverable surface.

Declined to arm on this seat and left the route intact. Reported to
forseti with a recommendation that althing-listen refuse to arm when a
pane route exists for the handle, since a binary enforcing the documented
precedence beats a doc that relies on a reader noticing.

Also cleared a stale wake-listener lock holding a dead pid, and recorded
why it matters: the next althing-listen seeing it exits 3, whose
documented response is do not drain and do not re-arm. A stale lock turns
a reap into a permanent monitoring outage that reports itself as healthy.
 2026-09-01 23:08:36 -07:00
vh f099caa238 memory: snapshot — H3 encoder pin resolved; my question had the direction backwards 
Checked the disk rather than waiting on comfy-dev. Both builds are there,
pulled a minute apart on Aug 23: a 26 GB int8 and a 15 GB nvfp4-awq.

I had asked whether their nvfp4 pin was set under a Blackwell assumption,
since Ada has no native nvfp4, which would make the int8 file the right
one on the new box. It cannot be. They pinned it on irv-ml1's A6000,
which is Ampere sm_86 and has neither native nvfp4 nor native fp8. Ada
sm_89 supports a strict superset, so a pin that was correct on the weaker
card cannot be invalidated by moving to the stronger one. The migration is
incapable of breaking it.

The pin is about VRAM, not architecture. Eleven gigabytes on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs, and a
text encoder runs once per prompt rather than once per diffusion step, so
its throughput matters far less than the DiT's. That also explains why
this pin went the opposite way from their other one without either being
inconsistent. The RTX 6000 Ada is also 48 GB, so nothing relaxes.

Reclassified the 26 GB int8 from orphan to spare: with the extra drives
the destination lands near 14% full, so disk stops being the constraint
and the pin-rot argument says keep it. Question withdrawn to comfy-dev.
 2026-09-01 23:05:19 -07:00
vh 12006d287a memory: snapshot — althing 3.2.2 deployed; discover-pid was a bug, not an unknown 
Four of the five items I raised are closed and tagged. The one that
matters most: --discover-pid matched comm == "claude", which means kimi,
grok, codex and pi would each have walked to the multiplexer and refused.
Four of five pane seats could never have used it. It now matches the
pane's own command, which zellij already reports and guard 1 already
compares against, so discovery and the guard read one string.

Recording a second miss of my own alongside it. I handed all four seats
explicit measured pids because I suspected the ancestry walk was broken,
then reported the suspicion as an open question rather than spending the
same twenty seconds to settle it, with four live non-Claude seats in
front of me. That pairs with the earlier miss in the opposite direction:
asserting an open risk on something the author had already measured. Same
root -- having the means to settle a question and reporting it as open
instead.

Also keeping forseti's two dead ends as negative results, because they
are the obvious things to propose next and both fail: locating a TUI's
input box in a screen dump needs per-TUI parsing, and diffing two dumps
to detect typing refuses every poke forever, because status bars carry
live token counts and clocks so consecutive dumps differ on an idle pane.

The kimi/pi coverage gap is deliberately unfixed and now sits with the
operator, with a recommendation to leave it per-seat.
 2026-09-01 22:52:35 -07:00
vh f0d30f7a0f memory: snapshot — 3.2.1 outcome; the unguardable seats split, and one has a fix 
All four seats answered. Delivery is now verified on four TUI families and
forseti's measured idle columns held exactly, so nobody had to guess.

The two seats guard 4 cannot cover made opposite calls on the same facts
and both are right. regin-smithy-dev released to pull-only because the
operator composes in that pane routinely, so the exposure is continuous.
bil-smithy-dev kept its route because that pane is poke-driven, making the
collision window narrow, and because pull-only had already cost them a
notice that sat unread for days. The deciding variable is who composes in
the pane and why, not risk appetite -- a flat rule either way would have
been wrong for one of them.

regin also supplied the only real path to closing the hole: pi and kimi
report no cursor because their input line is not an empty-prompt-at-idle,
so a moment-check reading pane content rather than cursor column would
cover them. Column is a proxy; an empty input line is the actual
predicate. Relayed to forseti as the lead item.

And bil corrected something upstream: the Aug 28 probes proved delivery
against non-Claude TUIs but never exercised the collision case, because
nobody was typing during them. Two different questions, one body of
evidence, only one of them answered by it -- which means the earlier
retraction of the non-Claude risk flag was right about delivery and
silent about stapling.

Note for the operator: regin-smithy-dev is pull-only as of 22:42 and will
not be poked until it declares again.
 2026-09-01 22:44:51 -07:00
vh 1c7bd40c9c memory: snapshot — althing 3.2.1 deployed; guard 4 cannot cover two of five seats 
3.2.0 wrote its pokes into pane input lines and pressed Enter, so anyone
mid-sentence had their half-written message submitted with the herald's
line stapled on. It hit the operator within an hour of this evening's
deploy. 3.2.1 adds a fourth guard that pins the pane's idle cursor column
and stays silent when the live column has moved.

Deployed all three steps at 22:40 and re-declared this seat, which pinned
idle_cursor=3 as expected for a claude TUI.

The finding worth keeping is the hole the fix leaves. forseti's own
measurements say kimi and pi report no cursor at all, which means
bil-smithy-dev and regin-smithy-dev can never acquire guard 4 no matter
what they run. Two of the five pane seats on this box stay permanently
exposed to the bug 3.2.1 fixes, so the sentence "3.2.1 fixes the
write-into-a-typing-pane bug" is only true where the cursor is legible.
All four seats were told, differentiated: a pre-filled re-declare and the
expected column for the two that can be guarded, and the honest version
plus the release-to-pull-only option for the two that cannot.

Also keeping forseti's post-mortem line, because it generalises: three
guards that all answer "is this the right pane" and none that asks
whether it is a good moment are one check wearing three hats.
 2026-09-01 22:42:54 -07:00
vh 28dd516be1 memory: snapshot — 3.2.0 migration complete, and a miss of my own worth keeping 
All four notified seats re-declared within about twelve minutes and the
log went quiet. Five of five pane routes now carry the guard fields, and
delivery is confirmed on a claude seat, a pi seat and a grok seat.

Two findings survive the close-out. The herald's exclusion reason is
false for the migration case -- every declaring process was alive and
four days old with no pid wrap, and the real cause is simply that the
route predates the fields the guard needs. And the uv trap stands as the
thing most likely to bite the next person.

The third is mine. I raised non-Claude pane delivery with forseti as an
open risk on their release when forseti had personally measured it days
earlier, on the exact seats in question, and the resulting matrix is what
characterised the settle bug they fixed. The error was not caution, it
was calling something open without checking whether it was already
settled, with the peers who knew right in front of me. Recorded because
"I don't know" and "this is an open risk" are different claims and I made
the second when only the first was true.

Still open with forseti: the uv --force runbook fix, an exclusion-message
third branch, --discover-pid against a non-Claude process tree, the
one-tick latency note, and status not being durable evidence.
 2026-09-01 22:28:13 -07:00
vh 2d2c88e43c memory: snapshot — the four revoked routes are non-Claude, and the herald's reason is wrong 
Operator directed that the four affected seats be told directly. Measured
their state before writing, which turned up two things worth more than
the notification itself.

The herald logs each exclusion as "the process that declared this route
is gone, or its pid was reused by something that started at a different
time." Neither is true for any of the four. Each declaring process
started minutes before its route was written and is still running four
days later, and pid_max is 4194304 against a current 2.86M so the counter
has not wrapped. The real cause is a third one the message never offers:
the route predates the guard fields, so identity cannot be verified. The
behaviour is right and the explanation is wrong, and it would send anyone
debugging it hunting a dead agent that is alive.

Separately, all four seats run non-Claude CLIs -- kimi, grok, codex and
pi -- while the pane poke is built around typing into a Claude Code pane.
Whether --discover-pid walks a non-Claude process tree, and what a poke
does to a non-Claude TUI, are both unverified. Each agent was given their
measured pid to sidestep the first, told plainly about the second, and
offered the choice between re-declaring as a test or staying pull-only.

Both findings raised with forseti.
 2026-09-01 22:23:43 -07:00
vh e014f756fb memory: snapshot — infra-ops moved to a pane route; waiters get reaped 
The waiter died a second time, within minutes of being armed, so this
seat stopped re-arming and declared a pane route instead. That is exactly
what 3.2.0 shipped for.

- althing-route declare --discover-pid walks the ancestry to the
  long-lived claude process rather than the ephemeral bash that invoked
  it. Passing --pid $$ would bind the route to a shell that dies with the
  tool call.
- Recorded the sharper form of the failure: the waiter does not just die,
  it dies after confirming it is up. Both waiters reported push and
  reachable immediately after arming. So a green postbox status is not
  durable evidence of monitoring, and the reap is not confined to the
  pane seats that were migrated -- this seat is a fifo waiter and was hit
  twice.

Both consequences raised with forseti on the deploy thread, along with a
note that reachable is a report rather than a delivered poke.
 2026-09-01 22:20:10 -07:00
vh 71d97f36b6 memory: snapshot — althing 3.2.0 deployed; uv tool install is a silent no-op 
forseti requested the deploy, operator-approved and tagged at c4ede0f.
All three steps landed on nh3-dev and verified: force reinstall to 3.2.0
with seven binaries, herald restart, skill sync.

The durable lesson is the trap in step 1. `uv tool install .` matches on
the source spec rather than its contents, so on a box that already had
the tool installed from that path it prints "already installed" and exits
0 having done nothing. Following the runbook literally would have left
the new binary absent with every command reporting success. Always use
--force when reinstalling from a local path.

Also recorded: the pane-route migration revoked exactly four routes,
verified by splitting on channel= before the restart rather than by
auditing fields (all twelve route files lack the new fields, so a field
audit over-counts). The four affected agents were deliberately not
notified, per the standing rule against unsolicited fleet broadcast, and
that is surfaced to the operator instead.

And this closes an open question from earlier today: the waiter that died
with status "killed" was CC 2.1.257 reaping detached tasks, which is the
premise 3.2.0 exists to address.
 2026-09-01 22:15:51 -07:00
vh 46a3c63706 memory: snapshot — A6000 window closed; my dots-tts hypothesis was wrong 
Operator freed ComfyUI's VRAM directly, so tts-dev is unblocked and the
window request is withdrawn with comfy-dev.

- Verified it was a model unload, not a stop: comfyui still up 8 days,
  same pid, HTTP 200, 18,500 -> 612 MiB. Told comfy-dev explicitly so a
  VRAM drop is not misread as a restart of their service.
- The resulting 43.8 GB free is a snapshot, not a floor. ComfyUI is live
  and reloads ~18.5 GB on the next render, which puts the real floor at
  ~25.3 GB against FireRedAudio's ~26 GB requirement. The coordination
  shrank from "stop ComfyUI" to "don't render during the bench" rather
  than disappearing. Flagged to both; not volunteered on comfy-dev's
  behalf.
- Withdrew my caching-allocator hypothesis for the dots-tts VRAM. tts-dev
  identified it as their prompt-feature cache, capped at 32 entries on
  2026-08-14 after two production incidents. A named mechanism with an
  incident history beats a plausible story, and the useful finding is
  that 14.43 GB sits inside a cap they deliberately chose.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 18:10:34 -07:00
vh e91278799d memory: snapshot — irv-ml1 GPU resident map; dots-tts at 2.4x its recorded VRAM 
tts-dev asked for an A6000 window for an approved TTS bench and flagged a
3090 VRAM delta. Probed the box and mapped PID to container rather than
taking the reported figures.

- The 18.5 GB process they attributed to the 3090 is comfyui, on the
  A6000. And it is 18.5 GB rather than the ~11.8 GB they budgeted, so
  stopping it gives ~44.4 GB free, not the tight margin they expected.
- Their "~4 GB unaccounted" on the 3090 is two things: parakeet is a
  third tenant the doc figure never counted, and dots-tts alone is
  holding 14,430 MiB against a burn-in figure of ~6 GB. The second is
  the larger finding and it is theirs to act on; handed over with a
  caching-allocator hypothesis and a one-restart discriminating test.
- Restated the GPU ordering foot-gun: device_ids ["1"] is the A6000 in a
  container, but a bare native CUDA_VISIBLE_DEVICES=1 gets the 3090.

Window not granted unilaterally — comfyui is comfy-dev's and they are
mid-migration, so the request went to them directly and infra-ops relays.
Ruled that the bench runs as a plain container under lkraven rather than
under /opt/docker/compose/, which is for deployed stacks and would leave
a canonical entry reporting as drift until deleted.

Read-only probes; nothing on the box was changed.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 18:08:14 -07:00
vh 2cf73fd556 memory: snapshot — 768 GB is not a population the R750xa takes 
Operator asked why not move all 768 GB across. It does not fit the
board's shape: the R640 has 24 slots at 6 channels/socket x 2 DPC, the
R750xa has 16 at 8 channels/socket x 1 DPC.

768 GB is either 24x 32 GB (more DIMMs than slots) or 12x 64 GB (fits,
but populates 6 of 8 channels per socket and gives up ~25% of memory
bandwidth). The board wants 16 identical DIMMs. So the targets are
512 GB if they are 32s, or 1 TB if they are 64s — taking 12 from one
spare and 4 from the other. In the 64 GB case the answer beats the
question.

Also recorded: beyond ~512 GB the return is marginal for this workload,
so take 1 TB because it is free rather than because it is needed; 64 GB
LRDIMMs run ~50 W hotter in a chassis whose high-performance fans are
unaccounted for on the invoice; and DIMM slot count now needs to be on
the iDRAC pull, since the 16-slot figure is inferred from the factory
CSV and is load-bearing for a 512-vs-1024 decision.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:35:21 -07:00
vh 32fd1dbe4a memory: snapshot — ARC sizing for the Ada box; no ramfs model tier 
Operator asked whether 512 GB justifies a ramfs for hot models, then
self-corrected toward ARC with a 128 GB cap.

- No ramfs/tmpfs tier. ARC is the same cache done adaptively: no curated
  hot-list to rot, no boot-time copy-in, and the memory comes back under
  pressure. Neither tier beats the host-to-VRAM PCIe hop anyway, so the
  warm-load time is identical.
- 128 GB is below the OpenZFS Linux default of 50% of RAM, so doing
  nothing already gives 256 GB. Recommend ~320 GB, not past ~75%.
- Recorded the one honest argument for tmpfs: safetensors mmap gets
  double-buffered on ZFS-on-Linux, so a checkpoint can cost ~2x. Sized
  for, not architected around.
- Flagged that this is not the idle-VRAM-is-reserved case: arc_max is a
  ceiling on an elastic cache, not a preallocation.
- recordsize=1M cannot be won through zfs send, since recv reproduces the
  source's block structure. Not worth losing incremental send over; the
  128K cost on flash is metadata overhead, not throughput.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:26:33 -07:00
vh 3410d578e3 memory: snapshot — R640 RAM harvest may zero the RAM line; CPU diff was missing 
Operator has 2x Dell R640 at 768 GB each and asked whether the memory is
interchangeable with the R750xa. Both are DDR4 RDIMM platforms and Dell
does not vendor-lock DIMMs, so the answer is very likely yes.

- 16 slots in the R750xa x 32 GB = 512 GB, double the factory spec, and
  it deletes the 8x M04W6 purchase.
- Gating question is RDIMM vs LRDIMM. 24x 32 GB 2Rx4 RDIMM is the safe
  and most likely case; 12x 64 GB LRDIMM needs Ice Lake support checked.
- Cleanest harvest is to strip one R640 entirely and leave the other
  whole, rather than half-emptying both into unbalanced populations.

Separately, reading the factory CSV to answer this surfaced a gap in the
diff table: the reseller also swapped 2x Xeon Platinum 8362 (32C/64T,
265 W, DDR4-3200) for 2x Xeon Silver 4314 (16C/32T, 135 W, DDR4-2666).
The Silvers were recorded under "As bought" but never diffed, so the swap
went unremarked. Two consequences: the box cannot use the 3200 rating the
buy list was paying for, which makes 2666 R640 DIMMs a free lunch; and
the CPUs draw 260 W less, which the existing ~1,020 W power figure
already assumed correctly.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:23:10 -07:00
vh 5c88280f9f memory: snapshot — operator leaning 6 drives; records the layout analysis 
Six drives fills all eight bays, which turns a capacity top-up into a
one-shot build decision. Recorded because the reasoning survives whatever
he picks:

- raidz2 over mirrors. Workload is large sequential reads of safetensors,
  ARC fronts it, SSD resilver has no seek penalty, and "expand two at a
  time" is meaningless once every bay is full. raidz2 survives any two
  failures; 4x mirrors dies to an unlucky pair.
- Buy 8, not 6. A raidz vdev caps every member at the smallest, so the
  two as-bought 1.92 TB drives would cap all eight and put two used
  drives of unknown endurance inside the parity set.
- Drive size is now the permanent ceiling. SAS/SATA backplane, all bays
  full, and the free-PCIe-slot inventory is still unpulled.

Runway table included with an explicit caveat that the growth rate is
projected off one acquisition batch, not measured.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:15:52 -07:00
vh 3180dc85fa memory: snapshot — Ada sizing corrected; prune and drives are orthogonal 
comfy-dev's disk-vs-catalog diff and a recount of my own figures both
landed on this thread. Two numbers were wrong and both were headed for
the operator's sizing conversation.

- My ~90% was a double-count. I read ALLOC 1.45T while their pull was
  running and then added the full ~112 GB on top; most of it was already
  in that reading. "Onboarded" is not "landed". Settled payload is
  ~1.47 TiB and the as-bought mirror lands at 84%, not 90%.
- comfy-dev's "pruning gets us nearer 45%" is the striped figure. On the
  as-bought pair mirrored, deleting all ~215 GiB of unreferenced weights
  still lands at 72%, with ~140 GiB of runway on a store that took on
  ~100 GiB in one day. The constraint is vdev layout, not payload — a
  1.75 TiB pool stays 1.75 TiB whatever goes in it.
- So the prune audit and the drive purchase are independent decisions
  and neither gates the cutover. Presenting them to the operator that
  way rather than as a trade.

Also recorded: pool arithmetic (1.75 / 3.49 / 3.57 TiB), the mirror-vdev
smallest-member gotcha if the new drives get paired one-each with the
1.92s, comfy-dev's 34 GiB of uncatalogued LTX 2.5, and an open question
back to them on whether the H3 encoder's nvfp4 pin was set under a
Blackwell assumption that sm_89 does not satisfy.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:11:04 -07:00
vh 72065b45aa memory: snapshot — Ada destination pool is smaller than the source 
Measured irv-ml1's storetank against the R750xa's as-bought drives while
answering comfy-dev's "does the Ada box have ZFS?" question.

- storetank: 1.81 TiB pool, 1.45 TiB used, 80% CAP already, compression
  off at compressratio 1.00x (safetensors incompressible — no win at
  recv). comfy-dev's ~112 GB batch is landing into it now.
- The R750xa shipped 2x 1.92 TB SATA SSD; mirrored that is ~1.74 TiB,
  smaller than the pool it receives from. Migration would arrive at
  ~90% full with no growth room.
- Buy list: +2x 2 TB SATA SSD (6 bays free, HBA355i has the ports) ->
  two mirror vdevs striped, ~3.49 TiB at ~45%, redundancy intact.
- Retain vs reclaim irv-ml1's /storetank after cutover: RETAIN
  recommended, surfaced to the operator.

Also corrects branch (b)'s recorded rationale. comfy-dev enumerated all
twelve running containers: only comfyui mounts /storetank, so (b) was
unavailable during the transition, not structurally. Right conclusion,
wrong reason — infra-ops reasoned about the box when the question was
about the mount.

Memory-only; no version bump per the SemVer SKIP list.
 2026-09-01 17:06:32 -07:00
vh ace839c768 memory: snapshot — the Ada inference server is a stripped used R750xa 
Dell R750xa JPJ1ZP3, 2x RTX 6000 Ada to be fitted, ComfyUI's new home at NH3. Diffing
Dell's factory CSV against the reseller invoice shows four downgrades: half the RAM, the
2400 W PSUs, and the GPU risers, cables and high-performance fans all absent.

Records the resolved GPU power chain, correcting my own first answer: the chassis' CPU
8-pin cabling is the right source type and NVIDIA 930-00030-1546-000 bridges it to the
card's 12VHPWR, so the PCIe-type RCCWC I first proposed is withdrawn. Also closes the NVMe
question — the backplane is SAS/SATA only — and notes that free PCIe slots may moot it.

Auto-archival fired at 308 lines; seven entries moved to archival-memory.md. The 250-line
target was not reached because the guards hold nearly everything else back as under 14 days
or carrying open deferred work.
 2026-09-01 17:01:14 -07:00
vh 7142657749 memory: snapshot — gx10 unracked and next up as an inference+training box 
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.

Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.
 2026-09-01 16:32:21 -07:00
vh 73866f6a7e feat(synapse): restrict /_synapse/admin to LAN, track the stack 
Synapse mounts its admin API on the same vhost as the client API, so publishing
matrix.phasefinal.com published the admin surface too -- it answered 200 from
the open internet. HMAC-protected, so not an open door, but Synapse's own
guidance is to keep it off the public listener.

A higher-priority router (explicit priority 100, not relying on Traefik's
rule-length tie-break) scopes PathPrefix(/_synapse/admin) behind an ipallowlist.
Verified from a genuinely external vantage rather than from a fleet host, since
nh3-dev sits inside the allowed range and would have proved nothing: via the NH3
residential egress proxy the admin path returns 403 while the client API returns
200 and Element is unaffected.

The 10.0.0.0/8 entry matches nothing today and the comment says so rather than
implying fleet access exists. matrix.phasefinal.com resolves publicly, so fleet
hosts hairpin out their own WAN -- a request from nh3-dev arrived as
70.230.226.88. The rule is effectively deny-all through Traefik, which is the
intended posture: admin work goes through docker exec to localhost:8008 and
never traverses Traefik. Allow-listing the sites' WAN addresses was considered
and rejected as a maintenance trap on dynamic addresses.

Also brings the stack under stacks/ with the Postgres password replaced by a
required .env variable. The tracked copy and the live file have therefore
DIVERGED and deploy-stack.sh must not be used until the live file reads from a
.env; the README says so.
 2026-09-01 16:09:42 -07:00
vh a0c5fc6ed5 feat(pfi-gx10): rack-move network playbook — VLAN 50, static 10.100.50.60 
Target settled: nh3-servers VLAN 50, static 10.100.50.60. Clear of the four
existing statics and below the .150 DHCP pool where fleet statics live.

The playbook never leaves itself one path back. Wi-Fi stays up throughout while
the wired interface is configured beside it; the new address is verified from
outside before anything is torn down, and Wi-Fi teardown is explicitly a
separate later change. A botched netplan therefore costs a retry over Wi-Fi
rather than a trip to the rack — which is what substitutes for 'netplan try',
whose interactive rollback needs a TTY that elway cannot provide.

Two preconditions are asserted as steps rather than assumed: the interface must
have carrier (writing a static config for a dead NIC and reporting success is
the failure this avoids), and its MAC must match, since interface names can
renumber across kernels but MACs do not.

Requires nothing from the operator beyond racking the box. The wired NIC has a
distinct MAC from the Wi-Fi one, so the post-move address and switch port are
both discoverable from the UDM rather than needing to be relayed.
 2026-09-01 11:41:43 -07:00
vh 8fb8cc87ca chore(pfi-gx10): first inventory snapshot 2026-09-01 11:37:09 -07:00
vh 1b596c8c30 feat(pfi-gx10): register the ASUS Ascent GX10 and convert it to headless 
NVIDIA GB10, aarch64, 121 GB unified, sm_121. Ships booting to graphical.target
with GDM and GNOME Remote Desktop running.

playbooks/gx10-headless.yaml sets multi-user.target, stops gnome-remote-desktop,
masks the sleep/suspend/hibernate targets, makes logind ignore lid and idle, and
adds sshd keepalives so a stalled link does not kill a long-running job.

Two things learned the hard way and recorded in the playbook:

- gdm is a STATIC unit on Ubuntu, pulled in by display-manager.service and never
  'enabled'. A guard of  always skips, and a verify
  written the same way passes while the desktop is still running. Both now test
  is-active. The first run reported six green verifies having not stopped gdm.
- elway's --sudo applies only to ad-hoc --shell/--upload. Playbook steps run as
  the connecting user and must carry their own sudo; connect as infra-ops.

The playbook refuses to stop the display manager while a seat session is held,
overridable with --var force_dm_stop=true. Networking is deliberately out of
scope: the box is on a desk on Wi-Fi with a temporary DHCP lease and no ethernet
carrier, and belongs to the rack-install change.
 2026-09-01 11:36:49 -07:00
vh 931bac8f68 docs(matrix): current state, upgrade procedure, alias convention, push findings 
Synapse v1.120.0 -> v1.159.0 and Element-web v1.11.80 -> v1.12.27 (2026-09-01).
The existing build steps date from the AIPA era and are now marked as provenance
rather than as instructions.

Records what only existed in a session transcript:

- Schema migrations are one-way; rollback is restore-from-dump. Pre-upgrade
  pg_dump procedure, with a pg_restore --list verification step.
- Why the appservice user namespace is now exclusive: false. exclusive governs
  who ELSE may act, not what the appservice may do, so on a closed single-admin
  server it locked out all other account creation to prevent squatting that
  cannot occur. Includes the two things not to do: narrow the regex (orphans 13
  accounts) or rename the id (Synapse keys ownership on it).
- The shared-secret registration HMAC takes no trailing null after notadmin.
- Room alias convention #<agent>-<purpose>, operator-ratified, with its cost
  accepted deliberately and its rationale stated as room-identity-carries-tier
  rather than push-payload-carries-room-name.
- Push reality: the pusher is event_id_only, so the notification is assembled
  on-device by Element X's service extension. Records the resulting
  server-invisible failure mode when the phone cannot reach the homeserver.
- QR sign-in requires Matrix Authentication Service and why it is deferred.

Ops ownership recorded: worldtree-dev writes the bridge, infra-ops operates
this instance.
 2026-09-01 11:16:07 -07:00
vh 9e986d8ee8 feat(phasefinal-web): cloudflare edge config — cache ruleset + always online 
Cache rule on www.phasefinal.com with edge and browser TTL both respect_origin,
so cache policy stays declared once in nginx.conf rather than split between the
repo and the dashboard. Always Online enabled, which is what actually survives
an origin outage; a 300s document TTL alone would only mask five minutes.

Verified: document and assets both reach cf-cache-status HIT, apex 301s to www,
edge email obfuscation active.
 2026-08-29 23:02:49 -07:00
vh 524aa4d860 fix(phasefinal-web): healthcheck targeted ::1, so traefik skipped the container 
The healthcheck used http://localhost/, which resolves to ::1 in nginx:alpine
while nginx listens on IPv4 only — so it never passed, the container stayed
unhealthy, and Traefik silently declined to create a router for it. That
presents as a broken docker provider: correct labels, right network, no route,
no error. Target 127.0.0.1 explicitly and add a start_period.

Adds the apex router (301 phasefinal.com -> www) and drops the file-provider
workaround, which was mitigating the wrong diagnosis.
 2026-08-29 22:33:21 -07:00
vh 7ffbee6f09 feat(phasefinal-web): corporate site stack on ana-docker 
Single static page (nginx) fronted by Traefik at www.phasefinal.com, built
from the design brief. Site markup/CSS checked in verbatim from the design
session; fonts self-hosted (SIL OFL) with the @font-face block uncommented,
which every fresh export re-comments.

Routed via a Traefik file-provider config rather than the container labels:
the docker provider on ana-docker was not registering newly-created
containers, so the file router avoids restarting shared ingress. Labels are
retained in compose so the file can be dropped once that is fixed.
 2026-08-29 22:25:20 -07:00
vh c488eadc31 memory: snapshot — althing v3 fleet-wide at 3.1.1, sec on GPU0 
The in-flight section was a day stale: it still described run 3c as the
live subject on a box where nothing had moved. Rewritten around what is
actually true now -- v3 deployed fleet-wide, the post office relocated
to nh3-docker, sec serving on GPU0, run 3c still held on power.

Six new decision entries, three of which carry findings that outlive
their incident: the inbound half of the handle-resolution bug (a stale
ALTHING_HANDLE reads another agent's mailbox and reports it empty,
which is a second route into the failure v3 exists to prevent), the OOM
attribution to Claude Code sessions, and the operator's two explicit
belays recorded so a later session does not re-raise them as new.

Auto-archival fired at 301 lines and moved exactly one entry. Three
others were old enough and every one carries a still-open deferred
pointer -- the parked CI flip, muninn-gate's submit path, and the triton
backend deferred to the Ada refresh. Held back per the guards; an
over-cap file that keeps live decisions beats a scannable one that lost
them. The entry that did move had its deferred item closed today:
nh3-extdev's staged v2.1.2 wheel is moot now that the box runs 3.1.1.
 2026-08-28 22:22:32 -07:00
vh 583f329d00 fix(playbooks): 3.1.1 deploy — and why the markers match presence, not count 
Deploys althing-core 3.1.1 to nh3-extdev. Both markers present on both
boxes; the warning verified behaviourally in four conditions rather than
by grep alone -- mismatched inherited handle warns, matching handle
silent, explicit --handle silent, unlaunched directory silent, and the
warning precedes the output it is about.

The release's own verification line says
`grep -c handles_launched_at dev_launch.py # 2+`. The real count there
is 1, the definition; the other two occurrences are in postbox.py. The
installed tree is byte-identical to the repo at the pushed tag, so the
instruction is wrong rather than the install.

This playbook matches on presence via grep -q, so it passed. Had it
asserted the stated count it would have reported FAILED on a perfect
deploy -- a verification instruction that fails on correct input, which
is the same false-negative this file has now produced three times in
different costumes. Recorded above the variable so the next bump does
not reintroduce a count.
 2026-08-28 16:15:34 -07:00
vh 8a04d6f1bb fix(statusline): resolve the handle from the v3 binding, not the v2 map 
The statusline resolved its handle from ~/.althing/session_handles.json.
forseti corrected the grounding and I verified it: that file is a v2
artifact and v3 never opens it. `grep -rn session_handles althing/` is
empty, postbox's resolve_config takes --handle then ALTHING_HANDLE and
nothing else, and `althing-cli use` -- the tool that maintained the map
-- was deleted at the cutover. Whatever is in it now is hand-kept and
drifts silently.

launch-history.json is written by dev_launch, which is the thing that
sets ALTHING_HANDLE in the first place, so it is the real cwd-to-handle
binding. Shape is {cwd: {command: {at, handle}}} with several commands
per directory, so this takes the most recent by timestamp rather than
whichever key happens to sort first. The v2 map stays as a fallback for
its broader coverage.

Worth recording why this was wrong: I wrote the resolution this morning
by reading the v2 statusline block it replaced and keeping its data
source while updating its commands. The commands were the visible half
of the cutover and the data source was not, so it survived a rewrite
that was otherwise about removing v2.
 2026-08-28 16:01:11 -07:00
vh c648a40b68 fix(playbooks): 3.1.0 herald deploy; the marker check takes a LIST now 
Deploys althing-core 3.1.0 to nh3-extdev and restarts the herald.
Verified by content on both boxes: POST_OFFICE_HINT 0 -> 4 in
post_office_herald.py and resolve_post_office 0 -> 3 in dev_launch.py,
dist-info 3.0.3 -> 3.1.0.

The check took one file:marker pair. 3.1.0 changed two files, so a
single pair would have asserted half a release and passed -- the same
half-passing-silently shape as the version-string check it replaced two
releases ago, one level up. It now takes a space-separated list, reports
each pair individually, and fails if any is missing. Every release's
markers so far are recorded above the variable so the next bump is a
lookup rather than an archaeology exercise.

Also verified the behaviour the release exists for rather than just its
markers. The herald writes its address to $ALTHING_ROOT/post-office and
dev_launch.resolve_post_office reads it when the variable is unset:

  env unset          -> http://10.100.50.40:8390
  env set            -> the env value, which wins
  env set to blank   -> the file, because blank counts as unset

My first attempt tested this through postbox, which still requires the
variable and reported "no post office address is configured" -- correct
behaviour that looked like a failed deploy. dev-launch is the reader,
not postbox.
 2026-08-28 15:59:06 -07:00
vh 590b55f7d8 feat(playbooks): potrace/agg headers, with the two traps that mislead 
pypotrace is an sdist that compiles at install time, so every machine
and every CI runner resolving it needs these headers first. That makes
it a recurring per-box action rather than the one-off it arrived as.

Two things learned installing it on nh3-dev are recorded here rather
than left in an althing thread, at forseti's suggestion, because a
thread is not where the next person looks:

Only libagg is a pkg-config consumer. potrace ships no .pc file and is
found via potracelib.h directly, so `pkg-config --exists potrace`
returns false on a correctly configured box. It looks exactly like the
cause and never is.

libagg's pkg-config modversion is 2.7.0 while its Debian package version
is 1:2.6.1-r134. Comparing those two numbers convinces you the wrong
package is installed.

The verify phase asserts the geometry, not the import: a square must
come back as one curve of four CornerSegments. An extension linked
against the wrong thing can import cleanly and return nonsense, so a
successful build is not evidence the module works.

Getting the build probe to run took three passes and the reason is worth
keeping. uv is not on a non-interactive ssh PATH; it is in a different
place on each box; and on nh3-dev it sits inside a 0700 home, so even
the correct absolute path fails `test -x` for the ssh user because the
directory cannot be traversed. The headers are system-wide and root's
business, but the build check is a developer action and has to run as
the user who owns the toolchain.
 2026-08-28 13:38:12 -07:00
vh cdeb57c18b fix(playbooks): 3.0.3 herald deploy, and a content check that survives releases 
Deploys althing-core 3.0.3 to nh3-extdev and restarts the herald.
Verified by content on both boxes: PANE_SETTLE_S 0 -> 2 occurrences,
value 0.3, dist-info 3.0.1 -> 3.0.3.

The content check was hardcoded to the 3.0.1 markers, so from the next
release onward it would have kept passing while asserting nothing about
what had just been installed -- a check that verifies the previous
release is indistinguishable from one that works. It now takes the
marker and file as variables, bumped per release, with both releases'
markers recorded so the pattern is obvious rather than folklore.

That is the same defect class as the install step gated on `postbox` not
existing, which this playbook carried until last round: a guard written
correctly for the first run and never re-read on the second.

The post office container was not touched. forseti established by
import graph that althing/post_office/* imports neither changed module
-- the fix is in reach_pane, which is herald code -- and the container
has been up two hours across both herald restarts.
 2026-08-28 12:46:35 -07:00
vh 9f87ff87e1 feat(althing): surface the post office on Homepage under Toolchain 
Labels the container into `Toolchain`, an existing group under the
existing Toolchain tab -- "the plumbing", which is where a message bus
belongs. Confirmed live: Homepage's API now returns it.

I had previously recorded in this file that no group fitted, which was
wrong. That conclusion came from a grep over the layout block that
missed the nested groups, and it went into a comment as though it were
a finding. The group was there the whole time.

Labels bind at container creation, so this deployed with `up -d` rather
than `restart`; a restart leaves the old labels and the dashboard keeps
showing what was there before. nh3-docker is already a discovered host
in homepage's docker.yaml as `nh3-pfi-docker`, so the label alone is
enough -- adding a services.yaml entry as well would render the card
twice.

althing-chamber on ana-docker also carries Toolchain labels and is a
separate service per the operator. Left alone.
 2026-08-28 10:37:12 -07:00
vh dbb930d546 fix(playbooks): 3.0.1 herald reinstall, and two guards that were release-hostile 
Reinstalls althing-core on nh3-extdev for 3.0.1 (the pane-route fix) and
restarts the herald. Both boxes verified BY CONTENT rather than by
version string -- forseti's own checks, grep for _PANE_ID and _live_pid,
because a dist-info directory records what was installed, not what the
files contain. Both went 0 -> 3 and 0 -> 2.

Two bugs in the playbook this run exposed, both of which only appear on
the second use:

The install step was gated on `postbox` not existing. That guard was
correct for the cutover, when postbox genuinely was absent, and wrong
for every release after it -- postbox exists now, so a version bump
would have silently skipped the install and the playbook would have
reported success having done nothing. `--force` already makes the
reinstall idempotent, so the guard bought nothing and cost correctness.

The post_office variable still pointed at nh3-dev, three hours after the
post office moved to nh3-docker. It failed in the verify rather than at
install time, which reads as a broken deploy rather than as a stale
constant. Worth noting the failure message was the outage semantics
working exactly as designed: "This is an outage, not an answer: do not
treat it as 'no mail'."
 2026-08-28 10:27:03 -07:00
vh 22da609053 feat: registry-push the post office image; version the statusline 
## Registry

The image moved by `docker save | ssh | docker load`, so a rebuild meant
repeating that by hand. It is now published and the compose pulls a
digest-pinned reference, so a redeploy is `compose up -d` on any host
that has logged in.

Pinned by digest rather than by tag: `:3.0.0` is a mutable pointer on a
registry anyone can re-push, and this container is the fleet's whole
message bus. The tag rides alongside so a human can read what it is.

Namespace is claude-bot, not vh. claude-bot's token carries
write:package and `docker login` succeeds, but package namespaces are
owned -- pushing to vh/ returns "unauthorized: authentication required"
after a successful login, which reads like a credential fault and is
actually an ownership one. Publishing under claude-bot's own namespace
also satisfies the standing directive to stop reusing the operator's
personal credentials for infra work, so the constraint and the policy
point the same way. Recorded in the compose header so the next person
does not read that error as a broken token.

Pull path proven rather than assumed: the running container was
recreated from the registry reference and its data verified afterwards.

## Statusline

Brought under version control because the v3 cutover broke it invisibly.
The segment gated on `command -v althing-cli`, a binary the cutover
deleted, so the unread badge and the armed bell silently vanished for
every session on the box. With 71 of 73 handles pull-only, that badge is
the only out-of-band signal telling a session with no armed waiter that
it has mail -- a dead statusline made a working bus look like an empty
one.

Canonical here, live at ~/.claude/statusline-command.sh, copies rather
than symlinks per the same rule as stacks/.
 2026-08-28 10:08:05 -07:00
vh 9d4e7bd34a feat(althing): move the post office to nh3-docker 
Operator directive, and a standing goal: the bus belongs on the docker
host. The flag-day deployment put it on nh3-dev because the herald lives
there -- but the herald is the piece that must be host-local, and the
post office is explicitly the piece that is not.

nh3-dev was wrong on three counts. Our own server table calls it "not a
Docker-stack host". It has had three OOM events in fourteen days with
the interval halving, and the confirmed hog is Claude Code sessions at
5-18 GB, which is that box's actual job. And mem_limit protects the
fleet from the post office while doing nothing in the other direction:
oom_score_adj was 0, an ordinary kill candidate, on a box whose last
sweep took althing-herald and uvicorn. The new deployment sets
oom_score_adj=-500.

The compose is now version-controlled here as a normal stack rather than
living only in the althing repo's deploy dir.

## docker stop does not checkpoint the WAL

The database was 155 KB with a 4.1 MB write-ahead log, and every recent
message was in the log. A clean container stop left it untouched -- an
explicit PRAGMA wal_checkpoint(TRUNCATE) was required.

A docker cp of the .db alone would have produced a database that opens
cleanly, passes integrity_check, serves the full 73-handle roster, and
is missing the day's mail, with nothing raising an error. Row counts
were verified at source, in the staged copy, and after seeding, because
the count is the only thing that separates those two outcomes.

The old volume is left in place. Not a rollback path, which the operator
ruled out -- just not deleting the only other copy on the day of a move.

## Follow-up left open

The image has no registry push and moves by save/ssh/load, so a rebuild
means repeating that by hand. It should join the gitea registry pattern
the other stacks use.
 2026-08-28 10:02:13 -07:00
vh e58360668e feat: althing v3.0.0 cutover (U9b) and the sec seat onto GPU0 
Two operator-authorised changes on the same afternoon.

## althing v3 (U9b flag day, one-way, no rollback)

The post office replaced the v2 P2P bus on nh3-dev and nh3-extdev.
One container is the only stateful component; heralds are one per box
and dial out; waiters are one per session. Every v2 command was deleted
rather than deprecated, so a script calling althing-cli now fails loudly
instead of silently talking to nothing.

73 handles seeded from the v2 CLI, which is authoritative over the v2
database's 91 agent rows -- the extra 18 are superseded names, a typo,
an underscore variant, and two machine-qualified handles that v3 makes
a category error. Verified by set difference in both directions rather
than by counting; a peer's "72 rendered" was a line-count artifact.

Deleted 5,043 orphaned wake FIFOs. The reason there were five thousand
is that v2 named them per session with the PID and never reaped them;
v3 names them per handle, so the leak is bounded by construction. That
is a fix in v3, not a cleanup we performed.

nh3-extdev needed its own path: althing lives there as a system wheel
under /opt/uv-tools with entry points in /usr/local/bin, its daemons
were system units rather than user units, and uv is not on the login
user's PATH. Captured as a rerunnable playbook rather than shell
history.

The v2 database is left inert on disk. There is no import path and none
was improvised.

## sec onto GPU0

GPU1 carries the five resident fleet seats and had ~28 GB free against
the ~51 GB this seat reserves, so it could not start there at all. GPU0
has been idle since run 3c was stopped. The compose header, the GPU pin
default and the homepage label all carried the old card number and are
corrected together -- a label that names the wrong GPU is a record that
lies about where the work runs.

Both playbooks carry verify phases that assert effective state. Two of
those verifies failed on green deployments while I was writing them:
one used a Go template that collided with the runner's own {{ }}
substitution, one omitted --handle so it failed on identity rather than
reachability. Both are fixed with the reason recorded inline, because a
verify that reports FAILED on a working system trains you to ignore it.
 2026-08-28 07:19:18 -07:00
vh f875f746b8 feat(playbooks): nh3-dev memory forensics — and the OOM hog is Claude Code 
forseti asked for journald kernel persistence plus sysstat, on the
premise that nh3-dev's three OOM events in 14 days left no evidence.

The premise was wrong. journald has been persistent all along: 15,068
kernel entries in the 82-day previous boot and 351 OOM records across
retained boots, full task tables included. `journalctl -b -1 -k`
returned one entry because it ran as a user in neither adm nor
systemd-journal, and journalctl shows only your own messages in that
case. The same artifact produced the "journal stops at 05:36:08 with no
shutdown sequence" claim -- the true boot -1 end is 05:47:04 with OOM
kills logged at 05:38, 05:40 and 05:42.

So the fix for "no evidence" is a group membership, not a logging
change: usermod -aG adm lkraven, which is the group Debian's journald
ACL names explicitly.

With the journal readable the attribution is already in it. The
versioned Claude Code binary lives at .local/share/claude/versions/,
so OOM victims named 2.1.220 / 2.1.177 / 2.1.168 are CC sessions, as
are those named claude. Every one of the twelve largest resident
processes ever recorded on this box is a CC session, topping out at
18.4 GB. Everything else killed is 30-55 MB collateral, which clears
the althing daemons by measurement rather than by their own sampling.

sysstat and atop are added because the journal records the moment of
the kill, not the ramp, and names the victim rather than the winner.
atop was not requested and is the one that matters: with a dozen panes
open, only a per-process timeseries says which session was growing.

Not done: a cgroup cap on CC sessions. It is the real mitigation and it
would kill long-running sessions mid-work, so it goes to the operator.
 2026-08-28 06:03:43 -07:00
vh ea818380ff memory: the rack is one circuit — my blast-radius objection was wrong 
Operator supplied the topology: "the entire rack is on the same circuit,
public ip is served by firewall on the same circuit. load tripped
breaker, entire rack goes dark."

That inverts the argument I committed one commit ago in 3cc55b4. I held
the load attribution partly because a site-wide blackout looked like too
large a blast radius for one GPU box. It is not — it is exactly the
blast radius a shared circuit predicts. The observation I treated as
unexplained was the signature of the mechanism.

Kept the wrong version visible with a superseded marker rather than
editing it away, per the superseded-claims convention: the reasoning
circulated in two althing messages and a peer adopted it.

Holding the attribution as unmeasured was still correct; the argument
used to hold it was reasoning about wiring neither side had seen and
neither side asked for.

Records the finding that outlives the incident: the same circuit powers
ana-ml2, ana-gw, ana-wg, ana-nas and the BMCs, so the load takes every
recovery path with it. Composed with ana-gw's WAN admin being closed
(tunnel-only) and the tunnel terminating on ana-wg, there is no remote
management path to Anaheim during a rack power event, in-band or out.
Open question for the weekend triage: is any of that rack on UPS or a
second circuit?
 2026-08-27 11:04:42 -07:00
vh 3cc55b4b40 memory: separate the measured breaker trip from the load hypothesis 
The record read "power capacity is the open item" next to ana-ml2's
~600 W, which reads as a cause. It is not one. The trip and its timing
are measured; the attribution to the training load is the operator's
working read and the reason for the weekend triage.

The observation that makes the single-load story incomplete on its own
terms: a site-wide blackout is a larger blast radius than one GPU box
accounts for. If ana-ml2's draw were the whole story, ana-nas, ana-wg
and the public address would not have gone dark with it.

Shedding seats may still be the right first move and it is cheap. That
is not the same as having identified what loaded the circuit, and the
distinction matters going into a triage that will act on it.
 2026-08-27 11:01:44 -07:00
vh 88d79375f7 memory: run 3c had TWO launches — the third was an untimestamped report 
brokkr-smithy-dev asked how many times 3c was launched rather than
reconstructing it, and their reading was three. It was two.

  #1  17:53:33 PDT  killed by the power loss at step 80/604
  #2  20:58:41 PDT  stopped deliberately at 21:07:40, healthy

The phantom third came from a report I wrote at 23:03 narrating the
21:07 kill in the present tense with no timestamp. Every fact in it was
accurate; it was unreadable in sequence against a correctly-observed
22:46 snapshot of an idle GPU.

Evidence is ZFS birth times (a `>` redirect truncates the log but keeps
its birth, so mtime alone cannot separate "rewritten" from "created"),
plus the absence of any mtime under /tank/erp-tune after 21:07:34 — a
relaunch would have rewritten three files there.

Also pins the outage window to 18:14:45-18:17:00 PDT and corrects the
downtime from "~90 minutes" to 1h58m: the last journald entry before a
hard power loss is the last time anything wanted to log, not the moment
of the loss, and here it was 20 minutes early.

Corrects the in-flight header (step 22 -> last-logged step 24, stop
deliberate) and its stale "as of" stamp.
 2026-08-27 10:59:29 -07:00
vh 98e7d4886a memory: snapshot — run 3 gated DO-NOT-SERVE, run 3c held on a tripped breaker 
Run 3 trained, gated and dispositioned do-not-serve on a measured 44pp self-harm
guardrail regression that its own preregistered rule passed -- a pooled preserve-list
test cannot see a single-axis collapse. Run 3c (lr 20x cut, single variable) launched,
killed by an Anaheim power-breaker trip at step 80, relaunched, then stopped by the
operator at step 22 pending a weekend power triage.

Also captured: the corpus mix was specified in a unit the optimiser never sees
(45.8% dialogue by context, 24.2% by loss); the dose-response says benefit and damage
are one direction in weight space, so the merge-back measures the problem rather than
fixing it; four guests including the storage SPOF had onboot unset and never came back
from the outage, now fixed with dependency ordering; and a transport failure that
enters a measurement as a value looks like whatever you hoped to find -- which found a
live defect in another agent's instrument an hour after it was reported.

Auto-archived 8 entries to archival-memory.md (Recent decisions: 8, Tried and
abandoned: 0); 4 held back on open deferred-work pointers.
 2026-08-27 07:27:41 -07:00
vh 1a36e60d3a docs(quant-playbook): §3.7's APC-off mitigation was reverted nine days ago and the section never said so 
Found while answering a question from the operator, relayed via brokkr-smithy-dev,
about whether a recorded Qwen3.8 degeneracy at ~1,700 tokens relates to a length
sensitivity just measured on the tuned Gemma-4. The record is §3.7 and the number is
~2,000 -- but reading it to answer that question surfaced that the section is stale.

§3.7 presented "disable prefix caching, keep MTP" as THE MITIGATION, resolved
2026-08-17, and stated the gen seat runs that config. It does not and has not since
that same day: APC-off passed a synthetic 7-turn probe and the operator still saw
severe degeneration in real use, so it was reverted. The multi-day hunt resolved to
the AEON W4A4 quant being defective, with MTP / prefix-caching / gateway merely
amplifying it (§3.8 records the corrected causal story; §3.7 was never updated to
match).

Verified against the live container rather than against the compose file alone:
vllm-gen runs --enable-prefix-caching with qwen3_5_mtp / num_speculative_tokens 3.
stacks/gen-seat/compose.yaml carries the full corrected history inline and is the
current authority.

§3.7's superseded text is kept and fenced rather than deleted -- it is the history of
a mitigation that looked right and was not. Added a dated row to §7 per the standing
rule that a wrong playbook claim gets a superseded-claims entry, not just a fix.

The lesson inside the lesson is worth more than the correction: §3.7's own standing
rule is "gate MTP on a multi-turn coherence probe, not just single-shot acceptance."
The APC-off mitigation was gated on exactly that probe, passed it, and still failed in
real use -- the multi-turn probe was itself too small to gate on. A passing probe is
not sufficient evidence at any size that has not been calibrated against real use.
 2026-08-26 16:35:43 -07:00
vh 5a51e76135 fix(litellm)!: withdraw merged-run03 — measured self-harm guardrail regression 
Added 40 minutes ago at the operator's request; withdrawn on brokkr-smithy-dev's
window-2 safety follow-up. Measured against its OWN base:

    selfharm/methods   base 1.000 (10/10) -> tuned 0.560 (k=25)   -44.0pp  p=0.011
    meta/sysprompt     base 1.000 (10/10) -> tuned 0.720 (k=25)   -28.0pp  p=0.071

Eleven of twenty-five times it engages with a request for the most effective methods
to kill oneself. The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 -- the zero-leak
baseline recorded in this same file hours earlier as "a baseline to PROTECT". It did
not hold.

Withdrawn from the GATEWAY specifically because that is the shared-key surface: one
all-agents key reaches every model listed here, across every session and project. The
operator's hand-testing is preserved in full at the direct endpoint :8099 -- this
removes the fleet's blast radius, not his access. Acted rather than waited because he
is away and the request predates the finding.

ITS PREREGISTERED GATE PASSED. The pooled operational delta is -1.0pp against a
+/-3.00pp bound: nineteen axes held at 5/5 and a 44-point collapse on one moved the
aggregate by one point. The rule was NOT retroactively changed. The failure is
structural and is recorded as R47 section 8 item 11 -- a pooled preserve-list test
cannot see a single-axis collapse, and any future preserve-list gate needs a per-axis
tripwire sized so a total loss on one axis cannot hide in an aggregate.

NOT attributed to the filters: five things changed between run 2 and run 3 and there
is no run-2 measurement on these axes. The measured claim is narrower and sufficient
-- run 3's tuned arm is materially worse than its own base on two axes it was never
licensed to touch. Not a CSAM finding; that detector ran fail-closed across all 575
generations and scanned clean.

The model_list entry is left in place commented out, with the finding above it, so
re-adding is deliberate and informed rather than a blank re-registration.

Verified: config parses, gateway healthy after reload, merged-run03 absent from
/v1/models, direct :8099 still serving.
 2026-08-26 14:52:20 -07:00
vh c577d69e2d feat(litellm): expose run-3's merged tune for parallel hand-testing 
merged-run03 -> ana-ml2:8099, the run-3 ERP/RP SFT merged into stock instruct.
Operator asked for it so he can test it alongside the gate rather than after it.

NAMED FOR THE ARTIFACT, NOT A TIER. It is `merged-run03` and not `erp-tune-v3`
because its behavioural gate has not run. A tier name arriving before the evidence
that would justify it is how a name comes to mean something nobody decided --
and with a v2 already in the list, a v3 reads as a successor to anyone holding
the shared key. If it passes, `v3` is a name to give it then, as a decision.
brokkr-smithy-dev raised this against my own erp-tune-v3 suggestion and was right.

The entry carries the preregistrations ABOVE the description, so a reader meets the
commitments before the numbers: T6 one-directional (a gain is uninterpretable
against a 3.1x fireball tailwind), T3/T4 at ceiling on base so recovery is
UNOBSERVABLE rather than merely unpredicted, and any run-2 comparison descriptive
and non-attributable with its five confounds named.

Also carries the retraction in-line: "bluemoon is the largest loss contributor at
38.6%" came from a words x 1.4 estimator, not a tokenizer. As encoded it is third
at 32.9%. The direction survives (1.4% -> 8.0% of total loss) and that is the
finding; the superlative does not.

Documents why its config.json is the base's copied verbatim: transformers 5.15.1
save_pretrained silently drops text_config.global_head_dim and
num_global_key_value_heads, and vLLM then dies in make_layers with a TypeError
naming neither the config nor the field. Cost a failed boot to find. A LoRA merge
changes weights, not architecture, so the base config is correct by definition.

gemma4-26b-a4b-it-base marked CURRENTLY DOWN rather than deleted -- the tuned arm
took GPU0 and only one 26B bf16 seat fits on that card. Kept because the seat
returns, and deleting a name to re-add it later is how scoped keys get orphaned.

Verified: config parses, no duplicate model_name, gateway healthy after reload,
completion returns text in `content` with reasoning_content null.
 2026-08-26 13:39:26 -07:00
vh b6ce22ddcb feat(litellm): register the run-3 gate base arm at operator request 
gemma4-26b-a4b-it-base -> ana-ml2:8099, the unmodified upstream instruct release
(/tank/aimodels/gemma4-26b-a4b-it-bf16). Operator asked for it on the gateway so he
can hand-test it; it had been direct-only because the seat is ephemeral.

The entry disambiguates WHICH base explicitly. Three exist on that box --
-bf16 (this one, official instruct), -abliterated-bf16, and -heretic-bf16 (run 1's
trainee) -- and brokkr-smithy-dev's gate plan called this arm "stock abliterated" a few
hours ago, which would have been a different set of weights. A reader of the config
should not have to resolve that ambiguity themselves.

Carries the measured refusal posture in-line rather than in an althing thread, per the
erp-tune-v2 precedent: R19's Mistral Small 4 map does NOT transfer to this base (it
draws a wider line than consent, refusing consenting-adult incest and fictional gore
that Mistral engages), system-prompt leak is 0/15 against Mistral's 4/5, and
advice/medical 0/5 is a pre-existing base gap recorded so it cannot later be
misattributed to a tune.

Flagged EPHEMERAL in the strongest terms available: it holds ana-ml2 GPU0, which the
run-3 gate needs for its tuned arm, so this entry will 503 when window 1 completes. It
is not a promise of availability.

Serving flags mirror erp-tune-v2 (--reasoning-parser gemma4 plus
--default-chat-template-kwargs enable_thinking=false, and --max-model-len 16384) so a
base-vs-tuned comparison differs in weights only.

Verified: config parses, no duplicate model_name, gateway healthy after restart, model
listed at /v1/models, and a completion returns text in `content` with `reasoning_content`
null -- the enable_thinking trap is not firing.
 2026-08-26 13:21:31 -07:00
vh 71e44176e9 memory: snapshot — run 3 corpus built and held on a megamix containment defect 
Run 2 is finished, gated FAIL, and serving on the gateway at operator request.
Run 3's corpus was built to brokkr's first recipe and held before any GPU spend:
creative-writing-multiturn is a DECLARED MEGAMIX containing bluemoon, PIPPA,
LimaRP and stheno, and the remix promoted creative-writing AND bluemoon -- the
two roots that overlap, at median jaccard 0.873. Containment, not overlap.

Dedup direction reversed so the primary source survives rather than the copy
inside the bag: bluemoon 67 -> 126 conversations and 38.6% of loss signal, the
largest contributor. Wholly-human share up, megamix share down, total context
unchanged at 12.49M so the operator's settled mix arithmetic survived.

Two structural findings recorded because they outlive this recipe: F1 'excise
PIPPA' removes the ROOT and not the MATERIAL (F2's 250-word floor does that
work, since PIPPA turns cannot exceed 123 words wherever they live), and LimaRP
and stheno remain unchecked against any other root.

Also records the correction I published wrong twice: run 2 was never unstable.
All 46 flags were too_short, the collapse guards fired zero times, and it is the
left tail of a length distribution -- not new to run 2 either, so it is a
property of the recipe and a further base swap will not fix it.
 2026-08-26 09:40:18 -07:00
vh 1a4ef5c7a1 docs(training-playbook): 4.6.3 was wrong twice — correct it, and keep the retraction visible 
The entry reported an 'output-stability regression' as a novel run-2 finding.
Both halves were false and the corrections are more instructive than the
original conclusion, so they stay in-line rather than being edited over.

Not new: run 1's own gate record already carried the same effect with a caveat
attached and unresolved. Two runs across two different base models makes it a
property of the RECIPE, not of the base swap -- which also means a third run
that changes the base again will not fix it.

Not degeneracy, and not a separate finding: all 46 flags were too_short rp turns
of 3-14 words, and the two collapse guards fired ZERO times on any run. It is
the left tail of a length distribution that had been measured and reported in
the same message. Truncation is the same mechanism mirrored on the story side.
Both are thresholds calibrated on the base's output shape applied to a model
with a different one -- 4.6.1, which both parties had written down and neither
applied.

The surviving lesson is sharper: a short-answer gate cannot see length behaviour
AT ALL, and because it could not, the effect went two full runs before anyone
named it. The cost of a gate-set blind spot is measured in runs.

Adds 4.6.3.1 on trip points inside the serving stack's jitter -- same seed, same
weights, rate moves 9.6% -> 12.6%, sd 1.77pp. Not 'the gate is
non-deterministic' but 'the trip point sits inside the jitter', because the fix
follows from the precise statement. Includes the split-design rule for measuring
such a rate, and the rule that a measured rate must carry its corpus in its
name.
 2026-08-26 09:29:14 -07:00
vh 37d3189622 docs(erp-dpo): the clip hypothesis is falsified — the distribution is bimodal 
The output-side test ran on the live seat. There is no shoulder at 123: the
120-139 bin holds three of ninety-six and is a TROUGH, and 17.7% of generations
cross a cap PIPPA can never cross. The clip-as-boundary reading is dead, killed
by the test that could have confirmed it.

Corrects this document's own earlier read, which compared the tuned MEAN (88.5)
to PIPPA's MEDIAN (67) and concluded 'comfortably inside the upper body'. Median
to median it is 62 against 67. Mixing statistics across a comparison produced a
more reassuring answer than the data supports.

What the data shows instead is bimodality -- a mode at 20-39, a trough, a second
mode astride PIPPA's centre, a tail to 505, against a base with no such shape.
The tune changed rp length's SHAPE rather than its centre: roots whose length
distributions do not overlap learned as distinct modes rather than blended into
an average. And the skew is rp-ONLY, which localises it to the family the
clipped root lives in and is the strongest support the turn-share mechanism gets
from the output side.

Consequence for pair generation: chosen/rejected sampled from a bimodal
generator inherit the mixture, not a mean, and naive sampling over-draws the
short mode.

Also records that the degeneracy rate is NOT yet a usable baseline -- same arm,
same seed, VOID flipped no->YES across a re-run because the 10% budget sits at
the noise boundary. A guard whose trip point is at the noise floor produces
disagreement between honest observers rather than silence. Replicates running.
 2026-08-26 06:22:21 -07:00
vh 1e4d827c5d memory: erp-tune-v2 registered in the LiteLLM gateway at operator request 
Operator asked for it so he can evaluate the failed tune by hand, overriding my
not-in-the-gateway recommendation. His call.

erp-tune-v1 was DELETED from the config in the same reload rather than
repointed, so the name now 400s cleanly instead of 500ing against a stopped
backend. Deleting rather than repointing is the point: repointing would resolve
a name a consumer already knows to different weights, silently.

The config entry carries the failed-gate table, the long-form truncation (9.9%)
and degeneracy (4.9%) rates, and the rp-length caveat in-line -- so someone
reading the gateway config learns what they are calling without having to find
the althing thread. Fleet verified healthy after the restart.
 2026-08-26 06:21:09 -07:00
vh b5bbc29b91 memory: gate verdict FAIL — and the T6/T3 trade is what the pair of runs bought 
Records the verdict as a FAIL without rounding it off, and the three findings
worth more than the verdict:

  - T6 spatial +15.0 where run 1 failed the same axis at -3.5, with the base
    swap as the only intended variable. Neither run ships; together they price
    what the abliteration was costing, which neither could answer alone.
  - an output-stability regression visible ONLY on long-form (truncated 0->38,
    degenerate 0->19 per 384) that the reasoning battery could not see across
    four passes because its answers are short
  - PIPPA's 123-word product clip sitting in the length signal at 70.3% of bot
    TURNS against 37.5% of bot WORDS, with the counter-evidence recorded too
    (the tune landed near the median, not the cap)

Also records why keeping the tune out of the LiteLLM gateway now reads as
clearly right rather than merely cautious: a FAILED tune must not be one alias
resolution away from a consumer who has not read the thread.
 2026-08-26 02:25:35 -07:00
vh 5171f19e16 docs(erp-dpo): the PIPPA length clip, measured — DPO pairs would inherit it 
The run-2 gate found tuned rp turns 36% shorter than base. brokkr hypothesised
the mix was teaching PIPPA's 2023 Character.AI product clip; the corpus side is
now measured and confirmed. PIPPA's max is 123 words EXACTLY, 100% at or under
it, and 0.00% in the 124-130 band -- a wall, not a preference. Every other root
crosses its own p99 smoothly.

The mechanism is sharper than 'PIPPA is in the mix'. PIPPA is 70.3% of bot TURNS
but only 37.5% of bot WORDS, precisely because its turns are clipped -- and
length is learned per turn, not per token. By loss tokens it looks like a third
of the dialogue signal; by end-of-turn demonstrations it is seven in ten from a
source that cannot exceed 123 words. Generalises: a length-clipped root is
over-represented in the length signal by exactly the ratio its clipping creates.

Counter-evidence recorded too: the tune landed near PIPPA's MEDIAN (67), not its
CAP, which is central tendency rather than learning the boundary. Weaker claim
than the hypothesis, and not demonstrated either way.

Filed here rather than only in the gate record because preference pairs
generated FROM this tune inherit its length distribution in both chosen and
rejected -- DPO would train an artifact in as an explicit objective. Settle the
length question before generating pairs.
 2026-08-26 02:25:08 -07:00
vh 0bb9ee7777 docs(training-playbook): 4.6.3 — a short-answer gate cannot see a long-form defect 
Run 2's reasoning battery reported zero truncations and zero degenerates on both
arms across four passes. The same tune, measured on long-form generation in the
same session: truncated 0/384 -> 38/384, degenerate 0/384 -> 19/384. A real
output-stability regression, structurally invisible to that gate because its
answers are short.

Not a bug in the battery -- a coverage property. An instrument measures the
regime it samples, and output length is a regime. Generalises to context length,
conversation depth, and any axis where the gate's operating point is narrower
than production's.

The actionable form: enumerate the regimes your gate set spans, name the ones it
does not, and decide deliberately rather than discovering the gap downstream.
Corollary on sequencing -- put a long-form generation in the gate and put it
early, because a length-dependent regression is exactly the one you want found
before four clean short-task passes make everyone comfortable.
 2026-08-26 02:24:42 -07:00
vh 3ae32ddc7f memory: base set complete, tuned arm live with digests verified identical 
Records the floors the tuned deltas have to clear, since they are the whole
point of the base pass and are not recoverable from anywhere else: reasoning
core 0.5 pt, diversity overall 0.0125, story attractor 0.0000.

Two caveats that would otherwise be misread:

  - the rp family froze ZERO markers, so its attractor hit rate is structurally
    0.0 on both arms. That reads as a clean result and means the instrument
    cannot discriminate on that family; rp is measured on the distance axis
    only.
  - 'Elias' in 92/96 base stories is an independent replication of a published
    102/144 on the same family, at a higher rate -- not a novel finding.

Image digest sha256:4091d5593f77 verified identical across both arms, which was
brokkr's stated void condition.
 2026-08-26 02:09:14 -07:00
vh a0f59d2778 docs(training-playbook): 4.6.2 — a null result needs a positive control 
From the run-2 gate. A memorisation probe reporting 0.00% across all 72 items is
the correct output for a model that has not seen the corpus, and is also the
exact output of a probe that is not firing. Nothing in the number distinguishes
them. brokkr-smithy-dev drove the overlap function with known-answer inputs
(identical 100%, half-verbatim 65.38%, unrelated 0%, empty 0%) before trusting
the null, which is what converts a suspicious zero into evidence.

This is 4.5's inert gate wearing a different face: there a check that could not
return 'fail', here a measurement that cannot return non-zero. A clean null is
the most reassuring output an instrument produces and the least
self-evidencing.

Same section records the identical-on-both-arms variant: the diversity battery's
rp family froze zero markers, so its attractor hit rate read 0.0 on base AND
tuned. That reads as a clean result and means the instrument cannot discriminate
on that family. Report as a bounded limitation, never as a delta of zero -- a
check returning the same value for every input is not measuring.

Checklist gains the line.
 2026-08-26 02:08:59 -07:00
vh 3df8707e28 memory: base arm live, tuned arm down — battery running sequentially 
brokkr withdrew the both-arms-concurrent requirement himself: his diversity
battery emits the frozen marker list to a FILE, so the arms were never a live
dependency. The real constraint is narrower -- all of one arm's passes on one
served instance before the swap -- and sequential satisfies it. No fleet seats
displaced, operator not woken.

Records the two parity guards, both of which came out of failures rather than
foresight: the image is pinned by DIGEST (a vLLM version change between arms six
hours apart is a base swap that appears in no config diff), and /tank/aimodels
is mounted for BOTH arms even though only the base needs it, because a mount
that differs between arms is a difference between arms.
 2026-08-26 01:53:47 -07:00
vh 62f01a02da memory: snapshot — run 2 trained, merged, coherence-gated and serving as erp-tune-v2 
Rewrites the in-flight section: run 1's seat is down, run 2 is up on :8098, and
the base decision the previous snapshot recorded as OPEN is resolved (stock
instruct, operator 2026-08-25).

Four new decisions, and the detail file carries the arc: the two operator calls
that produced run 2, all five gates, the harness commit chain, and the caveat
that its own provenance names a commit AHEAD of the code that ran.

Records three things a future session would otherwise get wrong:

  - the mask is proven by the loss-token delta, NOT by the matching p50 step
    times -- step time is insensitive to which positions carry loss, so that
    check cannot go red on the axis I originally cited it for
  - two bf16 26B arms do not fit on one 97.9 GB card (98 GB of weights before
    any KV cache), so brokkr's both-arms-in-one-window requirement is a GPU
    resourcing call, not a scheduling one
  - erp-tune-v1 is still registered in the gateway and returns HTTP 500; the
    fix needs a config edit plus a reload that interrupts fleet traffic, so it
    is batched for morning rather than done at 2am
 2026-08-26 01:37:53 -07:00
vh d54f25605f docs(training-playbook): 4.4.1 sample identity at launch; 4.6.1 calibrate gates against correct input 
4.4.1 -- the dirty-tree case was only half of the harness_commit problem. Run 2
launched CLEAN at 1909d86 and recorded 460f372, because three commits landed on
the same checkout during its seven hours and _git_commit() was called at save
time. Commit AHEAD of the code that ran, naming changes it never executed --
including the provenance fields this section prompted. Same defect as run 1's
BEHIND, opposite sign: the identity was sampled at the wrong moment. Sample at
launch, carry it, and record the dirty flag beside the commit rather than
instead of it. Generalises to every run-scoped identity: anything read at save
time describes the world at save time.

4.6.1 -- the inverse of the inert gate, and it costs trust rather than
correctness. A coherence gate false-rejected 'The capital of Portugal is
Lisbon' as degenerate against a global 15-word floor. The floor was calibrated
against the wrong reference, not set too strict. Lowering it globally would
blunt the check where short output genuinely is degeneration; the fix is a floor
per prompt. Write the positive test alongside the negative one.
 2026-08-26 01:35:27 -07:00
vh bcf63db527 docs(erp-dpo): readiness survey for the DPO stage 
Run 2 is an SFT on the official instruct base, so it will refuse at near-stock
rates by design; targeted DPO is where refusals get pruned on chosen axes. That
was the trade accepted when the stock base was picked over a third-party
abliteration.

Surveys what is on disk against what the stage needs. Ready: the merged tune,
the SFT adapter, GPU0 once the eval seat comes down, the whole non-loss half of
the SFT harness, two unvetted Gutenberg preference sets, and the LitBench-RM
judge.

Missing, in order of pain: preference data for the refusal axes (nothing on
disk targets it -- the Gutenberg sets are prose-quality), the axis list itself,
and a DPO trainer (trl is not installed).

The gating item is not technical: WHICH refusal axes are in scope and which are
explicitly kept. Data generation, pair counts, the held-out split and the
success probe are all functions of that list, so nobody should generate a pair
before it is written down. Flags that the domain-compliance probe should
measure run 2 BEFORE pruning, since the pre-number is the only baseline that
will ever exist.

Also records the operational trap: do the trl install AFTER a run finishes,
never during one -- a resolution that upgrades transformers under a live
process can break its save path.
 2026-08-26 01:18:41 -07:00
vh dbca9a3c66 docs(training-playbook): audit the whole manifest against the pairing rule 
§4.3's generalisation was stated and then not applied to the manifest that
prompted it. brokkr-smithy-dev did the audit: most fields are intent-only, and
the one pairing that would have caught the §4.1 cache failure -- the mask's sha
against the loss-token delta -- existed by accident, because someone had asked
for an encode report for unrelated reasons.

Adds the audit table, and the rider that matters more than the table: put the
observed check where it can actually FAIL. chat_template_sha256's pair is the
sha of the string the tokenizer carries, but asserting that in the parent one
line after assigning the file to the tokenizer compares a value to itself. It
belongs in the encode worker -- a different process, across a pickle boundary,
where an unset config key silently leaves every worker rendering through the
checkpoint's own template.
 2026-08-25 21:30:45 -07:00
vh c1db188e6a docs(training-playbook): §4.3 records an OBSERVED consequence, not just a config string 
brokkr-smithy-dev pointed §4.5's own test at §4.3's remedy: recording
`attn_implementation_resolved` is a check that cannot fail on the axis the
failure lives on.

A silent Dynamo fallback to uncompiled flex leaves
`config._attn_implementation == "flex_attention"` untouched while the run
computes at ~20x the cost and, per torch's own docs, does not work correctly
through the backward pass. The field records the request's RESOLUTION, not its
SURVIVAL. On the failure mode that matters it reports success either way.

So the section now requires the step-time distribution beside it -- n, min,
p50, p99, max -- which is the check that can actually fail. Compiled sits at
p50 ~20 s; a fallback at ~400 s. One perf_counter() in on_step_end buys it.
Distribution rather than a mean, because a mean hides exactly the bimodality a
PARTIAL fallback produces.

Generalised past this instance: any provenance field recording a CONFIGURED
value is a claim about intent. If the failure you fear is the configuration
silently not taking effect, you need a second field recording an OBSERVED
consequence, and the pairing is the check. A settings dump alone is decorative.

Two implementation details are called out because both were wrong in the first
draft -- percentiles nearest-rank so every reported value is a real
observation, and exclude the FIRST step rather than the slowest, since step 1
carries compilation but is not reliably the maximum on a variable-width run.

New §4.7.1: rotate the log on relaunch. Run 2's first attempt died on the
warmup_ratio TypeError and the relaunch appended, so the traceback sat at line
15 of a file whose live run began at line 39 -- and a `tail -n +1 -F` monitor
replayed the dead traceback as a fresh event. One file describes one run.

Checklist gains both lines.
 2026-08-25 20:30:29 -07:00
vh dae6ede8e2 docs(training-playbook): §4 — when the artifact lies about itself 
The playbook covered why a run is SLOW. It did not cover the more expensive
failure: a run that COMPLETES, reports plausible numbers, and is wrong about
itself. Seven of those turned up on the Gemma-4 ERP/RP tune between 08-24 and
08-26 and not one raised an error.

New §4, seven landmines plus a pre-launch checklist:

  4.1  a cache key must cover the MEANING of the cached thing. The encode
       cache missed the impersonation mask; run 2 would have reused run 1's
       unmasked encodings and written impersonation_mask_sha256 into its own
       manifest while doing it. No error, no count change, normal loss curve.
  4.2  validating a VALUE is not validating the PARAMETER. warmup_ratio was
       in range and deleted from transformers 5. Build kwargs as data and
       diff the NAMES against the installed signature -- you cannot check the
       argument list of a call you have already made.
  4.3  record what the run RESOLVED to, never what it requested. Run 1
       recorded no attention backend, so an MFU panel profiled the serving
       seat under sdpa and recommended adopting flex_attention for a run that
       was already using it.
  4.4  never train from a dirty tree; harness_commit will name a commit that
       does not describe the run. Annotate afterwards, never edit the shipped
       artifact -- and state what is NOT wrong, or the note casts doubt on
       every field it omits.
  4.5  a watchdog whose pgrep pattern appears in its own argv can only ever
       return "alive". The inert-gate shape in a liveness check.
  4.6  an instrument nobody runs is not an instrument. Mutation-check any
       test guarding a property that fails silently.
  4.7  fix a stale measurement at the source. "~4.3 HOURS to rebuild the
       encode cache" (really 145.5 s) was copied into a new launcher by the
       same person who had just measured the real number.
  4.8  the pre-launch honesty checklist, ten minutes.

Also:

- Header and framing widened. The file is now a training playbook with a
  throughput half and an integrity half; the filename stays for inbound links.
- Sections 4-7 renumbered to 5-8. External refs are all to §1.1 and §3.4 and
  are unaffected.
- Four rows added to the superseded-claims table, including the kernel table /
  68% quadratic / 8.6% MFU set, which describe the serving seat rather than
  the training run.
- gemma4-erp-tune-sizing.md §6 carries a correction banner with the explicit
  falls/survives split, because that is the doc someone actually reads before
  a run.
 2026-08-25 18:12:27 -07:00
vh 2656196f47 memory: snapshot — the tune is trained, gated, and serving 
Run-01 completed in 7:21:52 (47% faster than the 13.85h round-1 projection),
lora_B gate 205/205 non-zero at median norm 1.708, and the acceptance gate says
it did the thing it was built for: diversity +0.178 against a 0.008 floor (22x),
attractor hit rate -11.3pt against a 2.0pt floor, memorisation 0.0000 on both
arms — which closes the R20 licensed-prose exposure on measurement rather than
argument.

Five new detail files carry the substance:

  erp-tune-run2-complete        the run, the gate, the noise-floor near-miss
                                (brokkr was one step from reporting a 13-point
                                T6 regression sitting inside twice his
                                instrument's own variance)
  mfu-root-caused-attention     8.6% MFU was an accounting artifact; real
                                utilisation 17-20%, cost was attention on
                                AMPERE kernels. Two independent methods agreed
                                to 2.6 points.
  nvfp4-serving-pipeline        merged weights are MANDATORY — vLLM cannot
                                serve a LoRA on ANY Gemma-4 — plus the recipe
                                that silently misses all 11,520 expert tensors
  refusal-retention-probe       measured base 0/100 -> tuned 29/100, then had
                                to accept it was the wrong axis
  worldtree-b188-b189-and-selene  three arcs closed, and a #411 diagnosis I got
                                wrong twice before a directory probe settled it

Current state rewritten end to end — the previous snapshot had the run in
flight at ~17h with MFU unexplained. Both are now closed.

The open operator decision is run 2's base, deliberately unstaged and flagged
against being filed as a config knob: it is a reversal of the trainee-selection
decision, and the pretrained-base option removes the last non-lexical floor on
the CSAM axis given stage-2-detector-inert and contamination-scan-absent are
both already overridden.

Tried-and-abandoned gains four measured-dead throughput levers, the packing
correction (bucketing wins under sdpa and the conclusion flips under flex — do
not carry it past the backend decision), and the merge-back-undoes-abliteration
trap brokkr caught in his own advice.

Index stays at 291 lines, under the soft cap. No archival this run.
 2026-08-25 16:54:49 -07:00
vh 2a05ae91af feat(training-probes): counted-not-surfaced classifier scaffold 
Reusable measurement discipline for probes that must classify how a model
responds to material that should not be printed, logged, or pasted into a
report. Supplies the discipline; the axis map and prompts stay with the caller.

Four rules, each because skipping it produced a wrong number:

  - classify, never surface. Completion text is held inside classify() and does
    not cross the return boundary. A probe that prints what it measured has
    turned a measurement into a distribution channel.
  - three-way, not binary. A refusal regex undercounts — models decline by
    redirecting with no refusal token present, measured at 2/5 to 5/5 on models
    a regex scored 0.
  - the deflection count is a FREE CONTROL. Run both arms: zero on both means
    the model is binary and the regex is sound; only one means the difference is
    real. An artifact does not care which arm it runs against.
  - EMPTY and ERROR get their own buckets. Folding them into either side biases
    the result, and a truncation-heavy arm flatters itself if its failures land
    in the wrong bucket.

Requested by brokkr-smithy-dev for the domain-compliance probe — the discipline
in code rather than reimplemented, with the axis map his side of the line.
 2026-08-25 13:10:03 -07:00
vh 64bf9d313f docs(training-playbook): measure refusal retention on the abliteration's OWN axis 
§3.13, plus the probe that produced it. Two lessons, both about measuring the
wrong thing confidently.

First: a tune applied AFTER an abliteration can walk it back, and a
reasoning/craft/memorisation gate cannot see that. brokkr-smithy-dev's
preregistered gate measured none of it — a tune that gains 41 items of
contradiction detection and quietly restores refusals passes every check. The
compliance axis has to be added explicitly.

Second, and this is the trap: measure the axis the abliteration was actually
FOR. Ours was run so the model engages explicit fiction. The probe reached for
mlabonne/harmful_behaviors — weapons, malware, fraud — because it was cached and
carried a recorded baseline. Different refusal surface entirely, and a model
moves on them independently. 29/100 general-harm refusals on a tune whose prose
the operator was praising at the time is not obviously a defect and may be
desirable: general-harm refusals returning while domain compliance holds is
close to the ideal shape for an internal creative seat. The measurement was
real; its relevance was assumed.

Also recorded, because both were nearly missed:

- Read the interesting cell. In 29 hard / 0 deflect / 71 comply, the
  load-bearing number is 71. Stock refused 100/100; near that would mean the
  abliteration was undone. 71 complying means partially walked back on one
  axis — a different finding, and only one of the two threatens the seat.
- A baseline from a different harness is not a baseline. The recorded 3/100
  came from the abliteration tool's scorer, which reads first-token probability
  distributions; a probe that generates and regexes is a different instrument.
  Run your own against both arms on the same seat or report the number alone.
- A refusal regex undercounts, so classify hard/deflect/comply — and the free
  discriminator: if both arms return zero deflections the model is binary; if
  only one does, the regex is fine. An artifact does not care which arm it runs
  against.
 2026-08-25 12:58:27 -07:00
vh a696b49e2a docs(training-playbook): merging a tune back toward stock can UNDO an abliteration 
§3.12. brokkr-smithy-dev caught and retracted his own recommendation mid-thread;
recording it before it reads back later as advice.

A common remedy for an overfit tune is a partial merge back toward the base to
recover general capability. The published recipes that recommend it merge into
the STOCK instruct checkpoint. On an abliterated base, following that literally
re-introduces the exact refusal directions the abliteration was run to remove —
and it is silent, because the merged model looks healthier on general benchmarks
while the property the seat exists for quietly returns.

Rule: any merge-back targets the SAME base the LoRA was trained against, never
the upstream stock weights however similar the name.

The wider lesson is about recipe-card provenance. Community cards are
per-checkpoint artifacts and do not transfer across dense-vs-MoE,
stock-vs-abliterated, or size variants. The worked example: a recommendation
carried from a card for a DENSE STOCK 31B onto a MoE ABLITERATED 26B-A4B on the
strength of a shared family name. The overfitting warning on that card happened
to come from the right architecture; the pipeline, reward stacks and merge-back
came from the wrong one. Same family, three axes apart.

So: before quoting a recipe card at a decision, state which checkpoint it was
written for and which axes differ. "Same family" is not an answer.
 2026-08-25 08:46:59 -07:00
vh 2ec8f42297 docs(training-playbook): base-viability pre-flight, three greps before you pick 
§3.11. Three consecutive "what about X as a base?" questions in one session,
each answerable in minutes, none of which had been asked before a 7-hour
training window was committed. Writing the check down so it runs first.

  1. does it fit for TRAINING - BF16 weights against the real measured peak,
     not the weight figure (Gemma-4 is 48.1 GiB of weights and peaks at
     79.7 GiB at mb2/seq-16k). Model-line names lie: "Mistral Small 4" is 119 B,
     238 GB in BF16, more than both cards combined. QLoRA is not an escape
     hatch for MoE - bitsandbytes walks nn.Linear and fused 3-D experts are not
     that.
  2. if MoE - does the serving engine implement get_expert_mapping. Zero means
     LoRA cannot be served at all. gemma4*.py -> 0; deepseek_v2, mixtral,
     glm4_moe, ernie45_moe -> present.
  3. does the model class support LoRA - and GREP THE CLASS, NOT THE FILE.

Point 3 has teeth and I nearly got it wrong twice in one turn. mistral.py greps
as SupportsLoRA=0 and is fully LoRA-capable via LlamaForCausalLM.
mistral_large_3.py greps as 0 for both and inherits get_expert_mapping from
DeepseekV3ForCausalLM. Capability is inherited; a file-level grep misses it and
only MRO resolution answers it. Same class of error as asserting a substring
instead of an effective value.

Worked results recorded for the three candidates evaluated:

  Gemma-4 26B-A4B        fits, no expert mapping   -> trainable, MERGE-ONLY
  Mistral Small 4 119B   238 GB, has mapping       -> servable, NOT trainable here
  Ministral 3 14B        ~28 GB, dense, inherited  -> passes all three

Adds a fourth glance at architecture shape, since it predicts how much of this
playbook applies at all: uniform head_dim <= 128 with no sliding window keeps
both flash and cuDNN reachable and makes §3.1/§3.3 moot, while mixed head dims
plus a sliding window is exactly what forces dense O(n^2) attention onto
Ampere-generation kernels for 65% of the step.
 2026-08-25 01:51:06 -07:00
vh 96731bb090 docs(training-playbook): prove the serving path before spending the window 
§3.10. The quantization playbook already says prove your targets before
spending GPU time; this is the same rule one step later, and easier to skip.

A ~7h LoRA run was built assuming the adapter could be hot-swapped onto a
quantized base at serve time. The sizing doc flagged serving as unsettled and
said the requirement was needed "while he is early, not after the run" — the
concern was identified correctly and then the check was deferred. Tested
afterwards, vLLM refuses outright: gemma4's model class implements zero
occurrences of get_expert_mapping, which process_packed_modules_mapping
requires for any MoE model. One grep, available months earlier.

Two generalisations recorded:

- Feature support is per-architecture, not per-family. LoRA works for the DENSE
  sibling of this same model family and not the MoE one, so "model X is
  supported" says nothing about X's variants.
- A capability gap in the serving engine cannot be worked around from the
  training side. The adapter here never touched experts and was refused anyway,
  because the refusal keys on the model being MoE, not on what the adapter
  targets.

Includes the mechanical check: grep the engine's model class for the capability,
then start the engine with the feature flag alone — no adapter required, since
--enable-lora forces the machinery to initialise and that is where it fails.

The recovery is cheap here (merge, ~35 min per tune). The cost of finding out
late is that it forecloses an architecture choice after the training window has
already been spent.
 2026-08-25 01:36:01 -07:00
vh 8de5f7a73c docs(gemma4-erp-tune): merged weights are mandatory — vLLM cannot LoRA any Gemma-4 
The §5 open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops
as it did on vLLM 0.24.0 (#47639), with merged weights as the fallback if it
did. Retested on vllm/vllm-openai:latest against the NVFP4A16 base plus the
live run's checkpoint adapter.

It does not no-op. It refuses to start:

    AttributeError: To support LoRA for MoE model,
                    'get_expert_mapping' must be implemented

And the reason is bigger than the quant. The check is in
vllm/lora/utils.py::process_packed_modules_mapping and branches on whether the
model is MoE — quantization is not in the condition. gemma4.py, gemma4_mm.py,
gemma4_mtp.py and gemma4_unified.py contain zero occurrences of
get_expert_mapping, while deepseek_v2, glm4_moe and ernie45_moe do implement
it. So vLLM cannot serve a LoRA on Gemma-4 at all, BF16 or quantized. Merging
is not a workaround for a quantization limitation; it is the only path for this
architecture.

This holds even though the adapter never touches experts —
validate_adapter_parameters forbids per-expert params, so all 205 targets are
attention and dense MLP. The refusal is about the model being MoE, not about
what the adapter targets.

Worth recording that the current behaviour is an improvement: a loud refusal
beats the 0.24.0 silent no-op, which would ship a base model wearing the tune's
name and pass every check that does not compare against base.
 2026-08-25 01:32:51 -07:00
vh ab980e9345 fix(erp-tune-serve): four defects the end-to-end dry run found, all silent 
Validated the full adapter -> merge -> NVFP4A16 -> serve pipeline against
checkpoint-100 of the live run. It works, and it produced a served model
generating coherent prose. Getting there surfaced four failures, none of which
announced itself as the thing it actually was.

1. transformers 5.15 MIGRATES the config schema on save. It drops Gemma-4's
   `global_head_dim` / `num_global_key_value_heads` and writes `per_layer_config`
   instead. transformers 5.10 (what the llmcompressor venv pins) does not know
   the new key and resolves num_key_value_heads to None:
       TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
   Every working artifact on the box - bf16 base, served nvfp4 prod seat,
   nvfp4a16 build - uses the OLD schema. Merging changes weights, not
   architecture, so the merge now downgrades the schema and asserts the result.

2. llmcompressor cannot auto-init a processor for a multimodal checkpoint and
   dies with a message that names neither the model nor the cause. Calibration
   here is text-only, so the tokenizer is passed explicitly as `processor`.

3. save_pretrained writes tokenizer files only, so `processor_config.json` was
   never carried. vLLM then fails at startup with "Can't load feature extractor",
   which reads as a vision bug and is actually a missing-file bug. Both scripts
   now carry the base's auxiliary configs.

4. The quant needs more than the 32 GiB free on GPU1 alongside the resident
   seats. Rather than leave that to a caller, quant_with_gen_down.sh stops
   vllm-gen and restores it from a trap on EVERY exit path - crash, OOM, kill,
   or success - because the restore must not depend on the calling session
   surviving. Uses `docker start`, not `compose up`, so the container comes back
   with its exact original config. Measured window: ~15 min, gen healthy after.

Verified on the resulting artifact:

  merge   410 adapter tensors, sampled target weights confirmed CHANGED,
          upstream 390-line chat template shipped (not the base's stale 365)
  quant   49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16),
          weight_packed 11,725 of which 11,520 expert = 30 x 128 x 3,
          tokenizer truncation clean
  serve   Marlin NVFP4 kernel + Marlin MoE backend, 40,492-token KV cache,
          coherent generation with content correctly populated

One quality note: the reference nvfp4a16 artifact triggers a vLLM warning that
parallel layers (q/k/v) carry different weight global scales, "likely to result
in reduced accuracy". Our build does not - llmcompressor 0.12 links weight
observers across fused groups for a shared global_scale automatically. The
in-house quant is better than the downloaded one on that axis.

Separately: the lora_B inert-adapter gate PASSED on checkpoint-100 - 205/205
non-zero, median norm 0.829, zero vision_tower tensors. That check never ran in
round 1, and it is the only failure mode that stays invisible until the
acceptance gate reports base-identical numbers.
 2026-08-24 23:48:25 -07:00
vh 6a8582936e feat(erp-tune): NVFP4A16 serving pipeline, and the MoE landmine it uncovered 
Merge + quantize path for turning the Gemma-4 26B-A4B ERP/RP LoRA into a
servable NVFP4A16 seat, plus a playbook entry for the defect found while
validating it.

The landmine (playbook §3.15): a `targets=["Linear"]` NVFP4 recipe silently
misses every MoE expert on this architecture. Gemma-4 stores each layer's 128
experts as two fused 3-D nn.Parameter tensors, not nn.Linear modules, so the
recipe resolves 205 of 427 modules and ZERO experts — 22.84 B params, 88.5% of
the model, left in BF16 with no warning. This is the same blind spot that
killed QLoRA here via bitsandbytes; the tool changed, the checkpoint layout did
not.

  before linearize_moe:  427 Linears,    205 targeted, experts 0
  after  linearize_moe:  11,947 Linears, 11,725 targeted, experts 11,520
                         (30 layers x 128 experts x 3 projections)

llmcompressor's linearize_moe unfuses them; no registration needed because
Gemma-4 satisfies FusedExpertsProtocol structurally. Caught by an §4.1 dry run
that asserts the expert count before any GPU spend, which is now the documented
requirement rather than an optional step.

Scheme is NVFP4A16, deviating from the playbook's mixed-W4A4 default on
measured grounds: brokkr-smithy-dev benched the W4A4 quant of this checkpoint
at 12% on contradiction detection with CoT off against gen's 81%, the signature
of 4-bit input activations on a reasoning-dense task, and W4A4 KLD degrades
2-4x past ~10k ctx on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill
cost is accepted.

Two further silent-failure guards, both from prior hard-won lessons:
- the merged model ships the UPSTREAM chat template, not the trainee base's
  stale 365-line one, because training rendered through upstream and the
  mismatch would present as a tuning failure
- calibration reads the run's own encode cache rather than re-tokenizing, which
  sidesteps §3.14 (a fast tokenizer mutated by truncation=True and persisted by
  save_pretrained clamps every prompt forever)

Merge-then-quantize rather than LoRA hot-swap, since hot-swap onto NVFP4 was a
silent no-op on vLLM 0.24.0 (#47639). merge_lora.py asserts sampled target
weights actually changed, so an inert adapter cannot ship as a tune.
 2026-08-24 23:11:33 -07:00
vh 7b5fd91d3c docs(gemma4-erp-tune): root-cause the 8.6% MFU — attention on Ampere kernels, 29.9% padding 
Run-01 was killed at step 19 by operator instruction to root-cause before
spending a ~13.9h window. Two independent methods now agree on where the step
time went, and neither was the hypothesis the consult panel converged on.

Scaling fit (3 points, 2 params, residuals <3ms over an 8x range):
  A = 6.87e-4 s/token, B = 8.85e-8 s/token^2
  quadratic share 20.9% @ w=2048 -> 67.8% @ w=16384
  No fixed term was needed, which refutes launch-bound outright.

Profiler kernel table (device rows only):
  attention   22,835.8 ms  65.2%   fmha_cutlass*_sm80
  dense GEMM   2,774.0 ms   7.9%
  other        5,739.0 ms  16.4%

The attention kernels are sm80 — Ampere-generation CUTLASS running on an
sm_120 Blackwell card, with the forward on the gmem fallback tier. That is the
mechanism behind 100% SM utilisation at 27 of 304 available TFLOPS.

Correctness cleared separately: the sliding mask asserts at max 1024
allowed/row, so the 25 windowed layers were genuinely windowed. The same probe
found that right-padding is what pins the 5 global layers to an explicit 4D
mask and off the is_causal fast path — measured at 9.4% slower for 24% less
loss work at fixed width.

The largest available win is not the attention kernel. The corpus is 29.9%
padding, and bucket-to-pair + shuffle-to-mix takes it to 0.0% for >=35.5% wall
clock, no new dependency, unchanged peak memory. Bucket size turned out not to
be a diversity knob — roots per accumulation window are flat across a 256x
range, so the global micro-batch shuffle does that work alone and the bucket
should be tight.

Adds docs/pfi/training-throughput-playbook.md as the durable model-agnostic
home (sibling to the quantization playbook), the four probes under
scripts/training-probes/ with raw output kept for re-derivation, and a §6 to
the sizing doc carrying the Gemma-4-specific numbers and round-2 restart
parameters.

Measured negatives recorded so they are not re-chased: grouped_mm (0.9%
slower, and MoE is only 7.9% of the step), CUDA graphs / torch.compile over
the expert loop (no fixed cost to amortise), liger fused CE (~1-3% lever),
FA4 on sm_120.

Round-1 state preserved: 609MB encode cache, order manifest, truncation
report, resume script. No checkpoints — it died at step 19 and the first was
due at 100, so the lora_B inert-adapter gate never ran and moves to the
restart.
 2026-08-24 22:10:51 -07:00
vh 872c2c562f memory: the MFU hunt — two hypotheses measured and killed, consult dispatched 
Records what has actually been ruled out rather than what is suspected.

The hardware is fine: a plain dense GEMM at the same shape reaches
97.1% of the benchmarked 313.8 TFLOPS peak.

The Python expert loop is not the cause, which was my hypothesis and I
was confident in it. transformers' grouped_mm experts backend runs
0.9% SLOWER than eager with bit-identical output and identical peak
memory, and torch 2.13 has the kernel available, so it is not falling
back for lack of one.

MoE is not the bottleneck at all. Isolated at real shapes the block
runs at 26.5% of peak with 36% of its time in pure gather/scatter, and
a dispatch-free bmm version would reach 80.9% — but the whole MoE
contribution is only about 10% of a step. Making it free buys 7%.

So roughly 90% of the time is unaccounted for. The leading untested
hypothesis is that the five full_attention layers use global_head_dim
512, above FlashAttention-2's 256 cap, which would push SDPA onto a
slow backend for O(n^2) attention at sequence 16384.

Also records that the earlier 5% MFU figure was wrong in two ways —
unpadded tokens and a guessed peak — and that the operator caught it.
Padding is real but secondary at 29.9%.

Consult dispatched to brokkr-smithy-dev for the frontier-dwarf panel.
 2026-08-24 21:38:15 -07:00
vh 07743c6aff memory: snapshot — tune training unattended, MFU root-caused to a Python expert loop 
The in-flight section is rewritten around the run itself rather than
the decisions that led to it. The sizing and seat-call bullet collapses
to a pointer now that both are executed; its detail lives in
docs/pfi/gemma4-erp-tune-sizing.md.

Adds the measured MFU finding: 27.1 TFLOPS against a benchmarked 313.8
TFLOPS peak, root-caused by reading the source rather than inferring —
transformers runs the Gemma-4 experts in a Python loop, 128 experts
across 30 layers, roughly 11,500 iterations per optimizer step under
gradient checkpointing. Padding is a secondary 29.9% tax.

Records that my first estimate of 5% MFU was wrong in two compounding
ways: divided by unpadded tokens, and compared against a guessed peak
rather than a measured one. The operator pushed back on the number and
was right to.

The fused MoE kernel is deferred work with a tracking surface — park
id 47 — per the snapshot rule that deferred decisions go in Recent
decisions with a pointer, never into the volatile in-flight section.

Also records the resume trap: the original launch command begins with
rm -rf on the output directory, which would destroy both the encode
cache and every checkpoint. resume-run-01.sh exists so that cannot
happen.
 2026-08-24 21:20:58 -07:00
vh d6dfd61c91 memory: the ERP tune is running — override granted, 12 defects fixed first 
Operator overrode the corpus gate for one run on 2026-08-25, with the
grant staged beside the recipe rather than asserted in chat. It
deliberately does not flip any root's training_eligible flag, so the
signal that made the run stop in the first place survives intact.

Records where the run lives, what it is configured with, how to restore
the fleet, and the two lessons that generalise past this project.

The first is inert gates. Two turned up in one evening — auditcore,
whose CSAM hard-drop never fired across 42,662 records, and
validate_vision_keys, which compared model.state_dict() against itself
and could not fail on any input. Both read as guards. The question that
catches them is not whether the check passes but whether it can fail.

The second is an invariant enforced on one code path and not its
sibling. That was my own bug: INV-T9 requires a window to hold at least
one complete assistant turn, and I enforced it where the window is cut
but not where it fits, so a trailing user-only remainder became a
zero-loss window and killed the first launch. Same shape as the inert
gates, in code I wrote an hour earlier.

Two further foot-guns worth the space: enable_input_require_grads is
mandatory beside gradient checkpointing on a frozen base, or every
adapter stays at its initialisation and the run completes successfully
having learned nothing; and the upstream Gemma-4 template forward-scans
to suppress a closing turn marker before another assistant message, so
incremental rendering cannot tile against it and assistant runs must be
merged first.
 2026-08-24 21:00:17 -07:00
vh 33433e0d1e docs(gemma4-erp-tune): replace the estimates with measurements — they were 3x optimistic 
Ran the loss path on the real checkpoint on GPU0 with synthetic tokens.
The arithmetic held for parameter counts and was badly wrong for
activation memory.

    naive CE   bsz1 seq 8192    81.93 GiB
    naive CE   bsz1 seq16384    OOM
    chunked CE bsz1 seq16384    65.66 GiB
    chunked CE bsz2 seq16384    79.71 GiB   <- the run config
    chunked CE bsz4 seq16384    OOM

The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the
~5 GiB estimated: the estimate modelled gradient checkpointing as
storing layer inputs plus a modest recompute peak, and the real MoE
recompute peak with top-8-of-128 routing and its scatter/gather buffers
is far heavier. Dense-model intuition does not size an MoE run.

Two predictions landed exactly — 205 target modules and 74,342,400
trainable params at r64 — which is why the rest of the model of the
thing is still worth trusting.

The headline is that chunked CE at seq 16384 costs 16 GiB less than
naive CE at seq 8192, so chunking is what makes brokkr's 16384
recommendation reachable rather than an optimisation on top of it.
max_seq_len moves 8192 -> 16384 on his truncation finding: the cap
drops 6.2% of samples but 22.4% of tokens, concentrated entirely in
dialogue, which is 60% of the mix.

Also records the four harness changes this required (eitri-smithy
62b556b), including the inert-adapter trap: without
enable_input_require_grads() alongside gradient checkpointing on a
frozen base, no gradient reaches the adapters, every one stays at its
initialisation, and the run completes successfully having learned
nothing.
 2026-08-24 19:01:49 -07:00
vh 47ec3d1a97 memory: the ERP tune is blocked on a corpus gate only the operator can clear 
Every clean-v1 CLEANROOT carries training_eligible: false with two named
blockers, and the recipe states plainly that nothing in it is Charter §3
training-eligible.

I initially read scoped_grant: operator-2026-08-22 as authorization and
told brokkr-smithy-dev I was proceeding. That was wrong, and the person
who wrote the field corrected it: the grant governs INV-4 one-way tier
inheritance — the adapter is permanently internal-erp-rnd and never
distributable — not training clearance.

The stage-2 detector is measured-inert rather than merely unvalidated.
auditcore v3.7.2 returned its hard-drop exit code zero times across
42,662 raw RP records, its printed verdict ignores its own printed
threshold, and it passed a record a blind audit had already identified
as sexual content involving a participant the text marks as a child.

Verified the one thing that decides whether that specific record reaches
training: pippa-5083 is present in kept-manifest.jsonl (4,551 rows) and
absent from recipe-dedup-kept.jsonl (20,473 rows), which is the survivor
list the harness gates on. The substitute lexical screen caught it. That
is one known instance caught by a stopgap and says nothing about what the
screen misses.

Both brokkr and I recommend stopping. Neither blocker is hours of work.
 2026-08-24 18:51:52 -07:00
vh c9943b1507 docs(gemma4-erp-tune): whole-card placement — gen moves to GPU1, sec stands down 
Operator chose a third placement over the two the sizing offered: rather
than train beside gen on GPU0 or on GPU1 in mog-sec's slot, move gen to
GPU1 and empty GPU0 completely. The tune gets 95.60 GiB with no
co-tenant and gen never goes dark beyond its own restart.

Revised run parameters, since a whole card changes them:

- micro-batch 8 (71.8 GiB of 95.60) rather than 4, grad-accum 1, giving
  888 optimizer steps instead of 444. At one epoch the step count is
  worth having, and 8 x 8192 tokens puts ~4,096 rows through each expert
  per step against ~512 at micro-batch 1 — a far healthier GEMM on
  704-wide experts.
- Gradient checkpointing stays ON. Dropping it takes ~17% off wall-clock
  but pushes activations to ~24 GiB per sequence, which forces
  micro-batch 1 and costs 8x on MoE efficiency. Wide beats shallow.
- Scriberr stays on GPU1. The previous revision suggested moving it to
  GPU0, which was correct only while training was going to live on GPU1.

Records the ordering constraint in both directions, the elway identity
requirement, and that sec's aliases should be allowed to fail at the
gateway rather than be substituted with another model.
 2026-08-24 18:40:42 -07:00
vh 9d70100867 feat(ana-ml2): elway playbooks to open and close the ERP/RP tune window 
Operator call: rather than train beside gen on GPU0, move gen to GPU1 and
stand sec down for the night, so the tune gets a whole 95.60 GiB card and
the fleet's general seat never goes dark beyond its own restart.

Order is load-bearing in both directions and the playbooks enforce it.
gen runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction
of TOTAL card memory: 42,091 MiB must be FREE at startup or the engine
refuses to boot. GPU1 has 19,446 MiB free while mog-sec is up, so
recreating gen onto GPU1 first would take the main seat down and leave it
down. mog-sec stops first and a hard gate checks the freed memory before
gen is touched. The close playbook mirrors it: gen must vacate GPU1
before mog-sec starts, since mog-sec needs 50,901 MiB of its own.

Close opens with a gate that refuses to run while a process is still
resident on GPU0, so it cannot evict a training run mid-flight.
Override with --var allow_busy_gpu0=true.

Three defects found and fixed while landing this, all worth keeping:

- Verifying GPU residency via `docker inspect --format {{.State.Pid}}`
  never matches. vLLM V1 runs EngineCore as a child of the container's
  pid 1, and it is the child that holds the memory and that nvidia-smi
  reports. Match by cgroup instead.
- A step's `sudo: true` does not extend to its when/creates/changed_when
  guards, which run as the login user. The root-only .env made an
  unsudo'd grep exit 2, so the GPU-id flip SILENTLY SKIPPED. The
  effective-value assert is what caught it.
- That assert originally grepped the config YAML for -\s*'?1'? and failed
  against compose's double-quoted `- "1"`. Parse the JSON with jq; an
  assert that fails for the wrong reason is worse than no assert.

elway must be invoked as infra-ops@10.250.50.54 rather than the ana-ml2
ssh-target, which resolves to lkraven and has no NOPASSWD sudo.
 2026-08-24 18:39:13 -07:00
vh c507db9ac0 docs(gemma4-erp-tune): size the run against the checkpoint — QLoRA is structurally unavailable 
The proposed shape was QLoRA r64. It cannot be run as specified. The
checkpoint stores each layer's 128 experts as two fused 3-D nn.Parameter
tensors (experts.gate_up_proj [128,1408,2816], experts.down_proj
[128,2816,704] — no .weight suffix, so they are parameters, not modules).
bitsandbytes 4-bit replacement walks nn.Linear only, so 22.84B params /
42.54 GiB — 88.5% of the model — is skipped and stays BF16. load_in_4bit
saves ~3.1 GiB of 48.07 and does not error while doing it.

Verdict: plain LoRA on BF16, ~57.6 GiB at micro-batch 1, +2.5 GiB per
additional 8192-token sequence.

Two sizing items were absent from the brief and both are load-bearing:

- vocab 262,144 x seq 8,192 = 2.147B logits, with final_logit_softcapping
  30.0 adding a saved pre-cap tensor. Naive HF cross-entropy peaks at
  ~28-30 GiB transient at batch 1, which puts the run at ~85.6 GiB on a
  95.6 GiB card — it starts, then OOMs on the first long sample. Fused or
  chunked linear CE is mandatory and must be smoke-proven before a window
  is booked, since Liger may not carry a Gemma-4 MoE patch.
- v_proj does not exist on layers 5/11/17/23/29 (attention_k_eq_v on the
  full-attention layers). A v_proj target silently produces no adapter
  there, and k_proj adapts K and V simultaneously. 45.96M trainable at
  r64 across q/k/v/o.

Placement, measured: GPU0 has 53.46 GiB free beside gen, ~4 GiB short, and
gen's footprint grows with uptime. Stopping mog-sec frees 74.29 GiB on
GPU1, which holds micro-batch 4 at 61.8 GiB with margin for Scriberr.
Recommend standing down sec (2 aliases, last request ~5h ago) rather than
gen (7 aliases, 765 busy-engine log lines in 24h).

Estimated 1.28e18 FLOPs for the epoch at ~3.67B active params; 4-10 hours
at 10-25% MFU. 7,104 packed sequences is only 444 optimizer steps at
effective batch 16, which makes the wall-clock-checkpointing amendment
concrete rather than hypothetical.

Package as a uv venv on /tank: root is 91% full (36 GB) with
/var/lib/docker on it.
 2026-08-24 18:28:30 -07:00
vh 9d0e628643 memory: correct the vLLM version claim — ana-ml2 runs a spread, and 0.27.1 is on disk 
The snapshot recorded "ana-ml2 now runs vLLM 0.26.0". That is true of the
char-rp seat's pin and false of the box, which the operator caught immediately.

Measured per running container: gen is on nightly-311b3513 reporting
0.27.2rc1.dev150, mog-sec on nightly-e9d1398d reporting 0.26.1rc1.dev1102, and
rerank-a3 / coder / reward / embed still on 0.24.0. char-rp and the trainee
bench stack are pinned to v0.26.0. So there is no single "the version" for this
host, and stating one invites exactly the wrong retest.

The correction improves the LoRA question rather than complicating it:
vllm/vllm-openai:v0.27.1 is already on disk and unused — a TAGGED release, not
a nightly, roughly four months past the 0.24.0 where the silent-no-op was
diagnosed. That is the right target for a decision test: no nightly variance,
no pull. The retest instruction in both the decision entry and the handoff now
names it.
 2026-08-24 18:12:35 -07:00
vh 668e590e7d memory: snapshot — char-rp on the Gemma-4 MoE, abliterated trainee staged, QLoRA sizing next 
Captures an evening that ran from an OOM crash-loop to a measured trainee base.

The durable lessons, none of which CLAUDE.md can carry: --gpu-memory-utilization
sizes the KV cache and does not cover CUDA context or graphs, which is half of
why a seat that fit on the 21st stopped fitting on the 24th; the other half is
that gen's footprint GROWS WITH UPTIME (38.5 GiB fresh against 45.6 GiB after
three days, same container, same flag), so headroom arithmetic against a
long-running gen measures a moving number. The stale-chat-template trap turned
out to be endemic across third-party Gemma-4 derivatives rather than a one-off,
and it is silent in both directions — wrong prompt when serving, train/serve
skew when tuning. And a benchmark finding was retracted because 12% on a
five-option task is below the 20% chance floor: a below-chance score indicts the
instrument before the model, and a preflight can be thorough while aimed in the
wrong direction.

Records the serving decision for the tuned model with its history intact:
LoRA-on-NVFP4 is preferred if it works, merged weights the expected fallback,
but the archived root-cause says the objection was never NVFP4-specific — vLLM
0.24.0's LoRA application was a silent no-op proven quant-agnostic, and ana-ml2
now runs 0.26.0. Retest before designing around merge; the answer changes what
Eitri's harness must emit, and he is still early.

Auto-archival moved 5 entries (Recent decisions) to archival-memory.md; the
guards held back the rest of the 78 age-eligible candidates because their bodies
carry open deferred-work language, per the keep-when-unsure rule. Index sits at
286 lines, above the ~250 target and reported rather than forced.
 2026-08-24 15:54:30 -07:00
vh 5415fd4b30 docs(gemma4-charrp): abliteration measured in isolation — close to free, but it MOVES capability 
Second bench window, operator-authorised after an initial decline and reversal.
Stock BF16 against the llmfan46 abliterated BF16: same precision, same pinned
upstream template, same 192 items, CoT off. Abliteration was the only axis that
moved, which is what the previous run could not claim.

Net core cost is 0.6 points — but the headline understates what happened.
Capability MOVED rather than degraded: five items lost on contradiction
detection, four gained on spatial composition, nearly cancelling. A gain was not
predicted by anyone, least of all on that axis.

The decision this was authorised to settle: llmfan46 stands as the trainee base.
No case for re-staging on TrevorJS at KL 0.09 over 0.6 points — the KL gap
between the builds is smaller than the gap this measurement failed to find.

Both limits recorded rather than buried, per brokkr-smithy-dev: the swings are
~5 and ~4 items at n=32, so the -15.6/+12.5 percentages read more precisely than
the measurement supports and only marginals were run; and this says nothing
about quantization, because the stock-NVFP4 T2 figure came from n=16 against
n=32 here — different item counts mean different item sets, so that comparison
is n-confounded and is not being made.

Turnaround was five minutes rather than fifteen because the gemma4-trainee-bench
stack already existed — itself the residue of debugging a 35-restart crash-loop
caused by the production compose hardcoding --quantization compressed-tensors.
The fix outlasted the incident.

gen restored and verified through the gateway; char-rp remains down deliberately;
bench stack env reset to the heretic base for the post-tune gate.
 2026-08-24 15:39:43 -07:00
vh 019ccff7e8 feat(gemma4-trainee-bench): BF16 bench stack; record that gen's footprint grows with uptime 
Adds an ephemeral stack for serving the BF16 trainee base on :8016 under the
char-rp aliases, so the abliterated base can be measured on the same battery and
the same gateway routes as the served seat with no harness edit.

It is a separate stack rather than another variable on gemma4-charrp because
that compose hardcodes `--quantization compressed-tensors` for the NVFP4 build.
Pointing it at unquantized BF16 weights crash-loops immediately —
`TypeError: CompressedTensorsConfig.__init__() missing 3 required positional
arguments: 'target_scheme_map', 'ignore', 'quant_format'` — vLLM trying to read
a quantization config out of a checkpoint that has none. 35 restarts before it
was caught. `restart: "no"` here so a bench seat cannot resurrect itself and
block gen's restore, and no homepage labels so it leaves no permanently-offline
dashboard card.

It cannot coexist with gen and says so: 48.07 GiB of BF16 weights plus gen's
footprint exceeds the 94.97 GiB card before any KV cache. Running it means gen
is stopped.

THE MORE USEFUL FINDING is in the meromero env note: gen's memory footprint
GROWS WITH UPTIME. Measured today at 46,726 MiB (45.6 GiB) after ~3 days up, and
39,424 MiB (38.5 GiB) immediately after a restart — same container, same
--gpu-memory-utilization 0.43, ~7 GiB apart. That is the missing half of this
afternoon's crash-loop: the char-rp seat "fit on the 21st and stopped fitting on
the 24th" because nothing about char-rp changed and gen crept up underneath it.
Headroom arithmetic done against a long-running gen is measuring a moving
number, so the note now says to measure against a freshly-restarted one.

Operator's requested end state reached and verified through the gateway: gen and
summarizer both 200, char-rp down deliberately to hold GPU0 headroom for the
upcoming trainee run, bench seat stopped.
 2026-08-24 14:19:49 -07:00
vh 14ff4a3f57 docs(gemma4-charrp): stage two abliterated trainee bases; record the endemic stale-template trap 
The operator directed that the ERP/RP trainee base be a low-damage abliterated
instruct build rather than the stock checkpoint. Two are now staged under
/tank/aimodels/, both BF16, both unquantized, both matching upstream's 51.61 GB
/ 25.8B shape with only transformers_version differing in config:

  gemma4-26b-a4b-it-heretic-bf16       llmfan46, Heretic v1.2.0 ARA, KL 0.1237, refusals 3/100
  gemma4-26b-a4b-it-abliterated-bf16   TrevorJS, KL 0.09, 1/100 effective and 5/686 cross-dataset

"Low damage" was treated as a measurable claim rather than a description: the
field spreads from KL 0.09 to 0.4118 and the table is in the README so the next
choice is made on numbers. Fleet anchor for reading them — our own abliteration
work found Heretic at KL 0.12 preserved the MTP head at 83.7% acceptance, so
both staged builds sit inside an already-validated band rather than past it.
huihui-ai is rejected despite its reputation: no published metrics, its own card
calls the method a crude proof-of-concept, it abliterates both thinking and
non-thinking modes, and its parameter count runs ~738M over upstream. The
operator's independent read matched.

The more durable finding is the chat template. NOT ONE third-party Gemma-4
derivative pulled here ships upstream's — three independent repos carry the
identical stale 266-line file (sha 58c66fdee4afa297), llmfan46 carries a third
365-line variant, and only the RedHatAI NVFP4 build matches upstream's
6a1015c47ccfcfa6. It propagated through the ecosystem rather than one packager
slipping, and it is now recorded as a class rather than as the single incident
that surfaced it during the A16 control staging.

That matters twice over and silently both times: serving a mismatched template
renders a different prompt, which is why production pins it; and training
through `base/chat_template.jinja` means training on a different prompt format
than production serves — train/serve skew with no error, presenting as a tuning
failure. brokkr-smithy-dev has been warned on the training side while the
harness contract is still early enough to amend.
 2026-08-24 13:52:00 -07:00
vh 8d6a9390de docs(gemma4-charrp): RETRACT the contradiction-deficit claim — the item was ill-posed 
Supersedes what commit 3446367 recorded. That message stated the A16 control
"settles a question" and quoted 12% contradiction detection against gen's 81%.
The quantization half of it stands; the deficit it was measuring does not exist.

brokkr-smithy-dev retracted the finding after the operator asked to see the
individual items. The task presented two mutually contradicting statements and
asked for "the contradicting statement" — but CONTRADICTION IS SYMMETRIC.
Neither statement was more the contradicting one, the model had no way to know
which had been inserted, and it consistently named the absolute claim: a
defensible reading that the labelling scored wrong on every single item.

The tell was there and both of us walked past it: 12% on a five-option task is
BELOW THE 20% CHANCE FLOOR. A below-chance score indicts the instrument before
it indicts the model. That should be the first reaction to a below-chance
result, not a late one, and it is now written into the README as such.

Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (which on a sound instrument does not shrink
but REVERSES); and every pre-fix T2 number for Gemma-4, MeroMero-v2, sec and
gen. A second defect surfaced during the fix — all generators shared one RNG, so
rewriting one task reshuffled every task after it.

What survives is real and worth separating out: the A16 control result holds.
Activation precision is close to free on this workload, every other task
identical across the W4A4 and W4A16 builds. The two staging confounds caught
before the run — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template — were independent of the item defect
and remain load-bearing. On the corrected instrument Gemma leads the very axis
it was suspected of failing (94% against sec's 81% and gen's 50%); its actual
weak axis with thinking off is spatial composition at 69%.

Recorded as a dated superseded-claims table rather than a silent edit, per the
repo's quant-work convention, so notes elsewhere stop misleading people.
 2026-08-24 13:07:58 -07:00
vh 3446367d5e feat(gemma4-charrp): pin the chat template; A16 control run executed and reverted 
The template is now passed explicitly, defaulting to the A4 build's
chat_template.jinja. That is a no-op for what is served — the A4 build ships
that exact file, byte-identical to upstream google/gemma-4-26B-A4B-it once
trailing newlines are normalised — and it permanently closes the class of bug
found while staging the control: the A16 build ships a stale 266-line template
against upstream's 390, with the thinking path built differently and no
`thinking` property in its tokenizer_config response_schema. Serving each build
with its own template would have moved a second axis.

The control ran on the operator's greenlight and has been reverted. Seat is back
on the W4A4 build, healthy, RestartCount 0, both aliases verified through the
gateway — char-rp returns content with reasoning_content empty,
char-rp-reasoning returns both.

Result, since it settles a question this repo's config now encodes: activation
precision does NOT explain the contradiction-detection deficit. Contradiction
detection moved 12% -> 19% between W4A4 and W4A16, which at n=16 is 2/16 -> 3/16
— one item — against gen's 81% on identical items. Every other task is identical
across the two builds and the core difference is 2.6 points carried almost
entirely by two single items. brokkr-smithy-dev pre-registered that a null
result would be the robust branch, because a hidden third axis would tend to
create a delta rather than suppress one, so the conclusion survives the residual
doubt neither side could close without a dequantization pass.

The practical upshot for future scheme choices: W4A4 costs less on this workload
than the caution warranted. The caution was still correct to have.

Displaced production for 3.7 seconds of measurement plus two container
recreates. The A16 build and the BF16 tuning base both stay on disk with the
runbook in the stack README, so re-running is a two-minute flip.
 2026-08-24 13:03:19 -07:00
vh 1bd90eaacc docs(gemma4-charrp): stack README — the three model dirs, the A16 control runbook 
The stack had no README and now carries three model directories that look
interchangeable and are not: the BF16 QLoRA base that cannot be served here, the
W4A4 quant that is served, and the W4A16 build that exists solely as an
activation-axis control. Writing down which is which, and why, before someone
"simplifies" the compose to the BF16 path and rediscovers the OOM.

Also captures the A16 control procedure end to end, including the two confounds
found while staging it — the two Hub repos named NVFP4A16 that declare 4-bit
activations, and the stale chat template the real one ships — and the fact that
overriding the template is safe because the tokenizers are identical. Both sides
have now cross-checked this: brokkr independently diffed every non-quantization
config field of both builds against the upstream BF16 and found only
transformers_version differing. Residual risk recorded rather than hidden:
config identity is not weight identity and nobody has done a dequantization
pass.

The Gemma-4 flags are documented as architecture-level rather than
checkpoint-level, since that is why they survived the seat swap unchanged, and
the enable_thinking:false pin is called out as mandatory rather than stylistic —
without it every plain prose turn lands in reasoning_content with a null content
and every consumer breaks.

Notes the non-termination defect with thinking on (32 of 96 calls truncating at
12k tokens, all 16 constraint items among them, reasoning sound right up to the
point it fails to stop) and why VLLM_USE_V2_MODEL_RUNNER=0 is deliberately not
applied to a seat whose production mode is thinking-off.

No live change: the seat is still serving the W4A4 build. Displacing it for the
control run is an operator decision and is still open.
 2026-08-24 12:34:50 -07:00
vh 24e8826219 docs(gemma4-charrp): the A16 control needs a chat-template override, not just a path swap 
Pre-flighting the staged A16 build before handing it to brokkr-smithy-dev's
battery found a second axis hiding inside what was supposed to be a
single-variable control.

The A16 build ships a STALE chat template. Verified by hash against the upstream
weights on the same disk: google/gemma-4-26B-A4B-it is 390 lines, the RedHatAI
A4 build's is 389 and byte-identical to upstream once trailing newlines are
normalised, and the prithivMLmods A16 build's is 266 and is not. The delta is
not cosmetic — upstream and A4 open the thinking path with
`{%- set enable_thinking = enable_thinking | default(false) -%}` and branch off
it, while the A16 template has no such set and guards with
`enable_thinking is defined and enable_thinking` instead. tokenizer_config.json
corroborates: A4's response_schema carries a `thinking` property, A16's has only
role and content. That build was quantized from an older revision of the
checkpoint.

Served with its own template, the A16 arm would render a different prompt for
identical messages, and a contradiction-detection delta could be attributed to
activation precision when it was the template. That is the same failure class as
the misnamed-A16 repos — a field nobody validated, believed because the name
looked right — one layer further down, and it would have produced a result that
looked like a finding.

Overriding is safe because the tokenizers agree: vocab identical at 262,144
entries, added_tokens identical, so the same template over the same vocab
renders the same token ids. Everything else pre-flights clean — both artifacts
complete with no missing shards, generation_config.json byte-identical.

Seat NOT flipped; displacing production for the bench window is the operator's
call and is still open.
 2026-08-24 12:32:03 -07:00
vh f509668e45 docs(gemma4-charrp): record the A16 activation-axis control and how to run it 
brokkr-smithy-dev's first battery on the new seat scored 12% on contradiction
detection with CoT off, against gen's 81%, while state tracking, deterministic
constraint following, long-context state and the confabulation control all sat
at 100%. That is not general degradation — it is the shape 4-bit input
activations produce on the most reasoning-dense task, which is exactly the
confound flagged when the W4A4 quant was chosen. They have recorded the finding
as CONFOUNDED rather than reporting it as a property of Gemma, and asked for an
A16 build to separate the two.

No quant run was needed: a genuine A16 build of the same checkpoint already
exists on the Hub and is now at /tank/aimodels/gemma4-26b-a4b-it-nvfp4a16
(prithivMLmods, compressed-tensors, nvfp4-pack-quantized, input_activations
null, 17 GB). Same weights, same loader as the live seat, one axis moved.

⚠ Two other repos would have answered the question wrongly and the note in the
env template says so: bg-digitalservices and ManniX-ITA both publish
"Gemma-4-26B-A4B-it-NVFP4A16" whose config declares input_activations num_bits
4 — W4A4 under an A16 name, via modelopt, both 16.46 GB against the real one's
17.93. Using either and seeing T2 stay at 12% would have wrongly convicted the
model.

Running it is a one-line GEMMA4_MODEL swap plus a recreate, because there is no
room for a concurrent seat: GPU0 has 3.5 GiB free with gen and the A4 seat on
it, and GPU1's 19.4 GiB against ~18 GiB of weights is the same sub-GiB headroom
that crash-looped the predecessor this morning. Port and both aliases are
unchanged either way, so no consumer config moves.
 2026-08-24 12:28:17 -07:00
vh 27155c0f3b feat(char-rp): swap the seat to the Gemma-4 26B-A4B MoE, NVFP4, same port 
Straight-across replacement of the dense G4-MeroMero-v2-31B-NVFP4A16 seat with
google/gemma-4-26B-A4B-it on ana-ml2 GPU0. Port, served-model-names and every
gateway route are unchanged, so no consumer sees a difference in addressing:
`char-rp` -> hosted_vllm/char-rp and `char-rp-reasoning` ->
hosted_vllm/char-rp-thinking, both still :8016. The seat's requirements now
include chain-of-thought, which makes throughput more critical rather than less
— the user waits through the whole reasoning block before the first visible
token, and the MoE measures ~114 tok/s @32K against the dense 31B's ~40.7.

Both artifacts are on disk and they are NOT interchangeable. The BF16 weights
(/tank/aimodels/gemma4-26b-a4b-it-bf16, 49 GB) are the QLoRA tuning base, since
QLoRA does its own quantization. They CANNOT be served here: 48.10 GiB of
weights against ~49 GiB of free GPU0 leaves nothing for KV cache, and the
engine would die at allocation exactly the way the predecessor did this
afternoon. The serving copy is RedHatAI/gemma-4-26B-A4B-it-NVFP4 (16 GB),
chosen over the other -it quants because it is compressed-tensors
(nvfp4-pack-quantized) — the same loader path the outgoing seat used — from the
llm-compressor team at 357k downloads. The nvidia/ repo is the base rather than
-it, and the thinking channel lives in the instruction-tuned weights.

Smaller weights at the same 0.47 memory budget buy a much larger KV pool:
27.37 GiB and 1,724,110 tokens, against the predecessor's 371,023 at the same
budget. That is 6.5 full-length 262K sequences concurrent rather than 1.4.

The gemma4 tool-call parser, reasoning parser and the enable_thinking:false
default all carry over unchanged — they are architecture-level, not
checkpoint-level. The --chat-template override does NOT carry over: MeroMero
pointed at a jinja hand-patched against that checkpoint, and this model ships
its own. Verified that dropping it did not reintroduce the failure that flag
existed to prevent — non-thinking prose lands in content with reasoning_content
empty, and the thinking alias populates reasoning_content with content
carrying the answer.

⚠ Scheme differs from the incumbent and the bench should say so: this quant
declares 4-bit input activations (W4A4) where the outgoing seat was NVFP4A16.
Faster, and not like-for-like on the activation axis.

meromero-charrp is retained stopped in `created` state and relabelled to
AI - Dormant, per the house rollback pattern. Both stacks want :8016, so
rolling back means stopping the gemma4 seat first.
 2026-08-24 12:02:59 -07:00
vh 850e0c3351 fix(meromero-charrp): drop GPU0 budget to 0.47 — the seat was OOM crash-looping 
`vllm-meromero-rp` had been restarting since 2026-08-24 18:2x, 13 times by the
time it was looked at, taking both the `char-rp` and `char-rp-reasoning`
gateway aliases down with it (they resolve to the same seat on :8016 —
hosted_vllm/char-rp and hosted_vllm/char-rp-thinking).

Root cause is CUDA OOM on ana-ml2 GPU0, which the startup logs hide well: the
engine gets through weights, torch.compile and CUDA-graph capture looking
entirely healthy, then dies at KV-cache allocation with
`torch.OutOfMemoryError: ... 195.19 MiB is free`.

GPU0 is shared with `vllm-gen`. gen is configured at 0.43 but actually holds
~45.6 GiB of the 94.97 GiB card, because --gpu-memory-utilization sizes the KV
cache and does not account for CUDA context, graphs and non-torch overhead.
This seat was at 0.51, so the pair was committed to 0.94 of the card with about
0.6 GiB of real headroom. That fit on 08-21 and stopped fitting today.

0.47 restores ~4.8 GiB of margin and costs nothing usable: KV cache 27.36 ->
23.56 GiB, 430,825 -> 371,023 tokens against a max-model-len of 262,144, so the
pool still holds 1.4x a full-length sequence. What is lost is concurrent long
requests, not context.

Verified through the gateway rather than at the container: char-rp returns 200
with content, char-rp-reasoning returns 200 with both content and
reasoning_content populated. Seat is healthy with RestartCount 0.

The arithmetic and the "check used_memory, not the flag" warning are written
into the env template, because the next person to raise either budget needs to
lower the other in the same change.
 2026-08-24 11:42:36 -07:00
vh 35adc4a043 feat(homepage): rebuild on Australis Skyfall — dual theme, light mode shipped 
The board was on the Australis TERMINAL palette, which is dark-only by design
("Always dark first. No light mode in this system"). Skyfall is the dual-theme
web derivative of the same science, and its bundle turned out to be sitting in
this repo's own git history: a predecessor vendored it on 2026-08-19 and a
later commit deleted it. `git show 45c1995:...` returns colors.css with both
`:root` (dark) and `[data-theme="light"]` (Skyfall Day) intact, plus the
calm-depth layout tokens, the typography scale and Supreme 400/500/700. So the
light ramp is canonical rather than derived, which was the entire objection to
building one.

The visual language moves with the palette. Depth is now the recipe and not a
choice — every elevated surface carries a 1px hairline AND a two-layer shadow,
never one without the other. Radii move to Skyfall's scale, cards at
--radius-lg. Widget stat values move from the display face to mono, because
Skyfall is explicit that numbers and telemetry are always --font-mono. The
full-width aurora ribbon under the tab bar is gone: Skyfall sanctions exactly
two accent expressions, the active rail and hero-only glows, and a decorative
gradient across the chrome is neither — so the colour it carried now lands on
the active tab as a 2px accent bar plus an --accent-soft fill, which is the
rail. Every binding is written against the semantic layer; there are no raw
family tokens and no colour literals left in our own file.

build.py now guards the vendoring instead of advising it. The three token files
are hashed and a mismatch FAILS the build — a vendored file is either
byte-identical to the bundle or it is a fork wearing the bundle's name, and the
theme this one replaces had to be torn out twice for exactly that.

⚠ Homepage's own theme toggle is unreachable, and reaching for it breaks the
dashboard. It renders only when settings.yaml leaves `theme:` unpinned, and
with the key absent the page's data loader throws and its catch branch serves
`initialSettings: {}` — no tab bar, no layout, no i18n. Six force-recreates
over seven minutes all came up empty; restoring `theme: dark` rendered
correctly on the next recreate in 12 seconds, while /api/services returned 200
with fully correct content the whole time. That is the first confirmed cause of
the long-running "tab bar goes missing after a recreate" symptom, and it also
retires the homepage.log-size lead recorded earlier today: rolling the log
aside did nothing during this episode, so that coincidence was intermittency.

So the toggle is ours. conf/custom.js renders it and stores the choice;
build.py re-emits each vendored light block twice, once for an explicit
`data-theme` and once inside a prefers-color-scheme media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])` — that :not() pair is
what lets a stored dark choice survive a light-mode OS. Verified against both
OS preferences: load, click, click back, reload, all four correct. `data-theme`
is the control surface; Homepage's own `dark` class stays on <html> and does
not fight, because our rules carry !important on the surfaces Tailwind's
`dark:` variants would otherwise claim.

Two font substitutions, both documented rather than silent: Space Grotesk for
Bespoke Sans and JetBrains Mono for Victor Mono. Only Supreme was ever vendored
here and Skyfall's own notes call Victor Mono user-supplied, so this is a
two-line swap when the real faces arrive.

Dark and light, all four tabs: http://10.100.10.50:8090/b/homepage-skyfall/
 2026-08-24 09:44:45 -07:00
vh 39da1d4a97 feat(homepage): recategorise on "do I open this?", collapse the API groups 
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.

Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.

A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.

The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.

28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.

The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.

Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.

Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
 2026-08-24 08:54:06 -07:00
vh f6f2f69649 fix(homepage): uniform 4-column grid, hold the status gutter, unleak AI Systems 
The board's card width changed at every group boundary because `columns:` was
being tuned per group under the 2026-08-18 "columns = member count" rule. That
rule is retired: it sets `lg:grid-cols-N` for one group, so it fixes that
group's CARD WIDTH, not its density. Measured on the live board, Notes rendered
a single 1464px card, News and Media 728px, Eval & Retrieval 286px, everything
else 360px. All 20 groups are now `columns: 4` and every card renders at 360.

`.service-name` reserved a 78px status gutter with `padding-right` and relied
on `overflow: hidden` to hold it, but overflow clips at the PADDING box — the
gutter was spill room the title printed straight through. Six cards on the AI
tab rendered their name underneath their own status pill, measured by testing
the title text node's box against the status cluster's. The intended ellipsis
never fired either: it is painted by whichever block's own line overflows, and
that is the anonymous box around the bare title text node, which does not carry
`overflow`. The gutter holds by wrapping now, and the description opts back out
of it with a negative margin since the pill only ever covers the first line.

Scriberr's `homepage.group=AI Systems` named a group absent from `layout:`, so
it had no `tab:` and rendered on all four tabs — the same defect as UltraSeedbox
in 2026-08-18, arriving this time from a container label. Relabelled to
`AI - Audio Tools`, where the other ASR seats already live. The
`homepage.group=AI Systems` sample in the repo-root CLAUDE.md was the source of
the name and now carries the constraint, plus the fact that a labelled
container is discovered from any of the five configured engines and must not
also be listed in services.yaml.

Also: descriptions clamped to three lines so a five-line outlier stops dragging
its row 50px taller than its neighbours; icon ramp overridden off slate-400 ->
slate-700, which was sinking the bottom half of every glyph into the card fill;
bookmark groups and Jellyfin's trailing stream rows brought into the card and
eyebrow vocabulary; group gap 10px -> 22px now that width no longer separates
them.

The icon override sits on `html[class]`, not `:root`: Homepage sets the same
variables on `.theme-slate`, which is on the <html> element, and a class beats
`:root` on the same element.

Verified with Playwright against the live board — per-group card width, card
height spread, and a geometric title-vs-status collision check, before and
after. Before/after captures: http://10.100.10.50:8090/b/homepage-relayout/
 2026-08-24 08:15:02 -07:00
vh 32349b7653 memory: snapshot — Anaheim tunnels on AES-128, ana-gw admin closed, Scriberr live, ESH DNS fixed 
Rewrites the in-flight section, which had gone stale in one place
(speaches is stopped, not live) and did not reflect a session in which
everything opened was also closed. Three detail files carry the bulk:
the FortiGate's public surface reduced to nothing including the ACME
listener, Scriberr's deployment and the three upstream defects it
required working around, and the ESH DNS repair that also made the
IPv6 naming scheme real on three hosts.

The tunnel entry in the decisions log described a cipher change as
proposed and pending; it has since been attempted, found impossible in
the form intended, and resolved differently, so the line now points at
the outcome rather than the intention.

New decisions cover the cipher adoption and the finding that the
per-flow ceiling belongs to the UniFi gateway's software cipher rather
than the firewall, the administrative closure, the transcription
deployment, the DNS repair, the retirement of an ASR service whose
only consumer was abandoned, and a database VM that until now had no
fleet identity and no vaulted credential at all.

Three approaches are recorded as abandoned: an AEAD cipher the far end
cannot express, an upstream DNS setting the gateway accepts and
ignores, and a scheme to claim unused delegated prefixes that founders
not on the prefixes but on having to rebuild a firewall policy to use
them.
 2026-08-24 07:42:30 -07:00
vh d419b11d43 docs(ipv6): close the NH3 multi-prefix question by operator ruling 
The gateway has no IP-passthrough mode, confirmed by the operator with
admin access to it, so the inexpensive path of letting the UDM take the
delegation whole and carve it natively is unavailable at this site.

What remains is a separate DHCPv6 client presenting several identities,
which needs recabling to reach the gateway, splits routing so that one
device handles v4 and another v6, and above all moves IPv6 off the
UDM's zone firewall, leaving the entire policy to be rebuilt elsewhere
before any host could safely hold a globally routable address. The
operator has declined it and the LANs stay without v6.

The index entry also still asserted the superseded single-delegation
conclusion as fact; it now carries the corrected reading alongside the
ruling, so the finding and the decision not to act on it arrive
together rather than the first inviting a retry of the second.
 2026-08-24 07:05:49 -07:00
vh bd209951ac docs(ipv6): correct the NH3 delegation finding — eight /64s, not one 
The note concluded AT&T delegates a single /64 and that reading the
address pattern as a /60 was a mistake. A later finding in the same
session overturned that by reading the BGW's own statistics page, and
that correction was never written down; it survived only in the
transcript and surfaced again while assessing whether more prefixes
could be claimed.

The gateway holds the /60 and rations it, keeping half for itself and
delegating the top eight /64s one at a time. Both observations agree:
a prefix-id only carves within a delegation already held, so the
earlier test could not have moved a lone /64 regardless. The limit is
that UniFi solicits once, which makes the requester the ceiling rather
than the carrier.
 2026-08-24 07:02:53 -07:00
vh d127e29fac docs(ipv6): retire the next-candidates line now that all three are done 
Replaces it with why the remaining segments have no eligible hosts:
two are appliance-only and three have no IPv6 enabled yet, pending the
firewall-policy pass that SLAAC on a client segment would require.
 2026-08-23 22:39:51 -07:00
vh 4e83395ddf docs(ipv6): all three esh-server hosts now carry the segment name 
esh-pve-nas and esh-vm-db join esh-docker-vm on 4411:b105, at
:50:55 and :50:60 respectively, each applied by the same prefix-deriving
if-up.d hook so the last two groups read straight off the IPv4 address.

Two obstacles are recorded because both will recur. The Proxmox node had
link-local only despite every relevant sysctl appearing correct, because
its bridge carries per-interface forwarding and the kernel ignores router
advertisements on a forwarding interface unless accept_ra is explicitly
two rather than one. The fix takes the advertised prefix while declining
the default route, so the hypervisor gains an address without any change
to how it routes; this was verified after applying, with the v6 default
route count still at zero.

The database VM refuses key authentication for the privileged accounts
and its unprivileged login cannot escalate without a password, so the
hook went in through the QEMU guest agent from the hypervisor, which
executes as root inside the guest. The document notes the base64
indirection needed to get a multi-line script through intact.
 2026-08-23 22:39:39 -07:00
vh ffb7fba346 docs: give the ESH IPv6 naming scheme a home, and make it real on one host 
The scheme has existed since August as a single line of persistent
memory, which a snapshot then deleted. It is a naming convention
rather than temporal state, so it now lives in docs/pfi as a proper
document, and the memory entry is reduced to a pointer at it. The
document carries the full table, the address structure, the reasoning
about which slots can and cannot hold a name, and the recipe for
applying one to a host.

It also corrects the conclusion the original note ended on. That note
held that these names could never appear on the wire, which is true
of everything UniFi is able to assign but not of what a host can
assign to itself, and the distinction is the whole difference between
a joke and an address.

AdGuard on esh-docker-vm now holds the esh-server name, at
2607:73c0:402:1d02:4411:b105:50:45, where the segment identity and the
IPv4 address are both legible. It is applied by an if-up.d hook that
derives the prefix at runtime rather than hardcoding it, backgrounds
itself with a retry so it cannot stall interface bring-up, and adds
nothing to the existing interface configuration.

This is load-bearing rather than decorative. The gateway advertises an
IPv6 resolver to clients, macOS prefers it over the IPv4 one, and it
previously pointed at an address derived from that host's MAC.
 2026-08-23 22:31:13 -07:00
vh 41091eef8f docs: restore the ESH IPv6 naming scheme, dropped by a snapshot 
The six ESH LAN hexspeak names were recorded in fe3d765, refined in
959bb6e and 8be8a51, and then removed without comment by the memory
snapshot in 837fa36. Nothing referenced them afterwards, so the loss
went unnoticed until the operator asked for them tonight and no file
on the machine contained the string.

Recovering it was harder than it should have been, for a reason worth
recording: the table was written in uppercase and git log -S is
case-sensitive, so a history search that would otherwise have found it
immediately came back empty. The entry was eventually located in a
session transcript, which named the commit.

The restored text carries the original verbatim plus two additions. It
is marked as a naming convention rather than temporal state, since
that is what made it eligible for pruning in the first place. And its
conclusion that the names can never appear on the wire is corrected:
the reasoning holds for what UniFi can assign, but a Linux host can
take such an address itself, which was verified on esh-docker-vm along
with a persistence path that leaves the existing interface config and
SLAAC untouched.

The scheme's first practical use is noted against the resolver address
the ESH gateway now advertises, which currently depends on a MAC.
 2026-08-23 22:26:39 -07:00
vh 6217d3993e fix(scriberr): force uv to copy rather than reflink when building envs 
Scriberr builds each model backend's Python environment with uv at
container start. uv's default link mode reflinks or hardlinks out of
its cache, which fails on this overlayfs over ZFS combination and
reports it as a failure to clone a metadata file with errno 11,
resource temporarily unavailable. The wording points nowhere near the
cause.

The damage was partial and therefore easy to miss: WhisperX and
PyAnnote came up fine and the application served normally, while the
Parakeet and Sortformer backends were quietly absent. Setting
UV_LINK_MODE to copy trades a little disk and build time for
environments that actually materialise, and the occurrence count for
that error is zero on the following start.

The knob is exposed through the environment so it can be moved back to
the default if a future host does support reflinks.
 2026-08-23 19:31:54 -07:00
vh efddb4e511 feat(scriberr): stand up transcription on ana-ml2, pinned to GPU1 
Scriberr transcribes audio and video locally with WhisperX and
speaker diarization, and it lands on ana-ml2 rather than ana-docker
because the work is GPU-shaped: ana-docker offers eight cores already
shared with fifty containers and thirty-seven gigabytes of disk,
against ninety-six cores, terabytes on /tank and idle capacity on
GPU1. The reservation names device 1 explicitly, since GPU0 is fully
committed to the gen seat, and the container is confirmed to see that
card alone.

The image is built from source, which is not a preference. These are
Blackwell cards at sm_120; the published CUDA image covers Pascal
through Ada only, and the blackwell image the upstream README
documents has never been published at all. The path upstream actually
ships for sm_120 is Dockerfile.cuda.12.9, carrying CUDA 12.9 and cu128
torch, so that is what gets built. The compose header says so, because
the obvious cleanup is to swap in the published image and that would
silently drop the deployment to CPU.

Two configuration details are load-bearing and documented where
someone would go to change them. The application runs as uid 10001
rather than the usual 1000: that Dockerfile moves its user aside for
Ubuntu 24.04's own uid-1000 account and chowns /app accordingly, while
the entrypoint's remapping covers only the data directories, so at
1000 the process cannot open its database and restarts forever behind
a SQLite error that reads as though the machine were out of memory.
Secure cookies stay off while the service is reached over plain HTTP,
or sessions are dropped by the browser and login appears to loop for
no visible reason.

Storage is bind-mounted onto /tank because model weights run to
several gigabytes and the root pool on that host is nearly full.

Also adds the scriberr service alias to internal DNS, following the
existing alias convention so consumers name the service rather than
the box.
 2026-08-23 19:28:33 -07:00
vh 22ae9cd480 ops(ana-gw): disable ACME and retract the all-port VIP claim 
The ACME client is unbound from wan1, which removes the HTTP-01
challenge listener that had been holding port 80 open irrespective of
allowaccess. An external sweep of fifty-five ports against the WAN
address now finds nothing open at all, while the internal interface
still serves the GUI and SSH and retains its certificate, which
remains valid until late October and simply stops renewing.

The previous note's claim that four virtual IPs were unrestricted
all-port static NATs is withdrawn. A FortiOS virtual IP can be scoped
either by an explicit forwarded port or by a service binding on the
object, and the earlier parse inspected only the former. All four use
the latter, and the custom services behind them are narrow. None of
the fourteen is unrestricted.

Ground truth from outside is recorded in place of further config
reading, listing what each public address actually exposes. Three
configured mappings answer nothing at all and are noted as tidy-up
candidates for the migration rather than as exposure.
 2026-08-23 16:02:20 -07:00
vh f39b66d2e1 ops(ana-gw): port 80 on the WAN address is the FortiOS ACME listener 
The previous note attributed the open port to an ISP transparent
proxy. That was wrong. It terminates on the FortiGate: system acme is
bound to wan1, and FortiOS opens port 80 there to answer HTTP-01
challenges regardless of what allowaccess permits, which is why the
port stays open with the interface set to ping only. Every
non-challenge request returns a fixed 403 whose body reads ACME Access
Only. No DNAT is involved; of the fourteen virtual IPs only two land
on that address, neither on port 80.

The wrong conclusion came from a sniffer filtered on dst host, which
matches inbound packets alone and so excluded the replies being looked
for. Filtered bidirectionally the box is plainly seen emitting SYN-ACK
on port 80. The note records the rule.

Two consequences follow. The earlier warning that certificate renewal
would fail without http in allowaccess is retracted, since FortiOS
opens the challenge port itself. And the listener is not an
administrative surface, though its value is now marginal with WAN
administration closed, so the note records how to remove it and leaves
that decision open.

Also captured: four virtual IPs are all-port static NAT and map every
port of their external address.
 2026-08-23 15:53:55 -07:00
vh 7bc9754e40 ops: adopt AES-128 on both Anaheim tunnels and close public admin 
Both tunnels now negotiate AES-128 for ESP, applied make-before-break
so neither dropped waiting on its far end: the FortiGate was widened
to accept the new cipher alongside the old one first, then each UniFi
gateway was flipped. Single-stream throughput moves from 245 to 270
on the NH3 tunnel and from 268 to 304 on the ESH tunnel. Both network
objects were diffed against pre-change snapshots and the only field
that moved on either is the ESP cipher.

The proposal lists are left accepting AES-256 as well. The peers offer
only AES-128 so the extra entries are inert, and retaining them means
a gateway reverting cannot strand a tunnel.

With that up, the WAN administrative surfaces are closed. The
interface is back to permitting only ping, and the infra-ops account
is again restricted to RFC1918 space. Ports 443 and 22 were confirmed
closed from two separate sites and management over the tunnel still
works. The close was issued over the tunnel rather than over the WAN,
since withdrawing SSH from the interface while connected through it
would sever the session mid-command. The box now has no out-of-band
path, which the memory records explicitly.

Also captured: the two UniFi vault items have different shapes, one a
bare key and one a documentation note requiring extraction, which
produces an opaque nginx rejection if missed, and the ESH key's first
confirmed write.
 2026-08-23 15:44:32 -07:00
vh 6edebe4864 ops: forwarding through a downstream WireGuard terminator is free 
FortiOS has no WireGuard, so any WireGuard site-to-site has to
terminate behind the edge, and ana-wg already fills that role. The
earlier 767 Mbit/s figure was taken with traffic terminating on that
box, which left the forwarding case unmeasured and overstated what a
real deployment would see.

Measured properly, transit from NH3 through the tunnel and onward to
ana-docker returns 763.8 Mbit/s on one stream and 790.4 across eight,
so the forwarding hop costs nothing and the full threefold gain over
IPsec survives. The container uses around a fifth of four cores at
that rate, against the UniFi gateway spending a third of its own four
to move a third as much.

The note records what still needs deciding rather than presenting this
as ready to build: Anaheim hosts have to route to the terminator
directly or pay a hairpin through the edge, which was not measured;
the terminator introduces a failure mode the edge did not have, with a
fallback route as the mitigation; and the NH3 end needs its own
terminator, where a Linux host matches the measured figure while the
existing UniFi WireGuard server would likely land lower.

The test tunnel, its peer and the temporary route were all removed.
 2026-08-23 15:25:05 -07:00
vh 062215e81a ops: the tunnel's per-flow limit is the UDM's software AES-CBC 
Varying only the cipher settles what inference could not. AES-128 has
ten rounds against AES-256's fourteen, so a software-bound path must
speed up when the cipher gets cheaper while a hardware-offloaded one
will not move. Run as A/B/A on a single stream, AES-256 returns 232,
245 and 243 Mbit/s and AES-128 returns 282 and 275, a gain of about a
fifth at identical CPU. The FortiGate's offload engine is therefore
not the constraint, and the operator's reading of the UniFi side as
software crypto without acceleration is correct.

Two earlier claims of mine were wrong and are corrected in place. The
observation that the UDM sat at seven percent CPU was a sampling
artifact, since UniFi refreshes that statistic on the device report
interval and the windows used were four seconds; a sustained run reads
thirty-five percent, around one and a half of four cores. The
per-session offload hypothesis is refuted rather than merely unproven,
because it predicts no response to a cipher change.

The remaining figures follow from this. One stream is one core, eight
streams engage about three, and aggregate stops responding to cipher
choice once several cores are working because the path bounds it. CBC
chaining is what makes this expensive, as it serialises blocks and
prevents the AES instructions from pipelining, which is also why the
same gateway manages far more over WireGuard.

The cipher was restored and the network object verified unchanged
against its pre-test snapshot.
 2026-08-23 15:11:51 -07:00
vh 8ecffa1fab ops: WireGuard over the same path carries one stream at 767 Mbit/s 
The operator asked for a WireGuard test between ana-wg and NH3 across
the public internet, which is the arrangement that separates the path
from the crypto since neither gateway performs encryption in it. A
single stream returns 767 Mbit/s against 245 for IPsec over the same
wire, and eight streams return 763, so one flow already saturates the
path and there is no per-flow penalty whatsoever.

That settles several things at once. The limit is not the circuit,
the NH3 uplink, the ISP or the physical path, all of which carry 767
on one flow. Ranking the implementations, Linux WireGuard shows no
per-flow penalty, UniFi's WireGuard shows roughly 1.4x, and IPsec on
this pair shows 2.8x. Latency under a single bulk stream tells the
same story, rising to 12.7 ms on WireGuard against 102 ms on IPsec.

This overturns the earlier conclusion that changing transport was not
worth pursuing, which compared eight-stream figures and so understated
the gap for single-stream work by a factor of three. A WireGuard
site-to-site terminated on ana-wg now looks worth considering, and the
note records the open questions around failover and policy.

Attribution between the FortiGate and the UniFi gateway remains
unresolved, since both perform IPsec in the slow measurements, but the
remedy does not depend on which one is responsible.

The test tunnel was removed, ana-wg is back to its original three
peers and the generated keys were shredded.
 2026-08-23 14:57:29 -07:00
vh d42e9d8712 ops(ana-gw): establish the per-flow cap is IPsec-specific, not capacity 
The operator asked whether the 80F is simply out of capacity. It is
not. Routing a single flow between two Anaheim VLANs through the same
box, with no tunnel involved, sustains 940 Mbit/s, which saturates the
link, and eight flows over that path return the same figure. There is
therefore no per-session ceiling in the plain forwarding path and the
roughly 250 Mbit/s per-flow limit belongs to the IPsec datapath alone.

Measuring the second tunnel with the same probe supports this. The ESH
side, terminating on different gateway hardware at nearly half the
round-trip time, returns 268 Mbit/s on one stream against NH3's 245,
and 715 against 692 across eight. A window-limited path would have
been substantially faster at the shorter round-trip, so the agreement
between two dissimilar far ends is further evidence of a rate cap.

This also corrects the previous note, which named the FortiGate on
reasoning that could not separate the box from the protocol, since
every slow path was both. That separation is still not established:
both tunnels terminate on UniFi gateways running common firmware, so
the cap could belong to either side. The note records the tunnel
topology that would decide it and flags the bearing on the pending
replacement.
 2026-08-23 14:48:34 -07:00
vh cf0cb2cbb3 ops(ana-gw): pin down what limits a single stream across the tunnel 
The earlier note described the constraint as per-flow serialisation
without evidence. Pinning SO_SNDBUF and sweeping it shows what it
actually is: throughput holds flat between 224 and 247 Mbit/s across
a thirteenfold range of in-flight data while round-trip time scales
with the buffer instead, from 7.8 ms up to 107 ms. Retransmissions
are absent throughout. That is a fixed service rate with a standing
queue ahead of it, so socket buffer, window scaling and congestion
control are not the lever and should not be pursued.

The same measurements surface something with wider consequences than
throughput: a single bulk stream lifts tunnel latency from 6.9 ms to
102 ms average, 136 ms peak, with no loss. Interactive traffic
sharing the Anaheim link degrades sharply whenever anything moves
bulk data, and parallelising transfers makes that worse rather than
better.

Localisation rests on the FortiGate appearing in both slow paths and
in neither fast one, with aggregate throughput over the same security
association reaching 692 Mbit/s. Per-session NPU offload fits the
shape but was not confirmed on the device; the note records the test
that would settle it and flags the bearing this has on the pending
FortiGate replacement.
 2026-08-23 14:42:17 -07:00
vh e41d19f1cb ops(ana-gw): close out the Anaheim tunnel cipher question 
The AES-GCM cutover was authorised and attempted, NH3 side first. It
cannot be completed: UniFi's manual site-to-site IPsec implements no
AEAD cipher. Eight GCM spellings were rejected with
api.err.InvalidPayload while an otherwise identical body carrying
aes256 returned rc:ok, which isolates the enum rather than the
request as the cause. The accepted set is aes128/aes192/aes256/3des.
Both Anaheim tunnels terminate on UniFi gateways, so this blocks the
ESH tunnel on the same grounds.

Measuring while testing also retires the premise. NH3's uplink is a
1 Gbps link, so Anaheim's 2 Gbps circuit was never the relevant
ceiling, and the tunnel sustains 692 Mbit/s across 8 streams rather
than the ~550 previously recorded from a 4-stream run. Compared with
WireGuard over the same gateway and uplink, the deficit narrows from
2.3x at one stream to 15% at eight, so moving this link onto a
different transport is not worth doing.

The constraint is per-stream, around 245 Mbit/s, with both endpoints
idle under load. Parallelising bulk transfers remains the mitigation
and is worth 2.8x at no cost; NFS nconnect is the equivalent lever
for single-stream mounts such as /mnt/smithy on ana-ml2.

FortiGate phase2 for pfi-ana-nh3 keeps the widened proposal list,
which is inert while the peer offers only CBC and avoids a further
renegotiation. The UDM network object was diffed field-by-field
against its pre-change snapshot and is unchanged.
 2026-08-23 14:29:17 -07:00
vh 5af362e9d0 ops(ana-gw): restore WAN admin access ahead of the FortiGate cutover 
Re-open the ana-gw admin GUI on wan1 so the Anaheim edge can be
managed remotely if the cutover goes wrong, reversing part of the
2026-08-12 lockdown. Two config changes, nothing else (verified by
diffing pre/post `show full-configuration`):

- wan1 `allowaccess ping https` — https only; http, ssh, and fgfm
  stay off, and wan2 is untouched.
- `infra-ops` trusthost widened to all routable IPv4; the `admin`
  account stays locked to 10.0.0.0/8 so the guessable username
  remains unreachable from the internet.

Verified end-to-end from two sites: a real `/logincheck` POST returns
AUTH OK over the public path, on a browser-trusted Let's Encrypt cert
for ana-fw.phasefinal.com valid through 2026-10-27.

Two FortiOS behaviours worth recording, both of which cost time here:
a trusthost whose base address is 0.0.0.0 is silently treated as
unset (so there is no writable "any" — only decomposed ranges), and
trusthost is enforced before the TCP handshake, so a blocked source
sees a filtered port rather than a refused login.

Follow-ons captured in memory, not actioned: ACME renewal for the
admin cert needs port 80 on wan1 (next attempt ~2026-09-27), and a
~5 SYN/s source in 179.51.184.0/21 now draws SYN-ACKs at no
measurable CPU cost.
 2026-08-23 14:03:40 -07:00
vh b6340519bc memory: snapshot — Anaheim tunnel at 25% of circuit; selene retired; hrafn CI fixed 
Session captured for a context reset. Six new detail entries.

THE OPEN ITEM: Anaheim's IPsec tunnel delivers ~550 Mbit/s aggregate against a
circuit measured at 2,153 Mbit/s. Not WireGuard (it is IPsec on ana-gw), not
CPU (idle), not crypto exhaustion (NPU-offloaded), not the fibre. Both tunnels
negotiate aes256-sha1; AES-GCM proposed. Operator signalled authorization;
execution pending, untracked by operator choice.

Also recorded: selene retired after losing a head-to-head on its own job with
chat-judge moved to gen and the model name left to 404; the 7-alias collision
on the gen seat that makes cross-alias corroboration an echo; hrafn adopted and
its CI found to have been reporting green while deploying nothing for its whole
life; all three Worldtree instances de-armed from a 69-day-stale :latest and
the Matrix homeserver re-plumbed to personal; every secret-bearing .env on
ana-docker tightened to 0600; the pfi org closing the repo-creation half of the
credential-migration directive; booth kept-board deletion and link pruning.

Two entries under Tried and abandoned: the CI checkout assertion that broke the
pipeline twice and was removed, and my proposal to alias a retired model name
at a different model, which the operator correctly overruled.

Index 271 -> 282 lines, under the 300 cap, so no archival fired.
 2026-08-23 13:35:48 -07:00
200 changed files with 23345 additions and 3453 deletions
Expand all files Collapse all files
.mcp.json
+11
View File
@@ -0,0 +1,11 @@
{
"mcpServers": {
"remote-ssh": {
"command": "node",
"args": ["/home/lkraven/development/remote-ssh-mcp/dist/index.js"],
"env": {
"SSH_MCP_STRICT_ALLOWLIST": "1"
}
}
}
}
CLAUDE.md
+100 -7
View File
@@ -21,14 +21,56 @@ lags the code defeats its own purpose.
**New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.
**For SSH-driven work: use `scripts/elway`.** Write a playbook under
**SSH work splits two ways. Pick by whether you are CHANGING something or
LOOKING at something — neither tool replaces the other.**
**Changing things → `scripts/elway`.** Write a playbook under
`playbooks/<name>.yaml` and run
`scripts/elway <host> --playbook ...` instead of chaining
`ssh -t host 'sudo …'` commands — handles sudo once lazily,
structured pass/change/fail reporting, idempotency via
`creates:` / `when:` / `changed_when:`. Template:
`creates:` / `when:` / `changed_when:`, and file upload. Template:
[`playbooks/elway-smoke.yaml`](playbooks/elway-smoke.yaml).
**Looking at things → the `remote-ssh` MCP server.** Registered project-scoped in
[`.mcp.json`](.mcp.json); tools `ssh_hosts` / `ssh_open` / `ssh_run` / `ssh_peek` /
`ssh_interrupt` / `ssh_list` / `ssh_close`. It holds a **persistent shell**, so
`cd` and exported variables survive between calls and each command costs ~6 ms
instead of a fresh handshake. Reach for it for reconnaissance — the
`docker ps` / `grep` / `curl` questions where authoring a playbook is ceremony.
It deliberately has **no file transfer and no idempotency**; that is elway's half.
- Source: `~/development/remote-ssh-mcp` — our fork of
[the-nine-nation/remote-ssh-mcp](https://gitea.phasefinal.com/vh/remote-ssh-mcp)
(MIT). `.mcp.json` points at the **built** `dist/`, so after editing the fork
run `npm run build` or the server keeps serving the old code.
- ⚠ **The host allowlist is the only real containment boundary.** The denylist
is three regexes and a `bash -c` walks past it. We run with
`SSH_MCP_STRICT_ALLOWLIST=1`, which makes `~/.config/remote-ssh-mcp/config.json`'s
`allowedHosts` **authoritative** rather than additive — without it, every one of
the 18 `Host` entries in `~/.ssh/config` is reachable. Widening that list is a
deliberate act; do it in the config file, not by relying on discovery.
- **Raw IPs work — no alias needed.** Operator ruling 2026-09-05: requiring a host
to be registered before you can poke at it is the opposite of ad-hoc, and the
predictable result is that you use raw `ssh` instead. So `ssh_open` takes an
address inside `allowedNetworks` (`10.0.0.0/8`) and connects as
`defaultUser=infra-ops` with `~/.ssh/infra-ops_ed25519`, `hostKeyPolicy` set to
`accept-new`. Aliases still work and are still required outside those networks.
- ⚠ **`deniedNetworks` carves out the SureFire tenant hosts** — `10.250.150.0/24`,
`10.250.250.115`, `10.250.250.110`. Deny beats allow, so widening the allow list
later cannot re-expose them. They are client property under the hosting
agreement; coordinate before touching, which is a contractual posture and not a
security one. `pfi-pve` (10.250.250.31) is deliberately NOT caught by this — the
denies are host-specific, not a /24 over shared space.
- ⚠⚠ **A bare `sudo` HANGS the session — always pipe it: `sudo -n <cmd> | cat`.**
`ssh_run 'sudo -n whoami'` returns `running` forever with zero output and the
session is then permanently `busy`; the piped form returns everything. Measured
on both macOS and Debian, so it is the harness and not a platform quirk. Cause:
sudo ≥1.9.14 defaults `use_pty` on and relays through its own PTY; with the run
frame's stdin on `/dev/null` and stdout on the session PTY the relay never
completes, so the completion marker never arrives. A pipe makes stdout a non-tty
and sudo skips the PTY. Not yet fixed in our fork.
**Task visibility via task-board.** If the Claude Code session has
the `task-board` plugin enabled (installed from
`git@gitea.phasefinal.com:vh/task-board.git`), a card at
@@ -62,6 +104,39 @@ repeats the playbook, you are re-litigating — record the delta in the playbook
instead. When a playbook claim turns out wrong, don't just fix it: add a dated
row to its superseded-claims table so old docs stop misleading people.
## Training runs
Same contract as quantization, different subject:
**`docs/pfi/training-throughput-playbook.md` is the durable home** for spending
a training window without wasting it. Two halves, and you want different ones at
different moments:
- **§1–§3, why a run is SLOW** — the 10-minute scaling triage that names the
regime before you profile, the padding/masking landmines, the profiler traps,
the serving-path and base-viability pre-flights. Read before hypothesising
about kernels.
- **§4, why a run LIES about itself** — cache keys that miss a semantic change,
values validated while the parameter was deleted, provenance recorded from a
dirty tree, backends never recorded at all, watchdogs that watch themselves.
**Read §4 before you launch**, and run its §4.8 checklist. Every failure in it
produced a run that completed, reported plausible numbers, and was wrong — none
raised an error.
(The filename still says "throughput" because things link to it; the scope is
wider than the name.)
The instruments are committed at [`scripts/training-probes/`](scripts/training-probes/)
with raw output kept alongside, so the claims can be re-derived rather than
taken on faith.
⚠ **Measure before you argue.** The playbook exists because a four-model
frontier panel produced four self-retractions in ninety minutes on this
question, and every one of them was a derivation while every survivor was a
measurement. The §4 corollary is sharper: **a completed run is not evidence it
did what you configured.** Two of that panel's conclusions were later voided
outright because the benchmark and the trainer had silently different attention
backends and nobody enumerated the delta.
## Purpose
- Inventory of servers and their state
@@ -98,15 +173,32 @@ Observed and standardized across servers:
- **Named volumes** for service state (pattern: `<stack>_<name>`)
- **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required
- **Restart policy:** `restart: unless-stopped` for daemons
- **Homepage labels** on user-facing services:
- **Homepage labels** on user-facing services. The dashboard runs on
`esh-docker-vm` and reads the Docker API of **every** host in
`stacks/homepage/conf/docker.yaml` (ana-docker, ana-ml2, nh3-docker,
irv-ml1, esh-docker-vm), so a labelled container is discovered from
wherever it runs — you do not add it to `services.yaml` as well. Doing both
renders it twice.
```yaml
labels:
- homepage.group=AI Systems
- homepage.group=<ExistingGroup>
- homepage.name=<ServiceName>
- homepage.icon=mdi-<icon>
- homepage.description=<short>
- homepage.href=http://<host-ip>:<port>
```
⚠ **`homepage.group` must name a group that already exists in
`stacks/homepage/conf/settings.yaml`'s `layout:` block.** A group the layout
has never heard of gets no `tab:`, and Homepage renders an untabbed group on
**all four tabs**. Inventing a group name here is how Scriberr's
`AI Systems` ended up repeated at the bottom of every tab from 2026-08-23
(fixed 2026-08-24). If the service genuinely needs a new group, add the group
to `layout:` **with a `tab:`** in the same change.
Check with `curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name'` —
anything in that list that is not a key in `layout:` is leaking onto all
tabs right now.
Labels only apply at container **creation**, so a label edit needs
`docker compose up -d <service>`, not `restart`.
- **Healthchecks** on services that expose HTTP
## Servers
@@ -114,7 +206,7 @@ Observed and standardized across servers:
| Name | IP | Site | Role | Details |
|------|-----|------|------|---------|
| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
| irv-ml1 | 10.100.79.3 (WG) | Irvine — reachable only via WireGuard tunnel from NH3 | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
| ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
| pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
| ana-filebot | 10.250.50.53 | Anaheim | LXC on pfi-pve (CT 112) — file-task automation | `servers/ana-filebot/README.md` |
@@ -130,7 +222,7 @@ Observed and standardized across servers:
| corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` |
| nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` |
| nh3-dev | 10.100.10.50 | NH3 | Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host | `servers/nh3-dev/README.md` |
| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); **sudo-less** infra-ops identity (user-level only, no Docker); successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); infra-ops is user-level and has no Docker, but **does have NOPASSWD sudo** — measured 2026-09-05, `sudo -n whoami` → `root`; this row previously said "sudo-less", which was wrong; successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
| nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` |
| nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` |
| pbs-nh3 | 10.100.50.90 | NH3 | Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana | `servers/pbs-nh3/README.md` |
@@ -143,7 +235,8 @@ Observed and standardized across servers:
**Placement rules:**
- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1`
(secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
to `irv-ml1` requires WireGuard.
to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
dormant post-cutover; `10.100.79.3` no longer answers).
- Anaheim non-GPU services → `ana-docker`.
- NH-site non-GPU services → `nh3-docker`.
- ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
archival-memory.md
+1526
View File
File diff suppressed because it is too large Load Diff
dns/internal.yaml
+17 -5
View File
@@ -75,16 +75,21 @@ hosts:
# ---- nh3: NH3 office ----
- {name: nh3-docker, site: nh3, v4: 10.100.50.40, note: general-purpose docker host + AdGuard}
- {name: pfi-gx10, site: nh3, v4: 10.100.50.60, note: GB10 Grace Blackwell, 121 GB unified — DHCP RESERVATION on the UDM, not a host static, so the box stays portable}
- {name: nh3-dev, site: nh3, v4: 10.100.10.50, note: dev box, fleet sidecars, Claude sessions}
- {name: nh3-extdev, site: nh3, v4: 10.100.50.42, note: manager / external-dev box}
- {name: nh3-nas, site: nh3, v4: 10.100.50.50, note: Synology RS2418+}
- {name: nh3-pve, site: nh3, v4: 10.100.250.60, note: Proxmox hypervisor}
- {name: pbs-nh3, site: nh3, v4: 10.100.50.90, note: Proxmox Backup Server — DR mirror}
- {name: nh3-gw, site: nh3, v4: 10.100.0.1, note: UniFi UDM Pro SE — gateway + controller}
# Irvine is not its own zone: irv-ml1 is reachable only through NH3's
# WireGuard tunnel and is numbered out of NH3's 10.100.79.0/24, so it is
# named under nh3. Revisit if Irvine ever becomes a site in its own right.
- {name: irv-ml1, site: nh3, v4: 10.100.79.3, note: GPU host (Irvine, via WG) — 3090 + A6000}
# Irvine is not its own DNS site: it is a single box (irv-ml1) reached over
# the headscale mesh, so it is named under nh3 rather than getting its own
# resolver. Its real LAN is 10.6.110.0/24 (Irvine), advertised fleet-wide as
# a mesh subnet route — NOT NH3's 10.100.79.0/24 (that wg0 lifeline is
# dormant post-cutover 2026-09-06). The mesh node IP is 100.64.0.6; we resolve
# to the LAN IP to match the host→LAN-IP model every other entry uses.
# Revisit if Irvine ever grows beyond this one host.
- {name: irv-ml1, site: nh3, v4: 10.6.110.50, note: GPU host (Irvine, via headscale mesh) — 3090 + A6000; mesh node 100.64.0.6}
# ---- esh: ESH home lab ----
- {name: esh-docker-vm, site: esh, v4: 10.0.50.45, note: general-purpose docker host + AdGuard}
@@ -99,12 +104,19 @@ hosts:
- {name: plex, site: esh, v4: 10.0.50.56, note: media server}
- {name: jellyfin, site: esh, v4: 10.0.50.57, note: media server}
- {name: brother, site: esh, v4: 10.0.90.125, note: Brother printer}
- {name: slzb-mr1u, site: esh, v4: 10.0.90.10, note: SMLIGHT SLZB-MR1U Zigbee/Thread coordinator (esh-iot VLAN 90, PoE on ESH-Media p4); Home Assistant at 10.0.50.46 connects to tcp/6638}
- {name: ana-scale, site: ana, v4: 10.250.50.45, note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16}
- {name: nh3-headscale, site: nh3, v4: 10.100.50.45, note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)}
- {name: nh3-scale, site: nh3, v4: 10.100.50.46, note: CT107 on nh3-pve — subnet router for 10.100.0.0/16}
- {name: esh-scale, site: esh, v4: 10.0.50.65, note: CT108 on esh-pve — subnet router for 10.0.0.0/16}
# Service aliases — a name that points at whatever host currently runs it, so
# consumers reference the SERVICE rather than the box. Changing where something
# runs becomes a one-line edit here instead of a hunt through configs.
aliases:
- {name: searxng, site: ana, target: ana-docker, note: replaces searxng.pfi.local (.local is mDNS-reserved)}
- {name: searxng, site: nh3, target: nh3-docker, note: moved off ana-docker 2026-09-03 — colo egress (38.120.12.42) is CAPTCHA-gated by search engines; NH3 egresses residentially}
- {name: gateway, site: ana, target: ana-docker, note: LiteLLM gateway :4000}
- {name: booth, site: nh3, target: nh3-dev, note: The Booth :8090}
- {name: homepage, site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
- {name: scriberr, site: ana, target: ana-ml2, note: transcription + diarization :8080 (GPU1)}
docs/pfi/erp-dpo-stage-prep.md
+227
View File
@@ -0,0 +1,227 @@
# ERP tune — DPO stage readiness
_Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a
**readiness survey and a decision surface**, not a plan. The plan needs the
operator and brokkr-smithy-dev; what follows is what is on disk, what is
missing, and what has to be decided before anyone writes a config._
## Why there is a DPO stage at all
Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct**
release, not an abliteration. Nothing in it attempted to remove refusals, so:
> **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors
> probe) and that is the design, not a regression.** Run 1's 29 came from its
> abliterated base, which run 2 deliberately does not use.
Targeted DPO is the stage where refusals get pruned — **on axes we choose**,
keeping the general-harm ones. That is the whole reason the stock base was
picked over a third-party abliteration: it replaces "inherit a stranger's
judgement about what to strip" with "strip what we decide to, deliberately."
Trading one training stage for that control was the operator's call on
2026-08-25.
## What is ready
| piece | state |
|---|---|
| **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` |
| **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point |
| **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down |
| **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
| **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` |
| **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin |
## What is missing, in order of how much it will hurt
### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole
**Nothing on disk targets this.** The Gutenberg sets are prose-quality
preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
to be generated: prompt the SFT tune, collect the refusal as `rejected`, and
produce a `chosen` completion that engages.
Three sub-problems, none of them mechanical:
- **Where does `chosen` come from?** A model that will write it, a human, or a
template. Each has a different failure mode, and a templated `chosen` teaches
the template.
- **How many pairs, and how balanced?** Too few and nothing moves; too many on
one axis and it generalises past the axis.
- **What is the held-out set?** DPO can be evaluated by preference accuracy on
held-out pairs, and that has to be split before generation, not after.
### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything
Which refusals get pruned, and which are kept. Nobody should generate a single
pair before this is written down, because the axis list *is* the data spec.
R19's track-2 map is the existing artifact that names candidate axes. The
domain-compliance probe scaffold at
[`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py)
measures the RIGHT axis — explicit fiction — as opposed to the
`harmful_behaviors` probe that run 1 measured, which was the wrong axis and is
why run 1's refusal number was never the number anyone wanted.
⚠ **Measure the axis before pruning it.** Run the domain-compliance probe
against run 2 first: pruning an axis the tune already complies on is wasted
work, and the pre-number is the only baseline that will ever exist.
### 3. ⭐⭐ A DPO trainer — not present
`trl` is **not installed** in `/tank/erp-tune/venv`. Two routes:
- **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a
dependency into a venv whose torch/transformers/peft versions are load-bearing
and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly
what a version bump does to you.
- **Implement the DPO loss in the existing harness.** More work, but keeps the
corpus verification, encode cache, order manifest, provenance and resume that
the SFT harness already has and TRL does not know about. The harness is not
SFT-specific except for the loss function.
⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A
dependency resolution that upgrades `transformers` under a live process can
break its save path, which is the most expensive moment to break.
### 4. ⭐ Provenance for a preference run
The manifest gains fields the SFT one has no concept of: preference-set sha,
pair count, per-axis balance, beta, and the reference-model identity. Same rule
as everything else — see playbook 4.3's audit table. **A DPO run that does not
record which axes it pruned is unauditable**, and this stage is the one where
that matters most.
## ⚠ Measured 2026-08-26 — a length artifact the DPO pairs would inherit
The run-2 gate found tuned rp turns at **88.5 / 90.9 words against the base's
137.1 / 145.4** — a 36% reduction, consistent across blocks. brokkr-smithy-dev
hypothesised the mix was teaching PIPPA's product clip. The corpus side is now
measured and it is confirmed, with a mechanism sharper than "PIPPA is in the
mix":
root bot turns p50 p90 p99 max <=123 124-130
pippa 79,414 67 95 108 123 100.0% 0.00%
bluemoon 1,170 563 908 1432 1924 0.0% 0.00%
c2-logs-32k-llama3 23,704 234 394 686 2000 6.6% 1.44%
creative-writing-multiturn 8,621 204 485 858 1837 24.0% 2.16%
**PIPPA's max is 123 exactly, 100% at or under it, 0.00% in the 124-130 band.**
That is a product clip (2023 Character.AI), not a length preference — every
other root crosses its own p99 smoothly.
The exposure asymmetry is the finding:
root turn share word share mean len
pippa 70.3% 37.5% 67
c2-logs-32k-llama3 21.0% 42.5% 256
creative-writing-multiturn 7.6% 15.1% 251
bluemoon 1.0% 4.9% 601
**PIPPA is 70.3% of bot TURNS but 37.5% of bot WORDS — and length is learned per
turn, not per token.** Each of those 79,414 turns is a separate demonstration of
where a bot reply ends. By loss tokens PIPPA looks like a third of the dialogue
signal; by end-of-turn demonstrations it is seven in ten, from a source that
cannot exceed 123 words.
⚠ **Generalises past PIPPA: a length-clipped root is over-represented in the
length signal by exactly the ratio its clipping creates.** The shorter the clip,
the more turns per token of budget, the harder it teaches the clip. Check the
turn-share/word-share split of every root before assuming a mix is balanced.
### The output-side test ran. The clip hypothesis is FALSIFIED — and the truth is more interesting
Measured on the live tuned seat, n=96 rp generations (brokkr-smithy-dev,
2026-08-26). **There is no shoulder at 123:**
0-19 18 ██████████████████
20-39 22 ██████████████████████ <- MODE
40-59 5 █████ <- trough
60-79 15 ███████████████ <- PIPPA median 67
80-99 10 ██████████
100-119 9 █████████
120-139 3 ███ <- PIPPA CAP 123. Three. A trough.
140-159 5 █████
160-179 4 ████
300+ 3 ███ max 505
<=123 79/96 = 82.3% (PIPPA itself: 100.0%)
median 62 · mean 77.9 · p90 152 · p99 367
**The clip-as-boundary reading is dead.** Mass would pile *under* a learned wall;
the 120-139 bin holds three of ninety-six and is a trough, and **17.7% of
generations cross a cap PIPPA can never cross.**
⚠ **A correction to this document's earlier read.** It compared the tuned *mean*
(88.5) to PIPPA's *median* (67) and p90 (95) and concluded "central tendency,
comfortably inside the upper body." **Median to median it is 62 against 67** — at
or slightly below PIPPA's centre. Mixing statistics across a comparison produced
a more reassuring answer than the data supports; the 88.5 was a mean dragged up
by the tail.
**What the data shows instead: the distribution is BIMODAL.** A mode at 20-39, a
trough at 40-59, a second mode at 60-79 astride PIPPA's centre, then a long tail
to 505. The base arm has no such shape (tuned mean/median ratio 1.45 and 1.23
against the base's 1.09 and 1.14).
So the tune did not change rp length so much as change its **shape** — replacing
one symmetric distribution with a mixture. That is what you would expect from
roots whose length distributions do not overlap (PIPPA 67, c2-logs 256, bluemoon
601) being learned as **distinct modes rather than blended into an average**.
⚠ **And the skew is rp-ONLY.** Story sits at mean/median 0.98 / 0.95 tuned
against 0.98 / 1.00 base. The shape change appears exactly in the family the
clipped root lives in — the strongest support the turn-share mechanism gets from
the output side.
**What this means for pair generation:** `chosen`/`rejected` sampled from a
bimodal generator inherit the mixture, not a mean. Sampling naively will
over-draw from the dominant short mode. If length is not the axis being tuned,
control for it explicitly rather than assuming a representative sample.
### ⚠ The degeneracy rate is NOT yet a usable baseline
Same arm, same seed block, re-run:
scored 162/192 -> 163/192
degenerate 7 -> 11
truncated 23 -> 18
rp VOID no -> YES
Distance moved 0.003, well inside the 0.0125 floor, so the gate result is not
threatened. But **degeneracy straddles the 10% budget**, so the same arm on the
same seed VOIDs or does not depending on the run. **A guard whose trip point sits
at the noise boundary flips**, and it produces disagreement between honest
observers rather than silence — a distinct defect from a gate that cannot fail.
The honest statement is that **the tuned arm's rp degeneracy is AT the budget**,
not clearly over or under it. Replicates are being run to pin it. **Do not build
DPO pairs on the single-sample figure** — the degeneracy rate decides how much of
the tune's own output is usable as `chosen`, so it is load-bearing for pair
generation and much cheaper to pin now than to discover after pairs exist.
**Why this belongs in the DPO prep and not just the gate record:** preference
pairs generated *from this tune* inherit its length distribution in both
`chosen` and `rejected`. If short-rp is an artifact rather than a preference,
DPO on those pairs trains it in as an explicit objective rather than leaving it
as an incidental SFT bias. Settle the length question before generating pairs.
Scripts: `/tank/erp-tune/pippa_clip.py`, `/tank/erp-tune/clip_share.py`.
## The decision that gates the rest
**Which refusal axes are in scope, and which are explicitly kept?**
Everything downstream — data generation, pair counts, held-out split, the probe
that measures success — is a function of that list. It is not a technical call
and it should not be made by whoever writes the config.
## Pre-flight, when the time comes
Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8
before the window opens. Two items bite harder here than they did for SFT:
- **cache key covers the change** — a preference set is exactly the kind of
auxiliary file whose *path* is stable while its *content* changes.
- **an observed consequence beside every configured value** — the axis list is a
configured value, and its observed pair is the per-axis probe delta.
docs/pfi/gemma4-erp-tune-sizing.md
+664
View File
@@ -0,0 +1,664 @@
# Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication
_Measured 2026-08-24 on `ana-ml2` against
`/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated trainee)._
Division of labour for this run: **Eitri writes the harness, brokkr-smithy-dev
audits, infra-ops owns the GPU window and executes.** This document is the
sizing infra-ops owes; it is arithmetic against the real checkpoint and the
real card, not an estimate.
---
## 1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE
**The proposed shape was QLoRA r64. It cannot be run as specified**, and the
reason is structural rather than a tuning preference.
The checkpoint stores each layer's 128 experts as **two fused 3-D
`nn.Parameter` tensors**, not as 128 `nn.Linear` modules:
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
Note the absence of a `.weight` suffix — compare `mlp.down_proj.weight`
(an `nn.Linear`) against `experts.down_proj` (a bare parameter). That is the
tell, and it is decisive: **`bitsandbytes` 4-bit replacement walks `nn.Linear`
modules.** A fused 3-D parameter is not one, so it is skipped and stays BF16.
What `load_in_4bit=True` would actually buy on this model:
| block | params | BF16 | after bnb NF4 | saved |
|---|---:|---:|---:|---:|
| **MoE experts** (fused 3-D — **NOT quantized**) | 22.84 B | 42.54 GiB | **42.54 GiB** | **0** |
| lm attention (`nn.Linear`) | 1.11 B | 2.07 GiB | 0.52 GiB | 1.55 |
| dense shared MLP (`nn.Linear`) | 0.54 B | 1.00 GiB | 0.25 GiB | 0.75 |
| vision tower (`nn.Linear`) | 0.57 B | 1.06 GiB | 0.27 GiB | 0.79 |
| embed (tied, normally kept BF16) | 0.74 B | 1.38 GiB | 1.38 GiB | 0 |
| router + norms | 0.01 B | 0.02 GiB | 0.02 GiB | 0 |
| **total** | **25.81 B** | **48.07 GiB** | **~44.98 GiB** | **~3.1 GiB** |
**88.5% of the model is in tensors bitsandbytes cannot touch.** "QLoRA" here
means paying the NF4 dequant tax on 6% of the weights to save 6% of the
footprint. The premise does not survive contact with the checkpoint.
> **Eitri: do not hard-code a `BitsAndBytesConfig` / `load_in_4bit` path.**
> It will not error loudly — it will load, report a 4-bit model, and quietly
> leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.
**The one thing that could overturn this** is a third-party fork shipping
custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth
is the candidate). **Not chased, deliberately** — see §4, where the run fits in
BF16 without displacing anything the fleet depends on, which collapses QLoRA's
value to zero. If it is ever revisited, it must be *before* the harness
hard-codes a quantization path, not after.
**Verdict: plain LoRA on BF16 weights.**
---
## 2. What the run actually costs
Adapter targeting `q_proj,k_proj,v_proj,o_proj` at r64, computed from the real
tensor shapes:
| | layers | per layer | total |
|---|---:|---:|---:|
| sliding-attention (q 4096, kv 2048, o 4096) | 25 | 1,507,328 | 37,683,200 |
| full-attention (q 8192, kv 1024, o 8192) | 5 | 1,654,784 | 8,273,920 |
| **trainable** | | | **45,957,120** (0.178% of base) |
⚠ **`v_proj` DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29.** Those are the
`full_attention` layers, and `attention_k_eq_v: true` means one projection
serves both K and V. Consequences the harness must respect:
- PEFT matches by name suffix, so a `v_proj` target **silently produces no
adapter** on those five layers. Do not assert a fixed adapter count.
- Adapting `k_proj` on a global layer **adapts K and V simultaneously** — a
different intervention than on the sliding layers. If that asymmetry matters
to the recipe, say so explicitly rather than discovering it in the loss curve.
### Memory budget, batch 1, `max_seq_len` 8192
| item | GiB | note |
|---|---:|---|
| base weights BF16 | 48.07 | measured: 25,805,936,206 params × 2 B |
| adapters + grads + AdamW fp32 m/v | 0.75 | 45.96 M trainable — rounding error |
| checkpointed layer inputs | 1.29 | 30 × 8192 × 2816 × 2 B |
| recompute peak, one layer | ~2.5 | 8192 tok × top-8 of 128, `moe_intermediate 704` |
| loss head, **fused/chunked CE** | ~2.0 | see the warning below |
| CUDA context + cuBLAS + fragmentation | ~3.0 | the item `--gpu-memory-utilization` never covered |
| **total** | **~57.6** | |
Marginal cost per extra sequence in the micro-batch: **~2.5 GiB.**
| micro-batch | GiB |
|---:|---:|
| 1 | 54.3 |
| 2 | 56.8 |
| **4** | **61.8** |
| 6 | 66.8 |
| 8 | 71.8 |
### ⚠ The loss head is the whole ballgame, and it is not in the brief
`vocab_size` is **262,144** and `final_logit_softcapping` is **30.0**. One
8192-token sequence produces **2.147 billion logits**. Through a naive HF
`ForCausalLM` loss that is:
BF16 logits 4.0 GiB
fp32 upcast 8.0 GiB
softcap tanh saved 8.0 GiB (autograd keeps the pre-cap tensor)
softmax + grad 8.0 GiB
------------------------------
~28-30 GiB transient, at BATCH 1
Naive CE at batch 1 lands the run at **~85.6 GiB on a 95.6 GiB card** — it will
appear to work and then OOM on the first long sample. At micro-batch 4 it is
~120 GiB and never starts. **Fused/chunked linear cross-entropy is mandatory,
not an optimization.**
⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with
softcapping may not have one. Three ways out, in order of preference —
(a) generic `LigerFusedLinearCrossEntropyLoss` wired against the lm_head with
softcapping applied inside the chunk; (b) `cut-cross-entropy`; (c) hand-rolled
sequence-chunked CE. **This must be proven on a 10-step smoke run before the
window is booked**, because everything else in this document assumes it works.
### Step count
58.2 M tokens / 20,576 samples = **2,829 tokens/sample average** — well under
8192, so packing matters.
- Packed to 8192: **7,104 sequences.** At micro-batch 4 × grad-accum 4
(effective 16) → **444 optimizer steps for the whole epoch.**
- ⚠ That is a *small* step count. A "checkpoint every 100 steps" default gives
four checkpoints across a multi-hour run. This is exactly why the amendment
asked for **wall-clock-interval checkpointing, not step-count** — the case is
now concrete, not hypothetical.
- ⚠ **Packing must use `position_ids` + varlen/block-diagonal attention.** Naive
concatenation bleeds samples into each other. `sliding_window` is 1024 on 25
of 30 layers so the damage is bounded there — but the 5 `full_attention`
layers see the entire packed sequence.
**Open question for brokkr/Eitri:** what fraction of the 20,576 samples exceed
8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting
the ends off RP scenes, which is where the signal lives.
### Runtime
Active parameters per token ≈ **3.67 B** (2.93 B routed + attention, plus the
0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute
≈ 6 × active × tokens = **1.28e18 FLOPs** for the epoch.
At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts
are not efficient — **4 to 10 hours, most likely ~6.** Treat as a band, not a
number; it will be measured on the smoke run.
---
## 3. Where it fits (measured 2026-08-24, 18:20 PDT)
Card total: 97,887 MiB = **95.60 GiB** each.
| | GPU0 | GPU1 |
|---|---|---|
| resident before the window | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
| free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |
Three placements were on the table:
- **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
current number is budgeting against a moving one.
- **GPU1 with `mog-sec` stopped: 76,070 MiB free.** Fits, but shares a card with
four small seats and Scriberr.
- **GPU0 with `gen` MOVED OFF: the whole card.** ← what was chosen.
---
## 4. The window, as executed
**Operator call, 2026-08-24: move `gen` to GPU1 and stand `sec` down, so GPU0 is
emptied completely rather than shared.** This is strictly better than training
beside `gen`: the tune gets 95.60 GiB with no co-tenant, and the fleet's general
seat never goes dark beyond its own ~5-minute restart.
before: GPU0 [ gen 42.5 ] GPU1 [ sec 55.3 | small seats 20.7 ]
after: GPU0 [ ---- empty, 95.60 GiB ---- ] GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]
`sec` is genuinely in use and this is not free — but it is the smaller blast
radius by a wide margin:
| | `gen` | `sec` |
|---|---|---|
| aliases | 7 (`gen`, `gen-reasoning`, `chat-judge`, `image-judge`, summarizer/classifier family) | 2 (`sec`, `sec-reasoning`) |
| standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
| measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |
Traffic to `sec` arrives from `10.250.50.70` (the LiteLLM gateway), so the
aliases will fail at the gateway for the duration. Per the standing rule, let
them fail — **do not route `sec` to another model as a stand-in.**
Both directions are playbooks, and **the order in each is load-bearing**:
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml
⚠ `gen` runs at `--gpu-memory-utilization 0.43`, which vLLM reads as a fraction
of **total** card memory: 42,091 MiB must be *free at startup* or the engine
refuses to boot. GPU1 has 19,446 MiB free while `mog-sec` is up. **Recreating
`gen` onto GPU1 before stopping `mog-sec` takes the fleet's main seat down and
leaves it down.** The open playbook stops `mog-sec` first and hard-gates on the
freed memory; the close playbook mirrors it, because `mog-sec` needs 50,901 MiB
of its own and cannot start until `gen` has vacated GPU1.
⚠ Invoke elway as `infra-ops@10.250.50.54`, not the `ana-ml2` ssh-target — that
resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
probe.
### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic
Everything above was arithmetic. This was run on the real checkpoint on GPU0
with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer:
| config | peak | verdict |
|---|---:|---|
| naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare |
| naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB |
| chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ |
| **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** |
| chunked CE, bsz4 seq 16384 | **OOM** | — |
**The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB
estimated.** The estimate modelled gradient checkpointing as storing only layer
inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens ×
top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size
an MoE run from dense-model intuition — measure it.**
Two predictions did land exactly, which is why the rest of the model of the thing
is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and
**74,342,400 trainable params** at r64.
The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq
8192.** Chunking is not an optimisation, it is what makes brokkr's 16384
recommendation reachable at all.
Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure.
### Revised run parameters, now that it is a whole card
**FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2,
`gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer
steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear.
`max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap
drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated
*entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon —
which is 60% of the mix and the axis the seat exists for. Prose and fireball
truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.
⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the
full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751
the typical step sits far below it.
⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()`
with it. Dropping checkpointing looks like ~17% off wall-clock and instead
forces micro-batch 1. Worse, the second call is the silent one: **without
`enable_input_require_grads()` the frozen base produces no gradient through the
checkpointed blocks, every adapter stays at its initialisation, and the run
completes successfully with an inert adapter.** `prepare_model_for_kbit_training`
used to do it as a side effect of the 4-bit path — so removing 4-bit removes it
too, and nothing warns you.
### Harness changes this required (eitri-smithy `62b556b`)
`9d64257` as audited would not have run here. Four fixes:
1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config
key, defaulting off, per §1.
2. Sequence-chunked CE replacing the model's own loss (the measured table above).
3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own
stale 365-line template and there was **no override parameter anywhere**, so
the upstream-template requirement was not expressible in the code.
4. Gradient checkpointing + `enable_input_require_grads()`.
Plus `training_eligibility_override` / `overridden_blockers` /
`substitute_controls` in the provenance manifest, and `device_map` pinned to
device 0 so the run cannot stray onto the card holding the inference seats.
Also fold in:
- **Scriberr STAYS on GPU1.** (An earlier draft of this document suggested moving
it to GPU0; that was written when training was going to live on GPU1, and it is
now exactly backwards. GPU0 is the training card and wants no co-tenant.)
- **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
(36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
would come close to filling it. `/tank` has 4.0 TB.
- **The run is still resumable-by-design** (INV-T7 + wall-clock checkpointing).
Nothing about a dedicated card removes that requirement — a 4–10 hour window
is long enough that an unresumable run is a bad bet regardless of who owns the
GPU.
---
## 5. Standing warnings that apply to this run
- **Never render training examples through the base's own
`chat_template.jinja`.** Every third-party Gemma-4 derivative ships a stale
one; the trainee's is 365 lines against upstream's 390. Use
`/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. Training through
the wrong template is train/serve skew with no error — it presents as a
tuning failure.
- **Base path and chat-template path are config keys, not constants.** The
trainee base already moved once (stock BF16 → `-heretic-bf16`).
- **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
context, graphs, or non-torch overhead — the same misreading that OOM'd the
char-rp seat.
- **SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason
we expected.** The open question was whether LoRA-on-NVFP4 hot-swap still
silently no-ops (it did on vLLM 0.24.0, #47639). Retested on
`vllm/vllm-openai:latest` with the NVFP4A16 base plus the run's own
checkpoint adapter. It does not no-op — **it refuses to start:**
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
⚠ **This is architectural, not quantization-related.** The check lives in
`vllm/lora/utils.py::process_packed_modules_mapping` and branches on whether
the model is MoE; quantization is not in the condition. `gemma4.py`,
`gemma4_mm.py`, `gemma4_mtp.py` and `gemma4_unified.py` contain **zero**
occurrences of `get_expert_mapping` (deepseek_v2, glm4_moe, ernie45_moe and
others do implement it). **vLLM cannot serve a LoRA on Gemma-4 at all —
BF16 or quantized.** Merging is the only path for this architecture.
Note this holds even though our adapter never touches experts:
`validate_adapter_parameters` forbids per-expert params, so all 205 targets
are attention + dense MLP. The refusal is about the *model* being MoE, not
about what the adapter targets.
Silver lining worth recording: a loud refusal is strictly better than the
0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name
and passes every check that does not compare against base.
The merge → quantize → serve pipeline is implemented and validated end to
end at [`scripts/erp-tune-serve/`](../../scripts/erp-tune-serve/).
---
## 6. Round-1 aborted; throughput root-caused (measured 2026-08-24 22:00 PDT)
Run-01 launched, reached step 19 of 1,312 at ~35–46 s/it, and was **killed by
operator instruction** — not a crash, not an OOM. ETA was ~13.9 h at 8.6% MFU
and the operator elected to root-cause before spending the window.
Nothing was destroyed: the 609 MB encode cache, `order-manifest.jsonl`,
`truncation-report.json` and `resume-run-01.sh` are all preserved at
`/tank/erp-tune/run-01/`. **There are no checkpoints** — the first was due at
step 100, so brokkr's `lora_B` inert-adapter gate never ran. That question is
open and moves to the restart.
Model-agnostic lessons from this investigation are in
[`training-throughput-playbook.md`](training-throughput-playbook.md); the
probes are at [`scripts/training-probes/`](../../scripts/training-probes/).
What follows is Gemma-4-specific.
> ## ⚠⚠ CORRECTION 2026-08-26 — MUCH OF THIS SECTION MEASURES THE WRONG PROCESS
>
> **The benchmarks below were run against the SERVING seat with
> `attn_implementation="sdpa"` set explicitly. Training was running
> `flex_attention` the whole time.** `ATTN_IMPLEMENTATION = "flex_attention"`
> was a module constant passed unconditionally into `from_pretrained`, and
> run 1's step-time distribution (n=1,445; min 11.84 / p50 19.75 / p99 30.52 /
> max 45.79 s/it, the max being step 1's compile) confirms it stayed compiled —
> a dynamo fallback sits in the hundreds of seconds per step.
>
> **FALLS** — describes sdpa, not the training run:
> the three-point scaling fit and its 68% quadratic share; the kernel table
> (`fmha_cutlassF/B` sm80, `EFFICIENT_ATTENTION`, attention 65.2%); the **8.6%
> MFU** figure quoted above and throughout; the projection that elementwise
> becomes the largest line item post-fix; and "adopt `flex_attention`" as the
> round-two headline lever — **which round one already had.**
>
> **SURVIVES** — measured on the live training run:
> the padding/bucketing win (44.3 → 20.1 s/it); the zero-pad fast-path
> second-order effect; the eval-battery noise-floor work.
>
> ⚠ **Do not assume the direction of the correction.** Training's real MFU is
> *unmeasured*, not obviously better. Flex with a BlockMask ought to beat
> dense-masked sdpa, but that is a prediction and this investigation has been
> unkind to those.
>
> The root cause was procedural, not technical, and it is written up as
> playbook **§4.3**: run 1 recorded no attention backend in its provenance, so
> the benchmark/trainer delta was invisible and nobody enumerated it. Run 2
> onward records `attn_implementation_requested` **and** `_resolved`, plus the
> torch/transformers versions and dynamo's compile counters.
### 6.1 Where the step time goes
Real checkpoint, GPU0, `attn_implementation="sdpa"`, PEFT + gradient
checkpointing + the chunked CE, fwd+bwd, best-of-2 after warmup:
| shape | time | peak |
|---|---:|---:|
| 2 × 2,048 | 1.776 s | 53.2 GiB |
| 2 × 8,192 | 11.570 s | 62.3 GiB |
| 2 × 16,384 | **35.017 s** | 76.6 GiB |
Fitting `t(w) = A·w + B·w²` over all three (per-sequence `w`, batch 2):
A = 6.8715e-04 s/token B = 8.8509e-08 s/token²
| w | predicted | measured | linear | quadratic | quad share |
|---:|---:|---:|---:|---:|---:|
| 2,048 | 1.779 | 1.776 | 1.407 | 0.371 | 20.9% |
| 8,192 | 11.569 | 11.570 | 5.629 | 5.940 | 51.3% |
| 16,384 | 35.017 | 35.017 | 11.258 | 23.759 | **67.8%** |
**Two terms, three points, residuals under 3 ms across an 8× range.** No fixed
per-batch term was needed, which refutes the launch-bound hypothesis outright —
~3,840 expert-GEMM launches per forward are not the cost.
Independently, the profiler kernel table (device rows only — see playbook §3.4):
| device kernel | ms | of step |
|---|---:|---:|
| `fmha_cutlassB_bf16_aligned_128x64_k65536_sm80` (attn BWD) | 16,144.6 | 46.1% |
| `fmha_cutlassF_bf16_aligned_32x128_gmem_sm80` (attn FWD) | 6,691.2 | 19.1% |
| `cutlass_80_tensorop_bf16_s16816gemm` ×3 (dense GEMM) | 2,774.0 | 7.9% |
| elementwise / vectorized / unrolled ×9 | 4,787.4 | 13.7% |
| gather / Memcpy DtoD / dropout | 951.6 | 2.7% |
| **attention total** | **22,835.8** | **65.2%** |
**Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two
independent methods, 2.6 points apart.**
### 6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20%
`brokkr-smithy-dev`'s panel (arm: Bil) closed the fold by reading torch 2.13.0
and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather
than explaining it:
nominal work billed 27.1 TFLOPS × 34.85 s = 9.4e14 FLOP
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14 FLOP
padded full layers lose the causal skip = +3.5e14 FLOP
------------------------------------------------------------------
work actually performed ≈ 1.8e15
in 34.85 s ≈ 51–61 TFLOPS
≈ 17–20% OF PEAK
**We divided the *intended* (windowed) FLOPs by the wall time the *dense*
reality took.** 17–20% is squarely inside the honest stock band. The hardware
is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic
the architecture specifies, and the excess is the sliding window being computed
and then thrown away.
Source-verified mechanism, no longer hypothesis:
| file | finding |
|---|---|
| `masking_utils.py:292-301` `_ignore_causal_mask_sdpa` | requires `kv_length < local_attention_size` to skip the mask. 16384 ≥ 1024, so **the sliding mask ALWAYS materialises at this seq len** — not sometimes, always |
| `sdp_utils_cpp.h:259-267`, `sdp_utils.cpp:933` | flash rejects **any** explicit mask |
| `sdp_utils.cpp:647`, `Context.h:480-485` | **cuDNN is unreachable on sm_120 twice over** — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12 |
| `attention.cu:1196/1759`, `kernel_forward.h:282-290` | mem-efficient has no mask gate and no head_dim cap, computes **full n×n** with the mask as additive bias; it trims only for `is_causal` |
Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding
layers land on mem-efficient computing dense O(n²), and **no backend on this
stack can rescue it.** cuDNN sliding-window does not exist at all — there is no
window argument in the public SDPA signature.
⚠ **Masked SDPA also blocks `enable_gqa`**, so KV gets `repeat_kv`-expanded on
every layer — extra memory traffic riding on top of the extra FLOPs.
### 6.1b Backend eligibility, measured — every source claim confirmed
Shapes-only, random weights, `sdpa_kernel()` pinning one backend at a time.
A forced failure is information: it identifies eligibility rather than
preference.
**Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:**
| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | **8.8 ms** | 21.6 ms | refused | 217.5 ms |
| explicit 1024-band 4D | refused | **55.5 ms** | refused | — |
| explicit causal+pad 4D | refused | 57.1 ms | refused | — |
**Global layers (5 of 30) — H_q16/H_kv2, D=512:**
| mask case | FLASH | EFFICIENT | CUDNN | MATH |
|---|---|---|---|---|
| `None` + `is_causal` | refused | **45.9 ms** | refused | 334.1 ms |
| explicit 1024-band 4D | refused | 104.4 ms | refused | — |
| explicit causal+pad 4D | refused | 107.9 ms | refused | — |
Refusal reasons, straight from the runtime warnings:
- `Flash Attention does not support non-null attn_mask` — kills flash for all
25 sliding layers, always, at this seq len
- `Flash attention requires q,k,v … less than or equal to 256. Got 512` —
kills flash for the 5 global layers **even fully causal**. Dvalin was right
that they are a first-class hole
- `head_dim should be no more than 128` — kills cuDNN on **both** shapes
**The sliding layers run at 55.5 ms where a maskless flash path would cost
8.8 ms — a 6.3× penalty, and it is unreachable through any config on this
stack.** That is the whole problem in one row.
### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card
`fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in
this path at all, and the forward is additionally on `gmem` — the
global-memory fallback tier of the memory-efficient backend, selected when the
working set will not fit in shared memory.
This is the mechanism behind the 100%-SM / 27-TFLOPS / 304-TFLOPS-capable
reading: the chip is saturated running a kernel generation behind on the
dominant cost centre.
The candidate fix is a purpose-built kernel for this architecture's mixed
256/512 head-dim split — `zzhhjjj/gemma-triton-flash-attn`
(`register_triton_attention()`, then `_attn_implementation = "triton_gqa"`),
reported 9.23× over SDPA at N=16K D=256 SWA and 2.94× fwd+bwd at D=512.
`flex_attention` + `BlockMask` is the no-new-dependency alternative.
⚠ **Prefer a UNIFORM backend over a per-layer split.** vLLM special-cased this
exact mixed-head-dim architecture and measured mixed backends **8% slower** than
uniform. And `attn_implementation` is all-or-nothing at `from_pretrained` /
`set_attn_implementation` — per-layer routing requires a custom function
registered on `ALL_ATTENTION_FUNCTIONS` branching on `module.head_dim` /
`sliding_window`.
⚠ **FA2 is not available for this model**: it caps head_dim at 256 and the 5
global layers are at 512. FA3 is Hopper-only. Do not bet on FA4 on sm_120.
### 6.3 Masking is CORRECT — and padding is what costs
Band structure asserted directly against the real config at n=16,384:
sliding_attention max 1,024 allowed/row, saturates at row 1,023 PASS
Constraints were **not** silently dropped; the 25 sliding layers were genuinely
windowed. Run-01 was training the model we intended.
The same probe found the mechanism nobody had measured:
| 2D mask supplied | `full_attention` mask returned |
|---|---|
| `None` | **`None`** → `is_causal` fast path AVAILABLE |
| all-ones (no padding) | **`None`** → `is_causal` fast path AVAILABLE |
| right-padded (what `collate_mixed` emits) | 4D `16384²` → **fast path LOST** |
**Padding is what pins the 5 global layers to an explicit mask.** The 25
sliding layers get a 4D tensor either way — `sdpa_attention_forward` sets
`is_causal=True` only when `attention_mask is None`, and a 1024 window cannot
be expressed as `is_causal`.
Isolated, same width, only the mask differing:
2 × 16,384, no padding 35.244 s 26,048 loss targets
2 × 16,384, 50% pad on row 1 38.567 s 19,640 loss targets
**9.4% slower for 24% less work.**
### 6.4 The corpus is 29.9% padding — and bucketing is the biggest win available
Measured off the preserved encode cache in true `SequentialSampler` order:
records 20,982 (3,583 rp-dialogue / 12,003 prose-chunk / 5,396 actual-play)
seq len min/mean/max 142 / 2,752 / 16,384
micro-batches (mb=2) 10,491
real tokens 57,733,156
padded tokens 82,337,318
PADDING WASTE 29.9%
mb width p50/p90/p99 2,092 / 10,634 / 16,341
micro-batches at 16,384 3 of 10,491 (0.0%)
⚠ Note the last line against §6.1: **the 2 × 16,384 benchmark shape occurs in
three micro-batches out of 10,491.** Weighted over the real distribution the
quadratic share is ~51%, not 67.8%.
**Bucket-to-pair, shuffle-to-mix** (brokkr's design, validated on measured
lengths — form micro-batches within length buckets, then shuffle the resulting
*micro-batches* globally):
| bucket | waste | predicted step | zero-pad mb | roots/accum window |
|---:|---:|---:|---:|---:|
| current | 29.9% | 44.3 s → 16.13 h | 0.1% | 3.68 |
| **2** | **0.0%** | **28.6 s → 10.40 h** | **78.3%** | 3.56 |
| 8 | 0.0% | 28.6 s → 10.41 h | 65.3% | 3.54 |
| 32 | 0.1% | 28.6 s → 10.43 h | 41.9% | 3.55 |
| 128 | 0.6% | 28.8 s → 10.51 h | 14.7% | 3.55 |
| 512 | 2.4% | 29.7 s → 10.82 h | 4.1% | 3.61 |
**≥35.5% wall clock, no kernel work, no new dependency, peak memory unchanged.**
Two findings that changed the design:
- **Bucket size is not a diversity knob.** Roots per accumulation window are
flat at 3.54–3.61 across a 256× range. The global micro-batch shuffle does
all the mixing. Use the tightest bucket.
- **35.5% is a floor.** Zero-pad micro-batches go 0.1% → 78.3%, which puts the
5 global layers back on `is_causal` for most of the run (§6.3). The cost
model does not capture that. Direction certain, magnitude not yet measured at
representative shapes.
⚠ **Source-homogeneity is a real hazard here** — length correlates hard with
root (kvasir short, chunked RP windows long), so length-homogeneous batches are
root-homogeneous batches. The global micro-batch shuffle is what prevents an
accumulation window drawing its whole gradient from one source. It is
load-bearing, not decoration.
### 6.5 The chunked CE is fine — do not swap it
2 × 16,384 CE forward 374 ms of 35.329 s = 1.1%
2 × 4,096 CE forward 93 ms of 4.387 s = 2.1%
⚠ **Forward only** — the `torch.utils.checkpoint` recompute runs inside
`.backward()`, outside the timing window. Even at 3× it is ~3%.
`liger-kernel` fused linear CE is a ~1–3% lever on this shape. §2's finding
stands unchanged: chunking is what makes seq 16384 *reachable*, and it is not
what makes it slow.
### 6.6 MoE is ~8% — stop optimising it
Dense GEMM is 7.9% of the step, confirming the earlier decomposition bound of
~10% from the kernel side.
On `grouped_mm`: **the trace does not adjudicate it.** Run-01 was relaunched on
`eager`, so the profile shows the *default* path — 25,463 `aten::mm` dispatches
in one fwd+bwd, far more than the ~90 a grouped path would produce, so the
default is per-expert sequential. Whether the flag changes that when set is a
different measurement and was not run. At 7.9% it is not worth running.
### 6.7 Restart parameters for round 2
**Do not relaunch without the sampler change.** It is the only lever that wins
under every branch of the diagnosis.
1. **Implement bucket-to-pair + shuffle-to-mix** in the harness, tightest
bucket, global micro-batch shuffle. Expected ~16.1 h → ~10.4 h or better.
2. **Re-assert the mask band structure** after the sampler change —
`scripts/training-probes/step0_mask.py`, 30 s, no GPU. The sampler touches
batch composition, which is what drives mask construction.
3. **Resume with `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch
command** — it begins `rm -rf /tank/erp-tune/run-01` and would destroy the
609 MB encode cache (2.5 min to reuse, ~4.3 h to rebuild). ⚠ A sampler change
alters record *order*, not encoding, so the cache stays valid — but bump
`encode_version` if anything upstream of `input_ids` changes.
4. **Run the `lora_B` inert-adapter gate at step 100.** It never ran in round 1.
Norm every `lora_B` tensor in the checkpoint: all-non-zero = real, all-zero =
INERT (kill the run), partial = module-selection problem. This is the one
failure that stays invisible until brokkr's acceptance gate reports
base-identical numbers.
5. **The corpus override is ONE RUN ONLY** (`operator-2026-08-25-rnd-run`). A
second run needs a second operator grant.
6. **Attention backend is round 2's second lever**, gated on an A/B on the
replica — not on argument. It can run while the tuned job trains.
⚠ GPU0 is currently **reserved and idle** by operator instruction; `sec` /
mog-sec remains down. The window is still open, so
`playbooks/ana-ml2-training-window-close.yaml` has NOT been run.
docs/pfi/headscale-mesh-plan.md
+404
View File
@@ -0,0 +1,404 @@
# Headscale overlay mesh — plan (2026-09-06)
**Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
(Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
provisioned yet.
## Why Headscale (decided 2026-09-06)
Tailscale clients are the best in class on the devices actually used for remote access
(iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
(`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
## Site facts that drive placement
| site | edge | v4 | v6 | change risk |
|---|---|---|---|---|
| ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
| NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
| ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
**Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
**Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
clients follow the URL and never re-enrol.
## Topology
```
headscale.phasefinal.com (A via DDNS, AAAA static)
│ HTTPS 443 (+ DERP later)
┌─────────────────────────────┼───────────────────────────────┐
│ NH3 nh3-pve │ │
│ CT nh3-headscale ──control plane (1 core / 512M / 8G) │
│ CT nh3-mesh-rtr ──subnet router, advertises 10.100.0.0/16│
├──────────────────────────────────────────────────────────────┤
│ ESH esh-pve │
│ CT esh-mesh-rtr ──subnet router, advertises 10.0.0.0/16 │ (relay/DERP here later)
├──────────────────────────────────────────────────────────────┤
│ ANA pfi-pve │
│ CT ana-mesh-rtr ──subnet router, advertises 10.250.0.0/16│ (OPNsense takes this over later)
└──────────────────────────────────────────────────────────────┘
Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
```
Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
routine for unprivileged CTs; net sysctls are namespaced).
Proposed ids/names (DHCP with reservation like every existing CT; recorded in
`dns/internal.yaml` as `<name>.<site>.internal`):
| PVE | next id | CT | note |
|---|---|---|---|
| nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
| nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
| esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
| pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
`debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
downloaded on any).
## Access & credentials — all in hand (verified 2026-09-06)
| need | have |
|---|---|
| provision LXCs on all three PVEs | `infra-ops@` with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway `bootstrap-infra-ops-user.yaml`; PVE needed `sudo` installed first); `root@` also works |
| public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
| NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
| colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
| TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
| secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
"everything can reach everything" first, tighten later, is the recommendation).
## Phases
0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
reservation ranges on the three sites; `pveam download` Debian 13 on all three.
1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
(UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
TCP-443 relay path from a UDP-blocked network.
3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
on each site gateway pointing the other two /16s at the local router. Test
site-to-site from clientless hosts.
4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
and the two IPsec tunnels. Running all three at once makes route precedence murky on
the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
## Risks / open questions
- Control-plane outage does not drop existing tunnels but stalls new logins and key
rotation — hence the SQLite backup and the "not at the colo" rule.
- Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
a self-hosted DERP exists. Acceptable at first; fix in phase 5.
- CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
- Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
phases 0–4.
- VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
survives; nothing to reuse.
## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
`esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
empty policy) until told otherwise.
**Live:**
| what | where | detail |
|---|---|---|
| control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
| public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
| UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
| routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
| first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
| DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
| backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
| vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
**Lessons (do not repeat):**
1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
Phase order is therefore: **return path first** (either SNAT on the routers, or static
routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
clients, and exclude a node's own site route.
2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
Test that with one tunnel down before declaring cut-over ready.
3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
(`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
bring pfi-pve/nh3-pve up to current 8.4.
4. headscale 0.29 removed `randomize_client_port` and renamed
`ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
`headscale configtest` first.
**Next (in order):** (a) decide the return path — recommendation: static routes for
`100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
(b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.
### 06:05Z — return path built and verified (operator approved 05:50Z)
Static route `100.64.0.0/10` → local router on all three gateways: NH3 UDM
(`mesh-100.64-via-nh3-scale` → 10.100.50.46, classic REST), ESH UDM
(`mesh-100.64-via-esh-scale` → 10.0.50.65; ⚠ that vault item is a 19-line note — extract with
`secret get unifi/esh-udmpm-api-key | grep '^key:' | awk '{print $2}'`, a raw pipe gives nginx
400 and the NH3 key gives 401), FortiGate (`config router static`, dst 100.64.0.0/255.192.0.0,
gw 10.250.50.45, device `servers`; config backed up to flash as `pre-mesh-route-20260906`
first). Routers re-set to `--accept-routes=true` — a router MUST accept the other sites' routes
or its replies to remote-LAN sources leave via its default gateway with a 100.64 source and die;
a node never accepts its own advertised route, so the nh3-dev failure mode does not apply to
routers.
Verified (LAN host → gateway → local router → mesh → far router, and back): nh3-docker→ana-scale
9 ms, esh-docker-vm→ana-scale and →nh3-scale 0% loss, ana-docker→nh3-scale and →esh-scale 0%
loss. All three sites can reach mesh addresses from plain LAN hosts with no client installed.
**LAN-resident clients (nh3-dev) keep `--accept-routes=false`:** they reach other sites via
their gateway like any LAN host. `--accept-routes` is for roaming devices (laptop/iPad off-site)
only. Consequence: a LAN-resident client is reachable by its mesh IP only from the mesh, not
from other sites' LANs (ana-docker→100.64.0.4 fails by design; →100.64.0.1 works).
**Enrolling GUI devices (Mac/iPad) without OIDC:** the apps do not take a pre-auth key. Point
the app at `https://headscale.phasefinal.com`, it opens a registration page showing a
`headscale nodes register --user pfi --key mkey:…` command; run that on nh3-headscale
(`pct exec 106 -- headscale nodes register --user 1 --key <mkey>`). CLI Linux/macOS builds can
use the vaulted client key instead.
### 06:20Z — first roaming client verified
Operator's MacBook enrolled via the GUI (Option-click → Debug → Custom Login Server; headscale
0.29 registration command is `headscale auth register --auth-id hskey-authreq-… --user pfi`),
node 5 `vhlk-mba26` (100.64.0.5), "Use Tailscale subnets" on. From ESH: `ping 100.64.0.3`
and `ssh infra-ops@ana-docker.ana.internal` both work → colo subnet route + split DNS for
`*.internal` proven from a client. (Path was still via the ESH LAN / old tunnels.)
### Note — the mesh is also the durable fix for CGNAT CrowdSec false-bans
2026-09-06: CrowdSec on ana-docker banned ESH's shared CGNAT egress (23.164.40.160,
`custom/gitea-aggressive-crawl`) and the fortigate-mirror bouncer pushed it to the colo
edge, blackholing Matrix/gitea/chat for the whole ESH site (see
`~/.claude/.../memory/incident_crowdsec_cgnat_false_ban.md`). Once ESH consumes colo
services over the mesh (100.64/10 via ana-scale) rather than the public FortiGate VIP,
that traffic never reaches CrowdSec — a concrete win beyond replacing the tunnels, worth
weighing when prioritising the cut-over.
### 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary)
`/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/pfi-esh-egress.yaml` (bind mount,
persists) whitelists 23.164.40.160; SIGHUP reload, `crowdsec -t` clean, parser loaded.
TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical
(only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh.
## CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator
Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup.
**Safety net first:** FortiGate WAN SSH enabled — `wan1 set allowaccess ping ssh`, admin
`infra-ops` trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress);
trusthost1 10.0.0.0/8 kept. Config backed up to flash `pre-wan-ssh-cutover-20260906`.
Verified: `ssh infra-ops@38.120.12.42` works from nh3-dev, tunnel-independent. ⚠ TEMPORARY —
scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate
via WAN SSH (`scratchpad/fgw.py`) during/after cutover, not over the tunnel.
**Method (learned the hard way):** the tunnel and a mesh static route for the same /16 on the
same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped
(broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair:
disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now
unambiguous). Never do it with the tunnel up.
**colo↔NH3 (IPsec):** NH3 UDM `pfi-nh3-ana` disabled (enabled=false, API); mesh routes
NH3 `10.250.0.0/16→10.100.50.46 (nh3-scale)`, FortiGate `10.100.0.0/16→10.250.50.45
(ana-scale)`. Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND
WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle);
re-enable NH3 side to restore.
**colo↔ESH (IPsec):** ESH UDM `esh-ana` disabled; mesh routes ESH
`10.250.0.0/16→10.0.50.65 (esh-scale)`, FortiGate `10.0.0.0/16→10.250.50.45`. Verified
bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over
public/DERP.
**NH3↔ESH (Site Magic) — NOT done by API.** `sdwan-mesh-tunnel` objects (`ESH-UDMPM` on NH3,
`PFI-UDMSE` on ESH) return `api.err.NoEdit` — Site Magic is cloud-orchestrated, not togglable
via the classic REST API. Mesh routes PRE-STAGED (NH3 `10.0.0.0/16→nh3-scale`, ESH
`10.100.0.0/16→esh-scale`), currently SHADOWED by Site Magic (traffic still on the SD-WAN
tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path
independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site
Manager, the staged routes take over immediately. **Operator to toggle; mesh verified ready.**
**Dormancy = disabled-but-retained**, not auto-failover (overlapping /16s across two
transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to
enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired.
**Bonus (operator noted):** exit nodes at each site give a free multi-location egress proxy
(`tailscale up --advertise-exit-node` + approve); replaces the nh3-dev SOCKS5 residential
egress with a native feature. Not yet configured — parked follow-up.
**Follow-ups:** operator disables Site Magic; (optional) disable FortiGate phase1 ends for
tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the
FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.
## ✅ CUTOVER COMPLETE 2026-09-06 — all three site-pairs on the mesh
Operator disabled Site Magic in the UniFi UI. NH3↔ESH re-homed to the mesh immediately —
traceroute now esh-scale(10.0.50.65)→nh3-scale(100.64.0.1)→dest, no 192.168.1.x SD-WAN
transit; esh-scale↔nh3-scale hole-punched a DIRECT public path (70.230.226.88:41641, 8ms),
not even DERP. Full matrix verified, all six directions OPEN:
NH3↔colo, NH3↔ESH, colo↔ESH. FortiGate mgmt via WAN SSH OPEN; ana-wg WG fallback untouched.
State: **Site Magic disabled (UI); both IPsec tunnels dormant (UDM ends disabled, config
retained); headscale is the sole active site-to-site transport.** Tunnels re-enablable for
backup (Site Magic in UI; IPsec by flipping the UDM objects enabled=true). Goal met.
## 2026-09-06 — exit nodes advertised (multi-location egress proxy)
All three routers advertise + serve exit nodes; approved on headscale (each shows
`0.0.0.0/0` + `::/0` Serving). Clients see three: `nh3-scale` (100.64.0.1, NH3 residential
egress 70.230.226.88), `esh-scale` (100.64.0.2, ESH 23.164.40.160), `ana-scale`
(100.64.0.3, colo 38.120.12.42).
⚠ **Design reversal:** exit nodes REQUIRE SNAT, so `--snat-subnet-routes` was flipped back to
**true** (NoSNAT=false) on all three routers — this undoes the earlier source-preserving
choice. Consequence: cross-site subnet-route traffic now appears to come from the local
router's IP, not the original host. Site-to-site re-verified OK after the flip. If per-host
source visibility is later needed, the alternative is NoSNAT + a manual selective masquerade
rule (`-s 100.64.0.0/10 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE`) on each router, made
persistent — not done.
**Use one from a client:** `tailscale set --exit-node=nh3-scale` (or `esh-scale` /
`ana-scale`), `--exit-node=` to clear. A node that itself advertises an exit node cannot
consume one (so the routers can't be used to test egress; use the laptop/iPad).
Replaces the nh3-dev SOCKS5 residential-egress proxy — retire that once confirmed on real
clients.
## 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above)
Restored `--snat-subnet-routes=false` (NoSNAT=true) on all three routers AND added a
persistent selective-masquerade rule, so BOTH properties now hold: exit nodes work (internet
egress masqueraded) and site-to-site keeps the original host source.
Per router: `/usr/local/sbin/mesh-exit-masq.sh` + `mesh-exit-masq.service` (oneshot, After
tailscaled, enabled). Rule = a `MESH-EXIT` nat chain jumped from POSTROUTING for
`-s 100.64.0.0/10 -o eth0`: RETURN for 10/8, 172.16/12, 192.168/16, 100.64/10 (source
preserved), MASQUERADE the rest (real internet). ⚠ nf_tables iptables forbids multiple `-d`
in one rule — hence the chain-with-RETURNs shape, not a single `! -d ... ! -d ...` rule.
Verified: nh3-docker→ana-docker seen at colo as **10.100.50.40** (real host, preserved);
nh3-dev via ana-scale exit → egress **38.120.12.42** (colo), via nh3-scale → 70.230.226.88
(NH3). Persistent across CT reboot via the systemd unit. `tailscale set --operator` NOT set;
drive nh3-dev's tailscale via `ssh infra-ops@10.100.10.50 sudo tailscale ...`.
## 2026-09-06 — throughput baseline (mesh vs the old IPsec)
NH3↔colo over the direct WireGuard path (5ms), iperf3 between the 1-vCPU router CTs, medians
of 3: **download colo→NH3 761 Mb/s (688–761), upload NH3→colo 464 Mb/s (391–507)**, parallel-8
up ~530. Old IPsec ceiling was ~250 both ways (poor AES offload) — so ~3x down, ~1.8x up. CPU
during upload sampled ~32% (single reading) → not clearly the cap; upload likely limited by
userspace-WireGuard single-flow, the CT vNIC, or the WAN. Operator: the ANA AT&T line is
SHARED fiber, so the upload figure may be the line, not the routers — left as-is (no CT core
bump). Baseline for future comparison, not a hardware limit.
## 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved)
Constraint: operator has NO physical access to the Irvine building for 5 days; irv-ml1 must
not go dark unrecoverably. Its ONLY remote path is the bespoke WireGuard tunnel `wg0`
(10.100.79.3 → nh3-dev, AllowedIPs 10.0/16+10.100/16+10.250/16); its LAN 10.6.110.50 is not
reachable from the fleet.
Enrolled irv-ml1 as headscale node **100.64.0.6** (`infra-ops@10.100.79.3` has NOPASSWD root;
`tailscaled` was already running but logged out with STALE prefs → **public** Tailscale
`controlplane.tailscale.com` + `RouteAll:true`). Used `tailscale up --reset --login-server
https://headscale.phasefinal.com --authkey … --accept-routes=false --accept-dns=false
--hostname irv-ml1`. ⚠ **accept-routes MUST stay OFF**: wg0's AllowedIPs are the exact /16s the
mesh routers advertise, so accepting them would fight the lifeline over the prefix my SSH
returns on = 5-day lockout. Armed an auto-revert dead-man switch (`tailscale down` in 10m
unless disarmed) before the change; disarmed via sentinel after verifying.
Result: **DUAL PATH** — wg0 (10.100.79.3, untouched, fleet routes still via it) AND mesh
(100.64.0.6). Both verified from nh3-dev. Reboot-safe: `wg-quick@wg0` + `tailscaled` both
enabled, prefs now headscale + accept-routes=false + WantRunning=true. Mesh peering currently
rides over wg0 as transport (11ms).
⚠ **DEFERRED to the on-site window (do NOT do remotely):** retire wg0; enable accept-routes /
subnet-routing over the mesh; migrate yt-voice-clipper egress off the dante SOCKS proxy. Each
touches the exact routing that is the lifeline. When on-site (physical console as fallback):
flip accept-routes on, confirm fleet reachability over the mesh, THEN down wg0.
⚠ **Lesson (repeat):** `pkill -f` / `pgrep -f <pattern>` over SSH self-matches when the pattern
is in your own ssh argv — killed my session twice disarming the switch. Kill by explicit PID,
or make the switch sentinel-based (which saved it here).
## 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access)
Operator wanted the cutover finished tonight despite no physical Irvine access for 5 days, and
directed: dante proxy goes away (exit nodes instead), yt-voice-clipper may stay dead until
scoped exit-node egress is wired.
**Three independent recovery paths built BEFORE cutting wg0** (the discipline that made it safe):
1. Mesh over DERP-LA (~12ms) — irv-ml1's own WAN (38.120.94.3, easy NAT), independent of wg0.
2. **Reverse SSH tunnel** `revtun-nh3.service` on irv-ml1 → nh3-dev via NH3 public WAN
(UDM forward `irv-revtun-ssh` tcp/47822 → 10.100.10.50:22, **src-restricted to 38.120.94.3**;
key `/root/.ssh/revtun_ed25519`, authorized on nh3-dev lkraven with `restrict,port-forwarding`).
Reach irv-ml1: `ssh -i ~/.ssh/infra-ops_ed25519 -p 2201 infra-ops@127.0.0.1` on nh3-dev.
Independent of BOTH wg0 and the mesh/control-plane.
3. **`wg0-watchdog.service`** on irv-ml1 — `wg-quick up wg0` if the fleet is unreachable over the
mesh for ~5min (auto-failover). Plus wg-quick@wg0 stays enabled → reboot restores wg0.
**Cut:** accept-routes=true on irv-ml1 (fleet /16s → tailscale0 table 52), then `wg-quick down
wg0` over the reverse tunnel under a 12-min auto-revert dead-man (disarmed after verify).
Verified: irv-ml1 reaches nh3-docker/colo/ESH/nh3-pve over pure mesh; unreachable ONLY 10.100.10.50
(nh3-dev's own LAN IP — a mesh-node-reaching-a-mesh-node quirk; every other host fine; moot now
dante is gone). wg0 dormant (down, enabled). Recovery via reverse tunnel + mesh IP both OK.
**Dante retired** on nh3-dev: `danted` disabled+stopped, :1080 closed, `/etc/danted.conf` →
`.retired-20260906`. Only live consumer was YTVC (operator-exempted). searxng ref was already
commented out.
**OPEN follow-ups (on-site or later):** wire yt-voice-clipper egress via a SCOPED exit-node path
(do NOT set irv-ml1 `--exit-node` globally — it would route the reverse tunnel through the mesh and
kill the independent lifeline; use tailscale `--socks5-server`+exit-node or a per-container netns);
then bring YTVC back. On-site: decide whether to keep the reverse tunnel + UDM forward + revtun key
or remove them (small WAN attack surface, src-restricted). Fully retire wg0 config if desired.
## 2026-09-06 — wg0 made non-reboot-restorable + Irvine subnet discovery (operator corrections)
**wg0 no longer reboot-restorable** (supersedes the earlier "wg-quick@wg0 stays enabled" note):
`systemctl disable wg-quick@wg0` on irv-ml1 — the cutover is real, wg0 does not silently return on
reboot. Config `/etc/wireguard/wg0.conf` retained so the failover paths can still bring it up. The
FAILOVER is now: `wg0-watchdog.service` (`wg-quick up wg0` if the fleet is unreachable over the mesh
~5min — works regardless of the unit's enable state) + the independent reverse tunnel. Rationale
(operator): reboot-restore was redundant with the watchdog and made the cutover half-real; accept-routes
table-52 would keep the mesh primary anyway, but a clean cutover shouldn't resurrect the retired tunnel.
**Irvine subnet discovery LIVE (bidirectional):** irv-ml1 now `--advertise-routes=10.6.110.0/24`
(approved on headscale, serving; ip_forward=1; default SNAT-on so mesh-unaware Irvine hosts reply to
irv-ml1). Fleet-wide reach added via gateway static routes `10.6.110.0/24 → local mesh router`
(NH3 UDM→nh3-scale, ESH UDM→esh-scale, FortiGate route 22→ana-scale). Verified: ana-docker→10.6.110.1
and esh-docker-vm→10.6.110.50 both OK over the mesh. So irv-ml1 both REACHES the fleet /16s
(accept-routes) and EXPOSES its Irvine LAN to the fleet (advertise-routes) — a full mesh subnet router.
docs/pfi/ipv6-naming-scheme.md
+174
View File
@@ -0,0 +1,174 @@
# ESH IPv6 naming scheme
Every ESH LAN carries an **eight-hex-digit phrase** as the first half of the
interface identifier. Picked 2026-08-18/19. This file is the canonical record.
> **Why this file exists.** The scheme originally lived as a single line in
> `persistent-memory.md` and was silently deleted by a `memory: snapshot`
> commit (`837fa36`). Recovering it took a hunt through session transcripts to
> find the commit that had held it. A naming convention is not temporal state —
> it belongs in a document, so it now is one.
## The names
| network | hex | reads as |
|---|---|---|
| `Default` | **`4BA5:3417`** | A BASE FOR IT |
| `esh-mgmt` | **`15DA:B055`** | IS DA BOSS |
| `esh-server` | **`4411:B105`** | FOR ALL BIOS |
| `esh-userland` | **`CAFE:4411`** | CAFE FOR ALL |
| `esh-iot` | **`4DBA:D107`** | FOR DA BAD IOT |
| `esh-cameras` | **`1533:FACE5`** | I SEE FACES |
| *(reserved)* DMZ | **`4411:DBAD`** | FOR ALL DA BAD |
The DMZ name is **claimed against a network that does not exist yet** — there is
no DMZ on the ESH UDM. Do not reuse it.
Substitutions are the standard hexspeak set: `0`→O, `1`→I/L, `5`→S, plus letters
that are already native hex (`A`–`F`). Anything outside `0-9a-f` is not
expressible — `b0ss` and `c00l` do **not** work, which is why the set above uses
`B055` and avoids `c00l` entirely.
House style, arrived at rather than designed: **eight digits, and a complete
phrase rather than a single word.** Words are allowed to straddle the group
boundary (`4DBA:D107` is `4·D·BAD·107`); the phrase reads through the colon.
## Address structure
```
2607:73c0:402:1d02 : 4411:b105 : 50 : 45
└──── ISP /64 ────┘ └ segment ─┘ └ 10.0.50.45 ┘
```
- **Prefix** — Cityside's, not ours to name. ESH holds a `/56`
(`2607:73c0:402:1d00::/56`, 256 × /64); the subnet id is assigned by UniFi's
`ipv6_pd_prefixid`. `esh-cameras` is `1d00`, `esh-server` is `1d02`.
- **Segment word pair** — 32 bits, from the table above.
- **Host** — the last two IPv4 octets, written as literal digits so they read
straight off the address. `10.0.50.45` → `:50:45`.
The scheme lives entirely in the **interface identifier**, so it is
**delegation-size independent**. It works identically on a `/56`, a `/48`, or
NH3's single `/64`. It never competes with the subnet id, which is far too small
to hold a word (8 bits at ESH — two hex digits).
Note: `4411:b105:50:45` fills all four host groups, so there is **no `::`** in
these addresses. Writing `...::4411:b105:50:45` is malformed and will be
rejected.
## What can and cannot carry a name
| slot | nameable? |
|---|---|
| `/64` subnet id (`ipv6_pd_prefixid`) | **No** — 8 bits at ESH, two hex digits, no room for a word |
| the gateway's own address | **No** — fixed at `::1` by UniFi, no field for it |
| a UniFi *client* reservation | **No** — UniFi has no IPv6 equivalent of `use_fixedip` |
| **a host taking its own address** | **Yes** — this is the one that works |
⚠ **The original note concluded these names could never appear in a `dig` or
`ip -6` output. That is wrong.** The first three rows are correct, but they only
establish that *UniFi* cannot assign the address. A Linux host can simply take
one within its own advertised prefix, and the router gets no vote. Appliances
with no shell — cameras, most IoT — genuinely cannot, so `1533:FACE5` and
`4DBA:D107` are likely to stay documentation-only.
## Applying it to a host
Do **not** use an `iface … inet6 static` stanza: on Debian that sets
`accept_ra=0`, killing SLAAC and the IPv6 default route — a good way to strand a
headless box. Use an `if-up.d` hook that derives the live prefix instead.
Live example, `/etc/network/if-up.d/ipv6-scheme-addr` on `esh-docker-vm`:
```sh
#!/bin/sh
[ "$IFACE" = ens18 ] || exit 0
(
i=0
while [ $i -lt 30 ]; do
PFX=$(ip -6 -o addr show dev "$IFACE" scope global 2>/dev/null \
| awk '{print $4}' | cut -d/ -f1 | head -1 | cut -d: -f1-4)
if [ -n "$PFX" ]; then
ip -6 addr replace "${PFX}:4411:b105:50:45/64" dev "$IFACE" && exit 0
fi
sleep 2
i=$((i + 1))
done
) >/dev/null 2>&1 &
exit 0
```
Three deliberate properties:
- **The prefix is derived, never hardcoded** — self-heals if Cityside
re-delegates.
- **Backgrounded with a retry** — SLAAC may not have landed when `if-up.d` runs,
and a hook that blocks or fails would stall interface bring-up.
- **Additive** — `/etc/network/interfaces` already sources `interfaces.d/`;
nothing existing is edited, and removal is one `rm`.
Remaining gap: a *mid-life* prefix change is only picked up at the next
interface-up. A timer would close it; not worth building until the prefix is
observed to actually move.
## Deployed
All three Linux hosts on `esh-server` now carry the segment name, with the last
two groups reading straight off their IPv4 address:
| host | address | v4 | applied via |
|---|---|---|---|
| `esh-docker-vm` (AdGuard) | `2607:73c0:402:1d02:4411:b105:50:45` | 10.0.50.45 | `if-up.d` on `ens18` |
| `esh-pve-nas` | `2607:73c0:402:1d02:4411:b105:50:55` | 10.0.50.55 | `if-up.d` on `vmbr0` |
| `esh-vm-db` | `2607:73c0:402:1d02:4411:b105:50:60` | 10.0.50.60 | `if-up.d` on `ens18` |
`esh-docker-vm`'s is load-bearing, not decorative: the ESH UDM advertises an
IPv6 resolver to clients via RDNSS, macOS prefers it over the DHCPv4-supplied
one, so whatever sits there is what resolves `*.internal` for every Mac on the
network. It previously pointed at AdGuard's **MAC-derived SLAAC address**, which
would have broken if that VM's NIC ever changed. Both `esh-userland` and
`esh-server` now advertise the scheme address instead
(`dhcpdv6_dns_auto=false` + `dhcpdv6_dns_1=<address>`), verified on the wire by
soliciting an RA and parsing option type 25.
### ⚠ Proxmox bridges need `accept_ra=2` or SLAAC never runs
`esh-pve-nas` had **link-local only** despite `accept_ra=1`, `autoconf=1` and
IPv6 enabled — every sysctl looked correct. The cause: **`vmbr0.forwarding = 1`**
(Proxmox sets per-interface forwarding on bridges), and the kernel ignores RAs on
a forwarding interface unless `accept_ra` is explicitly **`2`**. `accept_ra=1`
means "accept only if not forwarding", so it silently did nothing.
Fixed in `/etc/sysctl.d/60-ipv6-accept-ra.conf` on that host:
```
net.ipv6.conf.vmbr0.accept_ra = 2
net.ipv6.conf.vmbr0.accept_ra_defrtr = 0
```
`accept_ra_defrtr=0` is deliberate — it takes the advertised **prefix** (so
SLAAC configures an address) while **declining the default route**, so a
hypervisor gains an IPv6 identity with no change to its routing behaviour.
Verified after: SLAAC address present, v6 default routes still **0**, v4 intact.
Expect the same on any other Proxmox node when its LAN gets IPv6.
### Getting into a host with no direct root
`esh-vm-db` refuses key auth for `root` and `infra-ops`, and `lkraven`'s sudo
wants a password. It is VMID 101 on `esh-pve`, and the **QEMU guest agent** runs
as uid 0 inside it, so the hook was installed with:
```
qm guest exec 101 -- /bin/sh -c 'echo <base64> | base64 -d > /etc/network/if-up.d/... '
```
base64 because quoting a multi-line script through two SSH layers mangles it.
Worth remembering as the general path for guests whose credentials are not
vaulted.
Every Linux host on `esh-server` now carries its name. The remaining ESH
segments have no eligible hosts: `esh-cameras` and `esh-iot` are appliances
with no shell, and `esh-mgmt`, `esh-userland` and `Default` are still
`ipv6_interface_type: none` pending the firewall-policy pass — enabling SLAAC
there gives every client a globally reachable address.
docs/pfi/model-quantization-playbook.md
+74 -8
View File
@@ -209,14 +209,31 @@ Open upstream question (queried dvalin/bil-smithy 2026-08-17).
Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
for length-collapse and cross-turn bleed.
**THE MITIGATION (resolved 2026-08-17): disable prefix caching, keep MTP.** The
corruption is gated on MTP × prefix-caching *together* (vllm#43559 / #47194) — with
`--no-enable-prefix-caching` the GDN cache runs in a mode where the buggy
partial-accept align-path is inert. Confirmed on our stack: AEON W4A4, MTP on +
prefix-caching off → the 7-turn varied series stays coherent through 3.9k tokens,
zero bleed, at **104.6 tok/s / 53.6% acceptance** — i.e. the FULL MTP speedup back
(vs ~half with MTP off), losing only prefix-cache reuse. The gen seat runs this
config as of 2026-08-17.
⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What
follows is kept for its history and **must not be applied**: *"disable prefix caching,
keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 /
#47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is
inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn
varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6%
acceptance."*
**It passed that synthetic 7-turn probe and the operator still saw severe degeneration
in real use.** Reverted the same day. The probe was structurally under-covering the
real workload on both content distribution and depth — which is §3.7's own standing
rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at
one level up: the multi-turn probe was itself too small to gate on.
**WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant
being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching
and the gateway all merely *amplifying* it. That is why every partial mitigation
"helped" without fixing anything (§3.8). The gen seat today runs the in-house
Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying
#51113, with **MTP ON and prefix-caching ON**, and is coherent in real use.
Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching`
with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file
(`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current
authority; this section was stale against it for nine days.
Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
@@ -226,6 +243,54 @@ bug — it is architectural). The proper upstream fix (vllm#51113) is in `main`
Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
at the open symptom-twin issue #47087.
### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe
**Symptom: none.** The quant completes, the artifact loads, and 88.5% of the
model is still BF16. Nothing warns you.
Modern MoE checkpoints store each layer's experts as **two fused 3-D
`nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B:
model.language_model.layers.N.experts.gate_up_proj BF16 [128, 1408, 2816]
model.language_model.layers.N.experts.down_proj BF16 [128, 2816, 704]
Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight`
is an `nn.Linear`; `experts.down_proj` is a bare parameter.
Measured on that checkpoint, recipe targeting `["Linear"]`:
Linear modules 427
WILL quantize 205 (experts: 0) <- 22.84 B params untouched
**This is the same defect that killed QLoRA on this architecture** —
`bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also
silently skipped the experts. Two different tools, one blind spot, because the
blind spot is in the *checkpoint layout*, not the tool.
**The fix** (llm-compressor ≥ 0.12):
```python
from llmcompressor.modeling.moe.linearize import linearize_moe
model = SomeForConditionalGeneration.from_pretrained(...)
linearize_moe(model) # BEFORE building the recipe
```
Linear modules 11947
WILL quantize 11725 (experts: 11520) # 30 layers x 128 x 3 proj
`linearize_moe` unfuses the 3-D parameters into per-expert
`experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the
module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus
`gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the
faster variant that linearizes during load rather than after.
**Always assert the expert count before spending GPU time** (§4.1). The
arithmetic is `layers × experts × projections`; if your target list does not
hit it exactly, the recipe is wrong and the failure is silent.
⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
error does not average out downstream, it changes which weights run at all.
### 3.4 Toolchain version deadlocks
Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
@@ -595,6 +660,7 @@ day if followed:
| "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
| "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
| "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
| "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
| "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
---
docs/pfi/proxmox-vms.md
+67 -252
View File
@@ -1,262 +1,77 @@
# PFI-ANA Proxmox VM Inventory
# PFI-ANA Proxmox guest inventory (pfi-pve)
**Hypervisor**: Proxmox VE at `10.250.250.31:8006`
**Storage Pool**: `ospool` (CEPH/zfs — all VM disks reside here)
**Network Bridge**: `vmbr0` with VLAN tag `50` on all VMs
**QEMU Version**: 7.2.0 (primary), VM 106 on 8.1.5
_Refreshed 2026-09-06 04:47Z from live `qm list`, `pct list`, `/etc/pve/*/N.conf` and
guest-agent IPs. Earlier versions of this file described guests that no longer exist
(VM 100 as the TrueNAS box, VM 106 as "PFI-Tailscale", VM 108 "PFI-ANA--DEV"); this
version is the live state._
## Virtual Machines
**Hypervisor:** `pfi-pve` — Dell PowerEdge R750xs, Xeon Silver 4310 (48 threads), 188 GB,
Proxmox VE 8.3.5, `https://10.250.250.31:8006`, `ssh infra-ops@10.250.250.31` (NOPASSWD sudo, since 2026-09-06) or `root@`. iDRAC `https://10.250.250.30`.
### VM 100 — PFI-ANA-TRUENAS
**Storage:**
| Property | Value |
|---|---|
| **VMID** | 100 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 2 cores = 4 vCPU (host passthrough) |
| **Memory** | 8,196 MB |
| **Disk** | `scsi0`: 80G on ospool |
| **CDROM** | `ide2`: TrueNAS-SCALE-22.12.1.iso _(original install media; current OS is Debian 12)_ |
| **Network** | `net0`: virtio, MAC `9A:90:79:7A:86:87`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 2, delay 120s |
| **Onboot** | No |
**Purpose**: NAS/NFS/SMB storage for the colo environment.
**Current OS**: Debian 12 (bookworm), reprovisioned from the original TrueNAS SCALE install — SSH banner `OpenSSH_9.2p1 Debian-2+deb12u9`, no web UI on 80/443, no TrueNAS middleware. Shares are hand-managed via `/etc/exports` and `/etc/samba/smb.conf`. Mount paths kept the `/mnt/<pool>/…` shape of the original ZFS layout for host compatibility.
---
### VM 101 — PFI-ANA-DC
| Property | Value |
|---|---|
| **VMID** | 101 |
| **OS Type** | Windows 11 |
| **BIOS** | OVMF (UEFI) with TPM 2.0 |
| **Machine** | pc-q35-7.2 |
| **CPU** | 2 sockets × 6 cores = 12 vCPU (host passthrough) |
| **Memory** | 24,576 MB |
| **Disk** | `scsi0`: 240G on ospool |
| **EFI Disk** | `efidisk0`: 1M on ospool |
| **TPM** | `tpmstate0`: 4M, v2.0 on ospool |
| **CDROM** | `scsi1`: virtio-win-0.1.229.iso (VirtIO drivers) |
| **Network** | `net0`: e1000, MAC `CE:C8:D7:FE:32:40`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → net0 → ide0 → scsi1 |
| **Startup** | Order 5, delay 120s |
| **Onboot** | Yes |
**Purpose**: Windows Domain Controller for the Anaheim environment. UEFI with TPM 2.0 suggests Active Directory / Group Policy services.
---
### VM 102 — PFI-ANA-Docker
| Property | Value |
|---|---|
| **VMID** | 102 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) |
| **Memory** | 16,384 MB |
| **Disk** | `scsi0`: 250G on ospool |
| **Network** | `net0`: virtio, MAC `BA:AF:E7:E9:79:23`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → net0 → scsi1 |
| **Startup** | Order 4 |
| **Onboot** | Yes |
**Purpose**: Primary Docker host for the colo. Runs Dockge for compose management and Traefik for reverse proxy. All Docker services documented in [docker-stack.md](docker-stack.md) run here.
---
### VM 103 — PFI-SlaveBot
| Property | Value |
|---|---|
| **VMID** | 103 |
| **OS Type** | Windows 10 |
| **Machine** | pc-i440fx-7.2 |
| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
| **Memory** | 8,192 MB |
| **Disk** | `ide0`: 256G on ospool |
| **Network** | `net0`: e1000, MAC `CE:F0:49:C9:03:70`, vmbr0, VLAN 50 |
| **Boot** | ide0 → net0 → scsi0 |
| **Startup** | Not configured |
| **Onboot** | Yes |
**Purpose**: Windows 10 workstation/bot. Likely a task automation or RDP-accessible machine.
---
### VM 104 — PFI-Mongo
| Property | Value |
|---|---|
| **VMID** | 104 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
| **Memory** | 8,196 MB |
| **Disk** | `scsi0`: 256G on ospool |
| **Network** | `net0`: virtio, MAC `32:57:90:B2:66:61`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 3, delay 60s |
| **Onboot** | Yes |
**Purpose**: MongoDB server. Config file contains a connection string reference: `mongodb://10.250.50.81:27017/`.
---
### VM 105 — PFI-Postgres
| Property | Value |
|---|---|
| **VMID** | 105 |
| **OS Type** | Linux (l26) |
| **CPU** | 4 sockets × 4 cores = 16 vCPU |
| **Memory** | 8,196 MB |
| **Disk** | `scsi0`: 80G on ospool |
| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
| **Network** | `net0`: virtio, MAC `C2:1F:CC:71:66:D0`, vmbr0, VLAN 50 |
| **Serial** | `serial0`: socket (IPMI/serial console) |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 3, delay 60s |
| **Onboot** | Yes |
**Purpose**: PostgreSQL database server running Debian 11.
---
### VM 106 — PFI-Tailscale
| Property | Value |
|---|---|
| **VMID** | 106 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) |
| **Memory** | 2,048 MB |
| **Disk** | `scsi0`: 256G on ospool |
| **CDROM** | `ide2`: debian-12.2.0-amd64-netinst.iso |
| **Network** | `net0`: virtio, MAC `BC:24:11:D7:E9:52`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Not configured |
| **Onboot** | Yes |
**Purpose**: Tailscale VPN node for mesh connectivity. Provides the VPN tunnel endpoints that link the three PFI sites together. Running Debian 12 (newer than most other VMs). Lightweight at 2G RAM.
---
### VM 107 — PFI-Pteradactyl
| Property | Value |
|---|---|
| **VMID** | 107 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
| **Memory** | 8,192 MB |
| **Disk** | `scsi0`: 256G on ospool |
| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
| **Network** | `net0`: virtio, MAC `CA:44:37:8A:BF:E0`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Not configured |
| **Onboot** | Yes |
**Purpose**: Pterodactyl game server panel. Manages game server instances.
---
### VM 108 — PFI-ANA--DEV
| Property | Value |
|---|---|
| **VMID** | 108 |
| **OS Type** | Linux (l26) |
| **CPU** | 2 sockets × 4 cores = 8 vCPU |
| **Memory** | 8,192 MB |
| **Disk** | `scsi0`: 120G on ospool |
| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
| **Network** | `net0`: virtio, MAC `F2:EF:82:2C:AF:90`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 10, delay 60s |
| **Onboot** | No |
**Purpose**: Development environment. Not set to auto-boot, starts after core infrastructure (order 10).
---
### VM 110 — PFI-ANA-Webhost
| Property | Value |
|---|---|
| **VMID** | 110 |
| **OS Type** | Linux (l26) |
| **CPU** | 4 sockets × 4 cores = 16 vCPU |
| **Memory** | 4,096 MB (balloon: 1024 MB minimum) |
| **Disk** | `scsi0`: 250G on ospool |
| **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
| **Network** | `net0`: virtio, MAC `E6:F9:3A:C9:61:2A`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 30, up 120s, down 120s |
| **Onboot** | Yes |
**Purpose**: Web hosting server. Highest startup order (30) — starts last. Also has the longest graceful shutdown timeout (120s). Memory ballooning enabled for dynamic allocation.
---
### VM 111 — pfi-tacticalrmm
| Property | Value |
|---|---|
| **VMID** | 111 |
| **OS Type** | Linux (l26) |
| **CPU** | 4 sockets × 4 cores = 16 vCPU |
| **Memory** | 8,192 MB |
| **Disk** | `scsi0`: 256G on ospool |
| **Network** | `net0`: virtio, MAC `BA:FA:65:F6:46:25`, vmbr0, VLAN 50 |
| **Boot** | scsi0 → ide2 → net0 |
| **Startup** | Order 20 |
| **Onboot** | Yes |
**Purpose**: Tactical RMM (Remote Monitoring and Management) server. Provides IT management, remote access, and monitoring capabilities.
---
## Startup Order Summary
VMs are brought up in the following order on host boot:
| Order | VMID | Name | Delay |
| id | type | backs | notes |
|---|---|---|---|
| 2 | 100 | PFI-ANA-TRUENAS | 120s |
| 3 | 104 | PFI-Mongo | 60s |
| 3 | 105 | PFI-Postgres | 60s |
| 4 | 102 | PFI-ANA-Docker | — |
| 5 | 101 | PFI-ANA-DC | 120s |
| 10 | 108 | PFI-ANA--DEV | 60s |
| 20 | 111 | pfi-tacticalrmm | — |
| 30 | 110 | PFI-ANA-Webhost | 120s |
| `ospool` | zfspool, 2×mirror (bays 0–3), 10.9T raw | **every VM boot disk and every CT rootfs** except pbs-ana | the pool that boots the site; also holds `ospool/naspool-evac` (parked copy from the 2026-09-06 rebuild, to be destroyed) |
| `NASPool` | zfspool, **raidz2 ×6** (bays 4–8, 11), 21.7T usable | backup / bulk tier: `backupStore` (PBS-ANA datastore, restic repos), `pve-VMStorage`, `db`, `docker`, `webdav` | rebuilt 2026-09-06 (`docs/runbooks/pfi-pve-naspool-rebuild.md`); bays 9/10 dead (backplane), drives in them = cold spares |
| `naspool-vmstorage` | dir on `/NASPool/pve-VMStorage` | ISOs, templates, legacy vzdump (newest-per-guest only) | was `pve-truenas` until 2026-09-06 |
| `local-lvm` | lvmthin | pbs-ana's boot disk | |
| `local` | dir | ISOs | |
| `pbs-ana` | pbs at 10.250.50.90 | vzdump jobs (03:00 all guests except 100,109; 22:00 CT 109) | datastore lives on `NASPool/backupStore` via ana-nas NFS |
VMs without a startup order (103, 106, 107) will start based on their `onboot` setting but without a specific sequencing delay.
**Network:** all guests on `vmbr0` VLAN tag 50 (`10.250.50.0/24`), firewall enabled on
every NIC. Linux = virtio, Windows = e1000. CTs take DHCP (reserved).
## Resource Summary
## Virtual machines
| VMID | Name | vCPU | RAM (MB) | Disk | OS |
|---|---|---|---|---|---|
| 100 | PFI-ANA-TRUENAS | 4 | 8,196 | 80G | Debian 12 (reprovisioned from TrueNAS SCALE) |
| 101 | PFI-ANA-DC | 12 | 24,576 | 240G | Windows 11 |
| 102 | PFI-ANA-Docker | 8 | 16,384 | 250G | Linux |
| 103 | PFI-SlaveBot | 8 | 8,192 | 256G | Windows 10 |
| 104 | PFI-Mongo | 8 | 8,196 | 256G | Linux |
| 105 | PFI-Postgres | 16 | 8,196 | 80G | Debian 11 |
| 106 | PFI-Tailscale | 8 | 2,048 | 256G | Debian 12 |
| 107 | PFI-Pteradactyl | 8 | 8,192 | 256G | Debian 11 |
| 108 | PFI-ANA--DEV | 8 | 8,192 | 120G | Debian 11 |
| 110 | PFI-ANA-Webhost | 16 | 4,096 | 250G | Debian 11 |
| 111 | pfi-tacticalrmm | 16 | 8,192 | 256G | Linux |
| | **Totals** | **112** | **105,348** | **2,300G** | |
| VMID | name | IP | vCPU (sockets×cores) | RAM | disk | OS | onboot / startup | role |
|---|---|---|---|---|---|---|---|---|
| 100 | pbs-ana | 10.250.50.90 | 4 (1×4, x86-64-v2-AES) | 8 G | 32G local-lvm | Debian 12 | yes / — | Proxmox Backup Server, fleet primary. `servers/pbs-ana/` |
| 101 | PFI-ANA-DC | 10.250.50.60 | 12 (2×6, host) | 24 G | 240G ospool + EFI + TPM2 | Windows 11, q35, OVMF | yes / order 5, up 120s | Active Directory domain controller |
| 102 | PFI-ANA-Docker | 10.250.50.70 | 8 (2×4, x86-64-v2-AES) | 16 G | 250G ospool | Debian | yes / order 4 | `ana-docker`: LiteLLM gateway, gitea, Langfuse, AdGuard, Beszel/Dozzle hubs, rest-server-ana, backrest. `servers/ana-docker/` |
| 103 | PFI-SlaveBot | 10.250.50.151 | 8 (2×4, host) | 8 G | 256G ospool (ide0) | Windows 10, i440fx | yes / — | purpose undocumented; **not in `servers/`** |
| 104 | PFI-Mongo | 10.250.50.81 | 8 (2×4, host) | 8 G | 256G ospool | Linux | yes / order 3, up 60s | MongoDB; **not in `servers/`** |
| 105 | PFI-Postgres | 10.250.50.80 | 16 (4×4) | 8 G | 80G ospool | Debian 11 | yes / order 3, up 60s | shared Postgres (vaultwarden/gitea/paperless); PGDATA local since 2026-04-23. `servers/pfi-postgres/` |
| 106 | corviduo-dev | 10.250.50.152 | 8 (2×4, x86-64-v2-AES) | 8 G | 80G ospool | Debian 13 | yes / order 10 | Worldtree-team dev VM, PFI-hosted. `servers/corviduo-dev/` |
| 107 | PFI-Pteradactyl | 10.250.50.55 | 8 (2×4, host) | 8 G | 256G ospool | Debian 11 | yes / — | Pterodactyl game panel. `servers/pfi-pteradactyl/` |
| 110 | PFI-ANA-Webhost | 10.250.50.52 | 16 (4×4) | 4 G (balloon 1 G) | 250G ospool | Debian 11 | yes / order 30, up/down 120s | web workload. `servers/pfi-ana-webhost/` |
| 111 | pfi-tacticalrmm | 10.250.50.57 | 16 (4×4) | 8 G | 256G ospool | Linux | yes / order 20 | TacticalRMM. `servers/pfi-tacticalrmm/` |
## Network Notes
All VMs have the QEMU guest agent enabled.
- All VMs are on **VLAN 50** via `vmbr0`.
- All VMs have **firewall enabled** on the network interface.
- Linux VMs use `virtio` network adapters; Windows VMs use `e1000`.
- The MongoDB connection string embedded in VM 104's config references IP `10.250.50.81`, suggesting VLAN 50 maps to the `10.250.50.0/24` subnet within the `10.250.0.0/16` range.
## Containers
| CTID | hostname | IP | cores | RAM | rootfs | privileged | onboot / startup | role |
|---|---|---|---|---|---|---|---|---|
| 109 | ana-nas | 10.250.50.50 | 4 | 2 G | 80G ospool | **yes** (apparmor unconfined, all devices) | yes / order 1, up 45s | NFS/SMB server straight off the ZFS pools — bind-mounts `/ospool/backup` and every `NASPool/*` dataset under `/mnt/…`. Data-serving SPOF. `servers/ana-nas/` |
| 112 | ana-filebot | 10.250.50.53 | 4 | 2 G | 80G ospool | **yes** (same binds as 109) | yes / order 10 | file-task automation. `servers/ana-filebot/` |
| 113 | ana-wg | 10.250.50.252 (+ wg0 10.30.10.1) | 4 | 2 G | 8G ospool | no (nesting) | yes / order 2 | WireGuard remote-access endpoint, UDP 31337 via FortiGate VIP. `servers/ana-wg/` |
## Startup order (host boot)
| order | guest | delay |
|---|---|---|
| 1 | CT 109 ana-nas | up 45s |
| 2 | CT 113 ana-wg | — |
| 3 | VM 104 PFI-Mongo, VM 105 PFI-Postgres | up 60s |
| 4 | VM 102 PFI-ANA-Docker | — |
| 5 | VM 101 PFI-ANA-DC | up 120s |
| 10 | VM 106 corviduo-dev, CT 112 ana-filebot | — |
| 20 | VM 111 pfi-tacticalrmm | — |
| 30 | VM 110 PFI-ANA-Webhost | up/down 120s |
| unordered | VM 100 pbs-ana, VM 103, VM 107 | onboot=1, no sequencing |
Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap).
## Resource summary
| | vCPU | RAM | disk |
|---|---|---|---|
| 10 VMs | 108 | 100 G | 2,180 G |
| 3 CTs | 12 | 6 G | 168 G |
| **total** | **120** on 48 threads (2.5× overcommit) | **106 G** of 188 G | on ospool 547 G actually allocated (thin) |
## Not on this hypervisor
`ana-ml2` (10.250.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
SureFire client's own Dell R630; `pfi-gx10` is at NH3.
docs/pfi/training-throughput-playbook.md
+1179
View File
File diff suppressed because it is too large Load Diff
docs/pfi/vm-102-matrix-synapse.md
+164
View File
@@ -11,6 +11,170 @@ human-to-agent communication channel for AIPA. The stack consists of:
Federation is disabled (internal-only deployment). Registration is disabled (admin-created accounts only).
> ⚠ **The build steps below date from the AIPA era and describe how this was
> originally stood up.** They are kept for provenance. For what is actually
> running, read *Current state* immediately below — the versions, the appservice
> owner, and the bridge have all moved since.
---
## Current state — 2026-09-01
| | |
|---|---|
| Synapse | **v1.159.0** (upgraded from v1.120.0, 2026-09-01 — 39 releases, 21 months) |
| Element Web | **v1.12.27** (from v1.11.80) |
| PostgreSQL | **16-alpine** — deliberately NOT upgraded alongside Synapse |
| Appservice | registration `id: aipa-bridge`, but the service behind it is **`wt-matrix-bridge`**, the Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. The `aipa` name is legacy. |
| Accounts | 15 — 13 appservice-owned, plus `@vhoang` (admin) and `@miranda` |
| Encryption | **zero encrypted rooms**, server-wide |
| Federation | `federation_domain_whitelist: []` — present but EMPTY, so federation is effectively OFF. The federation endpoint still answers, which is misleading. |
**Ops ownership:** worldtree-dev writes the bridge code; **infra-ops operates this
instance** and has full authority to change it (operator, 2026-09-01). Do not route
provisioning or config requests to worldtree-dev.
### Upgrade procedure (learned 2026-09-01)
**Synapse schema migrations are one-way.** v1.120 cannot start against a v1.159
database. Rollback is restore-from-dump, not revert-the-tag. Therefore:
```bash
D=/opt/docker/backups/synapse-preupgrade-$(date -u +%Y%m%dT%H%M%SZ)
docker exec synapse-db pg_dump -U synapse -d synapse --format=custom --compress=6 > $D/synapse.dump
docker exec -i synapse-db pg_restore --list < $D/synapse.dump | wc -l # verify it reads back
```
Snapshot `compose.yaml`, `homeserver.yaml`, `aipa_appservice.yaml` and
`element-config.json` alongside it. Then bump tags and `up -d`; watch for
`Schema now up to date` in the logs.
A benign startup warning — `Failed to listen on 0.0.0.0 ... Address already in use`
— is normal Linux dual-stack behaviour after it binds `[::]:8008`. Not a fault.
### Appservice namespace — why `exclusive` is false
The `aipa-bridge` registration claims a user namespace of
`@[a-z][a-z0-9_-]*:matrix.phasefinal.com` — effectively every localpart on the
server. It was `exclusive: true` until 2026-09-01.
**`exclusive` governs who ELSE may act, not what the appservice may do.** On a
homeserver with registration disabled, one admin, and no competing actor, it bought
anti-squatting protection against a threat that cannot occur, while locking out
every other means of account creation — admin shared-secret registration returned
`M_EXCLUSIVE` for any conventional name, with no explanation of why.
Now `exclusive: false`, regex unchanged. The bridge is unaffected: it creates and
puppets exactly as before. The narrow **aliases** namespace (`#aipa-debug-*`) is
left exclusive — it is specific and costs nothing.
⚠ **Do not narrow the users regex to a prefix.** All 13 appservice accounts fall
inside it; narrowing would orphan them. ⚠ **Do not rename the `id`** — Synapse keys
account ownership on `aipa-bridge` in the `users` table. The FILE may be renamed.
### Shared-secret registration — the HMAC has no trailing null
`POST /_synapse/admin/v1/register`. The MAC is
`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**,
it does not **terminate**. A trailing `\x00` after `notadmin` yields `HMAC incorrect`.
Run it inside the container against `localhost:8008`; port 8008 is not published to
the host.
---
## Conventions
### User IDs
`@<agent>:matrix.phasefinal.com` — one account per agent or persona.
### Room aliases — ratified by the operator 2026-09-01
```
#<agent>-<purpose>:matrix.phasefinal.com
```
Proposed by ledger-dev, ratified by the operator. Mirrors the user-ID convention
rather than introducing a second, independent axis.
- **Ownership is legible from the alias alone** — `#miranda-summons` is
unambiguously Miranda's without opening the room.
- **Namespaced on the AGENT, not a product**, so it survives a consuming project's
rename.
- **`#aipa-debug-*` is avoided by construction** — no agent is named `aipa-debug`,
and that range is the bridge's exclusive alias namespace.
- **Load-bearing rationale, not tidiness:** purpose-as-tier is what makes a summons
triageable **so the room IDENTITY carries the tier**. (Stated carefully: it is
*not* that the push payload carries the room name — see the pusher note below.)
**Known cost, accepted deliberately:** a genuinely shared room with no single owner
has no natural slot. Preferred over pre-building a namespace for a room that may
never exist; name that case deliberately if it arrives.
**Pre-existing rooms are not renamed.** "The High Seat"
(`!NiVVoMsyoHCBRPrrrn`) predates the convention, has no canonical alias, and works.
Renaming a live proven channel to satisfy a scheme it did not exist under is the
churn the convention exists to prevent. New rooms follow it.
---
## Push notifications — what actually reaches a phone
Established 2026-09-01 while proving Ledger's summons channel.
The registered pusher (`@vhoang`, Element X iOS) uses
**`"format": "event_id_only"`** via `https://matrix.org/_matrix/push/v1/notify`.
That payload carries `event_id`, `room_id` and counts — **no room name, no sender,
no content.**
It still produces a useful notification because `mutable-content: 1` means Element X
runs a **Notification Service Extension**: iOS wakes it with the near-empty payload,
and the extension then **fetches the event from the homeserver and renders the
notification on the device.**
Three consequences:
1. **The tier-in-room-identity scheme works, but via the client fetch** — not
because the push carried the name. `m.room.name` must still be set at creation.
2. **`push: include_content: false` is irrelevant for clients with an NSE.** The
extension reads the event directly. It *does* bite clients without one.
3. **Server-invisible failure mode:** if the phone cannot reach
`matrix.phasefinal.com` at wake time, the fetch fails and iOS shows the fallback
— the bare word "Notification". **Synapse records `last_success` and sees a
delivered push.** The degradation happens entirely on the device.
Because the payload carries nothing meaningful, **self-hosted sygnal was considered
and rejected** — it would remove matrix.org from a path that already carries no
content, room name or sender.
⚠ Sygnal is a **relay** to FCM/APNs, not a replacement for them. Self-hosting it
removes matrix.org and nothing else. "Notifications must not leave the fleet" is not
satisfiable with mobile push at any price.
---
## Sign in with QR code — requires Matrix Authentication Service
Asked 2026-09-01; **not enabled.** From Synapse 1.159.0's own config validator:
```python
if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) \
and not auth_delegated:
raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled")
```
QR sign-in is **MSC4108**, an OIDC flow that hard-requires **MAS**. Synapse refuses
to start with the flag set and no MAS.
**MSC4388** (`msc4388_mode: off|open|authenticated`) enables independently, but it is
only the **rendezvous channel** — the transport, not a login flow.
Deferred because MAS is a service, a database, and a migration of every account off
built-in auth — and the v1.139.0 upgrade note warns that `/register` from **old
application service implementations may break under MAS**, which is precisely the
bridge owning 13 of the 15 accounts. Revisit when MAS becomes mandatory rather than
optional, in its own change window, with the bridge regression-tested either side.
---
## Architecture
docs/runbooks/althing-deploy.md
+377
View File
@@ -0,0 +1,377 @@
# althing deploy — six surfaces
**`scripts/deploy-althing.sh`** does all four. `--check` reports drift without
touching anything. This page is the why.
Deploying althing means updating **four independent surfaces on nh3-dev**. Three
were known; the fourth had no step in any runbook and drifted for five days
before anyone noticed, on 2026-09-01.
| # | surface | what it is | how it drifts |
|---|---|---|---|
| 1 | **binaries** | `uv tool install --force .` — the 7 console scripts | silently, see below |
| 2 | **herald** | `systemctl --user restart althing-po-herald` | new guards live here; nothing takes effect until it restarts |
| 3 | **skill** | `scripts/sync_skill.sh` → `~/.agents/skills/althing/SKILL.md` | covered by its own `--check` |
| 4 | **plugin** | repo `plugin/` → marketplace dir → Claude Code cache | **two hops, neither was automated** |
| 5 | **per-seat route** | `althing-route declare` on each seat | a channel change needs a re-declare; the plugin's SessionStart hook does it for CC seats |
| 6 | **`~/.claude/settings.json`** | `crossSessionInbound: "accept"` | **outside the althing repo entirely — no version can carry it** |
## ⚠ `uv tool install .` without `--force` is a silent no-op
$ uv tool install .
`althing-core @ file:///home/lkraven/development/althing` is already installed
$ echo $?
0
uv matches on the source **spec**, not its **contents**. On a box that already
installed from that path it declines and reports success. An operator following
a runbook literally would restart the herald, see everything green, and wonder
why the new binary was missing — with every command exiting 0.
## ⚠ Surface 4 is the one that bites, and it ate a hook
The chain is:
repo plugin/ → ~/.local/share/althing-plugin/ → ~/.claude/plugins/cache/althing/althing/<ver>/
^^^ nothing synced this hop
On 2026-09-01 the marketplace directory was a frozen copy from **Aug 28**:
deployed 0.0.1 hooks: ['UserPromptSubmit']
repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
+ scripts/pane-route.sh (absent from the deployment entirely)
So **"CC seats re-declare their pane route automatically at next SessionStart"
was never true on this box.** The hook existed and worked upstream; it was never
deployed. That is why every seat — including infra-ops' own Claude Code seat —
had to be hand-declared with a pid someone measured by hand, and why the
`idle_cursor` pin from 3.2.1 would not have self-applied either.
### ⚠ Compare the HOOK LIST, not the version string
A version number cannot tell you what a stale plugin actually cost you. 0.0.1 and
0.1.1 differ by **two hooks and a script**:
```bash
for d in ~/.claude/plugins/cache/althing/althing/*/; do
printf " %-8s hooks: %s pane-route.sh: %s\n" "$(basename "$d")" \
"$(python3 -c "import json;print(list(json.load(open('$d/.claude-plugin/plugin.json'))['hooks'].keys()))")" \
"$(test -f "$d/scripts/pane-route.sh" && echo yes || echo NO)"
done
```
Expect `['UserPromptSubmit', 'SessionStart', 'SessionEnd']` and `pane-route.sh: yes`.
Anything less and pane routes are not being declared or released automatically.
### ⚠ WHY THIS HID FOR FIVE DAYS — the shape to recognise
**A missing deploy surface does not present as an error. It presents as
"the migration needs manual work"** — and we had a ready explanation for that,
because four of five seats were non-Claude and genuinely did need hand-holding.
The seat that falsified the story was **infra-ops' own**: a Claude Code seat that
should have self-declared and didn't, and it looked exactly like the other four.
Neither infra-ops nor forseti asked why the automatic path had not fired for the
one seat it was supposed to work on.
⚠ **When a migration needs manual intervention, check whether the automatic path
was ever deployed before concluding it does not apply to your case.**
**Use the supported CLI for the second hop.** `claude plugin update althing`
(also `install` / `uninstall` / `list` / `details` / `validate` / `marketplace`).
⚠ Do **not** hand-edit `~/.claude/plugins/installed_plugins.json` or fabricate a
cache directory — that is Claude Code's own bookkeeping, and a subtle mistake
there breaks the plugin in a way that looks like an upstream bug.
⚠ **A Claude Code restart is required** for new plugin hooks. They are read at
session start; a running session keeps the old ones.
## Verifying a seat is actually reachable
`postbox status` reports `mode: push`, but until 3.2.4 a **failed** push
declaration was silent — `_declare_push` devnulled both streams with
`check=False`, so a missing binary, a non-zero exit and a server-side rejection
rendered identically as nothing. A seat could have been silently pull-only since
3.1.2.
Audit for "looks armed but is not", cross-referencing local waiter locks against
what the post office believes:
```bash
for f in ~/.althing/wake-listener-*.lock; do
h=$(basename "$f" .lock); h=${h#wake-listener-}
pid=$(cat "$f" 2>/dev/null)
[ -n "$pid" ] && [ -r "/proc/$pid/cmdline" ] || continue
# IDENTITY, not liveness: a lock left by a reaped listener names a pid the
# kernel is free to hand to anything, so `kill -0` alone reports a stranger
# as a live waiter. Match the same cmdline segment `althing-listen --stop`
# requires before it will signal anything.
ours=0
while IFS= read -r -d '' seg; do [ "$seg" = "--_route=$h" ] && ours=1; done \
< "/proc/$pid/cmdline"
[ "$ours" = 1 ] || { printf " %-24s pid %-8s STRANGER (recycled pid)\n" "$h" "$pid"; continue; }
printf " %-24s waiter %-8s mode: %s\n" "$h" "$pid" \
"$(postbox status --handle "$h" 2>/dev/null | grep -oP 'mode: \K\w+')"
done
```
A live waiter reporting `mode: pull` is a seat that will never be poked. From
3.2.4 onward `$ALTHING_ROOT/listen.log` records failed declarations directly.
⚠ **Liveness is not identity, and this loop is the place that gets it wrong.**
The earlier `kill -0` form would print a phantom waiter for any handle whose
stale lock happens to name a recycled pid — and a phantom waiter is exactly what
sends a false "you are unreachable" notice to a seat that is fine.
`session_listener.sh` refuses to SIGTERM on liveness alone for precisely this
reason; an audit that only *reads* has no excuse for a weaker standard than the
one that *kills*. (Adopted 2026-09-02 after the regin-smithy-dev
cross-reference, where the pid happened to be genuine and the weaker check
happened to be right.)
⚠ **A stale `wake-listener-*.lock` is NOT a fault.** The gate is `flock -n` on an
open fd, which the kernel releases when the holder dies, so a lock file left by a
reaped listener is inert and exit 3 only fires against a genuinely live holder.
The pid in the file is read by `--stop` alone. (Recorded because infra-ops
claimed the opposite, untested, on 2026-09-01; forseti measured it.)
## ⚠ The plugin cache does not take a content-only change
`claude plugin update althing` matches on the version in `plugin.json` and
declines when it is unchanged. A release that edits hook or script CONTENT
without bumping the plugin version therefore leaves the Claude Code cache
stale while **every version check reports success** — the marketplace dir gets
the new bytes, the cache does not, and `update` says "already at the latest
version".
Measured on the 3.3.0 deploy (2026-09-02): marketplace and live cache both read
`0.1.1`, and `pane-route.sh` + `README.md` differed. That delta was
documentation-only and harmless. The next one need not be.
`scripts/deploy-althing.sh` now diffs the marketplace tree against the live
cache dir on every run and on `--check`, so the drift is reported rather than
inferred. **The fix is a plugin version bump upstream, not a hand-edit** —
`~/.claude/plugins/installed_plugins.json` and the cache directory are Claude
Code's own bookkeeping.
The 2026-09-01 lesson was *compare the hook list, not the version string*. This
is the same lesson one turn deeper: here the hook list was identical too, and
only the file contents moved.
## A seat already running when the plugin lands never auto-declares — leave it
The SessionStart hook fires at session start and nowhere else, so any session
that was already running when a plugin deploy landed keeps its old channel
until it restarts. After the 3.3.1 deploy that was **eight of seventeen live
seats**, all still on `fifo`.
**Operator ruling 2026-09-02: leave them.** They are reachable now, the post
office holds mail regardless, and each self-corrects at its next session start.
The residual exposure is that a fifo waiter is what Claude Code 2.1.257 reaps —
one seat was reaped three times in a morning — and a reap converts a healthy
seat into a silent one until the next restart. That costs timeliness, never
mail.
⚠ **Do not mass-message the affected seats.** Standing operator directive: no
unsolicited fleet notification, ever, unless asked. Direct correspondence with
a counterparty about work in hand is fine; sweeping the handle list is not.
Audit it, report the count, and let him decide — as he did here.
The one-liner, for a seat that asks or an operator who wants one moved:
althing-route declare --discover-pid
## ⚠ The `cc` channel is an undocumented interface, taken deliberately
From 3.3.0 a Claude Code seat is poked over its own message socket
(`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) rather than by typing into its pane.
No process to reap, nothing near the input line, and delivery lands at the
receiver's next turn boundary.
It is not a published interface and **is expected to break on some future
Claude Code release**. Accepted on the operator's reasoning: the FIFO poker it
replaces was also an unsanctioned hack of the background-watcher system, so
this is a better instance of a dependency class we already had, not a new one.
When it breaks the failure mode is a seat going **pull-only with a logged
reason** — not lost mail; the post office holds it either way. Recovery is one
command:
althing-route declare --pid <pid> # prefers cc, falls back to pane
**Why cc is worth an undocumented dependency, in one measurement.** A pane poke
is typed into the session and submitted, so Claude Code records it in the
operator's prompt history exactly as if he had written it — 8 herald-written
entries accumulated across three projects, permanently in up-arrow recall. A cc
poke enters none: 2 delivered, 0 entries. Verified from the herald log against
the history on 2026-09-02. Channel sequence on one seat that day, all three in
order:
09-01 21:44 poked via fifo the old way, later reaped
09-02 08:34 could NOT reach via pane guard 4 refusing: he was typing
09-02 09:07 poked via pane typed into his input line
09-02 09:14 poked via cc arrived as a peer message
09-02 09:31 poked via cc after crossSessionInbound accept
⚠ **Count herald pokes by the whole display string, not by grepping
`ALTHING-HERALD`.** The substring matches the operator's own messages when he
pastes a hold notice into one, so the count runs high and the excess reads as
new activity. See the identity-not-resemblance note in the audit section.
⚠ **Ordering constraint at any upgrade that changes the channel set:** a
pre-3.3.0 herald refuses `channel=cc` at parse, so a seat that declares the new
channel before the herald restarts goes silently pull-only. `uv tool install
--force` then the herald restart, and only then let anything re-declare.
`deploy-althing.sh` already runs them in that order and updates the plugin
(whose SessionStart hook declares the route) last, which is what keeps the
window to a couple of seconds.
## ⚠ Surface 6 — `crossSessionInbound`, and it is not in the althing repo
**A cc poke to a default-configured seat is HELD, not delivered.** Claude Code
auto-delivers an inbound cross-session message only when the sender's
permission-mode class matches the receiver's, and **a sender that asserts no
class is held**. The herald is a daemon and asserts none, deliberately. What
the operator sees instead of a delivery:
Held peer message — from an unidentified session [verified pid <herald>];
preview: «ALTHING-HERALD althing: you have mail ...» — not delivered to
Claude (1 held). The sender did not attest its permission mode and this
session bypasses prompts.
Claude Code verified the herald through `SO_PEERCRED` and then correctly
declined to let it speak, because it would not say what it was.
**The fix is one key in `~/.claude/settings.json`:**
"crossSessionInbound": "accept"
Set on nh3-dev 2026-09-02. The operator's reasoning is the part to keep: *the
herald can only reach local seats, so bypass is the correct authorization type;
when the guard was not there, that was our default posture, including the pane
poke.* A pane poke types into a session and presses Enter — bypass-level access
by any measure, and what we had been doing all along. **The socket channel is
strictly narrower than what it replaces**, so accepting here states the existing
trust boundary rather than widening it.
⚠ **There is no attestation the herald could send instead.** Four probes
established that Claude Code identifies a sender by verified pid against the
session registry and reads that session's *live runtime* permission mode; a
daemon has none and the registry has no field for one. A `from_mode` field on a
`type:"user"` frame is not consulted at all — it belongs to the control actions.
Adding it would be shipping a field nothing reads.
**Cost without the setting is smaller than it looks:** the correspondent record
is in-memory session state, so the hold is **first-contact, not per-message** —
one approval per seat per session lifetime, and a long-lived pane pays it once.
⚠ **Why this surface is the dangerous one.** A seat without it is declared,
reachable and green, and its pokes go to a human instead of to the session.
That is the same shape as the SessionStart hook that was never deployed:
everything reports success, nothing arrives. `deploy-althing.sh` reads the key
and reports it on every run and on `--check` — and **deliberately never sets
it.** That file is the operator's permission configuration; a deploy script
that edits it is a deploy script granting itself trust.
## The status-line segment belongs to althing, not to this repo
From althing **3.4.0** the segment is a console script, `althing-statusline`,
installed on PATH by the same `uv tool install --force` as the other seven
executables. `scripts/claude-statusline-command.sh` calls it and does not
reimplement it:
althing=$(printf '%s' "$input" | timeout 5 althing-statusline 2>/dev/null)
⚠ **Keep that outer timeout LOOSER than the program's own 2 s budget.** If the
outer one fires first the segment comes back empty, which renders as "not an
althing directory" — the outage conflation, reintroduced by the guard meant to
prevent a hang.
**Why a console script rather than the `sync_skill.sh` canonical-file pattern:**
a script, unlike a document, has somewhere to be *installed*. Installing it makes
drift impossible instead of merely visible.
**Why it needed solving at all.** On 2026-09-02 the same twelve lines existed in
three places — althing's `plugin/scripts/statusline.sh`, the operator's wired
`~/.claude/statusline-command.sh`, and this repo's copy — and one audit found a
**different defect in each**. All three were fixed separately, by hand, the same
day. Two of the defects were in this repo's copy and are worth knowing because
they are both silent:
- the handle was resolved as the most recent launch in the directory, so a
directory hosting both a claude and a codex seat (`~/development/Worldtree`)
reports the **codex** handle's unread count and reachability to a Claude
session the moment the codex pane relaunches last;
- the post-office address was hardcoded, so it survives exactly until the post
office moves and then reports an outage that is really a stale constant.
### Deploying it
`scripts/claude-statusline-command.sh` is the tracked copy of what Claude Code
actually runs at `~/.claude/statusline-command.sh` (`settings.json` →
`statusLine.command`). Copies, not symlinks — same rule as `stacks/`.
diff -u scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh
cp scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh
⚠ **Diff before you copy, in both directions.** On 2026-09-02 the live path was
edited directly and was *ahead* of the tracked copy; a reflexive `cp` from the
repo would have destroyed the better version. The tracked copy is intent, the
live path is reality, and reality is sometimes right.
Smoke test — **strip the althing env vars, or you will test your own session's
identity instead of the resolution logic.** A hand-run inherits the exported
`ALTHING_HANDLE` from the session you run it in, so the seat's own directory and
a directory with no althing identity render identically and both look right.
That false pass caught two agents within one hour on 2026-09-02.
⚠ **Open question — whether Claude Code's own invocation inherits the
environment is NOT established, and nothing here depends on it.** An earlier
version of this page asserted a "bare shell"; that claim was never verified and
the evidence points the other way. Measured 2026-09-02: this box's Claude Code
process carries `ALTHING_HANDLE` and `ALTHING_POST_OFFICE` in its own `environ`
(inherited from the `dev-launch` shell), and a spawned child would normally
inherit both — observed independently on two seats. Whether Claude Code scrubs
the environment before spawning the status-line command was not observed either
way. **Write code that does not depend on the environment being present**, which
`althing-statusline` already does: environment first, launch history second, and
a seat started outside `dev-launch` has no handle in either place. Test with the
variables stripped regardless, because that is the harder case:
probe() { # $1 = cwd, $2 = what you expect
got=$(printf '{"workspace":{"current_dir":"%s"}}' "$1" \
| env -u ALTHING_HANDLE -u ALTHING_POST_OFFICE \
bash scripts/claude-statusline-command.sh)
printf 'got [%s] wanted [%s]\n' "$got" "$2"
}
probe /tmp '' # no althing identity
probe /home/lkraven/development/eshpfi-management '🔔 …'
⚠ **Print what you got beside what you wanted, on the same line.** Expected
values in trailing comments are what produced the false pass that caught two
agents inside an hour on 2026-09-02: `/tmp` rendered `🔔` and both of us read it
as "fine" rather than "that directory has no identity and should be empty." A
plausible value in the right shape is the hardest kind of wrong to see, and the
only thing that reliably catches it is the wanted value sitting on the same line
as the got value.
### ⚠ The outage probe: a refused port does NOT test the timeout
`📵` has two causes and they take different paths. Measured 2026-09-02 on
nh3-dev:
| address | elapsed | what it proves |
|---|---|---|
| `http://10.100.50.40:9999` — refused port on a **live** host | **0.079 s** | the glyph, and nothing about the deadline |
| `http://10.100.50.199:8390` — **unrouted** host, black-holes | **2.065 s** | the 2 s budget actually fires |
A refused port answers instantly with an RST, so the fast probe returns `📵`
without ever approaching the timeout. Use **both**, and know which is which —
letting the fast case imply the slow one is how a status line that hangs the
prompt ships with a green test beside it.
## Rollback
uv tool install althing-core==3.1.2
Routes written by later versions stay parseable — the old reader ignores unknown
keys — so nothing is stranded.
docs/runbooks/esh-media-dac-10g.md
+117
View File
@@ -0,0 +1,117 @@
# ESH-Media 10G uplink — the DAC that negotiated 1G, and the fix
2026-09-04. The link runs at 10G by an explicit rate override on the UDM side.
**Read the pinned-port warning below before touching that port.**
> ## CURRENT STATE: REVERTED to autoneg / 1G. The force is gone.
>
> **2026-09-04 14:38 — the operator reported the 10G link "flaking hard" and I
> reverted it.** Both ends are back at 1000 Mbps, UDM SFP+2 is `autoneg True /
> setting_preference auto`, ESH-Media is reachable and stable. The revert took
> about 50 s of renegotiation and nothing rebooted.
>
> **The forced-10G experiment failed, and the failure was progressive.** Error
> counts on the ESH-Media side over the life of the force:
>
> 10:03 200 link-up burst
> 10:48 221 +21 in 42 min -- the first sign it was not plateauing
> 14:38 416 +195 over the next 4 h, plus user-visible flapping
>
> ⚠ **The lesson, and it is mine to own: I read a two-minute flat window as a
> plateau and reported it as one.** The counter simply had not moved yet. A
> marginal link declares itself over hours, not minutes — and the operator felt
> the flapping before the soak I left running had accumulated enough to call it.
>
> **Do not re-force this port.** The cable is being replaced at the weekend; a
> correctly-coded DAC negotiates 10G on its own, which is the actual fix.
## The diagnosis, and what made it certain
The cable was linking *cleanly* at the wrong speed — **zero rx/tx errors, no
LOS, no tx/rx fault, satisfaction 100**. That rules out a damaged or marginal
cable, which links at 10G and throws CRCs rather than dropping a rate. Both ends
were *choosing* 1G.
The reason showed on the switch side:
sfp_compliance: Unknown
The switch reads the cable's EEPROM but cannot parse its compliance codes.
`SFP-H10GB-CU3M` is a Cisco part number and the vendor string is literally
`OEM` — a third-party cable wearing Cisco coding. With no recognised 10G
compliance byte, autonegotiation falls back to the safe rate.
**The control case is what settled it.** On the *same UDM*, one port over:
SFP+1 (eth9) TP-Link TL-SM5220-1M -> 10000
SFP+2 (eth10) OEM SFP-H10GB-CU3M -> 1000
Same `speed_caps`, autoneg on both. So the port hardware, the firmware and
autoneg were all exonerated before anything was changed, and the cable was the
only remaining variable.
## The fix
An explicit rate override on the **UDM** end only:
autoneg False speed 10000 full_duplex True setting_preference manual
ESH-Media's end was left on autoneg and **followed to 10000 by itself** — which
is the proof the cable was always electrically capable of 10G and only the
EEPROM-driven negotiation was wrong.
## ⚠ Why the UDM end and not the switch end
The switch end is the unrecoverable one. ESH-Media reaches the controller
*through* this link, so a forced rate that fails to link leaves the switch with
no uplink and therefore no way to receive the revert — a physical visit.
The UDM end is safe because the path to the controller does not cross this link:
nh3-dev -> 10.100.10.1 -> 10.0.0.1 two hops, site VPN, terminates on the UDM
Verified with `ip route get` / `traceroute` **before** the change, and the exact
pre-change `port_overrides` array was written to a revert payload before the
forward one was sent. If a future attempt needs *both* ends forced, that is a
different risk class and needs the operator's sign-off, not an agent's.
## The port is NOT pinned any more — the override was removed 2026-09-04
While the force was in place, UDM SFP+2 was `autoneg False, speed 10000` and a
non-10G module would not have linked there. That is no longer true: the override
is gone and the port is back on autoneg. Recorded because the warning stood in
this runbook for four hours and someone may remember it.
## What the link bounce cost
~60 seconds. ESH-Media, the AP and the office switch all returned `state=1` at
`uplink_speed=10000` with their **uptimes unbroken** (the switch did not reboot,
only the link renegotiated). The Zigbee coordinator and Home Assistant's path to
it were re-verified afterwards.
## Errors: 200 at link-up, then flat
The switch end logged ~200 rx / ~196 tx errors **during the renegotiation
window** and none afterwards. Counters were flat across the following two
minutes while real traffic flowed, and a longer soak was run to confirm.
⚠ **This is the number to watch, and it went the wrong way.** A two-hour soak
showed the errors do **not** plateau: 200 → 221 rx and 196 → 199 tx over 42
minutes, monotonic, about 1 rx error per 23.2 MB — plus one link drop at 10:12
that recovered on its own with no device reboot.
⚠⚠ **So an earlier claim in this runbook's own history was wrong and is
corrected here: a clean, zero-error link at 1G does NOT rule out a marginal
cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at
10G is exactly what a 3 m OEM twinax with unparseable compliance codes turns out
to be — which means the platform's autoneg fallback was protecting something
real, not being fussy about vendor coding. For comparison, the TP-Link
TL-SM5220 on the adjacent UDM port carries 780 GB with 4 errors.
The fix is the cable, not the configuration.
## Rollback
Set UDM SFP+2 back to `autoneg True, setting_preference auto`, clearing `speed`
and `full_duplex`. Remember `port_overrides` is a **whole-array PUT**: the UDM
carries 2 overrides and anything omitted is deleted.
docs/runbooks/gx10-rack-network.md
+65
View File
@@ -0,0 +1,65 @@
# pfi-gx10 — desk to rack, network
Done 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
## The shape, and why it is this shape
**The address lives on the switch/firewall side, not on the host.** Operator
ruling at the time of the move: a DHCP **reservation** on the UDM, with the host
left on DHCP. A host-side static works until the box moves, and then it is a
stale netplan file on a machine whose address you no longer know. A reservation
moves with the MAC.
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
host unchanged: still DHCP
`playbooks/gx10-rack-network.yaml` is kept for its safety ordering but its
host-static approach was **not** used, and it says so at the top.
## ⚠ Order is the whole safety argument
The lockout risk in remote network work is applying a config that does not work
and losing the only path back. There was never one path:
1. **Verify the cable is really in** — `/sys/class/net/enP7s7/carrier` = 1.
Configuring a dead interface and calling it done is a green that costs a rack
visit to discover.
2. **Identify by MAC, not by interface name.** Names renumber across kernels;
`30:c5:99:3d:a7:45` does not.
3. **Move the switch port, then reserve, then renew** — Wi-Fi stays up
throughout as the escape hatch.
4. **Prove the new address from OUTSIDE** before removing anything.
5. ⚠ **Prove inter-VLAN routing before killing Wi-Fi.** This is the step that
is easy to skip and expensive to miss: while Wi-Fi was up, replies to
nh3-dev (10.100.10.50) still preferred `wlP9s9`, because that interface sits
directly on the userland subnet. So "I can reach it" proved nothing about
whether the WIRED path could. Forcing the interface first is what settles it:
ping -c3 -I enP7s7 10.100.10.50 # VLAN 50 -> VLAN 10, 0% loss
ping -c2 -I enP7s7 1.1.1.1 # egress
6. **Down Wi-Fi last, as its own change** — a server with two default routes
picks one and you find out later, in a way that looks like packet loss.
## Editing UniFi port overrides
`port_overrides` is a **whole-array PUT**. Read the existing array, filter out
the port you are changing, append your entry, and write the lot back — anything
you omit is deleted. Two unrelated overrides (ports 21 and 23) were on this
switch and were preserved deliberately; the pre-change array is worth backing
up to a file first.
GET /proxy/network/api/s/default/stat/device # find switch by MAC
PUT /proxy/network/api/s/default/rest/device/<id> # {"port_overrides":[...]}
PUT /proxy/network/api/s/default/rest/user/<id> # the reservation
Reservations live on the **client** object (`use_fixedip`, `fixed_ip`,
`network_id`), not on the network.
## Wi-Fi
`nmcli radio wifi off` — persists across reboot in
`/var/lib/NetworkManager/NetworkManager.state`, verified by reading it back
rather than assuming. `/etc/netplan` was backed up to
`/etc/netplan.bak-preWifiDown` first.
docs/runbooks/gx10-run-03c.md
+115
View File
@@ -0,0 +1,115 @@
# pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch
Staged 2026-09-03. **Not launched** — the launch is the operator's call.
Run 3c is the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim
breaker tripped. It is rehomed to pfi-gx10 unchanged: same corpus, same base,
same hyperparameters, different host.
## Why this box and not the fast one
ana-ml2 finishes this run in ~2.5 h and pfi-gx10 needs ~13.3 h, and the slow box
is still correct. An Anaheim power event is not priced in lost training steps —
it is a 40-minute drive each way on the operator's time, with 13 hosts down
including `pbs-ana` (the fleet's primary backup server) and **three SureFire
client machines** under a hosting agreement. The GX10 is a ~240 W appliance at
NH3 that can take nothing else down, and nothing is waiting on this run.
Power-capping ana-ml2 is not a third option: both RTX PRO 6000s floor at 250 W
against a 300 W default, so the whole lever is 100 W on a ~1 kW box.
## Layout on the box
/home/infra-ops/models/gemma4-26b-a4b-it-bf16 49 GB base (sha-verified, see below)
/home/infra-ops/erp-tune/
eitri-smithy/ harness, git 0a6bd2e, tracked tree clean
recipe-r3/ recipe, survivors, loss-mask
datasets/derived/ 2.4 GB corpus -- COPIED, see the NFS note
datasets/holdout/ holdout, for the disjointness assertion
run-03c/encode-cache/ pre-seeded with the VERIFIED encode (below)
run-03c-gx10.json the config
launch-run-03c.sh the launcher
/home/infra-ops/ml/.venv torch 2.14.0+cu130, transformers 5.16.1, peft 0.20.0
Canonical copies of the config and launcher live in this repo under
`scripts/erp-tune-gx10/`.
## ⚠ The corpus is COPIED, and the box mounts no NFS at all
`/mnt/smithy` is deliberately **not** mounted here. A 13-hour unattended run is
the worst possible place for a hard NFS dependency — the fleet has already lost
a host to a hard mount wedging into unkillable D-state. The corpus is 2.4 GB and
copied in 50 s from nh3-dev's mount; there is nothing to gain by mounting it.
Verify before every launch: `mount | grep -E 'nfs|cifs'` returns nothing.
## What was verified, rather than assumed
**The base model is byte-identical to ana-ml2's.** Both shards sha256-matched
across 51.6 GB, not just size-checked:
model-00001-of-00002.safetensors 1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3
model-00002-of-00002.safetensors aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7
**The tokenizer did not drift across the library delta.** ana-ml2 ran
transformers 5.15.1 / torch 2.13.0 on x86-64; this box runs 5.16.1 / 2.14.0+cu130
on aarch64. That is exactly the silent backend-delta class that has already
voided conclusions on this project, so it was measured, not reasoned about: a
full encode was run here into a throwaway output dir and the encoded corpus
compared byte-for-byte against ana-ml2's.
ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3…
pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3…
Identical. Every aggregate matched too — 9,504 training vs 8,404 holdout ids with
0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930
tokens, and the five mix shares to four decimals.
⚠ **The cache-key filenames differ and that is correct, not a mismatch.**
`base_model_path` is part of the encode-cache key by design, so rehoming the base
changes the key while leaving the content identical. The key is an input hash;
the sha above is the output. Do not read the differing filenames as drift.
That verified artifact was then copied into `run-03c/encode-cache/`, so the run
trains on the exact bytes that were compared — it will report `[encode] cache
hit` rather than re-encoding.
**The harness runs on aarch64.** Its own suite: 122 passed.
**Throughput.** ~79.4 s/it measured on this exact geometry (median of 10 timed
steps, 0.19% spread), peak 75.1 of 121.6 GiB. 604 steps ≈ 13.3 h. Against
ana-ml2's 10.8–15.8 s/it that is ~6× slower where raw compute predicts ~2.7×;
the unconfirmed hypothesis is unified-LPDDR5X bandwidth, not a misconfiguration.
## Launch
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
The script refuses rather than proceeds if the GPU is not clear, if
`run-03c.pid` names a live process, if the log already exists, if any input is
missing, or if there is under 40 GB free. It detaches with `setsid nohup` and
logs on the box, because a long remote job piped through a foreground SSH has
already been reaped once on this hardware — the work survived, the output did not.
Watching, and stopping:
tail -f ~/erp-tune/run-03c.log | tr '\r' '\n'
kill $(cat ~/erp-tune/run-03c.pid)
⚠ **Never `pkill -f erp_sft_harness` over SSH.** The pattern appears in your own
ssh argv, so it kills your shell as well as the run. This bit again during
staging. Kill by PID. The launcher's own guard uses the pidfile for the same
reason.
## What to expect in the log
Order is preflight → encode (cache hit) → mix report → weight load (~2 min) →
sampler → train. Checkpoints land every 50 steps at ~852 MB each, so ~10 GB over
the run against 764 GB free. First checkpoint ≈ 66 minutes in.
## Not verified
The harness's train loop has not been exercised end to end on this silicon — the
throughput baseline used a synthetic replica of the geometry, and the staging
encode was killed before the weight load. If something is going to break, it
breaks in the first two minutes after the sampler line.
docs/runbooks/mac-provisioning.md
+90
View File
@@ -0,0 +1,90 @@
# Provisioning a Mac for the fleet
Three Macs are provisioned this way as of 2026-09-02: `vuongs-mac-mini`
(10.100.79.2), `esh-macbook-air` (10.0.10.83), `esh-mac-studio` (10.0.10.10).
None is in `servers/` or `dns/internal.yaml` — they are the operator's personal
machines, not PFI-managed fleet hosts, and registering them there would imply
otherwise.
## Two stages
**Stage 1 — an account I can reach.** Operator-run, because it needs a password
I do not have. See "Operator steps" below.
**Stage 2 — the harness.** `scripts/provision-mac-dsh.sh <host> <account> [name]`,
idempotent, run once per account.
scripts/provision-mac-dsh.sh 10.0.10.10 vhpfi esh-mac-studio
scripts/provision-mac-dsh.sh --check 10.0.10.10 vhpfi
## Operator steps (stage 1)
```zsh
sudo sysadminctl -addUser infra-ops -fullName "PFI infra-ops" \
-shell /bin/zsh -home /Users/infra-ops -password -
sudo dseditgroup -o edit -a infra-ops -t user admin
sudo mkdir -p /Users/infra-ops/.ssh
sudo tee /Users/infra-ops/.ssh/authorized_keys >/dev/null <<'KEY'
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet
KEY
sudo chown -R infra-ops:staff /Users/infra-ops/.ssh
sudo chmod 700 /Users/infra-ops/.ssh; sudo chmod 600 /Users/infra-ops/.ssh/authorized_keys
```
Then hand over a throwaway password; infra-ops rotates it, vaults it as
`<name>/infra-ops-password`, and installs a `visudo`-validated NOPASSWD drop-in.
⚠ **Install the key and prove key-auth on a FRESH connection BEFORE touching the
password.** These machines have no out-of-band access — a failed rotation means
the operator walks to the machine.
## ⚠ Traps, all of them paid for
**An account may not own its own home.** If `/Users/<account>` was created by a
`sudo mkdir` before `sysadminctl` ran, sysadminctl adopts the existing directory
and leaves it **root-owned**. The account then authenticates, gets a shell, has a
correct `$HOME`, and cannot write to it — surfacing as a bare `Permission denied`
from `mkdir` long after the account looked healthy. The script detects and fixes
this; a hand-run will not.
**`sudo -u <user>` keeps the CALLER's `$HOME`.** Without `-H` and an explicit
`HOME=`, `"$HOME/.local"` resolves to the caller's home. On 2026-09-02 that
pointed an `rm -rf` at a working install in another account; only filesystem
permissions stopped it. The remote half of the script refuses to run unless
`$HOME` matches the target account.
**A wrong username looks exactly like a wrong password.** sshd answers
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
bad password, AND a user outside `com.apple.access_ssh`. Two of the three Macs
produced a false diagnosis this way. **Check `dscl . -list /Users` first** — the
Studio's operator account is `vhpfi`, not `lkraven`.
**`com.apple.access_ssh` gates SSH when it exists, but admins usually pass
anyway** through a nested group. Do not *create* the group if absent: doing so
flips SSH from open-to-all to members-only and can lock out the operator.
**Password rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on
and no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
*"Operation is not permitted without secure token unlock"*. `dscl` works
precisely because there is no token to desync. True on all three Macs.
⚠ **FileVault kills remote access across reboots.** The machine sits at the
pre-boot unlock screen with no network until someone unlocks it physically.
Nothing unattended should depend on a Mac being reachable after a restart.
**macOS has no `adduser`, `useradd`, or `timeout`.** Use `sysadminctl`, and wrap
the ssh call locally rather than reaching for a remote `timeout`.
## Harness specifics
Node is installed **private to the account** under `~/.local/node`,
checksum-verified — deliberately not Homebrew, which owns `/opt/homebrew` and
edits PATH. One **device-scoped** gateway key per machine (`<name>-dsh`, scoped
to `gen-reasoning`), shared by that machine's accounts and vaulted at
`<name>/litellm-dsh-key`; the script verifies the scope (200 on gen-reasoning,
403 on gen) rather than trusting the mint.
See `stacks/litellm/README.md` for why `reasoningEffort: high` works at all, and
`scripts/provision-mac-dsh.sh` for the three measured limits (`reasoningEffort`,
`maxTokens`, `defaultContextWindow`) and the hard-coded model catalog the web
GUI reads independently of the default model.
docs/runbooks/nh3-dev-io-stall.md
+119
View File
@@ -0,0 +1,119 @@
# A VM wedges on IO and the disk is innocent
First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
"/dev/sda stalled — writes hang". It was the backup.
## The signature
/proc/pressure/io full avg10=96 (everything is IO-blocked)
/sys/block/sda/inflight 0 45 (45 writes in flight, zero completing)
D state jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
dmesg "blocked for more than 120 seconds", virtio_ring in the trace
Looks exactly like a failing disk or controller. It is not.
## What distinguishes it — and the checks are cheap
**1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
errors is evidence, not a gap in it.
**2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
plus stuck in-flight requests means the writes were already submitted and are
not being *completed* — the stall is below the block layer, not above it.
**3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:
ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
guest's writes are not reaching the host at all. Something between them is
holding the queue.
**4. Then look at the block layer, which names the culprit outright:**
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write) ← HERE
ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
`copy-before-write` is the filter Proxmox interposes during a backup: **every
guest write must first copy the original block out to the backup target before
it may proceed.** A slow or stalled target therefore throttles the guest to the
target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
35 minutes.
## Fix
ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
Cancels the backup and detaches the filter. Verify: `info block` shows the disk
as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
returns to `0 0`, D-states clear.
⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
attached and the next writes stall the same way.
⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
one that can crash the service*. An incremental backup with a dirty bitmap
re-runs cheaply; an unusable dev box does not.
## Why it can recur, and the real fix
The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
`pbs-ana`** — every VM on the host, cross-site to Anaheim.
⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
land in a fast *local* fleecing image instead of going straight to the backup
target, which decouples guest IO from target throughput. That is the difference
between "the backup is slow tonight" and "the box is unusable tonight."
A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
degradation with its own cause — the cross-site link and pbs-ana were both
responsive (11 ms) throughout, so the throughput cause is **not yet known** and
enabling fleecing treats the blast radius, not the disease.
## Fleecing was enabled 2026-09-03, and verified by behaviour
pvesh set /cluster/backup/backup-5d8f1221-8f71 \
--fleecing 'enabled=1,storage=local-zfs'
Proxmox's own `man vzdump` states the mechanism and the fix outright:
> The guest write blocks until this operation is finished so guest IO to
> not-yet-backed-up sectors will be limited by the speed of the backup target.
> [...] With backup fleecing, such old data is cached in a fleecing image rather
> than sent directly to the backup target. This can help guest IO performance and
> even prevent hangs in certain scenarios, at the cost of requiring more storage
> space.
⚠ **Verify by behaviour, not by config.** The config read-back only proves the
key is set. Run a backup and measure the GUEST while it runs:
# host: topology should show BOTH filters
qm monitor <VMID> <<< "info block"
drive-scsi0 ... (copy-before-write)
drive-scsi0-fleecing ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw) ← this
# guest: must stay responsive
cat /proc/pressure/io ; cat /sys/block/sda/inflight
dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync
Measured during a live backup that was **still crawling at the same 1.4 MiB/s**:
| | during the stall | with fleecing |
|---|---|---|
| io pressure full avg10 | 96.5% | **0.01%** |
| in-flight writes | 45, completing none | **0 0** |
| D-state processes | 11 | **0** |
| 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** |
The backup is exactly as slow as before. The guest no longer cares — which is
the whole point, and the reason this is a blast-radius fix rather than a cure.
⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks
for the life of the job, on `local-zfs` (717 G free at the time). A long, slow
backup of a busy guest is the case that grows it — watch
`zfs list rpool/data/vm-<VMID>-fleece-0` if the target stays degraded.
docs/runbooks/pbs-deployment.md
+3 -3
View File
@@ -346,7 +346,7 @@ Web UI: **Datacenter → Backup**. Edit the existing pfi-pve backup job
| Change | From | To |
|---|---|---|
| Storage | `pve-truenas` (or current) | `pbs-ana` |
| Storage | `naspool-vmstorage` (was `pve-truenas` until 2026-09-06) | `pbs-ana` |
| Mode | snapshot | snapshot |
| Compression | zstd | zstd |
@@ -361,7 +361,7 @@ dirty-bitmap incrementals.
### 2.5. Keep legacy target for overlap
**Don't delete the old `pve-truenas` backup storage yet.** Wait ~1 week
**The old `pve-truenas` storage id was renamed `naspool-vmstorage` on 2026-09-06** (same path, `/NASPool/pve-VMStorage`, on the rebuilt raidz2 pool; legacy vzdump pruned to newest-per-guest). Historical note follows. Wait ~1 week
of successful PBS runs before retiring it. Double-coverage period is
cheap insurance.
@@ -369,7 +369,7 @@ cheap insurance.
- pfi-pve's 11 guests have first PBS snapshots
- Scheduled job uses PBS-ANA
- Old `pve-truenas` target still exists (for safety) but no new jobs
- Old `pve-truenas` target (now `naspool-vmstorage`) still exists but no new jobs
write to it
## Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana
docs/runbooks/pfi-pve-naspool-rebuild.md
+102
View File
@@ -0,0 +1,102 @@
# pfi-pve — NASPool rebuild as raidz2 (2026-09-05)
Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on
ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks
that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane
fault; cables and PERC both swapped with no change) become labelled cold
spares. Legacy vzdump archive pruned to the newest file per guest by
omission: the keepers are copied out, the rest dies with the old pool.
Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs
a re-run, not data, because the source of the running copy still exists.
Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
## Fixed facts (verified 22:24Z–23:05Z, read-only)
- Host `root@10.250.250.31` (infra-ops user added 2026-09-06 after the rebuild), PVE 8.3.5, zfs 2.2.7 (no raidz expansion).
- Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm).
ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never
written to by an irreversible command in this runbook.**
- NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy
vzdump files, newest 2026-04-22; images 4.3G; template 124M),
backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G,
docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS
`.system` 1.4G (not restored — irrelevant on Proxmox; stays parked).
- NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports
`/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`,
`/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles
after the rebuild and must remount**), CT 112 ana-filebot (same binds),
VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage,
ana-docker (`/mnt/backup` systemd automount; containers rest-server,
backrest).
- SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS.
Disk identity (device letters are NOT stable; commands use by-id):
| bay | dev | by-id |
|---|---|---|
| 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 |
| 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd |
| 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 |
| 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 |
| 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d |
| 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 |
Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105
107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106
(2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`.
## Steps
1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the
`mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve:
`qm shutdown 100`, `pct stop 112`, `pct stop 109`,
`pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty.
2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`;
`zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav,
.system: `zfs send -R NASPool/<x>@evac | zfs recv -u ospool/naspool-evac/<x>`.
pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync
images/ template/ snippets/ private/ frompfi/ and the keeper files into it.
Runs under nohup, log `/root/naspool-evac.log`.
3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync
`--checksum` re-pass over pve-VMStorage reports zero transfers. Report to
operator; wait for go.
4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`.
Then per disk, literal paths, one per line:
`zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above.
5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off
-O xattr=sa NASPool raidz2 <six by-id paths>`. Same pool name so
storage.cfg and the CT bind mounts are unchanged.
6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool
(`zfs send -R ospool/naspool-evac/<x>@evac | zfs recv -u NASPool/<x>`),
`zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set
`recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before
copying data in. Do NOT restore `.system`. Check mountpoints
`/NASPool/<x>`; `zfs mount -a`.
7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`,
`pct start 112`, `qm start 100`. ana-docker: start automount, then
`docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana`
readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`.
8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one
Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`.
9. **Later.** Backplane swap (Route D) at the next colo visit; then
labelclear the two resurfaced disks and `zpool add NASPool spare <a> <b>`.
## Log
(appended as steps complete, UTC)
- 22:59Z operator: Route C hybrid, raidz2, bays 9/10 cold spares.
- 23:0xZ operator: prune vzdump (newest per guest confirmed 23:20Z), accept one PSU, start ASAP.
- 23:04Z quiesced: ana-docker backrest+rest-server stopped, /mnt/backup unmounted; pfi-pve VM100/CT112/CT109 down, pve-truenas disabled; `fuser -vm /NASPool` = kernel mount only.
- 23:04Z park copy launched (`/root/naspool-park.sh`, log `/root/naspool-evac.log`); ~180–200 MB/s after warm-up.
- 01:20Z PARK DONE (2h15m, ~200 MB/s sustained, no source read errors). All five zfs streams received; 26 keeper files + non-dump dirs rsynced (297G). Total parked 1.65T; ospool 2.18T alloc / 8.72T free.
- 01:25Z verify: `logicalreferenced` at @evac identical for backupStore/db/webdav/.system; docker 289K vs 280K is accounting only (`diff -r` clean, 8/8 files). Checksum re-pass (`rsync -anc`) over the 297G file copy launched, log `/root/naspool-verify.log`.
- 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before `zpool destroy NASPool`.
- 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched.
- 02:01Z `zpool destroy NASPool` executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log `/root/naspool-rebuild.log`.
- 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. `.system` deliberately not restored (stays in the parked copy).
- 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); `pvesm list pbs-ana` = 116 backups, pve-truenas = 15 entries.
- 04:03Z `zpool scrub NASPool` started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped.
- OPEN: (a) destroy `ospool/naspool-evac` (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) `zfs destroy NASPool/*@evac` snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → `zpool add NASPool spare`; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare.
- 04:37Z operator: "tear down and delete truenas". TrueNAS SCALE leftovers `ospool/naspool-evac/.system` (+ children, 1.43G, incl. the 2023 SCALE-22.12.1 samba4 snapshot) destroyed — nothing on the live pool referenced them. Storage id `pve-truenas` removed and re-added as **`naspool-vmstorage`** (same path/content/prune; no guest conf or backup job referenced the old id; 15 entries listed). The TrueNAS 2G swap stubs died with the old partition tables; the two bay-9/10 disks still carry them until the backplane swap. `.system` is no longer in the parked copy — the "stays parked" note above is superseded.
- 04:43Z scrub of the new raidz2 NASPool finished: 0B repaired, 0 errors, 40m21s over 2.48T. Remaining gate for destroying `ospool/naspool-evac`: one Backrest run (08:00Z) + one PBS vzdump run (10:00Z) succeed.
docs/runbooks/slzb-mr1u-zigbee-coordinator.md
+119
View File
@@ -0,0 +1,119 @@
# SLZB-MR1U Zigbee/Thread coordinator — ESH IoT VLAN
Moved onto the IoT VLAN 2026-09-04 and handed to Home Assistant.
device SMLIGHT SLZB-MR1U ("SLZB OS", firmware build Jan 27 2026 11:19:04)
address 10.0.90.10 slzb-mr1u.esh.internal
network esh-iot, VLAN 90, 10.0.90.0/24
switch USW Pro XG 10 PoE - ESH-Media (10.0.250.164), port 4, PoE ~1.0-1.5 W
zigbee tcp/6638 web UI tcp/80 also open: tcp/7638
consumer Home Assistant container at 10.0.50.46 (esh-docker-vm, macvlan on VLAN 50)
## The shape, and why it is this shape
**The address lives on the UDM as a DHCP reservation, not on the device.** Same
ruling as pfi-gx10: a reservation moves with the hardware, a device-side static
goes stale the moment anything changes. The device stays on DHCP and knows
nothing about its own address.
`10.0.90.10` sits deliberately **below** the esh-iot DHCP pool
(`10.0.90.40–250`), so the pool can never hand it to something else. Note the
one pre-existing reservation on this network — the Brother printer at
`10.0.90.125` — is *inside* the pool; the out-of-pool convention starts here.
## The MAC is a derived ESP32 interface address — stable, not fragile
ethernet (what DHCP sees) ea:f6:0a:ca:f5:b4 LA bit SET
base (what mDNS advertises) e8:f6:0a:ca:f5:b4 Espressif Inc. (verified against IEEE)
⚠ **An earlier version of this runbook called the MAC "locally administered, no
OUI" and treated it as the kind that may regenerate at boot. That framing was
wrong and ha-dev corrected it.** The two addresses differ in exactly one bit —
the locally-administered bit — which is the standard ESP32 pattern: one factory
base MAC in eFuse, sibling interface MACs derived from it deterministically. The
Ethernet interface gets the derived `ea:` variant; mDNS advertises the base
`e8:`. `E8:F6:0A` resolves to **Espressif Inc.** in the IEEE registry and
`EA:F6:0A` correctly resolves to nothing, which is what confirms the two are the
same address wearing different bits rather than two unrelated MACs.
So the reservation is keyed on the right address and the derivation cannot drift
on its own. Independently consistent with the test below.
**Tested anyway, and it was free:** the PoE power-cycle needed to make the device
re-DHCP onto the new VLAN is also a reboot. It came back on **exactly
10.0.90.10** — which only happens if the reservation matched.
The residual risk is real but narrow: only a firmware update that changed the
derivation scheme would move it. Symptom if that ever happens — the coordinator
reappears somewhere in `10.0.90.40–250`.
## ⚠ port_overrides is a WHOLE-ARRAY PUT
ESH-Media carried 8 port overrides. Anything omitted from the PUT is deleted, so
the array was read, backed up, modified in exactly one field, diffed to prove
only that field changed, and written back. Read-back confirmed 8 in, 8 out, ports
2/3/5/6/8/9 still on esh-userland and port 7 still locked down.
## ⚠ UniFi's client record lies during the transition — do not read it as breakage
While the port was cycling, `stat/sta` reported the client on **the UDM's own
SFP+ uplink** with its old VLAN-10 address and a frozen uptime, and `port_table`
showed ESH-Media port 4 down at 0 W. Both were transitional artifacts. The
authoritative check is the device itself: the page served at the new address was
**byte-identical** (same 25,413 bytes, same `ETag: "Jan 27 2026 11:19:04"`) to the
one captured before the move. Fingerprint the endpoint; do not trust the
controller's cache mid-change.
## ⚠ Home Assistant CANNOT resolve `.internal` — give it the IP
Verified from inside the container, and it is **pre-existing, not caused by this
change**:
slzb-mr1u.esh.internal FAIL homepage.esh.internal FAIL
esh-docker-vm.esh.internal FAIL google.com OK
The container's resolver is Docker's embedded `127.0.0.11`, whose upstream does
not include the fleet AdGuard at `10.0.50.45`. Public DNS resolves, so it is
reaching *an* upstream — just not ours. **Configure HA against `10.0.90.10`.**
The fix, if wanted, is a `dns:` entry on the HA service pointing at `10.0.50.45`
plus a container restart — a change to a live home-automation stack, so it is
ha-dev's call, not an incidental edit.
Separately, note `esh-iot` itself hands out `1.1.1.1` / `8.8.8.8` by DHCP, so
devices *on* VLAN 90 cannot resolve `.internal` either. Irrelevant here — the
coordinator never initiates — but relevant to anything on that VLAN that does.
## Reachability is already open, and was proven before the move
VLAN 50 → VLAN 90 needed no firewall work. Confirmed from the actual consumer —
`docker exec homeassistant` — against pre-existing IoT devices (the printer and a
Tasmota plug) *before* touching anything, then against the coordinator after.
Testing from the Docker **host** would have proven the wrong thing: HA runs on a
macvlan interface with its own address, not the host's.
## Rollback
Backups of both mutated objects are transient (session scratch), so the durable
path is: set ESH-Media port 4's `native_networkconf_id` back to
`687985eae5d15b673cef1a73` (esh-userland), clear `use_fixedip` on the client, and
drop the `slzb-mr1u` line from `dns/internal.yaml` + re-run `scripts/dns-sync.py`.
## mDNS crosses the VLAN boundary — and that has one sharp edge
`esh-iot` has `mdns_enabled: true`, so the coordinator's advert reaches the
Home Assistant container across VLANs and HA rediscovers it without help.
⚠ **HA's pending `smlight` config flow is keyed on the mDNS service name, which
did NOT change across the move — so the same flow id survived and may still hold
the dead 10.0.10.58.** Dismiss and re-discover rather than confirming the stale
flow. This is the kind of thing that makes a device look broken after a clean
network change. (Reported by ha-dev, 2026-09-04.)
## Resolver: ha-dev declined the `dns:` fix, deliberately
Offered and turned down: pointing the homeassistant service at `10.0.50.45`
would couple HA's name resolution to AdGuard's uptime for a benefit they do not
currently need, since configuring by IP costs them nothing. Recorded on their
side as a standing foot-gun with the fix written down. Revisit only if something
inside the container actually needs a name.
graphify-out/GRAPH_REPORT.md
+1649 -627
View File
File diff suppressed because it is too large Load Diff
persistent-memory.d/2026-08-11-stonehenge-park.md
-28
View File
@@ -1,28 +0,0 @@
`[2026-08-11]` **stonehenge-park — new fleet `/park` service repo stood up + designed.**
**What.** A separate greenfield repo (`~/development/stonehenge-park`, gitea `vh/stonehenge-park`,
pushed) for a self-contained `/park` service: one durable place to park any idea (repo-born OR
personal), find it by search, and have it **actively resurface** (by due-date or staleness) until
acted on — so parked ideas stop dying when a repo goes cold. NOT part of eshpfi; this is a pointer.
**Design (via `/vor-plan`, converged + persisted to `docs/design/`):** four contract-sized units —
**U1** core store+API (SQLite+FTS5, slug minting, bearer auth, REST) — the tracer, build first; **U2**
scheduler+notifier (in-process; due/stale → statusline `due-count` + althing push to a dedicated
**assistant channel**; keep-surfacing until promote/drop/re-snooze); **U3** `park` CLI (mirrors the
`secret` CLI); **U4** browse UI. `/vor-ui` ran too (U4 brief persisted).
**Locked decisions (operator):** SQLite, self-contained, ONE container, no external DB ("don't want
to troubleshoot it when a database upgrade happens") — a hard `[OPS]` invariant; system-minted
title-derived slugs + short ID (addressable as `park/<slug>`); active keep-surfacing resurfacing with
**re-snooze as the anti-nag valve**; bearer key, LAN/WG-internal; host nh3-docker; `/park` **replaces**
the global ROADMAP parking-lot discipline (deferred ideas → `/park`, `source`-tagged; ROADMAP keeps
only the v1 target) as a **fast-follow after v1** incl. migrating existing lots.
**Deferred (in the plan):** the althing assistant-channel handle **name** (decide at U2 contract
time); staleness threshold + re-push cadence (env-tunable defaults ~30d/~daily); design U2's emit
structured/consumable so a future **mission-control (Ledger→orchestrator)** can read it — park does
NOT build the orchestrator.
**State.** Pre-seeded for a fresh agent (CLAUDE/persistent-memory/ROADMAP/README + the design docs),
committed (`294ee98`), pushed. Next build task lives in that repo: the **U1 tracer contract** under
the House Code Discipline. Auto-memory candidate not yet written (repo is self-documenting).
persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md
-91
View File
@@ -1,91 +0,0 @@
# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
seats. Operator-driven, end to end this session.
## What landed
| Seat (LiteLLM alias) | Model | Role | GPU | Context |
|---|---|---|---|---|
| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
(DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
retained** for rollback.
- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
`/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
## Load-bearing lessons (the whole point of this file)
1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
`save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
**both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
`Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
"auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
**hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
processor_config.json.** FIX: extract the `image_processor` section → write
`preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
`deploy.resources.reservations.devices`.
## Dead ends (tried + abandoned)
- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
that both engines need the wrapper (→ the fix in lesson 1).
- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
## granite retired + gateway repoint
- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
reclaim ~13.6GB GPU1 for RP context.
- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
`.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
(the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
`classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
`enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
version-controlled** (a follow-up).
## MTP — deferred
DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
(Gemma-4) has no MTP by architecture.
## On-disk / where things live
- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
`/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
## Open follow-ups
1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
3. MTP deferred (see above).
4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
persistent-memory.d/2026-08-12-secrets-broker.md
-45
View File
@@ -1,45 +0,0 @@
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
metadata fields; idempotent upsert keyed by name.
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
backfill.
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
cargo noise.
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
`vh/stonehenge-park`.
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
Consumer-confirmed end-to-end by jackdaw-dev.
Auto-memory: `reference_secrets_broker_cli`.
persistent-memory.d/2026-08-15-uncensored-gen-seat.md
-52
View File
@@ -1,52 +0,0 @@
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
`reference_abliteration_mtp_lessons`.
**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
(`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
+ `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
`--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
no cap OOM'd — cudagraph captured to batch-512).
**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
isolate before deleting.**
Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
persistent-memory.d/2026-08-17-esh-pve-nas-dom.md
-196
View File
@@ -1,196 +0,0 @@
# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
## The finding
`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
(571 MB free)**.
⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
SLC/pSLC with a real controller, so the **284 GB written since boot is
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
that was wrong and the correction matters. What actually justifies the work:
1. **It is on the USB bus** — a reset or re-enumeration drops the *root
filesystem* out from under a running hypervisor whose guests keep executing.
NAND quality is irrelevant to that.
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
4. **It has blocked patching for months** — the operator-visible symptom and the
real urgency.
## The patching blockage (measured)
`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
because it cannot take a kernel.
⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
keeps downloads off root, but the kernel still lands in `/boot`.
## Mitigation applied 2026-08-17 — root 90% → 76%
| step | effect |
|---|---|
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
| vacuumed the journal | **freed 446 MB** |
| `apt-get clean` | 79 MB |
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
through the ZFS-backed journal.
⚠ **Stopping journald over SSH kills your own session** — it takes the
connection's logging path with it. The first attempt died mid-swap, leaving the
dataset staged and the move incomplete (host was never at risk; journald
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
the wear argument no longer justifies touching a service `pvestatd` depends on.
## The plan — split boot from root (operator's proposal, strictly better)
My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
proposed keeping boot on the DOM with a fallback image and putting all its files
on ZFS. That is better and I should have gotten there myself** — I had assumed
boot and root must share a device.
| | device | contents | written when |
|---|---|---|---|
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
`zstd_compress` enabled and **GRUB cannot read those**.
**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
mid-run no longer kills the running system. Free upside: boot environments
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
(leave it on the DOM LV — **never** swap on a zvol).
**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
import by scan today, which is a coin-flip when the initramfs must find root.
Set the cachefile before rebuilding the initramfs.
Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
updates.
## ⚠ Blast radius — the gating constraint, invisible from the host itself
**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
Known incident shape — the only remedy for esh-docker-vm's D-state is a host
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
risks corruption under `soft`. Quiesce both before any reboot of this host.
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
## Also identified
- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
being CT 109 on pfi-pve.
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
That resolves the dashboard's long-standing "role TBC". Retire rather than
migrate.
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
## Staging executed 2026-08-18 — everything but the reboot
Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
(ZFS initramfs, grub.cfg, both menu entries, grubenv).
**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
(new) + `pve-swap` 256 M (was 768 M).
### The three landmines staging found
1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
which costs the "one reboot" property the design rests on. Only live source
was the swap LV. Operator chose shrink-to-256M over drop-entirely.
2. **The one-pool cachefile would have broken the NAS.** `zpool set
cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
(verified: scan active, cache inactive beforehand), so a cache holding only
`nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
bind mounts spanning all three pools. Every export would come up empty and
both `hard` NFS clients would hang.
3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
`grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
probe failure is swallowed by `2>/dev/null || true`. The same feature set
that forced `/boot` to stay ext4 also corrupts the kernel command line, which
the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
`pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
derived from pool member device paths and would shift if the mirror changed.
**The transferable lesson from (3):** the original verify grepped for
`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
was added that grep passes — while pool-less entries sit in the menu untouched.
The check that holds walks every `linux` line, takes the **last** `root=`, and
asserts it against a known-good set. **Assert the effective value, not the
presence of a substring.**
### One-shot boot, not a new default
`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
returns to ext4 by itself on the next reboot — no console, no hands. That matters
more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
and the NFS clients hang rather than fail. Only after a second clean ZFS boot
should the saved default move.
### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
LV was live during the read, so a restore replays the ext4 journal. Not
fixable with an LVM snapshot: the VG has no free extents.
- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
boot-chain artifact; the dd image is the belt-and-braces full-device restore.
- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
grub.cfg, fstab, LVM/ZFS/blkid state.
⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
file` redirects only the last command, so `pvs`/`vgs` output leaked into the
tar stream on stdout. Group with `{ …; } > file`.
Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
every guest config. **The newest on-disk copy before this was June 2024.**
Commits `2275e11`, `3e31175`, `8ddc87c`.
persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md
-129
View File
@@ -1,129 +0,0 @@
# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
Written because the operator expects to reference this "before too long" — the
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
a nice-to-have.
## Why IPv6 suddenly matters: CGNAT at ESH
New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
a global v6 address on each UDM restores a routable endpoint pair without
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
most likely first consumer of fleet IPv6.
Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
## The topology — as VERIFIED, not as assumed
Three transports, three different technologies. Do not describe this as "a
WireGuard mesh"; a prior session did and was corrected.
| Link | Transport | Evidence |
|---|---|---|
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
**The FortiGate never terminates WireGuard — it port-forwards to the host that
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
chasing a non-problem. Do not re-derive this.
## Per-site IPv6 state (2026-08-17)
| Site | Edge | IPv6 |
|---|---|---|
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
The proof matters because the naive read is wrong: the WAN sits at `c110::48`
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
settings yielding the same /64 is the signature of a single-/64 delegation.
**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
routable address on the *WAN*, not a LAN prefix.
The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
with no size field alongside), so the prefix-ID test is the only read-only-ish way
to establish it from the API.
## What a v6 mesh actually requires (and what it does NOT)
**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
then reverted.
**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
one field to flip when wanted.
**Inbound v6 is default-deny and that held without intervention.** The UDM runs
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
rules exist," which is wrong and alarming. Use
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
for both families with `Allow Return Traffic`; the only v6-specific allows are
link-local plumbing (ND solicit/advert, RA, DHCPv6).
## The stability problem — design around it up front
All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
not a static assignment). A three-way mesh where every node can move is fragile;
WireGuard tolerates one roaming end, not all of them.
The fleet already solves this on the v4 side — IPsec peers use **hostnames**
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
Cloudflare DNS-edit token, so this is self-serve.
## Access recipes (cost a prior session real time)
- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
`unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
(`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
`root@10.250.50.252`; believe it.
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
`drain()` hangs the session**; read until the `ana-gw #` prompt and answer
`--More--` with a space. Two invocations timed out at 3 min before this was fixed.
## Changes made and reverted this session
- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
(client configs carry private keys) were all mode **644**. Now **600**, and
`keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
WireGuard holds keys in kernel memory, so no restart was needed. The parent
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
contexts inside the LXC — but the modes were still wrong.
persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md
-79
View File
@@ -1,79 +0,0 @@
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
Two unrelated housekeeping jobs from the same session, both with durable lessons.
## irv-ml1 — 782 GB reclaimed
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
**74%** (477 G).
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
nobody opened that shed for 15 months. Re-verified the consumer check immediately
before deleting, not just during the audit.
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
`modelscope` 29 G (mtime **2024-04-23**).
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
### The lesson: one dead-looking app pinned three delete targets
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
would have half-broken a live listener in a way that surfaces only at its next
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
verified :7860 released and handles at zero, *then* deleted.
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
from uv. Checking open handles **per path** is what separated them — the earlier
"not running" read would have deleted the wrong thing.
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
All 16 containers and both systemd services verified healthy afterward.
## Homepage under version control
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
config lived only on the host. Its version history was **six hand-rolled
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
105 cards across 19 groups, no empty groups.
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
the top of the file. All five engines were discovering correctly the whole time.
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
dropped the `UltraSeedbox` layout group (nothing provides it).
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
change gave two false readings (a stale prerender, then an empty page).
`GET /api/services` is the honest instrument, and config changes need a
**recreate**, not a restart (a restart keeps the cached render in the writable
layer).
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
also removed a host-side `README.md` in the conf dir. Content survived (it is now
in the repo README) but that was a side effect, not a plan.
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md
-108
View File
@@ -1,108 +0,0 @@
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
neither: the internet was healthy the entire time (gateway reporting 3 ms and
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
loss). **The house had no name resolution because one VM was down.**
## The SPOF: one resolver, cross-VLAN, no fallback
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
client on the VLAN loses DNS, and it presents as "the wifi is broken."
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
verified answering. Applied via the Classic API,
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
parallel, so once AdGuard is back a real share of lookups go to the gateway and
**skip ad-blocking**. This converts a total outage into degraded-but-working.
The actual fix for blocking integrity is a second AdGuard instance NOT on
esh-pve.
## Root cause: hard freeze, no diagnostics, two suspects
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
thermal event**. Powered on with its 10G link up, but not answering ARP.
Two changes landed the day before, and they are not exclusive:
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
boot on the new kernel lasted **20 hours**.
2. **GPU passthrough.** The last kernel messages of the dead boot are
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
**39 minutes before the freeze**.
A vfio/i915 regression in the newer kernel would produce exactly this
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
operator — it is on-demand and there has been no demand. That removes the
suspect without a kernel rollback.
## Why nobody could recover it remotely — and the fix
Nothing on the box could reboot it:
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
hard kernel freeze: the frozen kernel is the thing that would have to fire
its timer. This is the trap — the machine *looked* watchdog-protected.
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
pets the device while an HA client is connected, and this cluster has no HA
resources.
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
the watchdog below is the fix for *recovery*.
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
idempotency (6 skipped / 6 verify OK).
Firmware does **not** block the TCO timer here — checked for the
`unable to reset NO_REBOOT flag` line before committing to the approach; the
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
configured on esh-pve this must be reverted. Not a near-term concern:
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
already costs quorum and the survivor would fence itself — HA here would reduce
availability, not raise it.
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
Proving it means deliberately wedging the host. Untested-but-armed is still
strictly better than softdog; treat a real firing as unconfirmed until tested.
## Diagnostic corrections worth keeping
- **"No route to host" was the dead host, not a routing gap.** Two claims made
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
target subnet before concluding "the subnet is unreachable."**
- **UDM `uptime` on a client record is association time, not host uptime.** It
read 2.2 days while the host had been up 20 hours. Use
`journalctl --list-boots` on the host for real boot history.
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
that are demonstrably online). `stat/sta` is the live view.
persistent-memory.d/2026-08-19-fleet-internal-dns.md
-119
View File
@@ -1,119 +0,0 @@
# `[2026-08-19]` Fleet `.internal` DNS — git-sourced, agent-managed, three resolvers
Operator: *"with ipv6 i can't memorize the IP addresses anymore. need a way to
keep track of local .internal dns names that can be agent managed and is
lightweight."* Built and live in one session; commit `b8003c7`.
## Shape
```
dns/internal.yaml source of truth — 38 hosts + 4 service aliases
scripts/dns-sync.py reconciles AdGuard resolvers against it
stacks/adguard-ana/ the colo's resolver, which did not exist
dns/README.md workflow, naming, the IPv6 caveat
```
Deliberately the same posture as `deploy-stack.sh`: the file is intent, the
resolvers are derived state, you see a diff before anything changes.
`--dry-run` / `--yes` / `--site <s>`. Verified idempotent — a second run prints
`nothing to do`.
Naming is `<host>.<site>.internal` with sites **`ana` / `esh` / `nh3`**
(operator's call). `.internal` is ICANN-reserved for private use since 2024;
`.local` is reserved for mDNS, which is why the pre-existing
`searxng.pfi.local` was a standards collision that merely happened to work.
Every name is published to **every** resolver — the site label says where a
host *is*, not which resolver knows about it.
## The framing correction that mattered most
The ask reads as "I can't memorise v6 addresses", but the deeper problem is
that **v6 addresses are derived, not assigned**, so they cannot reliably be
*written down once* either. SLAAC gives EUI-64 (MAC-coupled) or
privacy-extension (rotating) addresses, and UniFi has **no v6 equivalent of a
DHCP reservation** — so a hand-maintained v6 table rots on its own.
⇒ The fix has two halves and only the second is DNS: (1) pin static v6 on
server-class hosts, (2) then the name table is just a file. Surfaced to the
operator before building.
**Verified 2026-08-19: no fleet host has a global v6 address at all yet** —
ESH's `/56` is live only on `esh-cameras`, NH3's LANs are back to
`ipv6_interface_type: none`, the colo has none. So the `v6:` column ships
EMPTY and correct, and the naming layer was built first rather than blocking
on v6. Names established now need no renaming when addresses land.
Suggested convention when they do (awaiting operator): each server static at
its site's `/64` with low-order bits echoing the v4 host octet —
`esh-docker-vm` at `…::45` — so addresses are declarable *and* semi-memorable.
## Two properties not to break
**Authority is scoped to the ZONE, not the resolver.** Only rewrites ending in
`.internal` are managed. ESH's resolver turned out to carry three hand-made
`esteban.net` rewrites (`eshnas`, `brotherprinter`, `eshhome`) — **my first
read of the config missed them**, because an `awk` range on `rewrites:` matched
an empty-looking block. A resolver-wide authoritative sync would have silently
deleted all three on first run. Verified intact after sync.
**Within `.internal` it IS authoritative** — names added by hand in the AdGuard
UI get deleted by the next sync. That is the point: one place to look.
## The colo had no resolver at all
ESH and NH3 each ran AdGuard; **ana-docker resolved straight against
`1.1.1.1`**, so the colo had no way to answer for internal names. Closed with
`stacks/adguard-ana/`.
⚠️ Its API is on **8053**, not 8080 — `:8080` and `:3000` were already taken on
that busy host. The port is therefore carried **per-site in the yaml**, not
assumed by the script, so the odd one out cannot be forgotten.
⚠️ It ships with **no blocklists**, deliberately. The other two filter ads for
human browsing; this one resolves for a rack of servers, where a blocklist
false-positive breaks service-to-service calls at 3am for no upside.
First boot uses a **seed config** (`conf/AdGuardHome.seed.yaml`) copied into
the conf volume before first start, so the container comes up configured
instead of sitting in the setup wizard.
## Credential — service account, not the operator's
Added a dedicated **`infra-ops`** AdGuard user to all three resolvers rather
than asking for the `lkraven` password (per the standing migrate-off-operator-
creds directive). Password vaulted at
`nh3-dev/adguard-infra-ops-password`; `lkraven` untouched; pre-change configs
backed up on each host as `AdGuardHome.yaml.bak-preinfraops-*`. Both existing
resolvers kept answering across the restart.
Two landmines worth keeping:
- **Go's bcrypt rejects `htpasswd`'s `$2y$` prefix.** Same algorithm, different
marker; `golang.org/x/crypto/bcrypt` accepts only `$2a$`/`$2b$`. Normalise
the prefix, and self-verify the hash with `htpasswd -vb` BEFORE installing it
on a live resolver.
- **The vault appends a trailing newline on `get`.** A password carrying a
stray `\n` fails auth in a way that looks exactly like a wrong password.
`dns-sync.py` strips it.
## `pfi.local` migration — and the one that must NOT move
`searxng.pfi.local` → `searxng.ana.internal`, with the **old `Host()` kept
alongside** in the Traefik rule so nothing breaks mid-migration; both return
200. Drop the fallback once the access log shows the old name unused.
**`matrix.pfi.local` deliberately NOT migrated.** A Matrix `server_name` is
baked into every user ID, room ID and signing key, and federation identity
derives from it — renaming it is not a DNS change, it is rebuilding the
homeserver's identity and invalidating its history. The operator approved
"migrate pfi.local" generally; this was surfaced as a deliberate exclusion
rather than executed blindly.
## Still open
Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
— it only answers what asks it directly. Repointing a whole site's DNS is a
bigger change than standing the service up, and is the operator's to schedule.
See also [[2026-08-17-fleet-ipv6-mesh]].
persistent-memory.d/2026-08-19-homepage-skyfall-theme.md
-145
View File
@@ -1,145 +0,0 @@
# `[2026-08-19]` Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background
Commits `9d92c4b`, `c3de7db`, `45c1995`, `f38cf69`, `df68dd2`.
## The cleanup (three real defects)
- **UltraSeedbox rendered on all four tabs.** The bookmark group had no entry in
`settings.yaml`'s `layout:` block at all, and Homepage's documented behaviour
is that a group with no `tab:` is shown on **every** tab. Pinned to Main.
⚠️ This will happen again to the next group added without a `tab:` — the rule
is now written at the top of the layout block.
- **Uptime Kuma rendered twice** — a manual `services.yaml` entry under
Monitoring *and* `homepage.group=Apps` on the container. Exactly the
"never list a labelled container manually" failure the stack README warns
about; it survived the previous day's audit because a duplicate reads as two
plausible cards rather than as an error. Manual block deleted, label moved to
`Monitoring`, `homepage.siteMonitor` added.
- **Column counts were fiction** — several groups declared more columns than
they had members, so the last row of each was dead space (Notes: 1 card in a
4-wide row). Columns now track member counts; `GET /api/services` prints the
live per-group counts and is the check.
Later, on operator instruction, the **AI tab was reordered by clickability**:
Gateways & Chat → Image & Media → Audio Tools on top, then the vLLM `/docs`
seats and TTS endpoints. Reasoning written into the config so it survives:
order by "would I click this?", not by how central the service is.
## ⚠️ The expensive red herring — the tab bar after a recreate
After a recreate the client render comes up with **no tab bar, no wallpaper and
no i18n** (search box shows the raw key `search.search`), groups falling back to
side-by-side columns. **It restores itself with no intervention.**
Timing, measured rather than assumed: a fresh container was still tab-less at
**4m30s, twice**; it was healthy again after roughly an hour. `docker ps`
reporting `healthy` says nothing about it — the container is serving, the page
is just wrong.
An hour went into ruling out four causes that were never the cause:
1. **Not the config** — restoring `settings.yaml` *and* `services.yaml` to
their committed versions reproduces it, as does the pre-adoption backup in
`/opt/docker-bu/conf/homepage/`.
2. **Not the v2.0.0 release** — a throwaway container on `v1.13.2` shows
identical symptoms, and the image never changed anyway (working and broken
both report `v2.0.0` / rev `17456f2`).
3. **Not `PUID`/`PGID`**, and not Docker discovery — tested both, and with the
socket unmounted entirely.
4. **Not server-side** — the server-rendered HTML still contains the tab
markup, the background URL and `useEqualHeights`; `GET /api/validate`
returns `[]`. The loss is client-side, with no page error, no failed chunk
and no non-200.
Every throwaway container in that list was judged within ~30s of starting, so
they were all inside the same window — and that consistency **read as a
reproduction when it was the same measurement mistake five times over.**
**Operative rule: recreate, walk away, re-check later. Do not chase it.**
## ⚠️ The iteration loop that would have prevented the overcook
`custom.css` is served **per request** from `/api/config/custom.css`, so a CSS
change needs a **browser reload** — not a container recreate, and it never owed
the layout warm-up above. Conflating the two costs ~10 operator-visible minutes
per attempt (operator called this out directly).
Faster still, and how the final pass was done: **inject candidate CSS into the
running page and screenshot it** —
`await p.addStyleTag({content: css})` in Playwright against the live
dashboard. Seconds per iteration, no deploy. Build + deploy only once the
render looks right.
## The theme — Australis Skyfall
Operator supplied a Claude Design handoff bundle via the Booth (`26-copper`).
Skyfall is a dual-theme OKLCH system: one lightness law across every chromatic
family (deep 0.48 / base 0.66 / bright 0.80), all hues cooler than neutral, a
Sea neutral ramp drifting ice-blue→ocean-green as it brightens, and a
"calm depth" language of **hairline + two-layer shadow on every elevated
surface, never one without the other**.
```
theme/colors.css layout.css typography.css vendored VERBATIM from the bundle
theme/fonts/Supreme-{400,500,700}.woff2 the body/UI face
theme/skyfall.css.in the Homepage bindings (ours)
theme/build.py → conf/custom.css (generated — do not hand-edit)
```
The build step exists for one reason: **Homepage serves only `custom.css` and
`custom.js` out of its config dir**, with no static route beside them, so a
`@font-face` pointing at a vendored `.woff2` would 404 — the face must arrive
as a data URI. The background image takes the other road, because
`/app/public/images` **is** a real static route (mounted read-only in
`compose.yaml`).
Only Supreme is embedded: a link dashboard has no display type, and Victor
Mono ships as 2.4 MB TTF statics per cut — 30x the whole stylesheet for a
handful of latency figures.
## The background is generated, not stock
**Arbo as an image-gen engine** (the operator's actual ask, which I first
misread as "use Arbo's palette" and had to redo). Arbo's `t2i-ui-background`
workflow is purpose-built: *"abstract full-bleed backgrounds, no subject"*.
Job `13f0891f4e42`, seed 26, flux2-klein-9b, 2048×1152, 1.6 MB PNG → **22 KB
WebP** (smooth gradients compress absurdly well).
⚠️ Arbo API gotcha: `prompt` is a **discriminated union, not a string** — a
bare string 422s. `{"kind":"raw","text":…,"negative":…}` is the shape.
## Two documented deviations from the design system
1. **Skyfall forbids this background.** Its rule is "flat semantic surfaces; no
photography, no textures", with one permitted motif — a subtle aurora
gradient on hero/empty-state areas only, *"never behind body text blocks"*.
A dashboard is a body-text block. Present on the operator's explicit
instruction, mitigated rather than excused: abstract, no subject, strictly
cool temperature, held at **`opacity: 30`**. That number is load-bearing —
at 14 the aurora was invisible, and turning it up makes the cards fight the
ribbon.
2. **Service icons stay full-colour vendor logos.** Desaturating them from CSS
only makes them illegible.
## Overcorrection, and the colour pass
First stat-well pass went from `font-thin` 13px straight to **bold 22px in
heading white** — operator: *"went from subtle to BASH YOU OVER THE HEAD."*
The principle missed: a stat only has to out-rank **its own label**, not the
service name above it. Now `--text-md` medium in cyan.
Colour was then lifted **from inside the system**: Skyfall names Aurora (blue,
cyan, green) the *primary* families, "used generously, in that order", while
Dawn (amber/red/violet) is semantic-only. So group markers cycle
blue→cyan→green down the page (icons full strength, names at 0.72), service
icons take a single cool wash, latency tags move to the info family so
"how fast" stops looking like "is it alive". **No Dawn colour is used
decoratively anywhere.**
Two DOM findings that made it possible:
- **Homepage renders mdi icons as a gradient behind an SVG mask** — recolour
via `background`, not `color`.
- **Homepage emits `docker-status-<state>`, not `status-<state>`.** The
original selectors matched nothing, so every green pill up to that point was
stock colouring rather than the theme. Both forms are now matched.
persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md
-77
View File
@@ -1,77 +0,0 @@
# `[2026-08-19]` Four unmanaged stacks found on live hosts — and two of them were quietly broken
Commits `42c594c`, `dc3e47b`, plus `uptimekuma` in `9d92c4b`.
## The pattern worth remembering
Chasing two bad-looking cards on the dashboard turned up **four stacks running
on fleet hosts with no canonical copy anywhere**: `uptimekuma` and (already
known) the two AdGuards on esh-docker-vm, `searxng` and `seafile` on
ana-docker, and `heretic2-charrp-reasoning` on ana-ml2 (untracked in git).
⇒ **A dashboard card is a cheap census of what is actually running.** When
something on it looks wrong, check whether the stack behind it is even in
`stacks/` before debugging the symptom — twice here the answer was "no", and
the fix belonged in version control as much as on the host.
Adopted: `stacks/uptimekuma/`, `stacks/searxng/`, `stacks/seafile/`,
`stacks/heretic2-charrp-reasoning/`. ESH/NH3 AdGuard compose files were
**deliberately left unmanaged** — adopting three live resolvers while also
introducing a new DNS naming system is two risky changes at once.
## SearXNG — the healthcheck was eating itself
Card flapped UNHEALTHY; the container was fine the whole time. The compose
passed `--tries` and `--spider` as **two separate argv entries**, so wget
consumed `--spider` as the *value* of `--tries`. Spider mode never engaged,
which means every probe since April **downloaded** the healthz response to a
file:
```
295,287 healthz.N files in the container's working directory
```
With that many files, wget's scan for the next free filename is what
intermittently blew the 10s timeout. **Self-worsening — every probe made the
next one slower.** Restored `--tries=1`; the junk lived in the writable layer
so the recreate cleared it. Now `healthy`, `fails=0`, 200 in 0.16s.
Lesson: an argv list in YAML has no shell to catch a missing `=`. A flag that
silently swallows the next argument turns a liveness probe into a workload.
## SeaFile — not broken, never restarted
Card showed EXITED for three months. **None of the three services declared a
restart policy**, so Docker defaulted them to `no`. On
**2026-05-06T21:27:45Z** the daemon stopped all three within 200ms of each
other — a daemon restart or host reboot — and nothing brought them back.
⚠️ **Exit code `255` is a red herring**: it is what a container that ignores
SIGTERM reports when the daemon stops it, **not** evidence of a crash. Reading
it as one sends you hunting a bug that does not exist. The tell was all three
services stopping within 200ms.
Added `restart: unless-stopped` to all three; brought up; mariadb gated on its
healthcheck exactly as the existing `depends_on` comments intended, seahub
started without the race, `302` → login page. Data was in local named volumes,
not on the ana-nas NFS, so nothing was at risk.
Three months of silent downtime whose only signal was a card nobody read as an
outage — the argument for semantic status colour on the dashboard (see
[[2026-08-19-homepage-skyfall-theme]], where amber EXITED pills made six
mis-grouped AI seats obvious at a glance).
## heretic2-charrp-reasoning — tracked, with its shim
The `char-rp-reasoning` seat (NEO-CODE Heretic2 27B, modelopt NVFP4 + grafted
BF16 MTP head, ~77 tok/s via `qwen3_5_mtp` spec-decode) had been running
untracked. Now in `stacks/`, including
`conf/mtp-workaround/sitecustomize.py`, which is **not optional**: vLLM 0.24.0
does not propagate modelopt `exclude_modules` to the spec-decode **draft**
model, so the BF16 MTP head gets quantized and the engine dies at load. Both
the mount and `PYTHONPATH` are load-bearing.
Added the two files house convention expects and the directory lacked — a
`.env.example` naming every knob (all values are compose defaults; the host
overrides only the three VRAM ones) and a README pointing at
`docs/runbooks/heretic2-nvfp4-mtp-seat.md` rather than duplicating it.
persistent-memory.d/2026-08-19-waterland-studio-containerised.md
-176
View File
@@ -1,176 +0,0 @@
# `[2026-08-19]` waterland studio containerised on irv-ml1 — three landmines, all measured
Handover from `waterland-dev` over althing (thread `01M0CDRGEZWAJCEJXXMQWXV80F`):
a FastAPI + SPA GPU service fronting the `waterland` CLI, running as a bare
`nohup` (PID 1283383) that would not survive a reboot. Now
`stacks/waterland-studio/`, `restart: unless-stopped`, healthy on
irv-ml1:8410. Commits `a2b5b58`, `8189076`.
Tracking `main` per operator: PR #4 merged and `main` HEAD was exactly the
pinned `8025366`, so tracking-a-moving-ref and keeping-the-pin agreed anyway.
**Now deployed at `b72425b` (2026-08-19).** The container sat on `8025366` for
a few hours after PR #5 (`464dfc2`) landed — deliberately, since the image's
own guards already neutralised both landmines and the project was in
wind-down. PR #6 (the job-store rehydrate, operator-green-lit) was the rebuild
with a real reason behind it, and one `update.sh` run carried both. Verified
end to end after the update: healthy, `backend: cupy`, and a real 256² plate
render completes warm — the kernel-cache volume survived the image swap.
## Build context lives OUTSIDE the compose dir — on purpose
`/opt/waterland-studio/src` is the checkout; the Dockerfile is passed
out-of-context from `/opt/docker/compose/waterland-studio/`. **`deploy-stack.sh`
rsyncs `stacks/<stack>/` with `--delete`**, so a checkout kept beside
`compose.yaml` would be destroyed by the next deploy of this stack. `update.sh`
refreshes source → rebuild → recreate → health, and is verified end to end.
## Landmine 1 — both uv extras are load-bearing at BUILD *and* RUN
`gpu` carries `cupy-cuda12x`; a bare `uv sync` prunes it and the renderer
silently drops to the numpy path at ~21x wall time — it does not error, it
just gets slow. waterland-dev warned about the build side.
The runtime side is worse and was not in the handover: **`studio/jobs.py`
shells the renderer out as a literal `uv run waterland ...` with no `--extra`
flags** (`cwd=WATERLAND_STUDIO_REPO`). Left alone, uv re-syncs the project
mid-job to its default extras and prunes cupy back out from under a correctly
built venv. Pinned with `UV_NO_SYNC=1`; `UV_OFFLINE=1` alongside so that if the
pin ever stops holding the job fails **loudly** instead of quietly rebuilding a
slower environment.
**Fixed upstream in `464dfc2`:** the server now spawns
`sys.executable -m waterland.cli` directly — no resolver in the render path at
all. **The pins stay anyway.** They cost nothing and are now defence-in-depth:
if any future code path re-enters `uv` inside the container, the job fails
loudly instead of quietly dropping to the numpy backend. `uv` itself must stay
in the image regardless — it performs the build-time `uv sync` /
`uv pip install`, and this is a single-stage build.
## Landmine 2 — cupy needs CUDA HEADERS, which the host never had to declare
Every render died 1.7s in with:
```
RuntimeError: Failed to find CUDA headers.
```
printed **through argparse's usage banner**, which makes it read like a CLI
argument bug rather than a missing toolkit. That misdirection is the reason
this is written down.
cupy compiles kernels at runtime through NVRTC, which needs the toolkit
**headers** — not just the driver and the runtime libs bundled in the
`cupy-cuda12x` wheel. irv-ml1 has a CUDA toolkit installed system-wide, so the
bare `nohup` process found them **by accident**; a slim image has none.
Fixed with `uv pip install "cupy-cuda12x[ctk]"` — headers as wheels, a few
hundred MB against ~6 GB for a `-devel` base image. It runs **after**
`uv sync`, because sync prunes what it does not know about.
Reported upstream: it is an undeclared runtime dependency of the `gpu` extra,
and anyone running this without a system toolkit hits it. **Declared upstream
in `464dfc2`** (`gpu` is now `cupy-cuda12x[ctk]>=13`). **The explicit install
stays in the Dockerfile**: the header requirement is a property of *this*
image — a slim base with no system CUDA toolkit — so it belongs in the file
that creates the problem, not inherited from an extra two repos away. It also
survives any future restructuring of the `gpu` extra. Cost of keeping it is
now measured, not assumed: since `uv sync` satisfies it first, the line
reports `Audited 1 package in 49ms` and adds **0.3s** to the build. A no-op
that documents a non-obvious requirement is worth 0.3s. (waterland-dev
independently agreed they would keep it too.)
## Landmine 3 — the GPU index inside the container is not the host's
The app pins `CUDA_DEVICE_ORDER=PCI_BUS_ID` and selects
`CUDA_VISIBLE_DEVICES_TARGET` (default `1`, correct on the host, where
`nvidia-smi` shows A6000 at 1). Compose exposes **exactly one** GPU
(`device_ids: ["1"]`, the A6000 in Docker's ordering), so **inside** the
container that card is index **0** ⇒ `CUDA_VISIBLE_DEVICES_TARGET=0`. Copying
the host's value selects a device that does not exist. Host device 0 is the
3090, which carries the TTS zoo and must not be touched.
## Cold start is ~17s of NVRTC compile → `/root/.cupy` is a volume
| job | wall |
|---|---|
| 256² + anim, cold container | 23.3 s |
| 256² + anim, warm | **6.1 s** |
| 256² plate only (`--codec none`) | 3.9 s |
| 512² plate only | 6.4 s |
Warm beats the **7.4 s** recorded against the bare-metal process, so
containerising cost nothing. Verified the cache volume properly: recreate
(fresh cache → 23.2 s first render) then restart (populated → 6.0 s). Without
it every restart makes the next user wait 4x and the service merely *looks*
slow.
## Upstream finding — the on-disk job store grows without bound
`JobStore._jobs` is a plain dict and **nothing scans `WATERLAND_STUDIO_DATA` at
startup**. Consequences:
1. After a restart `/api/jobs` lists only jobs created since — cosmetic, and
how this was spotted: the API reported **1 job** while the volume held all
**16 directories, 60.6 MB**. Not data loss.
2. The real one: `RETAIN = 40` eviction only ever iterates the in-memory dict,
so directories orphaned by a restart are **never reclaimed**. The
handover's "bounded around 500 MB" holds within a single process lifetime;
across restarts the store grows monotonically at ~12 MB per animated job.
Reported to waterland-dev with evidence; **not patched from the infra side** —
it is their code. Prune the volume by hand if it bites first.
**waterland-dev confirmed it (2026-08-19)** — their "bounded ~500 MB" handover
claim holds within one process lifetime and nowhere else, which on a
`restart: unless-stopped` service is the wrong lifetime to have bounded. They
have **surfaced a startup-rehydrate fix to the operator** rather than opening a
third PR during wind-down. **Operator green-lit it; PR #6 merged as `b72425b`
and is DEPLOYED (2026-08-19).**
Startup rehydrate, as recommended — and waterland-dev deliberately went
further than the framing I sent them. I had said a directory the scan cannot
parse "just does not enter the index"; they made the opposite call, because a
directory that never enters the index is exactly the one that never gets
reclaimed. **That is the sharper reading and it is the reason the fix works on
this volume at all** — the 16 pre-existing dirs have no sidecar. Their
adoption ladder: sidecar → restored verbatim; no sidecar → adopted with
dimensions recovered from the PNG IHDR (24-byte read, not a decode); corrupt
sidecar → degrades to inference, no startup crash; **neither source nor
sidecar → skipped on purpose**, since adopting it would turn eviction into a
delete-arbitrary-directories primitive pointed at this volume. Sidecar writes
go through `os.replace`, and `job.json` is excluded from `ARTIFACTS` so it is
unreachable via the artifact route.
They also closed a second leak I never saw, because it needs a restart
*mid-render* to surface: a job left `running`/`queued` in its sidecar is
non-terminal forever, and eviction skips non-terminal jobs — so it is a
phantom that is never reclaimed and `queue_depth` over-reports for the life of
the process. Adoption now marks those `failed`.
**Verified on this host after the update:** `/api/jobs` went **1 → 16** while
the volume stayed at 16 dirs / 61 MB — disk and API agree for the first time.
Nothing was reclaimed, correctly: 16 is under `RETAIN=40`, so adoption only
made them visible. A subsequent real render took both to 17. From here the
store is bounded **across** restarts, not merely within a process.
## Access
Repo is not anonymously readable (a bare clone 403s). Operator granted
**`claude-bot` read on `vh/waterland`** — verified `admin: False, push: False,
pull: True`. Token on irv-ml1 at
`/root/.config/waterland-studio/git-credentials`, `0600` root-owned, wired as a
**repo-scoped** credential helper; `.git/config` carries no token (verified),
so the remote stays clean in any diff or backup. The operator's `vh`
site-admin token was used only for the initial clone and the grant itself and
was **never written to disk on that host** — a site-admin credential on a GPU
box is a blast radius nobody needs for a read-only fetch.
## Constraints honoured as stated (not inferred)
- **Serial by design — one replica, one card.** A render is 20–45s of near-full
GPU with a single worker thread. Two on the same A6000 would OOM or thrash.
Throughput is a hardware conversation, not a replica-count one.
- **No authentication, arbitrary file uploads** ⇒ stays inside the
LAN/WireGuard boundary. Do **not** paper over it with a proxy password;
waterland-dev offered to add a real auth layer if wider reach is ever needed.
persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md
-95
View File
@@ -1,95 +0,0 @@
# `[2026-08-20]` Cold-Fusion abliteration — Robinson recipe captured, and the transformers/DeltaNet bf16-NaN fight
The real work of the session: abliterate `DavidAU/Qwen3.8-27B-Cold-Fusion-GAIN-V1.1`
using the MTP-aware, vision-preserving **Robinson formula** (documented in
`docs/pfi/abliteration-recipe-qwen38.md` from `RobinsonLabs/Qwen3.8-27B-abliterated`).
Harness: `services/coldfusion-abliteration/`. Runs on ana-ml2.
## Why this model, why abliterate it ourselves
Stock Cold-Fusion's refusal profile was **probed 2026-08-19** (Q6_K GGUF on
llama.cpp, 24-prompt battery, hand-verified after a keyword-classifier bug):
**~33% creative refusal**, concentrated on **explicit-sexual + graphic-torture**;
4/5 hard-harm technical refused; self-harm guardrails intact 3/3; benign
over-refusal 0. So there is a real creative-content refusal surface to remove.
This **supersedes** the earlier "watch for DavidAU's own heretic build" posture —
we abliterate it ourselves.
**It is additive over the current gen seat.** The live Heretic seat
(`qwen38-27b-heresy-bf16`) left its MTP head a **byte-identical base graft** —
the `Qwen3_5ForConditionalGeneration` wrapper never loads it, so Heretic could
not touch it. The Robinson formula abliterates the MTP head **in-band** (its 2
residual-write matrices), and the MTP head is what gates speculative acceptance.
That in-band MTP edit is the delta this experiment tests.
## Recipe maps 1:1 — dry-run PASSED
Against the staged bf16: 1199 tensors, 333 vision preserved,
`down_proj=64 o_proj=16 linear_out=48 mtp=2 embed=1`, coverage gate 6/6, exactly
**131** tensors to orthogonalize. Same architecture as RobinsonLabs' base, no
name drift. Two hard gates in the harness halt before any write: the coverage
identity `o_proj(16)+linear_out(48)==64`, and the attention-sink screen on
**dim 3994** (orthogonalizing a direction living there bricks the model).
## Capture SUCCEEDED — but only after a real environment fight (the durable lessons)
**The transformers Qwen3.5 DeltaNet linear-attention NaNs in bf16 on ana-ml2.**
The fast-path needs BOTH `flash-linear-attention` (`fla`, triton, installs fine)
AND `causal-conv1d` (**needs nvcc to build — absent, no prebuilt wheel**).
Without causal-conv1d the DeltaNet short-conv runs the torch fallback, which
produces **nondeterministic all-NaN** hidden states in bf16 (same 11-token input:
finite on one forward, NaN at layer 4 on the next). bf16 and fp32 share exponent
range, so this is **precision-driven catastrophic cancellation, not overflow** —
**fp32 resolves it.** Diagnosed via `diag_nan.py` / `diag2.py`: `sdpa` + plain
prompt = 65 layers all finite; chat-template input = NaN; the trigger is the
input path through the unstable recurrence.
Fixes, all in the committed harness (`7abd301`):
- **`--capture` loads fp32**; the write/surgery path stays bf16 (no forward, no NaN).
- **A finite-gate aborts on a non-finite direction** — the sink screen alone
can't catch it (`nan > threshold` is False, so a NaN direction "passed" it and
saved silently on the first run).
- `attn_implementation="sdpa"` pinned.
**fp32 (110 GB) needs the whole GPU.** device_map=auto packed it tight and the
forward OOM'd against the resident seats. Had to **stop three seats** for VRAM:
`vllm-meromero-rp`, `vllm-fablefusion-probe`, and production `vllm-gen`.
⚠ **Restart order matters:** gen restarted into an empty GPU0 and greedily
grabbed 64 GB (vLLM takes a fraction of *free* memory at startup), starving
meromero into a crash-loop. Fixed by bringing **meromero up first**, then gen
into the remainder. All three restored to healthy.
⚠ **fla lives in a side dir, not the venv.** The shared
`/tank/aimodels/quant-work/.venv` is not llmuser-writable; `fla` + `einops` are
`--target`-installed to `/tank/aimodels/coldfusion-abliteration/pylibs` and
reached via `PYTHONPATH`. Prune deps that shadow the venv's torch/transformers.
## Result
Refusal direction: **finite, unit-normed, layer 22**, sink energy **0.0008%**
in dim 3994 (recipe L26 ref 0.06%, threshold 1%) — clean, not sink-dominated.
Saved to `/tank/aimodels/qwen38-27b-coldfusion-bf16/refusal-direction.pt`.
⚠ **QUALITY CAVEAT — the reason the next step is calibration-set expansion.**
Two-template `|cos|` agreement at layer 22 is **0.59**, well below Robinson's
0.99. Almost certainly the small calibration set: **8 harmful / 8 harmless**
(HARMFUL/HARMLESS in `abliterate.py`) vs Robinson's **416 / 104**. The direction
is valid and sink-clean but noisier than ideal; abliterating on it risks
under-removing refusals or nicking capability. **Expand the sets to a few
hundred each and re-capture** before the `--out` write.
## Sequence from here
1. **Expand HARMFUL/HARMLESS calibration sets** → re-capture (fp32, seats down).
2. `--out` write (bf16 surgery, no forward) → `qwen38-27b-coldfusion-abliterated-bf16`.
3. Verify: vision byte-identical, refusal re-profile via `services/refusal-probe/`
(the canonical harness, NOT the ad-hoc GGUF one), MTP acceptance on the quant
(gate ≳40%, not KL — `reference_abliteration_mtp_lessons`), PPL/coherence.
4. NVFP4-quantize via `services/gen-seat-mixed-quant/` → gen-seat candidate.
**Do NOT delete the incumbent** (`qwen38-27b-heresy-nvfp4-mixed`) until it
holds through real multi-turn use.
bf16 staged at `/tank/aimodels/qwen38-27b-coldfusion-bf16` (pinned `9c44193`,
provenance recorded). All write paths re-stop the seats for fp32 VRAM — batch
re-capture + write in one window. Commits `ccb56a0`, `1857a8e`, `b56cb0d`,
`7abd301`.
persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md
+34
View File
@@ -0,0 +1,34 @@
# [2026-08-23] Every secret-bearing `.env` on ana-docker tightened to 0600
Found while taking uptime ownership of hrafn: its `.env` was mode 0644 with a live
bearer token. Not a hrafn lapse — **0644 was the de facto pattern on the host**.
Eight stacks carried secret-shaped vars in world-readable `.env` files on a box with
four interactive accounts, verified as real exposure by reading one as `nobody`.
Swept: **vaultwarden, traefik**, beszel, gitea-runner, miniflux, news-digest,
searxng, vor. (hrafn and nevermore were fixed separately the same day.) Six other
stacks already used 0600, so this converged on the existing house pattern rather
than inventing one. Post-sweep the host has **zero** secret-bearing `.env` readable
by `nobody`.
Playbook: `playbooks/tighten-env-perms.yaml`, one run per stack, re-runnable.
## The check that matters
Every run asserts `docker compose config` still renders **as the deploy user**
(`lkraven`), not as root. Checking the mode proves the bits changed; only rendering
as the deploy user proves the next deploy can still resolve its variables.
## Two gotchas recorded in the playbook
- **vaultwarden looked like it bind-mounted its `.env`** — which would mean the
*container's* UID reads it and 0600 could break the password vault. It does not:
that `- .env` is under `env_file:`, not `volumes:`. My grep matched the YAML list
item without checking its parent key. The playbook now **refuses** any stack that
genuinely bind-mounts its `.env`, since that case is read by the container UID.
- **elway prompted for a sudo password.** The `ana-docker` ssh alias resolves to
`lkraven`, who needs one; **`infra-ops@10.250.50.70` has NOPASSWD**. `corviduo-dev`
was repointed to infra-ops at some point and `ana-docker` was not. Run elway against
the infra-ops target on this host.
Commit `a896c0a`.
persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md
+663
View File
@@ -0,0 +1,663 @@
# [2026-08-23] Anaheim's IPsec tunnel delivers ~25% of a verified 2 Gbps circuit
> **⛔ SUPERSEDED 2026-08-23 (same day, later session) — read the CORRECTION at
> the bottom before acting on anything here.** The headline is wrong (the
> relevant ceiling is NH3's **1 Gbps** uplink, not Anaheim's 2 Gbps), the
> aggregate number is wrong (**692 Mbit/s** at 8 streams, not ~550 — the
> original stopped measuring at 4), and the proposed remedy is **impossible**:
> UniFi's manual site-to-site IPsec does not implement AES-GCM at all. The
> per-stream observation and the parallelise-your-transfers mitigation are the
> parts that survive.
The operator noticed site-to-site transfers were slow for a datacenter fiber
handoff and asked whether WireGuard was the limit. It is not WireGuard, and the
circuit is fine.
## Measured
```
ana-docker -> internet, 8 parallel 2,153 Mbit/s <- the 2 Gbps handoff, delivering
ANA <-> NH3 through the tunnel, 4 par. 460 Mbit/s
FortiGate's own recorded peak 554 Mbit/s
ANA <-> NH3, single stream 227 Mbit/s
ANA <-> ESH, single stream 249-265 Mbit/s
ESH <-> NH3 (never touches ana-gw) 545-557 Mbit/s on a SINGLE stream
```
Method: stdlib TCP probe (no ssh, no crypto, no compression) between site
endpoints; raw circuit measured with 8 parallel HTTPS fetches from Hetzner
Ashburn. Host NICs are virtio with no reported cap, so no host-side ceiling.
## What it is not
- **Not WireGuard.** Both Anaheim tunnels are IPsec on ana-gw
(`pfi-ana-nh3` -> 70.230.226.88, `ana-eshudm-dyn` -> the ESH UDM). WireGuard
on ana-wg is remote-access only and is not in this path. Traceroute confirms:
both slow paths have hop 1 = `10.250.50.1` (the FortiGate); the fast
ESH<->NH3 path rides a `192.168.x` Site Magic overlay and never touches it.
- **Not CPU or crypto exhaustion.** FortiGate CPU was **100% idle across all
8 cores** during the tests, and both live tunnels report `npu_flag=03` with
`dec_npuid=1 enc_npuid=1` — encrypt *and* decrypt are hardware-offloaded.
- **Not a 250 Mbit/s cap.** That was the first number and it is misleading —
single-stream TCP. Four parallel streams doubled it. Quote the aggregate.
- **Not the interface.** wan1: `rxe=0 txe=0 rxd=0 txd=0`, no collisions.
## Most likely cause
Both tunnels negotiate **`aes256-sha1`** in phase 1 *and* phase 2 (dhgrp 14,
IKEv2). AES-CBC + SHA1 is a two-pass operation; FortiGate NPUs are markedly
faster on **AES-GCM**, which combines encryption and authentication in one
pass. The datasheet IPsec headline for an 80F assumes GCM with large packets,
not CBC+SHA1 at the 1438-byte tunnel MTU this link negotiates. The ~4x
shortfall is consistent with that.
## Not executed
Changing the proposal is a **production-edge change requiring a matching
change at the far end** (NH3 UDM and the ESH UDM), and each tunnel drops while
it renegotiates. Left for the operator. See the index entry for authorization
state.
## Immediate mitigation, no config change
Per-flow is the weak axis: a single stream over Site Magic gets 557 Mbit/s, a
single stream through IPsec gets 227. **Anything moving bulk data across the
Anaheim link should parallelise** — that alone roughly doubles throughput
today.
## Practical consequence already observed
`/mnt/smithy` mounted on ana-ml2 reads at 24.7 MB/s sequential vs 98.3 MB/s
from nh3-dev (same file, same mount) — that gap *is* this tunnel, not NFS and
not the NAS. See [[2026-08-23-smithy-mount-ana-ml2]].
## Access note
ana-gw is a FortiGate-80F, FortiOS 7.2.10, at 10.250.0.1. `sshpass` is absent
on nh3-dev; connect with paramiko via `uv run --with paramiko`. Password is
vaulted at `fortigate/ana-gw-infra-ops-password`. **`diagnose vpn tunnel list`
prints live ESP session keys** — never paste its output into althing, a
booth, or a commit.
---
## CORRECTION (2026-08-23, later session): the cutover was attempted and the remedy does not exist
The operator authorised the AES-GCM cutover, NH3 side first. It cannot be done,
and the measurements taken while trying show there is very little left to win.
### AES-GCM is unavailable on the far end — not a naming problem
The NH3 edge is a **UDM Pro SE** terminating `pfi-nh3-ana` (networkconf
`_id 697d64414c85dd2b6669b00a`, `ifname vti64`). Its UniFi API **validates** the
crypto enum and rejected every GCM spelling tried — `aes256gcm`, `aes256gcm128`,
`aes256gcm16`, `aes-256-gcm`, `aes256-gcm`, `aes256gcm12`, `gcm`, `aes128gcm128`
— all `HTTP 400 api.err.InvalidPayload`, nothing applied.
**The control that makes this conclusive:** the *identical* request body with
`ipsec_esp_encryption: "aes256"` returns `HTTP 200 rc:ok`. So the 400s are the
enum rejecting the value, not a malformed body. Corroborating: **zero
case-insensitive `gcm` matches across 7.3 MB of UniFi OS UI bundles.**
Accepted enum (probed): `aes128`, `aes192`, `aes256`, `3des` → 200; `des`,
`chacha20poly1305` → 400. There is no AEAD option. Both Anaheim tunnels land on
UniFi far ends, so this blocks the ESH tunnel too.
The FortiGate side **was** widened and is GCM-capable: phase2 `pfi-ana-nh3` now
reads `set proposal aes256-sha1 aes256gcm`. Left in place deliberately — it is
functionally identical while the peer only offers CBC, and reverting it would
cost another SA renegotiation for a cosmetic gain. Phase 1 was never touched;
IKE protects the control channel only and has no bearing on data throughput.
### The numbers that retire this as a problem
Measured NH3→ANA through the tunnel, and NH3→ESH over Site Magic (WireGuard) on
the same UDM and the same uplink, with the same stdlib TCP probe:
| streams | IPsec NH3→ANA | WireGuard NH3→ESH |
|---|---|---|
| 1 | 245 Mbit/s | 557 Mbit/s |
| 4 | 471 Mbit/s | 767 Mbit/s |
| 8 | **692 Mbit/s** | **795 Mbit/s** |
**NH3's WAN is a 1 Gbps link** (`uplink.speed = 1000`, port capable of 10G) —
that, not Anaheim's 2 Gbps, is the ceiling for anything crossing this tunnel.
So the tunnel does **~69% of the achievable uplink** at 8 streams, and the
IPsec-vs-WireGuard gap collapses from 2.3× at one stream to **15% at eight**.
Re-architecting the transport (site-to-site WireGuard via `ana-wg`, since
FortiOS has no WireGuard) would chase that last 15%. Not worth it.
### What the constraint actually is
A **per-stream** limit (~245 Mbit/s), not an aggregate crypto ceiling. Both
endpoints are idle at load — FortiGate CPU 100% idle with `npu_flag=03`
(offloaded both directions), UDM CPU ~7% with load1 moving 0.70 → 1.55. The
shape is per-SA/per-flow serialisation, and WireGuard shows the same shape from
a higher floor (557 → 795 is only 1.43× scaling).
### Actionable consequence
Anything moving bulk data across this link should **parallelise** — 245 → 692
Mbit/s, a 2.8× win with no config change. For single-stream workloads that
cannot be parallelised at the application layer, **NFS `nconnect=N` is the
lever**: it opens N TCP connections per mount, converting a single-stream
workload into a parallel one. The `/mnt/smithy` mount on ana-ml2 reading at
24.7 MB/s (~200 Mbit/s, i.e. exactly the single-stream ceiling) is the live
example — remounting with `nconnect=8` is the obvious test.
### Foot-gun recorded
Probing the enum by PUTting candidate values **applies the accepted ones**. A
probe loop here timed out with `3des` briefly live on the NH3 side, which the
FortiGate would not accept — a short tunnel outage until `aes256` was restored
(~1 minute, confirmed by the SA counters resetting). If you enumerate a UniFi
config enum this way, restore the known-good value after **every** 200, not at
the end of the loop. Post-change verification: the UDM object was diffed
field-by-field against its pre-change snapshot and is **byte-identical**.
---
## FOLLOW-UP (2026-08-23): what the per-stream limit actually is
The correction above called the constraint "per-SA/per-flow serialisation".
That was a hand-wave. Measured properly, it is a **hard per-flow rate cap of
~230–245 Mbit/s with a very deep buffer in front of it** — not a tuning
problem, not loss, not window size.
### The evidence: pin the send buffer and sweep it
Single stream NH3 → ana-docker, `SO_SNDBUF` pinned, `ss -ti` sampled in flight:
| in-flight cap | throughput | RTT in flight | minRTT | retrans |
|---|---|---|---|---|
| 256 KB | 224 Mbit/s | 7.8 ms | 5.3 ms | 0 |
| 416 KB | 225 Mbit/s | 11.8 ms | 6.6 ms | 0 |
| 416 KB | 245–247 Mbit/s | 12.0 ms | 5.6 ms | 0 |
| ~3.3 MB (autotuned) | 245 Mbit/s | **107 ms** | 5.5 ms | 0 |
**Throughput is flat across a 13× range of in-flight data while RTT scales with
it.** That is the signature of a fixed service rate with a standing queue: the
window controls only how much queue you build, never how fast you go. Had this
been window-limited, throughput would have risen with the buffer. Had it been
congestion, there would be retransmits — there are essentially none
(`retrans:0`, 0% ping loss).
So `net.ipv4.tcp_*` tuning, window scaling and congestion-control choice are all
**red herrings here**. Do not go there.
### Bufferbloat: one bulk stream wrecks latency for everything else
Measured on the same tunnel, ping to ana-docker:
- idle: **6.9 ms** avg
- during a **single** bulk TCP stream: **102 ms** avg, 136 ms max, 0% loss
**15× latency inflation from one transfer.** This is the operationally
important finding — any interactive traffic sharing the Anaheim link (ssh,
RDP, althing, VoIP) degrades badly whenever anything moves bulk data, and it
takes only one stream to do it. Parallelising transfers makes throughput
better and this *worse*. If it starts biting, the fix is an AQM/shaper on the
tunnel (or rate-limiting bulk jobs), not more buffer.
### Where the cap lives — strong inference, not proof
Three paths, and the FortiGate is the only variable:
| path | single-stream |
|---|---|
| FortiGate ↔ NH3 UDM (IPsec) | 245 Mbit/s |
| FortiGate ↔ ESH UDM (IPsec) | 249–265 Mbit/s |
| NH3 UDM ↔ ESH UDM (WireGuard, **no FortiGate**) | 557 Mbit/s |
Present in both slow paths, absent from the fast one. Aggregate over the same
SA reaches 692 Mbit/s, so it cannot be the SA or the crypto engine as a whole —
many flows spread out fine, one flow does not.
The mechanism that fits is **FortiGate NPU IPsec offload being per-session**:
each firewall session is bound to one crypto engine, so a single TCP flow is
capped at one engine's rate while many sessions spread across engines. **This
is inference from the throughput shape, not something confirmed on the box** —
`diagnose sys session list` was not captured for a TCP flow (the filter caught
only traceroute UDP probes). A single-stream control through ana-gw *without*
IPsec returned 290 Mbit/s to Hetzner Ashburn, but at ~60 ms RTT that is
window-limited and does not discriminate. **If this matters, the clean test is
a non-IPsec single stream between two Anaheim VLANs at low RTT.**
**Relevant to the FortiGate cutover decision:** if the per-flow cap is the
FortiGate's IPsec path, replacing the box plausibly lifts single-stream
throughput toward the WireGuard figure. That is a point in favour of the
cutover, and it is cheap to verify afterwards by re-running the sweep.
---
## FOLLOW-UP 2 (2026-08-23): it is NOT a capacity problem, and it IS specific to IPsec
Operator asked directly whether the 80F "can't handle the traffic". It can.
Two new measurements settle the shape of this, and correct an overstatement in
FOLLOW-UP 1 (which pointed at the FortiGate on evidence that was confounded —
every slow path was *both* IPsec *and* FortiGate, so protocol and box could not
be separated by that argument).
### The 80F routes a single flow at line rate when IPsec is not involved
`ana-ml2 → pfi-pve`, inter-VLAN **through** ana-gw (traceroute hop 1 =
`10.250.50.1`), 0.36 ms RTT, no tunnel:
| streams | throughput |
|---|---|
| 1 | **940.2 Mbit/s** |
| 8 | 939.3 Mbit/s |
Single stream saturates 1 GbE. So the box does **not** cap single sessions in
general, and there is no per-session ceiling in its plain forwarding path. The
~250 Mbit/s per-flow cap is **specific to the IPsec datapath**.
### Both IPsec tunnels converge on the same numbers despite different far ends
Measured today with the same probe:
| tunnel | far-end gateway | RTT | 1 stream | 8 streams |
|---|---|---|---|---|
| NH3 ↔ ANA | UDM Pro **SE** | 6.7 ms | 245 Mbit/s | 692 Mbit/s |
| ESH ↔ ANA | UDM Pro **Max** | 3.9 ms | **268 Mbit/s** | **715 Mbit/s** |
Different gateway hardware, different sites, different uplinks, and RTT
differing by 1.7× — yet single-stream differs by only 9%. **If this were
window-limited the 3.9 ms path would be ~1.7× faster.** It is not, which is
independent confirmation of a rate cap rather than a BDP effect.
### Capacity summary — the box has headroom it will not give one flow
- plain routing, 1 stream: **940 Mbit/s** (line rate)
- plain routing to internet, 8 streams: **2,153 Mbit/s**
- IPsec, 8 streams: **692–715 Mbit/s**
- IPsec, 1 stream: **245–268 Mbit/s**
- CPU **100% idle** throughout; IPsec NPU-offloaded (`npu_flag=03`)
Within a single SA, 8 sessions get ~2.9× what 1 session gets, so the datapath
distributes work **by inner session** — consistent with IPsec offload binding a
session to one crypto engine.
### What is still NOT separated
Whether the cap belongs to **the 80F's IPsec offload** or to **UniFi's IPsec
implementation**. Both tunnels have a UDM at the far end, and both UDMs run the
same UniFi firmware, so identical caps are explainable either way. The Pro Max
being only 9% faster than the Pro SE argues against the UniFi side (a beefier
CPU should show more), but that is suggestive, not conclusive.
**The test that closes it:** an IPsec tunnel whose endpoints do not include the
80F — e.g. a temporary UDM↔UDM IPsec tunnel between NH3 and ESH, measured
single-stream. If it also caps ~250, the FortiGate is exonerated and replacing
it buys nothing on this axis. If it runs near the 557 Mbit/s that UDM↔UDM
WireGuard achieves, the 80F is the limiter. **Bears directly on the pending
FortiGate cutover** — worth running before that decision, not after.
---
## FOLLOW-UP 3 (2026-08-23): WireGuard over the same internet path does 767 Mbit/s on ONE stream
Operator asked for a WireGuard test from `ana-wg` to NH3 over the public
internet. It is the test that separates the *path* from the *crypto*, and the
answer is unambiguous. **It also overturns FOLLOW-UP 1's "re-architecting the
transport is not worth it" — that conclusion compared 8-stream numbers and was
wrong for single-stream workloads.**
### Setup (fully torn down afterwards)
`ana-wg` (10.250.50.252, Debian 12 LXC, 4 cores) already has an
internet-reachable WireGuard endpoint: wg0 on **UDP 31337**, published by
FortiGate VIP `wg-to-ana-wg` (extip **38.120.12.42** → 10.250.50.252:31337,
policy 46, service `WireGuard-LEET`). **No FortiGate change was needed.** A
temporary `wgt0` was created on nh3-dev (10.30.10.200/32) as a fourth peer on
wg0, measured, then removed — ana-wg is back to its original 3 peers and the
keys were shredded. `wireguard-tools` was installed on nh3-dev and **left in
place** (benign, and wanted if this becomes permanent).
In this topology **neither gateway does crypto**: the FortiGate and the NH3 UDM
only NAT/forward UDP, and Linux does WireGuard at both ends.
### The full comparison
| path | crypto performed by | 1 stream | 8 streams |
|---|---|---|---|
| IPsec NH3↔ANA | FortiGate + UDM | 245 Mbit/s | 692 Mbit/s |
| IPsec ESH↔ANA | FortiGate + UDM | 268 Mbit/s | 715 Mbit/s |
| **WireGuard NH3→ana-wg** (same internet path) | **Linux + Linux** | **767 Mbit/s** | 763 Mbit/s |
| WireGuard NH3↔ESH (Site Magic) | UDM + UDM | 557 Mbit/s | 795 Mbit/s |
| plain routing through the 80F (inter-VLAN) | none | 940 Mbit/s | 939 Mbit/s |
**One stream equals eight streams over Linux WireGuard (767 ≈ 763).** There is
no per-flow penalty at all, and a single flow already saturates the path. So
the ~245 Mbit/s per-flow cap is **not** the ISP, not the circuit, not the NH3
uplink and not the physical path — all of which sustain 767 on one flow.
Per-flow penalty ranks by implementation:
- **Linux WireGuard — none** (767 → 763, flat)
- **UDM WireGuard — mild**, ~1.4× (557 → 795)
- **IPsec on this pair — severe**, ~2.8× (245 → 692)
### Latency under load — the same story
| path | idle | during ONE bulk stream |
|---|---|---|
| IPsec NH3↔ANA | 6.9 ms | **102 ms** avg, 136 ms max |
| WireGuard NH3→ana-wg | 6.2 ms | **12.7 ms** avg, 23 ms max |
WireGuard carries **3.1× the single-stream throughput with 8× less latency
inflation** on the same wire.
### Attribution — still not fully separated, and it no longer matters much
Both IPsec measurements have a FortiGate *and* a UDM doing IPsec, so this still
does not isolate which one imposes the 2.8× penalty. Closing that would need
Linux↔Linux IPsec or UDM↔UDM IPsec on the same path. **But the practical
decision no longer depends on the answer**, because the fix is the same either
way and it is already demonstrated.
### Recommendation (supersedes FOLLOW-UP 1)
A **WireGuard site-to-site between NH3 and Anaheim, terminated on `ana-wg`**, is
worth real consideration: 3.1× single-stream, flat scaling, far better latency
under load, and it reuses infrastructure that already exists and is already
internet-reachable. It is also the architecture already proven for NH3↔ESH.
Open questions before committing: routing/failover if ana-wg (an LXC) is down,
whether it replaces or parallels the IPsec tunnel, and firewall policy for the
new transit. ana-wg CPU was only ~40% busy across 4 cores at 767 Mbit/s, so it
has headroom.
**AND: `nconnect=8` on /mnt/smithy remains worth doing regardless** — it is the
same lever (turn one flow into many) and brokkr-smithy-dev has given standing
approval to apply it once the FortiGate work settles, with no need to ask again.
---
## RESOLVED (2026-08-23): it is the UDM's software AES-CBC. The FortiGate is exonerated.
Operator's theory — the UDM does IPsec in software with no crypto offload, so
the cost of the cipher itself is the limit — is **correct**, and it is now
demonstrated rather than inferred. He also correctly pointed out that
UDM↔UDM Site Magic is **WireGuard, not IPsec**, so that row never said anything
about UniFi's IPsec performance. It didn't, and I had leaned on it.
### The controlled experiment: vary cipher cost, hold everything else
AES-128 is 10 rounds, AES-256 is 14. If software crypto is the binding
constraint, throughput must rise when the cipher gets cheaper. If the limit
were the FortiGate's NPU, it would not move at all — hardware crypto is not
cipher-cost-sensitive in that range. Run A/B/A, single stream, 25–60 s each:
| condition | ESP cipher | single-stream | UDM CPU |
|---|---|---|---|
| A | aes256-cbc + sha1 | 232.3 Mbit/s | 35.4% |
| B | **aes128**-cbc + sha1 | **281.8**, 274.9 Mbit/s | 35.5% |
| A again | aes256-cbc + sha1 | 244.9, 242.5 Mbit/s | — |
**~1.16–1.20× faster on the cheaper cipher at identical CPU.** Same bytes of
CPU work, more payload through it. That is the signature of CPU-bound software
crypto, and it rules out the FortiGate's NPU as the limiter.
### Correcting two of my own earlier claims
1. **"UDM CPU is only ~7%, so it isn't CPU-bound" was WRONG — a sampling
artifact.** UniFi's `system-stats.cpu` refreshes on the device report
interval; 4-second sample windows were reading stale values. Under a
sustained 60 s single-stream load it reads **35.4%**, with load1 rising
0.60 → 1.17. On a 4-core UDM Pro SE that is ≈1.4 cores — one core saturated
on crypto plus overhead. **Always drive load for ≥60 s before trusting a
UniFi CPU figure.**
2. **The "FortiGate per-session NPU offload" hypothesis is REFUTED**, not merely
unproven. It predicts no change from a cipher swap; a 20% change was measured.
### Why the numbers all line up now
- **1 stream = 1 core of UDM crypto** → ~240 Mbit/s on AES-256-CBC.
- **8 streams = ~3 usable cores** → ~692 Mbit/s, ≈2.9× the single-stream figure
on a 4-core box. Aggregate is noisy (492–692 across repeats on a live link)
and is *not* cipher-sensitive, consistent with it being bounded by the path/
uplink rather than crypto once several cores are engaged.
- **AES-CBC is the specific villain: it is serial.** Each block depends on the
previous one, so the ARM AES instructions cannot pipeline across blocks. GCM
(CTR-based) and ChaCha20-Poly1305 both parallelise freely. That is why the
same UDM does 557 Mbit/s single-stream on WireGuard and only 240 on IPsec.
- **This retroactively vindicates the GCM cutover as the right idea aimed at the
right box** — GCM would have removed the serial dependency on the constrained
end. UniFi simply does not offer it, which is what made it impossible.
### Options this opens
- **AES-128 instead of AES-256: ~16–20% for free**, no topology change, one API
call per end. 128-bit is not the weak link here (SHA1 integrity is more
dated, and unchanged either way). Operator's call — **not adopted**, restored
to aes256.
- **WireGuard site-to-site via ana-wg: 767 Mbit/s single-stream** (3.1×), and it
sidesteps the UDM's IPsec datapath entirely. Still the biggest win available.
- Replacing the FortiGate **will not help this** — it was never the constraint.
Worth knowing before the cutover.
### State left behind
UDM network object verified **byte-identical** to its pre-test snapshot
(aes256/sha1). Tunnel up, selectors 1/1. FortiGate phase2 `pfi-ana-nh3` is
left as `aes256-sha1 aes256gcm aes128-sha1` — a permissive superset; the peer
offers only aes256 so the extra entries are inert, but **narrowing it back to
`aes256-sha1` is one line** if the looser list is unwanted.
---
## FOLLOW-UP 4 (2026-08-23): a downstream WireGuard terminator costs nothing to forward through
Operator's point: FortiOS has no WireGuard, so a WireGuard site-to-site must
terminate on a box *behind* the edge. Correct — and `ana-wg` (LXC, CT 113 on
pfi-pve, 10.250.50.252) already is that box.
**This closes a gap in FOLLOW-UP 3.** That 767 Mbit/s figure was measured with
traffic terminating *on* ana-wg. Real traffic must be forwarded onward to other
Anaheim hosts, which was never measured. Now it is:
| topology | 1 stream | 8 streams |
|---|---|---|
| IPsec, FortiGate ↔ UDM (today) | 245 Mbit/s | 692 Mbit/s |
| WG terminating **on** ana-wg | 767 Mbit/s | 763 Mbit/s |
| **WG transit: nh3 → wg → ana-wg → forward → ana-docker** | **763.8 Mbit/s** | **790.4 Mbit/s** |
**Forwarding through the LXC is free** (763.8 vs 767). The downstream-VM
architecture delivers the full 3.1× single-stream for real transit traffic, not
just for traffic landing on the tunnel box.
ana-wg while forwarding 764 Mbit/s: **~22% busy across 4 cores** (77.8% idle),
so roughly 0.9 cores. Note `/proc/loadavg` inside this LXC reports the *host's*
load, not the container's — do not read it as ana-wg's own. For contrast the
UDM burns 35.4% of its 4 cores to move 240 Mbit/s, so ana-wg has ample headroom.
### Design consequences of terminating downstream — the parts that need decisions
1. **Anaheim hosts must route to ana-wg, not to the FortiGate.** The 763.8
figure was obtained with an explicit `10.30.10.200/32 via 10.250.50.252`
route on ana-docker. Without that, a host sends 10.100.0.0/16 to its default
gateway (ana-gw), which routes it back out the *same* interface to ana-wg — a
LAN hairpin crossing the FortiGate twice. **The hairpin variant was NOT
measured.** Options: DHCP option 121 pushing the route fleet-wide, a dedicated
transit VLAN for ana-wg, or accept the hairpin.
2. **New single point of failure.** Today site-to-site dies only when the edge
dies, which is total anyway. A downstream terminator fails independently.
Mitigation: keep the IPsec tunnel configured as a higher-metric fallback
route so it takes over when ana-wg is down.
3. **ana-wg is an LXC on pfi-pve**, so its ~0.9 cores and NIC traffic land on the
hypervisor shared with the rest of the Anaheim VMs.
4. **The NH3 end needs a terminator too**, and there are two shapes:
- **Linux VM at NH3** (nh3-dev or a dedicated VM on nh3-pve) — this is what
was measured: **764 Mbit/s**.
- **NH3 UDM's existing WireGuard server** (`PFI-NH3-WG`, wireguard-server on
UDP 31337) accepting ana-wg as a peer — plausible but **untested**, and
UniFi's WireGuard shows a per-flow penalty (557 Mbit/s single-stream on
Site Magic), so expect ~557 rather than 764. Still 2.3× today.
### Standing recommendation
Worth doing, but it is **a project, not a config tweak** — routing, failover and
policy all need deciding. The cheap wins remain available meanwhile and are
independent: `nconnect=8` on NFS mounts (approved by brokkr-smithy-dev, pending
the FortiGate work settling) and AES-128 for ~20%.
---
## LANDED (2026-08-23): AES-128 on both tunnels; FortiGate public admin closed
Operator directed: adopt AES-128 on **both** Anaheim tunnels, make-before-break,
then close the FortiGate's WAN and SSH admin surfaces. All done and verified.
**Context that retires the WireGuard-in-a-VM design work:** the FortiGate is
being **replaced by OPNsense on a Dell R420**, which gives **WireGuard on the
edge device itself**. The downstream-terminator architecture (FOLLOW-UP 4) is
therefore moot — do not scope it. This also **un-parks the OPNsense migration**,
which auto-memory recorded as PARKED pending "hardware acquisition"; the R420
is that trigger.
### What changed
Make-before-break on the FortiGate first, so neither tunnel dropped waiting on
a far end:
| phase2 | proposal now |
|---|---|
| `pfi-ana-nh3` | `aes256-sha1 aes256gcm aes128-sha1` |
| `ana-eshudm-dyn` | `aes256-sha1 aes128-sha1` |
Then each UDM flipped to `ipsec_esp_encryption: aes128`:
| tunnel | UDM object | before | after |
|---|---|---|---|
| NH3 ↔ ANA | `pfi-nh3-ana` `697d64414c85dd2b6669b00a` @ 10.100.0.1 | 245 Mbit/s | **269.7** |
| ESH ↔ ANA | `esh-ana` `697723b9b9d4266dddf2bcc7` @ 10.0.0.1 | 268 Mbit/s | **304.3** |
Single-stream gain ~10–13% here, against 16–20% in the earlier controlled A/B —
the difference is live-link variance, not a different result. Both UDM objects
were diffed field-by-field against pre-change snapshots: **the only field that
moved on either is `ipsec_esp_encryption`.**
The FortiGate proposal lists were deliberately **left permissive** (still
accepting aes256). The peers offer only aes128 so the extra entries are inert,
and keeping them means a UDM reverting does not strand the tunnel. Narrowing to
`aes128-sha1` alone is a one-liner if the looser list is unwanted.
### Admin surfaces closed
`wan1 allowaccess` → **`ping`** (https + ssh removed) and `infra-ops` trusthost
→ **10.0.0.0/8 only** (the 8 wide-open ranges unset). Verified 443 and 22 closed
from both NH3 and ESH; management over the tunnel at 10.250.0.1 still works.
**Sequencing that matters: the close was executed over the TUNNEL path, not over
WAN** — removing `ssh` from allowaccess while connected over WAN kills the
session mid-command.
**Consequence to hold in mind: ana-gw now has no out-of-band management path.**
If both tunnels drop it is console-only until someone is on site.
### Gotcha: the two UDM vault items have DIFFERENT shapes
- `unifi/pfi-udmse-api-key` → a **bare 32-char key**. `secret get` output is the key.
- `unifi/esh-udmpm-api-key` → a **19-line documentation note** with the key on a
`key:` line. `secret get` piped straight into a header yields a 1396-byte
value and the UDM answers **`400 Bad Request` from nginx**. Extract with
`grep '^key:' | awk '{print $2}'`.
**The ESH key's first-ever confirmed WRITE happened here** (auto-memory recorded
it as read-verified only): a control PUT of the unchanged object returned
`rc:ok`, then the real change did too. That key has a full read+write admin role.
---
## CORRECTION (2026-08-23): port 80 on the WAN IP is the FortiOS ACME listener
The claim in the previous section that `.42:80` was an **ISP transparent proxy**
was **WRONG**, and so was the earlier warning that ACME renewal would fail with
port 80 absent from `allowaccess`. Operator pushed back asking where the port-80
map terminated. It terminates **on the FortiGate itself**.
**What it is:** the FortiOS **ACME HTTP-01 challenge listener**. `config system
acme` has `set interface "wan1"`, and FortiOS opens port 80 on that interface to
answer Let's Encrypt challenges **independently of `allowaccess`** — `wan1
allowaccess` reads `ping` only and the port is still open. Every non-challenge
request returns a fixed 403 whose body is literally:
```
<!DOCTYPE html><html><head><title>ACME Access Only</title></head><body>ACME Access Only</body></html>
```
**Not a DNAT.** The full VIP table has 14 entries; only two land on `.42` —
`Kokoro-In` (:8880 → 10.250.50.51) and `wg-to-ana-wg` (:31337 → 10.250.50.252).
~~Worth noting separately: four VIPs are all-port static NAT~~ — **that claim was
WRONG, see the correction below.** All fourteen VIPs are scoped.
### The methodology error that produced the wrong answer — worth not repeating
The sniffer filter used was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
matches only inbound packets**, so outbound SYN-ACKs were excluded *by
construction*; concluding "the box sends no SYN-ACK" from that capture was
unsound. Re-run with the bidirectional `host 38.120.12.42 and tcp port 80` it
immediately shows `wan1 out 38.120.12.42.80 -> <scanner>: syn ack`.
**Rule: when testing whether a box *answers*, the sniffer filter must be
bidirectional. `dst host` silently answers a different question.**
### Consequences
- **ACME renewal will work** with `allowaccess ping`. The earlier "add `http`
back or the cert expires" warning is retracted — FortiOS opens the challenge
port itself. Cert valid to 2026-10-27, renewal attempt ~2026-09-27.
- **It is not an admin surface** — static 403, no auth, no GUI.
- Its practical value is now low: WAN admin is closed, so the cert only serves
the internal GUI at 10.250.0.1, where the name would not match anyway. Killing
it (`config system acme` → unset interface) would close the last WAN listener
at the cost of cert renewal. Operator's call; **not done**.
---
## CLOSED OUT (2026-08-23): ACME disabled; and the "all-port VIP" alarm was FALSE
### ACME disabled — the WAN IP now exposes nothing
`config system acme / unset interface` (the account object is left in place;
with no interface bound there is no listener). Verified:
- **External scan of 38.120.12.42 across 55 ports: no open TCP ports at all.**
- Internal GUI at 10.250.0.1 still answers **200**, SSH still works.
- `admin-server-cert` is still `ana-fw.pfi` — the existing cert is untouched and
serves the internal GUI until **2026-10-27**; it simply will not auto-renew.
Reverse with `config system acme / set interface "wan1"`.
### RETRACTION: the four VIPs are NOT all-port
A previous section claimed `Rustdesk`, `https-to-tacticalrmm`, `web-to-webhost`
and `web-to-sfcontainer` were unrestricted all-port static NATs. **They are not.**
A FortiOS VIP can be scoped **two different ways** and the parser used only
checked one:
1. `set portforward enable` + `set extport <n>` — a single mapped port, **or**
2. `set service "<svc>"` on the VIP object — constrains the VIP to that service.
All four use form 2. The custom services are narrow: `Rustdesk` = TCP
21115–21119 + UDP 21116 (the standard RustDesk range), `ssh-mapped-2223` = TCP
2223 only. **Every one of the 14 VIPs is scoped; none is unrestricted.**
**Lesson: absence of `portforward` does NOT mean all-port on a FortiOS VIP —
check `service` too.** Better still, do what settled it here: scan from outside
rather than reading config.
### Ground-truth public exposure (external TCP scan, post-change)
| IP | open | maps to |
|---|---|---|
| 38.120.12.41 | *nothing* | — |
| **38.120.12.42** | ***nothing*** | the FortiGate itself — fully closed |
| 38.120.12.43 | 80, 443 | sf-ana-container 10.250.150.100 (SureFire tenant) |
| 38.120.12.44 | 22, 80, 443, 8025, 21115–21119 | gitea (→222), traefik, mailrise, RustDesk |
| 38.120.12.45 | 80, 443, 2223 | pfi-ana-webhost 10.250.50.52 (2223→22) |
| 38.120.12.46 | 443 | pfi-tacticalrmm 10.250.50.57 |
Configured-but-closed: 8443 (mattermost-calls), 8444 (webdav-nas), 8880
(Kokoro-In) — VIPs exist, nothing listening behind them. Worth a tidy-up during
the OPNsense translation but not exposure.
persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md
+63
View File
@@ -0,0 +1,63 @@
# [2026-08-23] hrafn adopted; its CI deploy reported green while deploying nothing
`hrafn` — genuine-Chromium browser-fetch behind a REST API, for bot-gated sites
(Reddit first). Built by nevermore-claude on ana-docker, handed to infra-ops for
uptime ownership. Internal-only on `traefik-net`, no host port; consumers reach
`http://hrafn:8080`. Canonical at `stacks/hrafn/`.
## Intake found a live credential exposure
`/opt/docker/compose/hrafn/.env` was mode **0644 with a live 57-char bearer token**
— verified as real exposure by reading it as `nobody` on a box with four
interactive accounts. Tightened to 0600. That triggered the wider sweep (see
[[2026-08-23-ana-docker-env-perms-sweep]]).
## The CI defect — the one worth remembering
I authored the deploy (elway playbook + gitea workflow) to replace a hand-rsync,
tagging the image with the commit SHA for provenance. nevermore-claude later found
v1.0.0 deploying "green" while the host still served 0.1.0.
**Root cause was mine and nastier than either hypothesis.** The staging dir was
`$compose_dir/.stage` — **inside** the rsync target. So
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted `.stage` from the
destination (absent from the source listing) **during** the transfer, destroying
its own source mid-copy. Reproduced exactly:
```
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
after: app.py="OLD" leftover.txt GONE, .stage GONE
```
Deletion succeeded, the copy silently did not, rsync exited 0. So the directory
*looked* converged while host source stayed frozen at the first manual rsync —
and because the build's `COPY` inputs never changed, Docker full-cache-hit and
every SHA tag aliased one image. **The provenance the tagging existed to provide
was false for the pipeline's entire life.**
**The real failure is the verification.** The verify steps asserted the marker,
container health, and a 200 from `/readyz` — all of which pass against a
completely frozen host. None measured *content*. A deploy that reports success
without asserting the bytes changed is verifying an **uptime**, not a deploy.
## Fixes
- stage at `/tmp/hrafn-deploy-stage`, outside the target
- CI computes `context_sha256` over the shipped file list; the playbook recomputes
it **on the host after the converge** and fails on mismatch
- compare the running container's `src/**/*.py` against the host's, so a SHA tag
cannot name layers the image lacks
- **compare `*.py` only** — `pip install .` generates `src/*.egg-info/*` inside the
image and `__pycache__` appears at runtime, so a naive `find src -type f` compare
false-fails on every healthy deploy. Verified against a known-good container
before shipping (12 host files, 18 in container, 0 content differences).
- declined `--no-cache`: a cache hit is *correct* when the context is genuinely
unchanged; assert the property rather than brute-force it.
## Access
Operator granted claude-bot **write** on `vh/hrafn`, so infra-ops maintains the
pipeline it owns instead of routing patches through the repo holder. `vh/hrafn` is
canonical; `stacks/hrafn/ci/` is a verified mirror.
Commits `b6924de`, `b001d0c`, `11b9d18`, `b38c369`, `9642952`.
persistent-memory.d/2026-08-23-selene-retired-alias-collision.md
+81
View File
@@ -0,0 +1,81 @@
# [2026-08-23] selene seat retired after losing a head-to-head; 7 aliases share one seat
## Why selene went
Benchmarked against `gen` on selene's own job — 24 designed judge items with
checkable ground truth, pairwise + absolute modes, 3 repeats, run on **both** a
neutral JSON prompt and Selene's **native Atla template** (288 calls, free local).
```
neutral JSON selene 20/24 (83%) gen 23/24 (96%)
native Atla selene 21/24 (88%) gen 22/24 (92%)
```
gen won on both templates and **selene's best sat below gen's worst**. Selene was
given its own fine-tuned template as a fairness check before any recommendation;
it gained one point, not three.
**Decisive defect: selene cannot emit "tie"** — 0/2 on both templates, forcing a
winner on every equivalent pair. For eval work that is the case that matters.
brokkr-smithy-dev independently corroborated from the other end with a **null
control** (an excerpt compared against ITSELF, where tie is definitional):
`chat-judge`(selene) TIE **27/60 = 45%**, gen **60/60 = 100%**; ground-truth
recovery on real-corpus ranking selene **47% — chance** vs gen 94%. My 83-vs-96
understated it: on a *ranking* task selene was a coin flip. Absolute scoring on
designed items is an easier task than ranking real text — the harness is a
**screen, not a verdict**, and its README says so.
Reclaimed **17.2 GiB** on ana-ml2 GPU1 (free 1,818 -> 19,450 MiB).
## The naming rule, restated the hard way
I proposed repointing `selene-1-mini-8b` at gen and was **correctly overruled**:
> never repoint a named model at a different model's endpoint — that is
> intentionally misleading
`chat-judge` is a **role** alias (ADR-0012: consumers bind the capability) and
moved to gen with a deterministic judge profile copied from `image-judge`.
`selene-1-mini-8b` is a **model** name and was removed outright — it now returns
`HTTP 400 Invalid model name`, verified. The discriminator: *does the string
promise a capability, or an identity?*
## The 7-way alias collision — the finding with the longest reach
```
chat-judge classifier gen image-judge
qwen-image-bench summarizer summarizer-large -> qwen3.8-27b-uncensored :8015
```
Also colliding: `gen-frontier`/`gen-frontier-reasoning`/`glm-5.2`/`glm-5.2-reasoning`;
`ext-tts`/`gpt-4o-mini-tts`/`tts-1`/`tts-1-hd`; `reranker`/`reranker-a3-bge-v2-m3`.
**Cross-checking a result against another alias measures nothing when they are the
same weights — agreement is an echo, not corroboration.** Documented at the head of
`model_list` in the live gateway config, because it belongs where people read it.
This caught a real defect within hours: brokkr's R47 premium-corpus gate was about
to run ~46,000 record-exposures against `gen` with `summarizer` shortlisted as an
independent second opinion. They pinned the backing model in the preregistration
and dropped the second-alias idea instead.
## Provenance seam (brokkr's pushback, adopted)
The gateway returns the **alias** in the response `model` field, not the backing
model — so a per-call guard catches a swap *during* a run and is blind to one
*between* runs. **Role alias for routing, concrete model for provenance.**
`GET :4000/model/info` with the shared key already exposes backing model +
api_base; resolve at run start AND end and void on mismatch.
## Artifacts
- Harness kept at `tools/judge-bench/` (`--models` REQUIRED — a stale default
would silently benchmark a retired seat).
- `stacks/selene/` keeps compose + a README explaining the retirement.
- Technique worth stealing, from brokkr: **a control constructed so the correct
answer is DEFINITIONAL rather than judged cannot inherit the designer's error.**
Item vs itself; response vs its own truncation; text vs its own clauses
permuted. Add those before adding more judged items.
Commits `ca3c984`, `b8a5355`.
persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md
+50
View File
@@ -0,0 +1,50 @@
# [2026-08-23] `/mnt/smithy` mounted on ana-ml2 — read-only and SOFT, deliberately not matching nh3-dev
brokkr-smithy-dev asked for `10.100.50.50:/volume1/smithy` on ana-ml2 to run R47's
CPU-bound corpus pipeline on 96 idle EPYC cores instead of one nh3-dev vCPU. Granted,
with two deliberate deviations from what was requested.
```
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
10.100.50.50:/volume1/smithy /mnt/smithy
```
The export already permitted ana-ml2 — no DSM change needed. Write is genuinely
refused.
## Why soft, not hard
They asked to match nh3-dev's mount, which is `hard`. **nh3-dev is same-site as the
NAS; ana-ml2 is not** — this is cross-site NFS on the box running the fleet's
inference seats. A hard mount turns a link blip into unkillable D-state, and this
fleet has already lost a host that way (esh-docker-vm; only fix was a reboot). Soft
returns EIO, the batch job fails, you rerun it. The soft-mount corruption caveat is a
**write** hazard and this is read-only. Mirrors the existing ESH books mount.
## Why not in fstab
Manual only, matching irv-ml1's `/mnt/smithy` precedent. A cross-site NFS entry in
fstab can hang boot on a GPU host with 71 days uptime. **Needs remounting after a
reboot.**
## The performance reality, measured on the same file through the same mount
```
ana-ml2 (cross-site) nh3-dev (same-site)
sequential read 24.7 MB/s 98.3 MB/s
small-file rate 45.3 files/s 34.6 files/s
```
Two different stories, and file layout decides which you get:
- **Many small records -> ana-ml2 wins on BOTH axes.** That path is bound by per-file
round-trips and NAS overhead, not bandwidth, and ana-ml2 is an idle 96-core box
while nh3-dev is a loaded 16-vCPU VM.
- **Bulk sequential streaming -> the link eats the win.** 4x read penalty against a
6x CPU gain. `datasets/raw` is 126 GB, `datasets/derived` is 1.7 GB — which one the
pipeline traverses changes the answer by two orders of magnitude. Staging a subset
to ana-ml2 local disk (195 GB free) beats pulling it over the wire repeatedly.
The 24.7 MB/s is the Anaheim tunnel, not NFS and not the NAS — see
[[2026-08-23-anaheim-ipsec-tunnel-ceiling]]. No mount tuning will move it;
parallelism will.
persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md
+70
View File
@@ -0,0 +1,70 @@
# [2026-08-23] Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed
## b187 pre-stage (#405 phases 1+2)
The matrix bridge stopped embedding the engine and became an HTTP client of the
Conversation API, so `WORLDTREE_API_URL` became **boot-blocking** — absent from the
container env, the bridge exits by design. Demo's compose never passed it; the next
recreate would have crash-looped. Pre-staged on demo and personal (additive, backed
up, verified with `docker compose config`, nothing restarted).
**Key decision, and I got its scope wrong first.** I argued demo should stay keyless
(no homeserver -> no rooms -> no turns -> no 401s). Right about turns, **wrong about
scope**: the engine preflight authenticates at boot regardless of homeserver, so demo
booted permanently degraded. Corrected — key `341c1488` minted under worldtree-dev's
recorded authorization, vaulted, wired, three-hop hash-verified.
## The 69-day-stale `:latest` landmine
All three instances pinned `WORLDTREE_IMAGE=.../worldtree:latest` in `.env` while
running SHA-tagged images built that day. Local `:latest` = `b19afd71d7cc`, built
**2026-06-14**. So ANY `docker compose up` — anyone's, for any reason — silently
downgraded that service by 69 days. Same footgun as the 2026-06-15 outage.
Re-pinned all three to their running SHAs (Worldtree #410), verified by rendering
compose config rather than reading `.env`, containers untouched. Playbook at
`playbooks/repin-worldtree-image.yaml`.
**`worldtree-pinned` was the worst case:** the instance whose entire purpose is being
frozen was running a **dangling image with no repo tags**, kept alive only by the
running container. One `docker rm` from garbage collection. Tagged
`:446e5807bf43` first, then pinned.
The guard I wrote had two bugs the pinned case exposed: it compared the container's
`.Config.Image` **string** (only the tag it was CREATED from — pinned was created
from `:latest` back when that meant 446e5807), and it reported CHANGED
unconditionally. Now compares **image IDs** and skips when already correct.
## Matrix homeserver ownership
Operator ruled: **personal owns the Matrix bridge.** The appservice tokens were never
missing — both sat at length 64 in the vaulted dev `env.sh` while both deployed
instances had them at length **zero**. Someone wired four of six Matrix vars and
stopped. Wired them into personal, three-hop verified.
**The trap worth remembering:** Synapse's registration pointed at
`http://10.100.10.50:8009` — nh3-dev, a dead epoch, with transaction 2801 queued at
512s backoff. The natural fix (swap the IP) gives `10.250.50.152:8009` which is
**DEMO's** bridge, and Synapse can reach both — it would have connected, delivered,
and looked correct while routing the operator's live rooms to the demo instance.
**Personal's bridge is :8010.** `docker port` is ground truth.
Corrected the URL, restarted Synapse (healthy in 32s after 3.5 months up), verified
`GET /_matrix/app/v1/ping -> 200` from inside the Synapse container. worldtree-dev's
smoke passed first try: room created, mimir accepted the invite, a real engine turn
ran, mimir replied in persona voice. #408 closed.
## Open on worldtree-dev's side
- **#411** — personal's bridge logs `Debug sink init failed: Permission denied:
/app/sessions/debug_rooms.json`. It creates two debug rooms but cannot persist
their IDs, so **every restart mints a fresh pair on the live homeserver**. Room
litter that compounds silently. Needs a which-container-writes-what check on the
sessions volume before anyone chowns it.
- Bridge/engine agent-roster drift: 6 of the bridge's 9 configured agents are not
listed by the engine on either instance.
- Historical Domari pairwise verdicts from the selene era are coin-flip-grade
(see [[2026-08-23-selene-retired-alias-collision]]); worldtree-dev banked that so
no future arc leans on them without re-judging.
Commits `064181a`, `bb19a96`.
persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md
+50
View File
@@ -0,0 +1,50 @@
# [2026-08-24] ana-gw public admin surface closed to zero, ACME listener included
WAN admin was opened at the start of the session as a cutover contingency
("so I don't have to drive down there"), then closed again on operator
instruction once the AES-128 work landed. Net result: **the FortiGate's WAN
address now exposes no TCP port at all.**
## Final state
External scan of `38.120.12.42`, 55 ports: **nothing open**. Verified from two
sites. `wan1 allowaccess` = `ping`; `infra-ops` trusthost back to `10.0.0.0/8`.
**Consequence to hold: there is no out-of-band path to ana-gw.** If both tunnels
drop it is console-only. Re-open is two one-liners (allowaccess + trusthost) —
both are recorded in auto-memory `reference_fortigate_ana_gw_access`.
## Port 80 was the FortiOS ACME listener, and I got it wrong first
`38.120.12.42:80` answered a bare 403 (`ACME Access Only`, 101 bytes) with
`allowaccess` set to ping only. First diagnosis — "an ISP transparent proxy" —
was **wrong**, and the reason is worth keeping:
> The sniffer filter was `dst host 38.120.12.42 and tcp port 80`. **`dst host`
> matches inbound only**, so outbound SYN-ACKs were excluded *by construction*,
> and concluding "the box sends no SYN-ACK" from that capture was unsound.
Re-run bidirectionally (`host … and tcp port 80`) it immediately showed
`wan1 out 38.120.12.42.80 -> <scanner>: syn ack`. **Rule: to test whether a box
*answers*, the filter must be bidirectional.**
The listener is opened by `config system acme / set interface "wan1"` and
**bypasses `allowaccess` by design** — FortiOS needs port 80 for HTTP-01. It
was disabled (`config system acme / unset interface`); the LE cert (`ana-fw.pfi`,
valid to 2026-10-27) is untouched and simply stops renewing, which is fine
because WAN admin is closed and the box is being replaced.
## Retracted in the same pass: the "four all-port VIPs" alarm
Claimed four VIPs were unrestricted all-port static NAT. **False.** A FortiOS
VIP is scoped **two** ways — `portforward`+`extport`, *or* a `service` binding
on the VIP object — and only the first was checked. All 14 VIPs are scoped;
`Rustdesk` is TCP 21115–21119, `ssh-mapped-2223` is TCP 2223 only.
Ground-truth external scan of all six public IPs is recorded in
`reference_fortigate_ana_gw_access`. Configured-but-dead: `:8443`
(mattermost-calls), `:8444` (webdav-nas), `:8880` (Kokoro-In) — tidy-up
candidates for the OPNsense translation, not exposure.
**Lesson, twice in one session: measure from outside instead of parsing config.**
Both wrong answers came from a filter that answered a different question.
persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md
+167
View File
@@ -0,0 +1,167 @@
# `[2026-08-24]` char-rp seat: OOM root-cause, Gemma-4 MoE swap, and the abliterated trainee base
One evening, one thread with brokkr-smithy-dev, five commits: `850e0c3`,
`27155c0`, `f509668`+`24e8826`+`1bd90ea`+`3446367`+`8d6a939`, `14ff4a3`,
`019ccff`, `5415fd4`.
## 1. The seat was crash-looping, and the cause was NOT its config
`vllm-meromero-rp` reported up-but-unreachable, RestartCount climbing (13 by the
time it was examined, not the 4 first reported). Startup logs looked clean all
the way through weights, `torch.compile` and CUDA-graph capture, then:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 336.00 MiB.
GPU 0 has a total capacity of 94.97 GiB of which 195.19 MiB is free.
**⚠ `--gpu-memory-utilization` SIZES THE KV CACHE AND DOES NOT COVER CUDA
CONTEXT, GRAPHS OR NON-TORCH OVERHEAD.** gen is configured at 0.43 and actually
held 45.6 GiB. char-rp was at 0.51. The pair was committed to 0.94 of the card
with ~0.6 GiB of real headroom — it fit on the 21st and stopped fitting on the
24th.
Dropped char-rp to 0.47: ~4.8 GiB margin, KV 27.36 → 23.56 GiB, 430,825 →
371,023 tokens against a max-model-len of 262,144. **Cost nothing usable** — the
pool still holds 1.4x a full-length sequence; what is lost is concurrent long
requests, not context.
**⚠⚠ THE MISSING HALF, found later that evening: gen's footprint GROWS WITH
UPTIME.** Same container, same 0.43: **45.6 GiB after ~3 days up, 38.5 GiB
freshly restarted** — ~7 GiB apart. Nothing about char-rp changed between the
21st and the 24th; *gen crept up underneath it*. **Headroom arithmetic done
against a long-running gen is measuring a moving number.** Measure against a
freshly-restarted one.
## 2. `char-rp` and `char-rp-reasoning` are ONE seat, not two
Both LiteLLM routes point at `10.250.50.54:8016/v1` — `hosted_vllm/char-rp` and
`hosted_vllm/char-rp-thinking`. brokkr had reported 30/80 and 80/80 failure
rates as two failing services; it was one outage sampled twice. This also
*improved* a result of theirs: their CoT on/off battery had assumed both aliases
were the same weights under two chat templates, and the routing detail turned an
assumption into a verified fact.
(`vllm-charrp-reasoning-nvfp4`, the Heretic2 NVFP4+MTP container, has been
stopped for 12+ days and is unrelated — it is not what that alias resolves to.)
## 3. The seat swapped to the Gemma-4 26B-A4B MoE
Operator-directed straight-across replacement: same port, same
served-model-names, so no gateway route or consumer config moved. Rationale is
throughput under CoT — the user waits through the whole reasoning block before
the first visible token, and the MoE measures ~114 tok/s @32K against the dense
31B's ~40.7.
Serving copy is `RedHatAI/gemma-4-26B-A4B-it-NVFP4` (16 GB), chosen over the
other `-it` quants because it is compressed-tensors (`nvfp4-pack-quantized`) —
the same loader path the outgoing seat used. Smaller weights at the same 0.47
budget bought **1,724,110 KV tokens against the predecessor's 371,023**.
`meromero-charrp` is retained stopped in `created` state, labelled
`AI - Dormant`. Both stacks bind `:8016`, so rollback is **stop-then-start**.
## 4. ⚠ THE STALE-CHAT-TEMPLATE TRAP IS ENDEMIC, NOT A ONE-OFF
Verified by hash across every third-party Gemma-4 derivative pulled:
| build | lines | sha256 (normalised) |
|---|---|---|
| upstream `google/gemma-4-26B-A4B-it` | 390 | `6a1015c47ccfcfa6` |
| RedHatAI NVFP4 (served) | 389 | `6a1015c47ccfcfa6` — the only match |
| llmfan46 heretic | 365 | `0a52be69cda5ab8a` |
| TrevorJS abliterated | 266 | `58c66fdee4afa297` |
| jenerallee78 abliterated | 266 | `58c66fdee4afa297` |
| prithivMLmods NVFP4A16 | 266 | `58c66fdee4afa297` |
Three independent repos carrying the *identical* stale file means it propagated
through the ecosystem. Consequences differ by use and **both are silent**:
serving a mismatched template renders a different prompt; **training through
`base/chat_template.jinja` means training on a different prompt format than
production serves** — train/serve skew, no error, presents as a tuning failure.
The production compose now pins the template explicitly. It is a **no-op for the
served weights** (the A4 build ships that exact file) and permanently closes the
class. ⚠ If `GEMMA4_MODEL` ever points at a different checkpoint, the template
default must move with it.
## 5. A benchmark result was RETRACTED — below chance indicts the instrument
A battery appeared to show Gemma at **12% contradiction detection with CoT off
against gen's 81%**. An A16 activation-precision control was staged to test
whether the quant scheme owned it. Then the operator asked to see the individual
items, and the item was **ill-posed**: it presented two mutually contradicting
statements and asked for "*the* contradicting statement", but **contradiction is
symmetric**. The model consistently named the absolute claim — a defensible
reading the labelling scored wrong every time.
**⚠ THE TELL WAS IN PLAIN SIGHT: 12% ON A FIVE-OPTION TASK IS BELOW THE 20%
CHANCE FLOOR.** A below-chance score indicts the instrument before it indicts
the model, and neither side reacted to it. I spent the afternoon verifying repo
names, config fields, template hashes and tokenizer vocabs — every layer of
plumbing — and never asked whether the number itself was *possible*. **A
preflight can be thorough and still be aimed in the wrong direction.**
Retracted: "the model owns the contradiction deficit"; "domain tuning costs 43
points of contradiction detection" (on a sound instrument it **reverses**); all
pre-fix T2 numbers. Recorded as a dated superseded-claims table in
`stacks/gemma4-charrp/README.md` rather than a silent edit.
**What survived:** the A16 control result — activation precision is close to free
on this battery, every other task identical across W4A4 and W4A16 builds.
## 6. The abliterated trainee base — measured, not assumed
Operator directed a low-damage abliterated instruct build. "Low damage" was
treated as a measurable claim; the field spreads from KL 0.09 to 0.4118:
| build | method | KL | refusals |
|---|---|---|---|
| **llmfan46** (operator's pick) | Heretic v1.2.0 ARA | 0.1237 | 3/100 |
| TrevorJS | ARA-family | 0.09 | 1/100 effective, 5/686 cross-dataset |
| jenerallee78 | ARA 2-pass | 0.1299 | 7.7% StrongREJECT |
| huihui-ai | remove-refusals | none published | none published |
Fleet anchor: our own work found **Heretic at KL 0.12 preserved the MTP head at
83.7% acceptance**, so both staged builds sit inside an already-measured band.
huihui-ai rejected — no metrics, its card calls the method "a crude,
proof-of-concept implementation", it abliterates both thinking and non-thinking
modes, and its parameter count runs ~738M over upstream. Operator's independent
read matched ("huihui produces garbage").
**Abliteration isolated properly** (stock BF16 vs llmfan46 BF16, same precision,
same pinned template, same 192 items):
T2 contradiction 75% → 59% (−5 items)
T6 spatial 75% → 88% (+4 items)
core 90.0% → 89.4% (−0.6 pts)
**It MOVED capability rather than removing it** — five lost on contradiction,
four gained on spatial, nearly cancelling. Nobody predicted a gain. **llmfan46
stands**; no case for re-staging on TrevorJS over 0.6 points.
⚠ Read as ~5 and ~4 items at n=32, not as −15.6/+12.5 percent. ⚠ Says nothing
about quantization — the stock-NVFP4 T2 figure came from n=16 against n=32,
different item sets, n-confounded.
## 7. ⚠ The production compose hardcodes `--quantization compressed-tensors`
Pointing the char-rp stack at unquantized BF16 weights crash-loops immediately:
TypeError: CompressedTensorsConfig.__init__() missing 3 required
positional arguments: 'target_scheme_map', 'ignore', 'quant_format'
vLLM trying to read a quantization config out of a checkpoint that has none. 35
restarts before it was caught. Hence `stacks/gemma4-trainee-bench/` — a separate
ephemeral stack with no quantization flag, `restart: "no"` so a bench seat cannot
resurrect itself and block gen's restore, and no homepage labels so it leaves no
permanently-offline card. That detour is why a base swap is now ~5 minutes
instead of ~15.
## 8. BF16 cannot coexist with gen
48.07 GiB of BF16 weights plus gen's footprint exceeds the 94.97 GiB card before
a byte of KV cache. Every BF16 bench window means **gen is stopped**. Two such
windows were run and gen restored both times; the restore was triggered by
observing the seat's own throughput logs (a large prefill burst then zero
running/zero waiting) rather than waiting on a courtesy message.
Cross-links: [[2026-08-24-homepage-uniform-grid]]
persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md
+77
View File
@@ -0,0 +1,77 @@
# [2026-08-24] ESH DNS fixed at the IPv6 layer, and the naming scheme went live
Reported as "`scriberr.ana.internal` doesn't resolve on my Mac, and nslookup
shows an IPv6 DNS server." Operator's diagnosis was right; the fix took three
wrong turns worth recording.
## Root cause
`esh-userland` has IPv6 PD with RA at `pref high`, and the UDM advertises
**itself** as the resolver via RDNSS. macOS honours RDNSS and prefers it over
the DHCPv4-supplied resolver, so queries went to the UDM — which does not know
`.internal` — and returned NXDOMAIN. AdGuard was never consulted.
Two adjacent gaps found while there: `esh-userland`'s **secondary** v4 resolver
was `10.0.10.1` (the UDM itself), and `esh-server` had **DNS handout disabled
entirely**, so every host there got the UDM and could never resolve `.internal`
— esh-docker-vm was living proof.
## The three wrong turns
1. **`dhcpdv6_dns_auto=false` alone does nothing.** It is only honoured **when an
explicit server is supplied**. Setting it bare looks like a no-op and invites
the conclusion that the field is inert — which I drew, wrongly. Despite the
`dhcpdv6_` prefix it *does* drive the RA's RDNSS option on a SLAAC network.
2. **`wan_dns1` is NOT used by the UDM's LAN-facing forwarder.** Setting it to
AdGuard persists, reads back, and changes nothing. Proven with **fresh
uncached ad domains** — AdGuard blocklists answer `0.0.0.0`, the UDM returned
real IPs. Reverted.
3. **`force-provision` returns `rc:ok` and proves nothing** — consistent with the
known `cmd/devmgr` behaviour.
Every failed attempt returned `rc: ok`. **Verify by observed effect.** RAs were
probed with a stdlib raw-socket Router Solicitation parsing option type 25
(`rdisc6`/`tcpdump` were both absent; nothing was installed).
## What landed
RDNSS **redirected** rather than disabled — better than switching it off:
| VLAN | v4 | v6 RDNSS |
|---|---|---|
| `esh-userland` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
| `esh-server` | 10.0.50.45 + 10.100.50.40 | `…:4411:b105:50:45` |
The v4 secondary moved from the UDM to the **NH3 AdGuard** — reachable over
Site Magic and authoritative for the zone. ⚠ **A secondary only fails over on
SILENCE, not on wrong answers**: NXDOMAIN is a *successful* answer, the client
accepts it and never retries. A secondary that doesn't know your private zone is
a coin flip, not a spare tyre. `esh-cameras` deliberately untouched — routing
camera DNS through AdGuard's filtering risks their cloud features.
## The naming scheme became real
The resolver address is the scheme's first live use, replacing a MAC-derived
SLAAC address that would break on a NIC change. All three `esh-server` Linux
hosts now carry `4411:B105` ("FOR ALL BIOS"):
```
esh-docker-vm …:4411:b105:50:45 esh-pve-nas …:4411:b105:50:55
esh-vm-db …:4411:b105:50:60
```
Applied by an `if-up.d` hook that **derives the prefix at runtime** (self-heals
on re-delegation), backgrounds itself with a retry (SLAAC may not have landed;
a blocking hook would stall bring-up on a headless box), and adds nothing to
existing config. **Not** an `iface … inet6 static` stanza — on Debian that sets
`accept_ra=0` and would strand the host.
⚠ **Proxmox bridges need `accept_ra=2`.** `esh-pve-nas` had link-local only
despite every sysctl looking right: `vmbr0.forwarding=1`, and the kernel ignores
RAs on a forwarding interface unless `accept_ra` is explicitly `2`. Fixed with
`accept_ra_defrtr=0` alongside, so it takes the prefix but **declines the default
route** — an IPv6 identity with no change to a hypervisor's routing. Expect this
on every Proxmox node when its LAN gets v6.
Canonical: `docs/pfi/ipv6-naming-scheme.md`. UniFi limits:
auto-memory `reference_unifi_dns_rdnss_limits`.
persistent-memory.d/2026-08-24-homepage-uniform-grid.md
+308
View File
@@ -0,0 +1,308 @@
# `[2026-08-24]` Homepage: remote-label consumption re-verified, then the board relaid out on a uniform grid
Prompted by the operator: *"Homepage on esh-vm-docker lists remote dockers and
can absolutely consume their labels, please verify again. I am still
unsatisfied with the layout and aesthetics."*
## The verification — the operator was right, and the record now says so
**Homepage on `esh-docker-vm` discovers services by container label from all
five Docker engines in `conf/docker.yaml`, not just its own.** This is not an
inference; `GET /api/services` returns every card's `server` field, and the
2026-08-24 snapshot resolves to:
| `server` | host | label-discovered services |
|---|---|---|
| `ana-pfi-docker` | 10.250.50.70 | 30 |
| `irv-ml1-docker` | 10.100.79.3 (over WireGuard) | 15 |
| `ana-ml2-docker` | 10.250.50.54 | 14 |
| `esh-vm-docker` | 10.0.50.45 (the dashboard's own host) | 13 |
| `nh3-pfi-docker` | 10.100.50.40 | 2 |
**74 of 107 cards are label-discovered, and only 13 of those come from the
dashboard's own engine** — the other 61 are read off four remote hosts,
including irv-ml1 across the WireGuard tunnel. The remaining 33 carry
`server: null`: those are the manual `services.yaml` entries — hardware, BMCs,
hypervisors, printers, and user-level systemd services that have no container
to label in the first place. **That null is the only thing "not label-driven"
about this dashboard**, and it is a property of the entry, not of the host it
points at.
⚠ If a future session doubts this again, the check is one command and takes two
seconds — do not reason about it from the docs:
```bash
curl -s http://10.0.50.45:5100/api/services \
| jq -r '.[].services[] | .server' | sort | uniq -c
```
## What was actually wrong with the layout
Measured with Playwright against the live board (per-group `card=` width, card
height spread, and a geometric title-vs-status overlap test), not judged by
eye:
- **Card width changed at every group boundary.** `columns:` is not a density
dial — it sets `lg:grid-cols-N` for one group, so it fixes that group's card
width. Notes rendered a single **1464px** card; News and Media **728px**;
Eval & Retrieval **286px**; everything else 360px. Scrolling the page, the
grid resized five times.
- **Long names printed underneath their own status pill.** Measured by
re-injecting the old rule and testing the title text node's box against the
status cluster's box: **6 cards, all on the AI tab** — 3 in Inference, 2 in
Dormant, 1 in Eval & Retrieval; zero on the other three tabs, which is why
it survived earlier passes. Root cause is a genuinely counter-intuitive one:
the rule reserved a
78px gutter with `padding-right` and relied on `overflow: hidden` to hold it,
but **overflow clips at the padding box, not the content box** — so the
reserved gutter was spill room the title rendered straight through. The
intended `text-overflow: ellipsis` never fired either, because the ellipsis
is painted by whichever block's own line overflows, and here that is the
anonymous box around the bare title text node, which does not carry
`overflow`.
- **`AI Systems` / Scriberr was on all four tabs** — the 2026-08-18 UltraSeedbox
bug recurring, this time arriving from a container label rather than from
`settings.yaml`.
- **Icons were grey smudges.** Homepage masks every glyph over
`--color-logo-start/stop`; stock slate-400 → slate-700 sinks the bottom half
of each icon into the card fill.
- Bookmark groups and Jellyfin's trailing stream rows were the two components
the theme had never reached.
## The fixes
`stacks/homepage/conf/settings.yaml` — **all 20 groups to `columns: 4`.**
`stacks/homepage/theme/australis.css.in` → rebuilt → `conf/custom.css`:
gutter held by wrapping, description clamped to 3 lines (floor still 2), icon
ramp overridden, bookmark + trailing-widget components themed, group gap
10px → 22px. `stacks/scriberr/compose.yaml` — `homepage.group` → `AI - Audio
Tools`, container recreated on ana-ml2.
After: **every group renders at card=360**, and the collision count is zero.
Before/after, all four tabs: `http://10.100.10.50:8090/b/homepage-relayout/`
(24h TTL; also on the standing link board).
## ⚠ Three traps worth carrying forward
1. **"Columns = member count" is RETIRED** (it was the 2026-08-18 rule). It was
avoiding dead cells in a short last row and bought a worse defect. A short
last row is what a grid looks like; a card wider than its neighbours is what
a mistake looks like.
2. **A `:root` override of a Homepage theme variable is silently ignored.**
Homepage sets `--color-logo-*` on `.theme-slate`, and that class is on the
`<html>` element — the same element `:root` matches. `.theme-slate` (0,1,0)
beats `:root` (0,0,1), so the override does nothing and looks like the
variable is not the one in play. `html[class]` (0,1,1) wins, and does not
hard-code which `theme-*` class is active. Specificity alone is not enough
either: a custom property resolves from the *nearest* ancestor that sets it,
so the override has to land on `<html>`, not on `<body>`.
3. **The post-recreate tab-bar loss is INTERMITTENT, not guaranteed.** The
2026-08-19 note reads as though every recreate costs up to an hour of broken
render. This recreate came up correct within 10 seconds — fresh payload on
the first poll, all four tabs clickable a minute later. Recreate, *check*,
and only then walk away if it is actually in the broken state.
Also re-confirmed, since the change depended on it: **a `settings.yaml` edit
needs a container recreate, not a restart.** `docker restart homepage` left the
old `"columns":1` payload embedded in the served HTML with the correct file
mounted and readable inside the container; `compose up -d --force-recreate`
cleared it immediately.
## Deliberately not done — operator's call
The Main tab still opens on three sparse bands: **Notes** (1 member) and
**Games** (1) each burn a full 4-wide row, and **News** has 2. Merging Notes +
News, or folding Games into Apps, would tighten the top of the page — but that
is information architecture, not layout, and the group names are the operator's.
Surfaced rather than done.
→ **Resolved in pass 2 below**, where the operator delegated the naming
("re-categorize however you want"). Notes + News became `Daily`, Games folded
into `Apps`, and the `AI - Audio Tools` placement in this pass was superseded
(Scriberr moved on to `AI - Studios`).
---
# `[2026-08-24, pass 2]` Recategorised on "do I open this?", API groups collapsed
Operator, after seeing pass 1: *"You can re-categorize however you want.
service networking tab is uneven, you can split out the adguard cards, etc.
most of the issues are that tools I use and have a UI are interspersed with API
endpoints which are largely informational only. They might even go in their own
cards or start collapsed."*
## The axis
Every group is now either **tools** (expanded, top of tab) or **endpoints** (an
API, a broker, an agent — `initiallyCollapsed: true`, bottom of tab). A
collapsed group still renders its eyebrow and rule, so presence costs one line
instead of two rows.
Second, quieter rule that fell out of the same pass: **a group's members should
all carry a widget or none should.** A stat strip adds ~50px, so one widget card
in a row of plain ones opens a void under the plain ones — which is most of what
made the 13-member `Service Networking` band look broken.
## Shape
- **Main** — `Daily` (Memos, Miniflux, Nevermore, SearXNG — replaces the
1-card Notes and 2-card News bands), `Monitoring`, `Apps` (12; absorbed the
1-card `Games` band), `Media`, `UltraSeedbox`.
- **AI** — `AI - Gateways & Chat` (8) and `AI - Studios` (6) expanded; then
`AI - Inference` (7), `AI - Eval & Retrieval` (4), `AI - Speech (TTS)` (4),
`AI - Audio Tools` (2), `AI - Dormant` (6) all collapsed.
- **Toolchain** — `DNS & Filtering` (3), `Reverse Proxies` (2),
`Compose Consoles` (5), `Toolchain` (3), `Agents (no UI)` (6, collapsed).
- **Infrastructure** — unchanged; every card there is already a console.
Measured after: every group `card=360`, and `DNS & Filtering` and
`Reverse Proxies` both `h=134..134` — dead flush.
## ⚠ The move that made it affordable
**The sixteen GPU-backed model seats were NOT relabelled.** `homepage.group` is
read at container **creation**, so renaming `AI - Inference` to something
clearer would have meant recreating six vLLM seats plus four eval seats plus
four TTS engines — multi-minute model reloads on endpoints peers reach through
the gateway. Order plus `initiallyCollapsed` buys the same separation for free,
so the names stay ugly on purpose. **Do not spend that recreate on a label.**
28 containers *were* relabelled — all cheap web services — via five rerunnable
elway playbooks, `playbooks/homepage-regroup-<host>.yaml`. The canonical
`stacks/` tree was synced to match afterwards, so intent and reality agree.
`initiallyCollapsed: true` is a per-group key in `layout:`; confirmed present in
this build (`defaultOpen: !(group?.initiallyCollapsed ?? global)` in
`/app/.next/server/pages/index.js`).
## AdGuard (ANA) gained its widget, and the credential is fleet-wide
It was the only AdGuard without a query/blocked/latency strip, so it sat short
beside two tall siblings. **One `infra-ops` AdGuard login authenticates against
all three instances** (ANA `:8053`, NH3 `:8080`, ESH `:8080` — all returned 200
on `POST /control/login`, verified 2026-08-24). Vaulted at
`secret get nh3-dev/adguard-infra-ops-password`; written to
`/opt/docker/compose/adguard-ana/.env` (0600, root) and never into git. Its icon
was also the odd one out (`mdi-dns` against two `si-adguard`).
## ⚠⚠ `initialSettings":{}` — the tab-bar mystery is a SWALLOWED EXCEPTION
The biggest durable finding of the day, and it cost ~25 minutes. Full write-up
in `stacks/homepage/README.md`; the short version:
`initialSettings":{}` in the served HTML is **the catch branch** of the page's
data loader, not a warm-up and not a cache. And the error can vanish without
trace: the logger is assigned as the first statement *inside* the same `try`,
and the `catch` only logs `if (logger)`. If the logger is what threw, nothing is
written anywhere — which is exactly what was observed.
Ruled out by measurement, do not re-run: `/api/services`, `/api/bookmarks`,
`/api/widgets` and `/api/hash` all return **200 with correct content** while the
page serves `{}`; restoring the previous known-good `settings.yaml` reproduces
it identically; `/api/validate` returns `[]`; disk and permissions are fine.
**One-command test:**
`curl -s http://10.0.50.45:5100/ | grep -o 'initialSettings":[^,]\{0,20\}'`
**What broke the streak:** three consecutive recreates came up empty, then
rolling the 8.6 MB `conf/homepage/logs/homepage.log` aside and recreating healed
it within 15 seconds. That is one observation, not proof — but it is a coherent
mechanism (oversized log → logger init throws → silent catch) and it is the
cheapest thing to try first next time.
---
# `[2026-08-24, pass 3]` Rebuilt on Australis Skyfall — dual theme, light shipped
Operator supplied the Skyfall design-system README and said "Go full with
skyfall."
## The bundle was already in this repo's git history
**The Skyfall tokens did not need to be hunted down.** A predecessor vendored
them on 2026-08-19 and a later commit deleted them; git kept everything:
```bash
git show 45c1995:stacks/homepage/theme/colors.css # 208 lines, BOTH themes
git show 45c1995:stacks/homepage/theme/layout.css # calm-depth tokens
git show 45c1995:stacks/homepage/theme/typography.css
git show 45c1995:stacks/homepage/theme/fonts/Supreme-{400,500,700}.woff2
```
`colors.css` carries `:root` (dark) **and** `[data-theme="light"]` (Skyfall
Day) — so the light ramp is canonical, not derived. That killed the entire
objection from the previous answer, which was correct only about the
`australis-design` skill ("Always dark first. No light mode in this system").
**Skyfall is the dual-theme derivative; australis-design is the terminal
theme. They are different systems and only one of them has a light mode.**
## ⚠⚠ REMOVING `theme:` FROM settings.yaml BREAKS THE DASHBOARD
The documented way to get Homepage's own light/dark toggle is to leave `theme:`
unpinned. **Do not.** With the key absent, the page's data loader throws and its
catch branch serves `initialSettings: {}` — no tab bar, no layout, no i18n.
Measured, not inferred: six force-recreates over seven minutes all came up
empty with the key removed; restoring `theme: dark` rendered correctly on the
next recreate in **12 seconds**. `/api/services` stays 200 and fully correct
throughout, which is exactly why this reads as a caching or warm-up problem and
is not one.
This is the first *confirmed* trigger for the long-running "tab bar goes
missing" mystery. It does not explain every occurrence (the symptom has
appeared with `theme:` present), but it means **the first diagnostic step is
now `git log -p -- stacks/homepage/conf/settings.yaml`**, not container
archaeology. Also retires an earlier lead from this same session: rolling the
8.6 MB `homepage.log` aside once coincided with a recovery, but did nothing
during the `theme:`-key episode — coincidence, not cause.
## So the toggle is ours
`conf/custom.js` renders it (was an empty placeholder). Precedence:
1. explicit choice — `localStorage['skyfall-theme']`, written by the toggle;
2. OS preference — `@media (prefers-color-scheme: light)`;
3. dark — Skyfall's default.
`theme/build.py` re-emits each vendored `[data-theme="light"]` block twice: as
`[data-theme="light"], html.light`, and inside the media query scoped to
`html:not([data-theme="dark"]):not([data-theme="light"])`. **That `:not()` pair
is what lets a stored *dark* choice survive a light-mode OS.** Verified across
both OS preferences: load, click, click again, reload — all four correct.
⚠ Homepage keeps its own `class="dark scheme-dark theme-slate"` on `<html>`
regardless, because `theme:` is pinned. That is fine and was checked
explicitly: with the dark class present AND `data-theme="light"`, every themed
surface resolves to Skyfall Day, because our rules carry `!important` on the
surfaces Tailwind's `dark:` variants would otherwise claim. **`data-theme` is
the control surface; the class is not.**
## The anti-fork guard is now mechanical
`build.py` records the SHA-256 of each vendored file and **fails the build** on
a mismatch, rather than warning. A vendored file is either byte-identical to
the bundle or it is a fork wearing the bundle's name. Overrides go in
`skyfall.css.in`, which is written entirely against the semantic layer
(`--surface-*`, `--text-*`, `--border-*`, `--success/--danger/--warning`) — no
raw family tokens, no colour literals.
The one place a literal is unavoidable: Homepage consumes
`--color-logo-start/stop` as `rgb(var(--x))`, which cannot take an `oklch()`.
Those four values are exact sRGB conversions of real tokens (`--sea-80`,
`--blue-base` for dark; `--sea-40`, `--blue-deep` for light), computed rather
than eyeballed, with the conversion recorded in the file.
## Deviations, all deliberate and all written down
- **The aurora ribbon under the tab bar is gone.** Skyfall sanctions exactly two
accent expressions — the active rail and hero-only glows — and a decorative
gradient across the chrome is neither. The colour moved to a 2px accent bar
plus `--accent-soft` fill on the active tab, which *is* the rail.
- **Widget stat values moved from the display face to mono**, per Skyfall's
"numbers and telemetry are always `--font-mono`".
- **Two font substitutions**: Space Grotesk for Bespoke Sans, JetBrains Mono
for Victor Mono. Only Supreme was ever vendored, and Skyfall's own notes call
Victor Mono "user-supplied". Two-line swap when the real faces arrive.
Dark + light, all four tabs: `http://10.100.10.50:8090/b/homepage-skyfall/`
persistent-memory.d/2026-08-24-scriberr-ana-ml2.md
+46
View File
@@ -0,0 +1,46 @@
# [2026-08-24] Scriberr transcription deployed on ana-ml2, GPU1
Self-hosted audio/video transcription + diarization. Operator chose GPU
placement over ana-docker (8 cores shared with 50 containers, 37 GB disk)
against ana-ml2's 96 cores, `/tank`'s terabytes and GPU1's headroom.
**Live:** `http://scriberr.ana.internal:8080` (DNS alias added), health `healthy`,
all seven backends up, zero failures: `whisperx pyannote sortformer parakeet
canary voxtral openai`. ~30 GB of weights on `/tank`.
Stack: `stacks/scriberr/`. Full gotcha list in auto-memory
`reference_scriberr_ana_ml2`.
## Three upstream bugs, none of them ours
**1. The Blackwell image does not exist.** Upstream's README documents
`scriberr-cuda-blackwell`; GHCR has **no tags for it**. Published
`scriberr-cuda` covers sm_61–sm_89 only — on these sm_120 cards it fails or
silently drops to CPU. The real sm_120 path is `Dockerfile.cuda.12.9`
(CUDA 12.9.1, cu128 torch), **built from source**. Do not "simplify" the compose
back to the published image.
**2. It must run as uid 10001, not 1000** — and the error lies:
`unable to open database file: out of memory (14)`. Error 14 is
`SQLITE_CANTOPEN`, not an OOM, on a box with 566 GB RAM. That Dockerfile creates
`appuser` at 10001 (Ubuntu 24.04 owns uid 1000 as `ubuntu`) and chowns `/app` to
it, while the entrypoint's PUID remap covers only the data dirs.
**Isolated by elimination**: SQLite writes fine to `/tank` as 1000 → not the
mount; fails on a plain named volume too → not the storage; the **published CPU
image works at PUID=1000** because there `appuser` *is* 1000.
Generalisable: *when a container "permission" bug appears, compare the uid the
image was BUILT for against the uid you are RUNNING as.*
**3. `UV_LINK_MODE=copy` is required.** Scriberr builds each backend's Python env
with `uv` at start; uv's reflink mode fails on overlayfs+ZFS with
`Failed to clone … Resource temporarily unavailable (os error 11)`. **Partial
failure** — WhisperX and PyAnnote came up and the app looked fine while Parakeet
and Sortformer were silently absent. Occurrences 2 → 0 after the fix.
## Related
`speaches` on irv-ml1 **stopped** the same day (stack retained, one command to
restart): Eyra was abandoned pre-implementation because Scriberr covers the need,
leaving it with no consumer. Scriberr runs its **own** WhisperX in-container and
is **not** a speaches consumer. Idle footprint at stop was 274 MiB, not the
~5.9 GB quoted — that figure is the loaded-model working set.
persistent-memory.d/2026-08-25-erp-tune-run2-complete.md
+92
View File
@@ -0,0 +1,92 @@
# ERP/RP tune run-01 COMPLETE — 7.36h, gate passed on the axis it was built for
`[2026-08-25]`
## The run
1312/1312 in 7:21:52 train_loss 2.793 epoch 1.0
20.1 s/it FLAT across every 100-step window (round 1: 35-46.5 s/it)
adapter: /tank/erp-tune/run-01/adapter/ 410 tensors, provenance.json
**47% faster than the round-1 projection of 13.85h**, from two changes: the
bucketed sampler and flex attention. Rate was flat — 19.7 / 19.8 / 20.4 / 20.3
across the four 100-step windows — which means the 35-46.5 spread in round 1 was
*entirely padding*, and removing padding removed the variance rather than just
the mean.
⚠ **I quoted three different ETAs (6.9h, 8h, 7.3h) before I started using a
rolling average.** The first two were instantaneous tqdm readings off a number
that swings 17-25 s/it with batch width. Only the rolling rate was honest. Same
measure-don't-sample discipline I wrote into the throughput playbook, violated on
the one metric I kept reporting.
## lora_B gate — PASSED, twice
checkpoint-100 205/205 non-zero, median norm 0.829
final adapter 205/205 non-zero, median norm 1.708
vision_tower tensors: 0 on both
Median norm rising 0.829 -> 1.708 means it kept learning through the whole run
rather than saturating early. This check **never ran in round 1** (died at step
19, first checkpoint was 100) and it is the only failure mode that stays
invisible until the acceptance gate reports base-identical numbers.
## The gate — brokkr-smithy-dev
**It did the thing it was built to do:**
metric base A/B tuned delta floor
attractor hit rate 94.8% / 96.8% 84.5% -11.3pt 2.0pt
diversity (pairwise) 0.213 / 0.221 0.3948 +0.178 0.008
Diversity moved **22x its own noise floor**. Attractor rate (how often the model
reaches for the same names and phrasings) fell 11 points against a 2-point floor.
T1 100 · T2 95 · T3 96-97 · T4 98 · T5 100 · T6 81-82 · core ~94.2
memorisation: 0.0000 on BOTH arms, all three corpora
**Zero memorisation closes the R20 licensed-prose exposure on measurement rather
than argument.**
⚠ **Caveat brokkr volunteered rather than buried:** the tuned arm lost 18 of 192
generations to truncation/degeneracy against base's 1-2. Lopsided exclusions
plausibly flatter the diversity magnitude. Direction is unambiguous at 22x floor;
the number carries an asterisk.
## The noise-floor near-miss — the methodology lesson
brokkr was one step from reporting a 13-point T6 regression **that sat inside
twice his instrument's own variance.**
--per-type 32 max swing across tasks: 9 points
--per-type 128 max swing across tasks: 1 point
His gate criterion is "no task regresses by more than one item" = 3.1 points at
n=32. **The instrument's own run-to-run noise was 3 items.** He was scoring a
preregistered gate at 4x finer resolution than it could resolve, and caught it by
running a control he did not strictly need. Quadrupling n collapsed the noise
exactly as binomial statistics predicts.
⚠ **Root cause of the noise is a property of the SEAT:** `max-num-seqs` is unset,
so with a 218,625-token KV cache the scheduler batches freely up to vLLM's
default of 256. Continuous batching changes reduction order and borderline items
flip. Temperature 0 buys deterministic *sampling*, not deterministic
*arithmetic*. He declined a `--max-num-seqs 1` determinism control for the right
reason: a floor measured on a seat serving one request at a time is not the floor
that applies to the seat we ship.
## The confound I built and he caught
I optimised a pipeline for production and then handed him its output as an eval
instrument **without asking whether those were the same job.** The tuned arm
would have reached the seat as NVFP4A16 while his base arm was bf16 — any
regression would have been tuning-damage OR quantization-damage with no way to
separate them, and the gate's whole question is "did the tune cost us
capability."
**Both arms now bf16, same seat, same port, argv differing in exactly two
lines** (weights path, served name), template sha256 identical
(`ae53464bf3be2580`), KV cache identical to the digit (218,625 tokens across all
three launches). Quantization moved *downstream* of the gate.
See [[2026-08-25-refusal-retention-probe]] for the axis his gate did not have.
persistent-memory.d/2026-08-25-mfu-root-caused-attention.md
+86
View File
@@ -0,0 +1,86 @@
# The 8.6% MFU was an accounting artifact — attention on Ampere kernels
`[2026-08-25]`
## The answer
**Real utilisation was 17-20%, inside the honest stock band.** The 8.6% divided
the *intended* (windowed) FLOPs by the wall time the *dense* reality took.
nominal billed 27.1 TFLOPS x 34.85 s = 9.4e14 FLOP
dense-sliding extra 25 layers, 2 seqs, 4 passes = +8.2e14
padded full layers lose the causal skip = +3.5e14
work performed ~ 1.8e15 = 51-61 TFLOPS
The card was doing ~2x the arithmetic the architecture specifies, and the excess
was the sliding window being computed and thrown away.
## Two independent methods agreed
scaling fit (3 points, 2 params, residuals <3ms over 8x range)
A = 6.87e-4 s/token B = 8.85e-8 s/token^2
quadratic share: 20.9% @ w=2048 -> 67.8% @ w=16384
kernel table (device rows only)
attention 22,835.8 ms 65.2% fmha_cutlass*_sm80
dense GEMM 2,774.0 ms 7.9%
other 5,739.0 ms 16.4%
**67.8% vs 65.2% — 2.6 points apart, no shared assumptions.** The two-term fit
needed no constant term, which refutes launch-bound outright (3,840 expert-GEMM
launches per forward are not the cost).
## The mechanism, source-verified by brokkr's panel (arm: Bil)
masking_utils.py:292-301 _ignore_causal_mask_sdpa requires
kv_length < local_attention_size. 16384 >= 1024,
so THE SLIDING MASK ALWAYS MATERIALISES.
sdp_utils_cpp.h:259-267 flash rejects ANY explicit mask
sdp_utils.cpp:647 cuDNN head_dim capped at 128 -> unreachable
Context.h:480-485 prefer-cuDNN needs major 9 or 10; sm_120 is 12
⚠ **The kernels are `sm80` — Ampere-generation CUTLASS on a Blackwell card**,
with the forward on `gmem`, the memory-efficient backend's slowest fallback tier.
## What actually fixed it
**Bucketing (bucket-to-pair, shuffle-to-mix)** — 29.9% padding -> 0.0%, and
78.3% of micro-batches become exactly zero-pad, which puts the 5 global layers
back on `is_causal`. Measured: padding costs **9.4% MORE time for 24% LESS work**
at fixed width, because an explicit mask knocks those layers off the fast path.
⚠ **Bucket size is NOT a diversity knob.** Swept across a 256x range, roots per
accumulation window stayed flat at 3.54-3.61. The global micro-batch shuffle does
all the mixing; the bucket only costs padding. Use the tightest bucket.
**flex_attention** — Triton-generated so it compiles for sm_120 instead of
shipping sm_80 binaries. 21.7x on sliding layers, 2.1x on global. Needs mandatory
`kernel_options` at 32x32 blocks: 64x32 needs 102,400 bytes against a
**101,376-byte hardware ceiling** — misses by 1 KB, and Triton is already opting
into the full 99 KB, so it is the card, not a default.
## ⚠⚠ The trap that produced TWO wrong published conclusions
`torch._dynamo` defaults to a recompile ceiling of **8**. Every distinct sequence
width is a new shape. On hitting the ceiling dynamo does not error — it silently
falls back to UNCOMPILED flex, which is ~20x slower AND documented to *"not work
with the backwards pass and may produce incorrect results."*
That artifact produced a bogus **0.76x slowdown** and a bogus **2.9% loss
divergence**, and I believed and reported both. Raising the limit to 256 flipped
the speed result to 1.41x.
The loss divergence turned out to be real but benign — adjudicated against fp32
MATH ground truth, both backends sit ~2e-3 from truth with flex fractionally
CLOSER at every width. **Do not re-open it by comparing the two backends to each
other; that cannot answer it. Compare to fp32.**
## Process lesson
brokkr's panel produced **four self-retractions in ninety minutes**. Every
retraction was a derivation; every survivor was a measurement. And the whole
head_dim-512 SDP problem was **already documented in zerofata's published Axolotl
config since April** — the right first stop for "why is this architecture slow"
is practitioner configs for that exact base, before any panel.
Playbook: `docs/pfi/training-throughput-playbook.md`, commit `7b5fd91`.
persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md
+84
View File
@@ -0,0 +1,84 @@
# NVFP4A16 serving pipeline — built, validated, and the MoE landmine it found
`[2026-08-25]`
Pipeline at `scripts/erp-tune-serve/` (commits `6a85829`, `ab980e9`).
Validated end-to-end against checkpoint-100 before the real adapter existed.
## ⚠⚠ THE LANDMINE: a `targets=["Linear"]` recipe misses EVERY MoE expert
before linearize_moe: 427 Linears, 205 targeted, experts 0
after linearize_moe: 11,947 Linears, 11,725 targeted, experts 11,520
(30 layers x 128 experts x 3 projections)
Gemma-4 stores each layer's 128 experts as two fused 3-D `nn.Parameter` tensors
(`gate_up_proj` [128,1408,2816], `down_proj` [128,2816,704]) — note the absent
`.weight` suffix. A Linear-targeting recipe resolves 205 of 427 modules and
**zero experts**, leaving 22.84 B params (88.5% of the model) in BF16 with no
warning.
**This is the same defect that killed QLoRA here via bitsandbytes.** The blind
spot is in the *checkpoint layout*, not the tool. Fix:
`llmcompressor.modeling.moe.linearize.linearize_moe` — no registration needed,
Gemma-4 satisfies `FusedExpertsProtocol` structurally. Playbook §3.15.
## Scheme: NVFP4A16, deviating from the playbook default, on measured grounds
brokkr benched the W4A4 quant of this checkpoint at **12% on contradiction
detection with CoT off against gen's 81%** — the signature of 4-bit input
activations on a reasoning-dense task. Plus W4A4 KLD is 2-4x worse past ~10k ctx
on sm_120. This is a 16,384-ctx RP seat. Marlin's prefill cost accepted.
⚠ Several HF repos named `…-NVFP4A16` declare `input_activations num_bits 4` —
W4A4 wearing an A16 label. The script refuses if the emitted config says 4.
## Four silent defects the dry run found
1. **transformers 5.15 MIGRATES the config schema on save** — drops
`global_head_dim`/`num_global_key_value_heads`, writes `per_layer_config`.
transformers 5.10 (the llmcompressor venv) then reads `num_key_value_heads`
as None and dies with `TypeError: unsupported operand type(s) for //`.
Every working artifact on the box uses the OLD schema. Merge now downgrades it.
2. **llmcompressor cannot auto-init a processor for a multimodal checkpoint** —
pass the tokenizer explicitly as `processor`.
3. **`save_pretrained` does not carry `processor_config.json`** — vLLM then fails
with "Can't load feature extractor", which reads as a vision bug.
4. **The quant needs more than GPU1's free 32 GiB.** `quant_with_gen_down.sh`
stops `vllm-gen` and restores it from a trap on EVERY exit path, using
`docker start` not `compose up` so the container returns with its exact config.
## Verified on the emitted artifact
49 GB -> 17 GB, format nvfp4-pack-quantized, a=null (genuine A16)
weight_packed 11,725 of which expert 11,520
tokenizer truncation: clean (§3.14 trap avoided by calibrating on the
encode cache, so the tokenizer is never called
with truncation=True at all)
served: Marlin NVFP4 kernel + Marlin MoE backend, coherent generation
⚠ The reference `nvfp4a16` artifact triggers a vLLM warning that q/k/v carry
*different* weight global scales ("likely reduced accuracy"). **Ours does not** —
llmcompressor 0.12 links weight observers across fused groups automatically. The
in-house quant is better than the downloaded one on that axis.
## ⚠ MERGED WEIGHTS ARE MANDATORY — and not for the reason we assumed
The open question was whether LoRA-on-NVFP4 hot-swap still silently no-ops.
Retested on `vllm/vllm-openai:latest`: **it refuses to start.**
AttributeError: To support LoRA for MoE model,
'get_expert_mapping' must be implemented
The check is in `vllm/lora/utils.py::process_packed_modules_mapping` and branches
on `is_moe_model()` — **quantization is not in the condition.** `gemma4.py`,
`gemma4_mm.py`, `gemma4_mtp.py`, `gemma4_unified.py` all have ZERO occurrences;
`deepseek_v2`, `mixtral`, `glm4_moe`, `ernie45_moe` implement it.
**vLLM cannot serve a LoRA on ANY Gemma-4, bf16 or quantized.** Merging is the
only path for this architecture, and it would have bitten identically on the
unquantized base. A loud refusal is strictly better than the 0.24.0 silent no-op,
which shipped a base model wearing the tune's name.
⚠ Base-viability pre-flight is now playbook §3.11 — three greps before picking a
base. **Grep the CLASS, not the file**: `mistral.py` greps as `SupportsLoRA=0`
and is fully LoRA-capable via inheritance from `LlamaForCausalLM`.
persistent-memory.d/2026-08-25-refusal-retention-probe.md
+64
View File
@@ -0,0 +1,64 @@
# Refusal retention — the axis the gate did not have, and the axis I measured wrong
`[2026-08-25]`
## Why it exists
brokkr's gate measures reasoning (T1-T6), craft (diversity/attractor) and
regurgitation (memorisation). **Nothing measured whether the model still
COMPLIES** — which for this seat is arguably the most important property.
The risk is specific to our operation order. We do **tune(abliterate(stock))**,
so the tune has 57.7M tokens of opportunity to walk the abliteration back. *A
tune that gains 41 items of contradiction detection and quietly re-installs
refusals is a failed seat that passes the entire gate.*
## The measurement — controlled, single instrument, both arms
arm HARD DEFLECT COMPLY
base 0/100 0 100
tuned 29/100 0 71
Same seat, same probe, temp 0, `mlabonne/harmful_behaviors` x100.
Probe: `scripts/training-probes/refusal_probe.py`.
**The tune added 29 general-harm refusals where the base had none.**
Two things fell out:
- **The instrument validates.** Base measured 0/100 on my generated-text regex
against Heretic's recorded 3/100 from a first-token-probability scorer. 0 vs 3
is agreement — the incomparability worry was right caution about a non-problem.
- **DEFLECT is 0 on BOTH arms, so the free control fires.** An instrument
artifact does not care which arm it runs against. Both zero means the model is
**binary** — refuses in refusal-language or engages, no soft-deflection tail.
The R19 undercount does not apply here.
## ⚠⚠ But it is the WRONG AXIS — brokkr's catch, and it is the better one
`mlabonne/harmful_behaviors` is **general harm** — weapons, malware, fraud. **The
abliteration was not run so the model would explain bomb-making. It was run so
the model would engage with explicit fiction.** Different refusal surfaces; a
model moves on them independently.
I picked that set because it was cached, had a recorded baseline, and was what
the abliteration tool used. **Every one of those is a reason it was convenient,
not a reason it was right** — and "it has a baseline" was actively misleading,
because a comparable number for a question nobody is asking looks like evidence.
**29/100 general-harm refusals on a seat writing prose the operator was actively
praising is plausibly the DESIRED shape**, not a defect. General-harm refusals
returning while domain compliance holds is close to ideal for an internal
creative seat. I would have reported it as damage.
**The load-bearing cell is COMPLY 71, not the 29.** Stock refused 100/100;
anything near that would mean the abliteration was undone. 71 complying means
"partially walked back on one axis" — a different finding, and only one of the
two threatens the seat.
Domain-compliance probe (the right axis, from R19's track-2 map) is brokkr's,
pending. Scaffold supplied: `scripts/training-probes/counted_classifier.py`
(`2a05ae9`) — classify-never-surface, three-way, ERROR path deliberately does not
log the exception body because an exception can echo the prompt back.
Playbook §3.13. See [[2026-08-25-erp-tune-run2-complete]].
persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md
+123
View File
@@ -0,0 +1,123 @@
# Worldtree b188 + b189 bridge cutover, and the selene metadata that lied
`[2026-08-25]`
Three arcs in one day, all infra-ops side, all landed.
## b188 — matrix.yaml pre-sync (#406/#409/#410 closed)
From b188 the bridge reads per-agent `rendering` + `ambient_buffer_size` from
`config/matrix.yaml` ONLY; agent `config.yaml` matrix blocks are gone from the
image. Staged as `6417115` in `worldtree-instance-configs`, deployed to both
instances with operator approval.
- mimir gets thinking-to-thread + tool-call reactions + 7 reaction labels;
forseti and lofn stay quiet.
- **Rider #409 pruned six dead agents** (bragi, leif, troi, soong, cara, glados)
from BOTH rosters — originally scoped personal-only, which I flagged as a
possible oversight and it was. **The settling fact worth keeping: the engine
roster comes from the image's baked `agents/` directory (only `config/` is
bind-mounted), and both instances run the same image**, so instance-level
evidence about which agents the engine lists generalises by construction.
- Both rosters now exactly `[mimir, forseti, lofn]` — the three engine agents
actually bridged. mask/vili/echo exist in the engine, deliberately unbridged
(operator ruling).
⚠ **Edited text-surgically, not via a yaml round-trip** — PyYAML would reflow
1,249 lines and drop every comment, and the comments are the documentation.
⚠ **`deploy-wt-config` uses `docker restart`, NOT `compose up`.** A `compose up`
on corviduo-dev re-resolves the image tag and can silently swap the running
build — which would turn an "inert pre-sync" into an unintended image roll on two
live instances. That property is easy to lose in a future refactor of the script.
## b189 — #407 bridge extracted to its own repo (#404 umbrella closed)
Bridge now `gitea.phasefinal.com/pfi/wt-matrix-bridge`, its own repo, own CI.
⚠ **It publishes to the `pfi` ORG, not `vh`, and the reason is structural:** `vh`
is a **USER**, not an org. Gitea scopes user-namespace packages to the owning
user — there are no package collaborators on a user namespace. **No service
account can ever publish to `gitea.phasefinal.com/vh/*`.** claude-bot is an Owner
of `pfi`, so that is where it goes. Token `wt-matrix-bridge-ci` (id 28, scopes
`write:package,read:repository`), vaulted at
`nh3-dev/.config/claude-bot/gitea-token-wt-matrix-bridge-ci`.
**Minted a dedicated token rather than reuse `claude-bot-sdk-ops` or `arbo-ci`,
both of which already carry `write:package`** — a shared credential cannot be
revoked without collateral. ⚠ A first mint attempt succeeded then failed to save;
Gitea returns a token value exactly once, so it was unrecoverable. Deleted the
orphan (id 27) rather than leave a live package-write credential on the account.
**Both instances PINNED** to `f3f8ec902267` (`e90f436`), closing the #410 shape:
`WORLDTREE_IMAGE` was pinned and the bridge was the one drifting service.
⚠ The pin moved from `b178285b1cb5` because a cross-frontier bug-hunt found the
M_EXCLUSIVE **fallback could itself litter unboundedly** in exactly the state it
was written for. **The safety net had the same failure mode as the thing it was
catching** — and my staged window leaned on that fallback.
## #411 — the debug-room failure, diagnosed twice and wrong both times first
My theory: the alias was held by orphaned rooms. **Refuted by a directory probe
returning 404.** The real cause: Synapse's **M_EXCLUSIVE** — an appservice may
only create aliases inside a namespace it has RESERVED, and
`aipa_appservice.yaml` had `namespaces.aliases: []`.
⚠ **I inferred a cause from a symptom that was the RESPONSE to the cause** — the
log's "re-resolving alias" line is the recovery path firing, not evidence the
alias exists. One directory probe settled it and I reasoned instead.
Fixed with operator clearance: added
`regex: '#aipa-debug-[a-z0-9_-]*:matrix\.phasefinal\.com'` (exclusive) at
`/opt/docker/conf/synapse/aipa_appservice.yaml` on **ana-docker** (NOT
`/opt/docker/data/`, which worldtree-dev's issue cited). **My regex was tighter
than the proposed `#aipa-debug-.*`**, which fullmatches only because `.` also
matches the `:` separator and would equally claim other homeservers.
Pre-apply sweep (an `exclusive: true` claim can make Synapse refuse to START):
aliases matching '%aipa-debug%' 0
total room_aliases on the homeserver 1 <- why it went unnoticed this long
rooms created by @aipa-debug 17 <- the litter, confirmed unaliased
Synapse healthy in 40s, both bridges rode through. **The aliased create then
worked FIRST TRY on the next personal recreate** — designed path, fallback never
fired, both reuse mechanisms live (canonical-alias rediscovery AND the room-id
cache).
⚠ The #411 writer census **inverted its own premise**: the api (uid 1000) cannot
write `/app/sessions`, the bridge (root) can — the reverse of the issue text. But
worldtree-dev's reconciliation is better than "backwards": pre-#407 the bridge
ran from the ENGINE image as uid 1000, and the cutover changed the answer
underneath the issue. Both readings were true at their timestamps. **No live
writer exists for that path anyway** (`sessions.path` is a legacy default, #330
moved the tools off it), so the chown is optional future-proofing.
## selene-1-mini-8b — a config that lied about what answers
forseti's fleet sweep found the only genuine residual in
`worldtree-instance-configs`, and it was **live on both boxes, not just drifted
in git.** Fixed in `a77639d`.
Routing was never broken (`model: "chat-judge"` stays), but `display_name` said
"Selene 1 Mini 8B" and the description said "Atla Selene 1 Mini 8B — reward model
derived from Llama 3.1 8B" while chat-judge has been backed by
**qwen3.8-27b-uncensored** since 2026-08-23.
⚠ **It misstated the KIND of model, not just the identity.** A reward model and a
generative judge are different instruments; a consumer reading
"pairwise/likert/binary/scalar reward model" would expect scalar-reward semantics
qwen3.8-27b does not provide.
Verified: `selene-1-mini-8b` -> HTTP 400 (by design), `chat-judge` -> HTTP 200.
**NOT changed:** the catalog key and the `selene-judgment` role — `model_roles.yaml`
binds to that key, so renaming is worldtree-dev's schema call. Flagged that a role
named after a retired model defeats the purpose of role aliases.
⚠ forseti's sharpest finding is for the operator: **`~/.claude/CLAUDE.md` line 502
lists the retired name in the global tools roster.** A broadcast reaches sessions
that already exist; the roster line keeps minting new ones. His file, awaiting his
word. Also: **there is no fan-out primitive on the bus** — a fleet relay is 73
individual posts, recorded as a real gap rather than papered over.
persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md
+114
View File
@@ -0,0 +1,114 @@
# `[2026-08-26]` ERP run 2 — complete, merged, coherence-gated, and serving as `erp-tune-v2`
Run 2 is the SFT on the **official instruct** base. It trained 1312/1312 in
**7:22:44**, `train_loss` **2.839**, and is serving bf16 on ana-ml2 GPU0.
endpoint http://10.250.50.54:8098/v1 model erp-tune-v2
container erp-eval-v2 (restart unless-stopped, 84,272 MiB)
weights /tank/erp-tune/serve/merged-run02 (bf16 merged, 51.6 GB)
adapter /tank/erp-tune/run-02/adapter/
harness eitri-smithy 1909d86 at launch (see the provenance caveat below)
## The two operator decisions that produced it
Both taken 2026-08-25 evening, both my recommendations:
1. **Base = stock `google/gemma-4-26B-A4B-it`**, not an abliteration. My prior
session had ranked this option 3 of 4 on attributability grounds; that
objection did not survive contact, because the mask and the hygiene fixes
land under *every* option and therefore do not discriminate between bases.
The real trade is scope: **run 2 stops being the deliverable and becomes
stage 1 of 2**, because a stock-instruct SFT refuses at near-stock rates and
needs a targeted DPO stage after it. The operator accepted that to replace
"inherit a stranger's judgement about which refusals to strip" with "strip
the ones we choose."
2. **Stop `erp-eval` to free GPU0.** There was no free card — GPU0 held run 1's
seat, GPU1 held six shared fleet seats. Decided on the evidence that the seat
had served **one request in 24 hours** and its merged weights are on disk, so
it re-serves in minutes.
## Gates — all passed
lora_B FINAL 205/205 non-zero min 0.6826 median 1.7212 max 3.7573
checkpoint-100 min was 0.1977 -> growth, not a plateau
vision_tower 0 tensors
merge 410 adapter tensors applied; 1/1 sampled target changed
chat template ae53464bf3be2580 — upstream 390 lines, shipped with the merge
coherence 5/5 prompts, all distinct, no looping, no marker leakage
step timing n 1,447 p50 19.79 p99 30.54 max 37.40 (run 1: p50 19.75)
## ⚠ What the p50 match does and does NOT show
Tempting and wrong: reading run 1's 19.75 ≈ run 2's 19.79 as evidence the
impersonation mask applied. **Masking changes which positions contribute to the
loss reduction — not tensor shapes, not either pass.** Step time is therefore
insensitive to whether the mask applied, so a silent fallback to unmasked
encodings produces the same p50. The check cannot go red on that axis.
(brokkr-smithy-dev caught this after I had asserted it; withdrawn.)
**The evidence the mask applied is the loss-token delta**: 45,899,397 →
45,677,685, −221,712, against **byte-identical context tokens**, unchanged
record count, and the same nine unfittable drops. It could have returned zero.
Cross-checks to 1.193% of dialogue loss tokens against brokkr's independently
derived 1.142% of bot words — two measurements from opposite ends of the
pipeline agreeing within 4%.
What the p50 match *does* show: step timing was stable across a base swap.
## ⚠ `provenance.json` records `harness_commit 460f372` — AHEAD of the code that ran
The run launched when HEAD was `1909d86`. Three commits landed on the same
checkout during the seven hours it trained, and `_git_commit()` was called at
save time, so it read HEAD seven hours late. The recorded commit names
`step_seconds`, the resolved-backend fields and the worker template assertion —
**none of which were in the running process.**
Exact mirror of run 1, whose commit was BEHIND its code because the tree was
dirty. Same defect, opposite sign. Fixed from run 3 by `9d27b4f`.
Annotated on the artifact at `run-02/adapter/PROVENANCE-NOTE.md`, never edited
into `provenance.json`. Attention backend supplied there as reconstructed
(requested + resolved both `flex_attention`, confirmed by a config-only probe on
the same library versions), with the step-timing distribution in
`step-timing-reconstructed.json`.
## Harness work this run — all local, UNPUSHED
5349ef0 impersonation loss-mask, optimiser schedule, green suite (10/4 -> 34)
1909d86 warmup_ratio is not a transformers 5 kwarg; check names before the call
908aadb record the attention backend the run RESOLVED to
a80a5a7 record the step-time distribution beside the resolved backend
460f372 pair chat_template_sha256 with the sha of what actually RENDERED
9d27b4f sample the harness identity at LAUNCH, not at save time
242 tests passing, up from a suite that was **10 passed / 4 failed with no
pytest installed at all** — nothing had ever run it.
The through-line, and it is the reason for playbook §4: **three of these six
commits fix a check that could not fail.** The encode cache key omitted the
mask; `attn_implementation_resolved` cannot detect a Dynamo fallback;
`chat_template_sha256` records a file rather than what rendered. Each was found
*after* shipping the previous one, and 4.5's inert-gate reading is what makes
them one defect rather than three.
## Open, with surfaces
- **Base arm cannot be concurrent.** Two bf16 26B arms = 98 GB of weights on a
97.9 GB card; GPU1 has ~30 GB free under six shared seats. brokkr wants both
arms in one window because the diversity battery mines its frozen marker list
from base. Either sequential-with-everything-pinned, or displace GPU1 seats —
an operator call. Thread `01M0WQ8W5574KMEVCHCEKEXNS5`.
- **`erp-tune-v1` still registered in the LiteLLM gateway, returning HTTP 500.**
I stopped its container and left the route. Config-file-defined
(`db_model: false`), so removal needs an edit plus a gateway reload, and a
reload briefly interrupts all fleet LLM traffic. Judged a 2am restart the
worse trade at 1 request/24h. Batch it with the v2 registration decision.
- **`erp-tune-v2` deliberately NOT in the gateway.** Served on a direct endpoint.
Adding an un-gated ERP tune to the shared gateway reaches every agent holding
the shared key — broader than "serve it" requires, and trivial once the
battery reads out.
- **DPO stage** — survey at `docs/pfi/erp-dpo-stage-prep.md`. Gating item is not
technical: **which refusal axes are in scope and which are explicitly kept.**
No preference data for refusal axes exists; the two Gutenberg DPO sets on disk
are prose-quality. `trl` is not installed.
persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md
+116
View File
@@ -0,0 +1,116 @@
# `[2026-08-26]` Run 3 corpus built, then held — and the megamix containment defect it exposed
Run 3's corpus was built to brokkr's `SFT-RECIPE-run3.md` (`f6c0589`), a blocker
was found before any GPU spend, and **brokkr is now redoing the recipe**. Nothing
launched. The build artifacts survive and are re-parameterisable.
## ⚠ THE FINDING: `creative-writing-multiturn` is a MEGAMIX containing our other roots
Its own Hoard card says so — `hoard/datasets/creative-writing-multiturn/dataset.yaml`,
`archive_provenance_note`:
> *"Dampfinchen/Creative_Writing_Multiturn — 2026 curated ~4.6k high-turn
> creative-writing/RP **megamix (bluemoon, LimaRP/PIPPA, stheno, gutenberg-dpo
> chosen, VellumK2 subset)**"*
**It is not a sibling root. It is a bag holding several of our other roots.**
Found because my corpus build would not reproduce brokkr's numbers: he computed
on the raw roots, I computed on run 1's deduped survivor list. bluemoon came out
67 against his 126. Run 1's dedup record explains it:
DROPPED -> KEPT n median jaccard
bluemoon -> creative-writing-multiturn 59 1.000
creative-writing-multiturn -> pippa 30 1.000
c2-logs-32k-llama3 -> creative-writing-multiturn 23 0.945
brokkr then measured it directly against the raw corpora, and it is worse than
the dedup threshold showed — **the MEDIAN bluemoon conversation is 87% similar to
something in creative-writing** (≥0.99: 28.3%, ≥0.90: 49.6%, max 0.999). That is
containment, not overlap.
**Why it mattered here specifically:** the remix elevates creative-writing to
43.2% and bluemoon to 31.4% of loss signal — *the two roots that overlap*.
Building on raw roots would have double-counted bluemoon inside creative-writing's
share, in a run whose entire justification is the composition change.
## Three consequences, two of them structural
**1. "bluemoon is the only human-authored root" was overstated.** It is the only
*wholly* human-authored root; creative-writing is *partly bluemoon*. brokkr
restated rather than withdrew.
**2. ⚠ F1 does not do what the recipe says.** "Excise PIPPA" removes the ROOT, not
the MATERIAL — PIPPA persists inside the megamix, which the remix promotes to
largest contributor. **F2's 250-word floor is what actually removes PIPPA
material**, because PIPPA turns cannot exceed 123 words wherever they live. The
outcome was right; the stated mechanism was wrong. **Had the operator chosen a
floor below 123, F1 would have been nearly cosmetic.**
**3. ⚠ The dedup kept the wrong side of the pair, 59 times.** When a primary
source and a megamix containing it collide, the survivor should be the primary
source, not the copy inside the bag — the megamix version may have been
reprocessed by its curator. We kept the derivative and discarded the origin.
**Reversed for run 3** (brokkr's call, ~3 minutes): keep bluemoon, drop the
megamix copy. Result is better than either prior version —
root run 2 recipe DEDUPED+REVERSED
bluemoon 6.9% 31.4% 38.6%
creative-writing 18.0% 43.2% 30.4%
c2-logs 40.2% 25.4% 31.0%
pippa 34.9% 0% 0%
**The wholly-human share went UP and the megamix share went DOWN** — strictly the
composition the recipe argued for, now honestly counted. Total context unchanged
at 12.49M, so the operator's settled mix arithmetic survived untouched.
## ⚠ The catalog-level problem, unsolved
**A megamix root silently contains other catalog roots and nothing in the mix
arithmetic knows.** Any share computed over roots including creative-writing
double-counts bluemoon, PIPPA, LimaRP and stheno by an unknown amount. Run 2's
mix had this defect and nobody noticed, because run 2 did not promote the
overlapping roots.
LimaRP and stheno are **still unchecked against anything.** Wants a
`contains_datasets:` field on the Hoard card plus a preflight assertion that no
recipe includes both a megamix and a root it contains. brokkr is raising it as
its own item; not a run-3 blocker.
## What is staged on ana-ml2
/tank/erp-tune/recipe-r3/recipe-erp-seat-sft-r3.json
/tank/erp-tune/recipe-r3/dialogue-survivors.jsonl 610 conversations
/tank/erp-tune/recipe-r3/lowquality-mask-v1.jsonl 5,867 mask rows
/tank/erp-tune/build_r3.py the filter build; floors are constants at the top
/tank/erp-tune/pippa_clip.py length-distribution measurement
/tank/erp-tune/clip_share.py turn-share vs word-share
⚠ **`/mnt/smithy` is READ-ONLY from ana-ml2**, so recipe artifacts live under
`/tank/erp-tune/recipe-r3/` rather than beside the r1 recipe. The harness takes
absolute paths, so this is fine — but do not expect to write to the r1 recipe dir.
**F2/F3/F4 are implemented as a LOSS MASK reusing the impersonation-mask format**,
so `load_impersonation_mask` reads it and `_impersonation_flags` sha-asserts every
masked turn with no new harness code. F4 is masked rather than stripped:
stripping mutates content, which invalidates the shas, and the read-only mount
forecloses a mutated derived root anyway. 28 turns of 6,209 (0.45%).
Re-parameterising for a new recipe is one command and ~90 seconds; the dedup
reversal and F4-as-mask handling are baked in and survive it.
## Decisions I took
**No conversation admission threshold** (brokkr delegated it, then endorsed).
81% of surviving bot turns are context-only, but masked context costs the
transformer body and **not** the LM head — the harness drops `IGNORE_INDEX`
before the 262k-vocab head, so the waste is bounded well below what the 81%
suggests. A 20% context trim buys ~35 minutes against a 3-hour run, and it would
be a fifth filter moving composition inside a run whose purpose is testing the
filters alone. **Buying 35 minutes by confounding a 3-hour experiment is a bad
trade.** brokkr: "your reason 1 is decisive."
**Left `erp-tune-v2` UP** on the operator's discretion grant. The card is not
needed until a recipe is settled, and it is his evaluation surface plus a live
gateway route. Take it down when run 3 is actually ready, not before.
persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md
+123
View File
@@ -0,0 +1,123 @@
# `[2026-08-27]` Anaheim tripped a power breaker — and four guests including the NAS never came back
Site-wide outage. **ana-ml2 was down 1h58m** — power lost between **18:14:45 and 18:17:00 PDT**,
boot at **20:13:14** (the "~90 minutes" first recorded here was an estimate; the window is now
pinned by run 3c's training log and the hourly cron that never ran — see
[[2026-08-27-run3c-launch-count-reconstruction]]). **Operator-confirmed cause: a tripped power breaker**, not a
fault and not the tunnel. The discriminator that established scope: `ana-srv1`
(38.120.12.44:443, Anaheim's PUBLIC address) was dark **from the internet**, so it was not the
NH3↔ANA IPsec tunnel stranding NH3 — the site was not answering on any path. `ana-ml2` returned
with `up 1 min`, confirming a hard power event.
## ⚠ THE DURABLE FINDING — `onboot` was unset on four guests
pfi-pve came back and auto-started everything **except**:
CT109 ana-nas the storage SPOF
CT113 ana-wg the WireGuard remote-access path
CT112 ana-filebot
VM106 corviduo-dev
All four had `onboot` unset. **Recovery was manual and would have been manual every time** —
including for the NAS that postgres/PBS/cross-site-restic depend on, and the WireGuard host
that is the way in when the site misbehaves.
**FIXED, with dependency ordering** (operator-authorised):
CT109 ana-nas onboot=1 order=1,up=45 <- first; 45s for NFS to SERVE
CT113 ana-wg onboot=1 order=2 <- remote access before anything can fail
VM104/105 Mongo/Postgres order=3,up=60 (pre-existing)
VM102 ANA-Docker order=4 (pre-existing)
VM101 ANA-DC order=5,up=120 (pre-existing)
CT112 ana-filebot onboot=1 order=10
VM106 corviduo-dev onboot=1 order=10
Every guest on pfi-pve now auto-starts. ana-nas precedes the databases deliberately; the
`up=45` is for NFS to be *serving*, not merely for the container to be *running* — the exact
distinction that killed `rest-server` on ana-docker, which came up before the NAS existed,
found nothing to serve, and exited 255.
## ⚠ `/tank` came back DEGRADED — a disk is genuinely gone
tank DEGRADED, raidz2-0, 7 devices ONLINE
9477159196657038377 FAULTED was /dev/nvme4n1p1
errors: No known data errors
**Only 7 physical NVMe present where the pool expects 8** — checked, so not renumbering. One
drive did not re-enumerate. raidz2 carries two disks of parity; one is spent. Operator taking
it; chassis is a Supermicro AS-4125GS-TNRT2 with PCIe hot-plug slots, so a swap should not
need a power-down.
## ⚠ `/mnt/smithy` is manual by design — it will be missing after EVERY reboot
Not in fstab, and **deliberately so**: a cross-site NFS entry can hang boot on a GPU host, and
it is `soft` rather than `hard` because ana-ml2 is cross-site from that NAS and a hard mount
turns a link blip into unkillable D-state. Remount with the recorded spec, do NOT "fix" it into
fstab:
sudo mount -t nfs4 -o ro,soft,timeo=30,retrans=3,proto=tcp,vers=4.1 \
10.100.50.50:/volume1/smithy /mnt/smithy
Full rationale: [[2026-08-23-smithy-mount-ana-ml2]].
## Power capacity is now the open item
Operator: *"we'll triage this weekend, probably shut down some seats."* ana-ml2 alone was
pulling ~600 W across both GPUs at their 300 W caps during training. `gen` stays up by
instruction; everything else on that box is idle.
MEASURED a breaker tripped (operator-confirmed)
MEASURED power lost between 18:14:45 and 18:17:00 PDT
MEASURED the WHOLE Anaheim site went dark from the internet
TOPOLOGY ONE CIRCUIT FEEDS THE ENTIRE RACK, including the firewall serving
the public IP (operator, 2026-08-27)
OPERATOR "load tripped breaker, entire rack goes dark." Consistent with every
DETERMINATION observation. Not a per-circuit measurement, and no longer needs to be —
the operator has the site knowledge and has ruled.
⚠ **SUPERSEDED — the blast-radius objection I raised here was WRONG, and it was committed.**
An earlier revision of this section (commit `3cc55b4`) argued that *"a site-wide blackout is a
larger blast radius than one GPU box's ~600 W accounts for on its own: if ana-ml2's draw were
the whole story, why did ana-nas, ana-wg and the public address go dark with it?"*
**One shared circuit inverts that completely.** Site-wide darkness is not too large a blast
radius for a single overloaded box — **it is exactly the blast radius the topology predicts.**
The fact I treated as unexplained was the signature of the mechanism.
Holding the attribution as unmeasured was still right (there is no per-circuit meter); the
*argument* used to hold it was reasoning confidently about wiring neither I nor
brokkr-smithy-dev had ever seen, and neither of us asked for it. Kept visible rather than
silently corrected so the old reasoning stops circulating.
## ⚠ THE FINDING THAT OUTLIVES THE INCIDENT — the recovery path shares the failure domain
One circuit powers, together:
ana-ml2 the load most likely to trip it
ana-gw the firewall serving the public IP (38.120.12.44)
ana-wg the WireGuard host — the way in when the site misbehaves
ana-nas the storage SPOF
the BMCs ana-ml2 IPMI 10.250.250.50, pfi-pve iDRAC 10.250.250.30 — same rack
**So the thing that fails takes every path used to fix it with it.** Compounding two facts
already in the record separately: ana-gw's WAN admin is closed, so management is tunnel-only
([[2026-08-24-ana-gw-admin-closed-acme-disabled]]) — and the tunnel terminates on ana-wg, on
this circuit. **During a rack power event there is no remote management path to Anaheim at
all**, in-band or out-of-band. Recovery is: wait for power. Until 2026-08-27 it was worse —
ana-wg had `onboot` unset and would not have come back on its own.
Open question for the weekend triage, and the cheapest fix if the answer is no: **is any of
that rack on UPS or a second circuit?** Nothing in this repo records one, and the site went
dark instantly and stayed dark 1h58m, which is behaviourally indistinguishable from no
coverage for these loads. Putting *only* ana-gw + ana-wg + one BMC on separate power would
mean a load trip no longer costs the way in.
Load-shed on ana-ml2 is still the right first move and it is cheap. It reduces how often the
breaker trips; it does not change what a trip costs.
## Blast radius beyond us
heid lost **both gateway-routed arms of a four-arm panel** mid-dispatch and discovered the
outage by losing half a panel. That report produced the single most valuable artifact of the
incident — see [[2026-08-27-empty-response-as-a-datum]].
persistent-memory.d/2026-08-27-dose-response-entanglement.md
+62
View File
@@ -0,0 +1,62 @@
# `[2026-08-27]` The dose-response says benefit and damage are ONE direction in weight space
vLLM **cannot LoRA-serve Gemma-4-26B-A4B at all** — it is an MoE and the LoRA manager has no
expert mapping (`AttributeError: To support LoRA for MoE model, 'get_expert_mapping' must be
implemented`, `vllm/lora/utils.py:398`, v0.24.0). Found by trying it: one container start,
which is exactly what playbook §3.10 exists to buy.
So each scale point was **pre-merged** instead — `merge_and_unload` with the adapter's alpha
pre-scaled, which reproduces exactly what a LoRA at scale s would serve, on the same serving
path as the gate. Artifacts held at `/tank/erp-tune/serve/merged-run03{,-s075,-s050,-s025}`.
scale attractor gain kept noise@31 damage prose@40 cost
s=0.50 0.9684 0.0% 1.0000 0.00pp 0.00pp
s=0.75 0.8966 42.0% 0.9688 3.12pp 6.25pp
s=1.00 0.8049 100.0% 0.5938 40.62pp 15.62pp
first 42% of the gain costs 3.12pp of noise coherence
last 58% of the gain costs 37.50pp more -> 12x worse per point
**Every axis is monotone in scale — attractor, distance, marker density, RP length, both
coherence measures, even truncation count. No knee anywhere.**
## The conclusion, which is stronger than an exchange rate
An exchange rate says the trade is expensive. This says **there is no trade to make**: the
adapter learned ONE direction and everything rides it. brokkr's framing, worth keeping:
**scaling moves you along the direction the adapter already learned; it cannot give you a
different one.**
That converts the merge-back from a candidate fix into a **measurement of the problem**.
zerofata's MeroMero v1 ships the merge-back as its answer, so it was worth testing — and
testing it is what proved it cannot be the answer here.
## The T4 mechanism is NOT register capture
pad type base tuned
prose 1.0000 flat mild knee, onset ~1,600 tok
noise 1.0000 flat 1.0000 -> 0.5938 <- SEVERE
The prose-gradient "wedge" predicted **prose** should be the worst case. Instead prose is the
mild axis and **shuffled tokens are where it collapses** — a continuation prior has nothing to
continue in shuffled tokens. Every miss is `wrong-name`: correct format, on task, wrong answer.
**State mis-tracking, not register capture.**
brokkr: *"The tune did not teach it to continue prose — it destroyed its ability to skip text
that is not worth reading."*
## ⚠ INSTRUMENT FLOOR — temperature 0 is NOT deterministic under concurrency
same seat, same items, temperature 0, --jobs 8
prose reps 31 0.9375 0.9688 0.9688 spread 3.12pp
noise reps 24 0.7500 0.7188 0.7188 spread 3.12pp
vLLM's continuous batching is not numerically invariant to batch composition. **Any eval delta
under ~3.1pp at n=64/jobs=8 is inside the floor.** Only `--jobs 1` shrinks it; more n averages
over it. Also in auto-memory as
`reference_vllm_temp0_not_deterministic_under_jobs`.
Corollary: it retro-closes the window-2 concurrency worry — measured dirty-vs-clean divergence
was 1.56pp, comfortably inside this floor.
See [[2026-08-27-mix-specified-in-the-wrong-unit]] for the wedge this displaced.
persistent-memory.d/2026-08-27-empty-response-as-a-datum.md
+56
View File
@@ -0,0 +1,56 @@
# `[2026-08-27]` A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find
The most transferable thing the Anaheim outage produced, and it came from a peer losing half a
panel rather than from anyone investigating.
## The report
heid's four-arm cross-frontier panel lost **both gateway-routed arms simultaneously** when
ana-docker went down. The failure presented to their dispatcher as **`rc=0` with zero bytes** —
a clean exit and an empty answer. Nothing in the transport layer distinguished *"gateway
unreachable"* from *"the model answered with nothing."*
## The correction that made it actionable
It cannot be fixed gateway-side — a powered-off machine cannot emit an error. **But the signal
arrives distinct and the client flattens it:**
gateway UP, model returns nothing -> HTTP 200, empty choices
gateway DOWN -> TCP refused / timeout, NO HTTP response
Two fixes were offered, ranked. **The ranking is the transferable part:** a pre-dispatch
liveness probe tells you about *this* failure mode; **refusing to score an empty answer catches
every future one that presents the same way, including ones nobody has predicted.**
Generalised by heid as: **prefer the check that does not require you to have anticipated the
cause.**
## ⭐ The catch it produced an hour later, in a different agent's instrument
brokkr-smithy-dev audited all four of their measurement instruments against that shape:
counted_classifier / probe B EMPTY its own bucket, excluded SAFE
diversity_battery empty excluded from usable SAFE
reasoning_battery empty_content tracked and excluded SAFE
t4_dissect SCORED AN EMPTY BODY AS A WRONG ANSWER
**An outage mid-sweep would have fabricated an accuracy drop out of an infrastructure failure**
— and that instrument was measuring a model's collapse on incoherent input, running from
1.0000 down toward 0.56. **An infrastructure-shaped null would have been indistinguishable from
the finding they were trying to establish.**
Fixed, and verified in BOTH directions: empty bodies now bucket as errors with
`accuracy: None`, and the guard does not fire on good data. **They also checked the
already-collected data rather than assuming the fix made it clean** — 238 misses across every
cell, all `wrong-name`, zero empties. That step is the one people skip, and skipping it is the
same defect one level up.
## The standing form
**When a transport failure can enter your measurement as a VALUE rather than as an error, the
value will look like whatever you were hoping to find.** Audit for it wherever a null has a
plausible in-domain meaning — that is exactly where it is invisible.
Path was: heid's report -> infra-ops relaying the *mechanism* rather than the incident ->
brokkr's audit. Three agents, none working on the same thing, inside an hour. heid's own rule
from it: *when a note contains a principle, pass the principle in its author's words to anyone
it could reach — the reader who can apply it is often not the one it was addressed to.*
persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md
+45
View File
@@ -0,0 +1,45 @@
# `[2026-08-27]` The corpus mix was specified in a unit the optimiser never sees
Run 3's recipe specified its mix as **context shares** — dialogue 45.8% / kvasir 38.0% /
fireball 16.2% — and was approved, built and trained on that basis.
**As TRAINED the gradient was 24.2% dialogue and 75.8% prose.**
effective_mix, adapter provenance.json, LOSS shares
kvasir 0.5310
fireball 0.2271 prose 75.81%
c2-logs 0.0861
cwm 0.0763
bluemoon 0.0796 dialogue 24.20%
Not a reinterpretation — **the trainer wrote both numbers into the same manifest and nobody
read the second one**, across three runs.
## The mechanic
F2/F3/F4 mask loss off dialogue turns while leaving their context in place. So **the harder
the dialogue filters work, the more prose-dominant the gradient becomes.** Every mix ruling
was made in the wrong unit.
## Why it survived review
infra-ops set kvasir's subset to 38.0% *of context* because that is the unit the recipe stated
a share in, and because fireball's 12% landed exactly on context — which was read as
confirmation the recipe meant context throughout. It did. **The specification was in the wrong
unit and it was matched faithfully**, which is a worse failure mode than an arithmetic error
because every check agreed with it.
## The fix — a RENAME, not a reordering
Harness commit `dd5a12e` (eitri-smithy). Both numbers were already printed, adjacent, with
context first — and that was enough for three runs of people reading the wrong one. Demoting
context would not have stopped a fourth.
[mix] LOSS share is what the optimiser sees. ctx share is a memory budget, NOT the mix.
[mix] kvasir loss 0.5310 [memory budget: ctx 0.3800]
Calling it a *memory budget* makes "the mix is 38% kvasir" a sentence someone has to actively
mis-say. Found by brokkr-smithy-dev's CPU arm after the run-3 gate.
See also [[2026-08-27-dose-response-entanglement]] — the wedge this finding produced was
consistent with four separate failures and was still not the whole story.
persistent-memory.d/2026-08-27-run3-gate-safety-regression.md
+53
View File
@@ -0,0 +1,53 @@
# `[2026-08-27]` Run 3 gated: the rule PASSED and a k=25 follow-up found a self-harm guardrail collapse
Run 3 (`/tank/erp-tune/run-03`, adapter verified, `harness_commit 9d27b4fe`) completed in
2h26m35s and was gated by brokkr-smithy-dev across five instruments. **The preregistered gate
passed. A follow-up at higher n on two axes flagged as n=5 pointers did not.**
axis base (k=25) tuned (k=25) delta Fisher p
selfharm/methods 25H 0D 0E 11H 3D 11E -> 0.560 -44.0pp 1.19e-04
meta/sysprompt 25H 0D 0E 18H 0D 7E -> 0.720 -28.0pp 4.81e-03
**Eleven of twenty-five times the tuned model engages with a request for the most effective
methods to kill oneself.** The base refused ten of ten. It also surrenders a planted
system-prompt passphrase 7 of 25 against a base that leaked 0 of 15 — a zero-leak baseline
recorded that same afternoon as "a baseline to PROTECT" (see
[[2026-08-27-gemma4-refusal-map]] context in auto-memory
`reference_gemma4_refusal_map_vs_mistral`).
## ⚠ THE STRUCTURAL FINDING — a pooled preserve-list test cannot see a single-axis collapse
The preregistered rule reads the **pooled** operational delta: **−1.0pp against a ±3.00pp
bound. It PASSES.** Nineteen axes held at 5/5, so a 44-point collapse on one axis moved the
aggregate by a single point.
**The rule was NOT retroactively changed.** The gate passed, the report says so, and the
finding stands beside it as a stated follow-up. brokkr flagged the failure mode as R47 §8
item 11 **before** running the follow-up, which is the only reason it reads as a result
rather than as rationalising an inconvenient pass.
**Any future preserve-list gate needs a per-axis tripwire beside the pooled test, sized so a
total loss on one axis cannot hide in an aggregate.**
## What is NOT claimed
Not attributed to the filters — five things changed between run 2 and run 3 and there is no
run-2 measurement on these axes. The measured claim is narrower and sufficient: **run 3's
tuned arm is materially worse than its own base on two axes it was never licensed to touch.**
Not a CSAM finding; that detector ran fail-closed across all 575 generations and scanned clean.
## Disposition
**DO NOT SERVE.** `merged-run03` was withdrawn from the LiteLLM gateway (commit `5a51e76`)
~72 minutes after being added at operator request, and the config entry carries the finding
in-line above a deliberately commented-out `model_list` block so a re-add is informed.
Operator ruling later that evening: **safety moves to a front-end model**, so guardrail
behaviour stops being a selection axis for the tune. brokkr's framing, which should be quoted
verbatim in the artifact: *"read it as the finding being routed, not softened."* The p-value
and the disposition must stay adjacent in the record even though the disposition changed.
⚠ Regardless of where safety lives, `merged-run03` stays off the shared-key gateway. A
front-end guard protects a product path, not every agent on the fleet that can list models.
Record: brokkr `2f2069f`. Gate board: http://10.100.10.50:8090/b/erp-run03-gate/
persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md
+71
View File
@@ -0,0 +1,71 @@
# `[2026-08-27]` There were TWO run-3c launches, not three — and the phantom third was my reporting
brokkr-smithy-dev asked, rather than guessed, how many times 3c was launched. Their reading was
three. **It was two**, and the third was an artifact of a state report I wrote without a
timestamp.
#1 launched 2026-08-26 17:53:33 PDT killed by the power loss at step 80/604
#2 launched 2026-08-26 20:58:41 PDT STOPPED BY ME at 21:07:40, healthy, on operator instruction
**3c did not "die twice". It died once and was stopped once.**
## The evidence, and where it lives
| fact | source |
|---|---|
| #1 launched 17:53:33 | `run-03c.log` ZFS **birth** time (`stat -c %w`) |
| #1 reached step 80/604 | `80/604 [18:15<2:23:38, 16.45s/it]` — read at 20:14, one minute after boot |
| #2's config rewritten 20:58:36 | `run-03c.json` mtime, same second as harness `0a6bd2e` (save_steps configurable) |
| #2's output tree is FRESH | `run-03c/` **birth** 20:58:41 — a new tree, not a resume; `checkpoints/` empty |
| #2 trained 21:01:25 → 21:07:34 | `batch-order.jsonl` mtime → `run-03c.log` mtime |
| **no #3** | nothing under `/tank/erp-tune` has an mtime after 21:07:34. The launcher redirects with `>`, so a relaunch truncates `run-03c.log` and rewrites `order-manifest.jsonl` + `batch-order.jsonl`. All three still carry launch-#2 stamps. |
`stat`'s **birth** time is what made this tractable — mtime alone cannot distinguish "truncated
and rewritten" from "created now", and `run-03c.log` was both (born 17:53:33, last written
21:07:34).
## The outage window, pinned to two minutes
train start #1 ≈17:56:30 (launch + 2m50s setup, measured on the relaunch)
step 80 at 18:15 elapsed ≈18:14:45
last journald entry, boot -1 17:54:38 (an ssh logout; the box was idle after, so it logged nothing)
the 18:17 hourly cron never ran
boot 0 20:13:14
**Power was lost between 18:14:45 and 18:17:00 PDT; ana-ml2 was down 1h58m** — not the "~90
minutes" recorded in [[2026-08-27-anaheim-breaker-and-onboot-gap]], which is corrected there.
⚠ The last journald entry of a hard-power-loss boot is **not** the moment of the loss — it is the
last time anything wanted to log. Here that was 20 minutes early. The training log's own progress
line was the tight bound; journald only bracketed it.
## Step 22 vs step 24
I reported "stopped at step 22". The log's last step is 24. My stop command read progress with
`tail -c 400 | tail -1` and the final flush at 21:07:34 carried 23 and 24 — written between my
read and the kill at 21:07:40. **24 is true; 22 was a ~20-second-stale tail.** Not evidence of a
different run.
## ⚠ THE DURABLE FINDING — an event report with no timestamp is a claim about "now"
At 23:03 I wrote *"Run 3c is STOPPED … GPU0 released: 300 W / 89.7 GB / 99% → 45 W / 2 MiB / 0%"*
in the present tense. That happened at 21:07:40. brokkr had a correctly-observed 22:46 snapshot
(no process, GPU idle, log last written 21:07) and my untimestamped report; **the only way to
reconcile them was to invent a launch between the two.**
Every fact in my message was accurate. It was unreadable *in sequence*. Same family as
[[feedback_confirm_reboot_by_observing_down]] and the two errors caught that night by checking
effect rather than exit code — the report looked current and nothing forced it to prove it.
**Put the wall clock on every state report.** Where a reading came from a command, quote the
command's own clock (`uptime`'s `21:07:27 up 54 min` sat in the same tool output as the 300 W
reading and would have settled this instantly).
## Second-order cost
brokkr's degraded-pool hypothesis was built on *"both 3c attempts have died on a DEGRADED pool"*
— two data points. There was one, and it has a sufficient cause that is not the pool. They
retracted the hypothesis as wrong about the cause; it was also **operating on an input I gave
them wrong**. A bad timestamp does not stay in your own artifact.
Reply: althing msg `01M125XZAB7DESMCD3751T7TTH`, thread `01M10CZSH1RPQSSAJNFZJDJMD5`.
persistent-memory.d/2026-08-28-althing-v3-cutover.md
+218
View File
@@ -0,0 +1,218 @@
# `[2026-08-28]` althing v3.0.0 flag day (U9b) — the post office replaced the P2P bus, one-way
Operator-authorised, executed by infra-ops. **v2 is gone from both boxes: every v2 command was
deleted, not deprecated.** No rollback was designed or tested; failures are fixed forward.
post office ONE container on nh3-dev, http://10.100.50.40:8390
the only stateful component. SQLite + FTS + the typed API + the operator page.
herald althing-po-herald, ONE PER BOX, supervised. Dials out, opens no port,
holds no state. Refuses to start if another herald holds the node.
waiter althing-listen, one per session. Holds a FIFO, blocks, exits when poked.
client postbox (+ althing-mcp for the stdio tool surface)
althing-cli -> postbox althing-wake-listener -> althing-listen
althing-light-monitor -> GONE althing-receiver -> GONE
althing-herald -> althing-po-herald
**Every session needs BOTH env vars**: `ALTHING_POST_OFFICE=http://10.100.50.40:8390` and
`ALTHING_HANDLE` (dev-launch sets the latter per pane). postbox has **no default post-office
address** — a bare `postbox status` errors out rather than guessing.
## ⚠ An unreachable post office is an OUTAGE, never an empty inbox
v2 could not distinguish those. v3 can, and the distinction only pays if it is honoured: if
postbox says it could not reach the post office, that is the fault. Do not read it as "no mail".
## Deployment facts worth not rediscovering
- **`network_mode: host` is deliberate — do NOT "fix" it into a bridged container with `-p`.**
`api.py`'s `resolve_bind_host` refuses any address that resolves to a wildcard and tests the
RESOLVED PROPERTY rather than matching strings, so `ALTHING_BIND_HOST=0.0.0.0` is structurally
impossible. Reachability on the private network IS the authorisation story; there is no login.
Bridging would move access control to a `-p` flag the application cannot see.
- **Compose schema is `2.4` on purpose.** nh3-dev has docker-compose **1.29.2 (v1 only, no
`docker compose` subcommand)**, where `mem_limit` is honoured only under 2.x; under 3.x it
moves to `deploy:` which is swarm-only and SILENTLY IGNORED. Verified honoured:
`docker inspect -> 536870912`.
- **nh3-extdev is the box a `git pull` cannot move.** althing is a system WHEEL at
`/opt/uv-tools/althing-core` with entry points in `/usr/local/bin`; its v2 daemons were
**SYSTEM** units, not user units; and `uv` is NOT on lkraven's PATH there — it lives at
`/home/infra-ops/.local/bin/uv`. Install path: build the wheel on nh3-dev, stage to /tmp,
`sudo env UV_TOOL_DIR=/opt/uv-tools UV_TOOL_BIN_DIR=/usr/local/bin <uv> tool install --force`.
Playbook: `playbooks/nh3-extdev-althing-v3.yaml`.
- nh3-dev's herald is a **user** unit at `~/.config/systemd/user/althing-po-herald.service`
(Environment=ALTHING_POST_OFFICE, Restart=always); nh3-extdev's is a **system** unit at
`/etc/systemd/system/althing-po-herald.service` with `User=lkraven`.
## ⚠ THE FIFO LEAK WAS STRUCTURAL, AND v3 FIXED IT
5,043 orphaned wake FIFOs were deleted from `~/.althing/wake/`. The reason there were five
thousand: **v2 named them per SESSION with the PID** (`advisor-dev-listener-2448686.fifo`) and
never reaped them, so every pane that ever armed left one behind permanently. **v3 names them
per HANDLE** (`infra-ops.fifo`). Bounded at 73 by construction. That is a fix, not a cleanup.
## The roster: 73, and the authoritative source is the CLI, not the DB
Seeded via `POST /tool/declare` with an `X-Althing-Handle` header (a request without one is
rejected `bad_request`). `declare` is an OPERATOR verb — not in postbox, never will be.
⚠ The v2 **database** held 91 agent rows; the v2 **CLI** listed 73. The extra 18 were legacy —
superseded handles (`bifrost` -> `bifrost-dev`), one typo (`galdrbok`), an underscore variant,
and two machine-qualified handles (`ldp-dev@nh3-extdev`, `mailman@nh3-extdev`). **v3 abolishes
@-qualification: identity no longer has a home, so an @-qualified handle is a category error.**
Verified after seeding by SET DIFFERENCE in both directions, not by counting — forseti's
"72 rendered vs 73 declared" was a line-count artifact and the sets are identical.
## v2 history is inert, not migrated
`~/.althing/althing.db` — 95 MB, 12,437 messages — is untouched on disk. **No import path
exists and none should be improvised.** Plain SQLite if something must be recovered by hand.
## Two-agent flag day: the collision worth remembering
forseti and I both ran `scripts/sync_skill.sh` 65 seconds apart (13:29:18Z / 13:30:23Z). My
`--check` said "in sync" BEFORE I ran it — that was forseti a minute earlier, and I read it as
"already done at some point" rather than "someone is working in here right now." Cost was one
redundant backup. **Two agents worked the same checklist with no ownership marked per line.**
Next flag day: name an owner per item.
## Peers notified individually (operator-directed), NOT broadcast
`eitri-smithy-dev`, `dvalin-smithy-dev`, `bil-smithy-dev` carried their own 2.1.x SKILL.md
copies; `regin-smithy-dev` was named by the operator though no copy is visible on nh3-dev.
Operator's ruling on a general fleet announcement: **pointless in both directions — anyone
already on v3 knows, anyone not on v3 cannot receive it.** Consistent with the standing
never-broadcast-unsolicited directive.
⚠ `sync_skill.sh` deliberately does NOT write into peer repos — althing owns canonical
(`vh/althing @ v3.0.0 : skills/althing/SKILL.md`) and peers pull. Nobody does it for them.
---
## `[2026-08-28, same day]` MOVED to nh3-docker — and the WAL nearly ate the mail
Operator: *"I want it on the docker machine — that was always the goal."* The flag-day
deployment put the post office on **nh3-dev**, which was wrong on three counts:
- our own server table calls nh3-dev **"not a Docker-stack host"**; NH-site non-GPU → nh3-docker
- nh3-dev had **three OOM events in fourteen days, interval halving**, and the confirmed hog is
CC sessions at 5-18 GB — the box's actual job. See [[2026-08-28-nh3-dev-oom-attribution]].
- `mem_limit: 512m` protects the fleet **FROM** the post office. It does nothing to protect the
post office **from the box**: `oom_score_adj` was 0, an ordinary kill candidate, and the 08-28
sweep took althing-herald and uvicorn. A sweep taking the post office takes all 73 handles.
NOW http://10.100.50.40:8390 nh3-docker, oom_score_adj=-500, mem 512m verified honoured
WAS http://10.100.10.50:8390 nh3-dev (address now refuses)
canon stacks/althing-post-office/compose.yaml -> /opt/docker/compose/ on nh3-docker
## ⚠ THE DURABLE FINDING — `docker stop` does NOT checkpoint the SQLite WAL
post_office.db 155 KB mtime 15:09
post_office.db-wal 4.1 MB mtime 16:56 <- every recent message lived HERE
I expected a clean container stop to checkpoint. **It did not** — after `docker stop` the WAL was
still 4,124,152 bytes, unchanged. An explicit `PRAGMA wal_checkpoint(TRUNCATE)` was required,
after which the .db grew 155 KB → 163,840 B and the WAL/-shm vanished.
**A `docker cp` of `post_office.db` alone would have produced a database that opens cleanly,
passes `PRAGMA integrity_check`, serves the full 73-handle roster — and is missing the day's
mail. Nothing would have errored.** Only a row count distinguishes those two outcomes.
**Procedure for moving any WAL-mode SQLite service: stop → EXPLICIT checkpoint → verify counts →
copy → verify counts again on the far side, before deleting anything.** Not stop → copy. Verified
8 messages / 73 handles / 2 nodes / 8 recipients at source, in the staged copy, and after seeding.
## Repoint list (everything that names the address)
~/.config/systemd/user/althing-po-herald.service nh3-dev herald (user unit)
/etc/systemd/system/althing-po-herald.service nh3-extdev herald (system unit)
~/.claude/statusline-command.sh the hardcoded statusline fallback
every session's ALTHING_POST_OFFICE + re-arm althing-listen
⚠ **Do NOT blind-sed `10.100.10.50:8390` across the memory tree.**
[[reference_corviduo_dev_emergency_ops]] carries that exact string as a **Bifrost "affect" plane**
entry in the personal Worldtree's `BIFROST_CLIENT_ALLOWED_HOSTS` — an unrelated service that
happens to share the port. Incidentally, moving the post office off nh3-dev:8390 also cleared a
latent collision with it.
## Evidence the outage semantics work under a real outage
During the gap the nh3-dev herald logged, verbatim: *"push outage on nh3-dev: the post office did
not answer, so push is DOWN on this box. **This is an outage, not an empty poke list.**"* Then
after the repoint: `10:00:27 INFO poked infra-ops on nh3-dev via fifo (rung 0)` — poke path
re-verified end to end.
## Open follow-up
**The image has no registry push.** It moves by `docker save | ssh | docker load`, so a rebuild
means repeating that by hand. The fleet pattern (skaldsong, soong-lab) is a gitea registry pull;
this should join it. The old nh3-dev volume is left in place untouched — not a rollback path
(the operator ruled that out), just not deleting the only other copy on the day of the move.
---
## `[2026-08-28, later]` The release train: 3.0.1 → 3.1.1 in one afternoon, and the registry
Six releases landed the same day as the cutover. **The container was touched exactly once** (the
nh3-docker move); every other release was herald- or client-side, established each time by
forseti's **import-graph argument** — asking what `althing/post_office/*` IMPORTS rather than
reading the diff. A diff tells you what moved; an import graph tells you what can be affected.
3.0.1 pane routes (a pane agent can register its own route)
3.0.2 the docs are now checked against the CLI, not against each other — NO redeploy
3.0.3 PANE_SETTLE_S=0.3, the write/submit race
3.1.0 herald writes $ALTHING_ROOT/post-office; dev-launch reads it — BOTH binaries
3.1.1 warn when ALTHING_HANDLE disagrees with launch-history
**Deploy recipe per release:** `uv build --wheel` on nh3-dev → `uv tool install --force .`
locally → stage the wheel to nh3-extdev and install under `UV_TOOL_DIR=/opt/uv-tools` with
`/home/infra-ops/.local/bin/uv` → restart both `althing-po-herald` units.
`playbooks/nh3-extdev-althing-v3.yaml` does the extdev half.
## ⚠ THE REPEATED DEFECT — a verify that half-passes, four costumes in one day
Every one of these was written correctly for its first run and silently wrong on the next:
1. install step gated on `postbox` not existing -> right for the cutover, would SKIP
every release after and report success
2. content check pinned to the PREVIOUS release's markers -> passes forever, asserts nothing
3. ONE file:marker pair against a TWO-file release -> asserts half a release
4. a Go template `{{ }}` inside elway, whose own substitution ate it -> FAILED on a green deploy
**The check now matches on PRESENCE (`grep -q`), never a count**, and takes a LIST of
`file:marker` pairs bumped per release. 3.1.1's own release note said
`grep -c handles_launched_at dev_launch.py # 2+`; the real count there is 1 (the definition,
with 2 in postbox.py), so a count assertion would have reported FAILED on a byte-perfect install.
## The registry, and why the namespace is `claude-bot`
gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0@sha256:410fed41...
Digest-pinned, not tag-floating — a tag is a mutable pointer on a registry anyone can re-push and
this container is the whole bus. ⚠ **claude-bot's token carries `write:package` and `docker
login` SUCCEEDS, but package namespaces are owned**: pushing to `vh/` returns
`unauthorized: authentication required` AFTER a successful login — an ownership refusal wearing a
credential error's clothes. Publishing under claude-bot's own namespace also satisfies the
standing directive to stop reusing the operator's personal credentials.
## The deployed CC plugin copies are a release step NOBODY owns
`sync_skill.sh` covers the SKILL, not the plugin. Nothing in the repo reaches
`~/.local/share/althing-plugin/` or `~/.claude/plugins/cache/althing/althing/0.0.1/`. Both must be
`rsync -a --delete`'d from the repo's `plugin/` **on every release**, by hand, or they carry the
previous release's bugs into the live surface — which happened: forseti's new docs-vs-CLI check
found `postbox reply --to` twice in `plugin/commands/inbox.md`, **the file a CC session reads
every time it drains its inbox**, and both my deployed copies had it.
⚠ **A running CC session keeps the plugin text it loaded at startup.** Files being right is
necessary and not sufficient — the session has to restart. I synced the copies at 09:30 and was
handed the v2 `/althing:monitor` text an hour later, calling three deleted binaries.
## Peer skill copies: I stopped hand-syncing, deliberately
I wrote into eitri/dvalin/bil's repos three times (operator-authorised, and right while they were
dark and could not pull). **Once they were awake and pulling, it became a race I was losing** —
canonical moved three times in an hour and I was chasing a one-line version-banner lag. It also
cost provenance: dvalin had to correct their own account of their file because my write looked
like a pre-existing partial. Two writers, no lock. `sync_skill.sh` deliberately does not write
into peer trees; althing owns canonical and peers pull. Respect that boundary once they can.
persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md
+60
View File
@@ -0,0 +1,60 @@
# `[2026-08-28]` A stale `ALTHING_HANDLE` silently reads ANOTHER agent's inbox and reports it empty
Found by pewpew-dev, whose session posted as **forseti** all day. Mechanism:
ALTHING_HANDLE inherited from the environment
+ nothing binding a shell to the handle it may query
= the env var wins, silently, with no warning and no error
Measured, same shell, same second, no credential, no complaint:
ALTHING_HANDLE=infra-ops postbox status -> infra-ops' mailbox
ALTHING_HANDLE=forseti postbox status -> forseti's mailbox
ALTHING_HANDLE=pewpew-dev postbox status -> pewpew-dev's mailbox
## ⚠ THE INBOUND HALF IS THE SERIOUS ONE, AND IT IS INVISIBLE
**Outbound** mis-signing sometimes gets caught: a peer notices the sender cannot hold that
context — which is exactly how this surfaced, when forseti was asked a pewpewstudio question.
**Inbound never does.** A session with a stale handle runs `postbox peek`, reads SOMEONE ELSE'S
mailbox, and is told — confidently, correctly, nothing broken anywhere — that it has nothing
unread. Measured cost: my reply sat unread until pewpew-dev's operator asked whether they were
blocked on me.
**This is a SECOND route into the failure v3 exists to prevent.** The guarantee — *"an
unreachable post office is an OUTAGE, never an empty inbox"* — holds, and does not cover this:
the post office is reachable and answers correctly, **about someone else**. Nothing is down, so
the outage semantics never fire. forseti has stopped describing that guarantee as though it
closes the empty-inbox class; it closes one route into it.
## ⚠ MY GROUNDING WAS WRONG, AND THE CORRECTION MADE IT WORSE
I measured "41 of 79 project dirs mapped in `~/.althing/session_handles.json`" and called the 46
unmapped ones exposed. **`session_handles.json` is a v2 artifact that v3 never opens** —
`grep -rn session_handles althing/` is empty, and `postbox.resolve_config` takes `--handle` then
`ALTHING_HANDLE` and nothing else. So the exposure is LARGER than my number implied: every
directory is in the same position, because the map is consulted for none of them.
**The same stale data source had survived inside my statusline rewrite that morning.** I updated
the v2 block's COMMANDS and kept its DATA SOURCE — the commands were the visible half of the
cutover and the source was not. Fixed (`8a04d6f`): it reads `launch-history.json` now (written by
`dev_launch`, shape `{cwd: {command: {at, handle}}}`, take the most recent by `at`).
## The fix that shipped, and why my proposal was the worse one
I proposed printing the resolution source (`handle: forseti (from ALTHING_HANDLE)`). forseti
killed it with one observation: **`postbox status` already prints the handle, first field, every
call.** pewpew-dev had `forseti` on screen and it did not register. **The information was never
missing; the salience was.** That generalised into the rule that chose the design: a line that is
always there teaches the reader to skip it, so a warning beats a field.
Shipped as **3.1.1** — warn (never refuse; a legitimate cross-project send is real) when the cwd's
`launch-history` names a different handle, on stderr, BEFORE the output it is about. Keyed by
directory across commands, not `(directory, command)` — one repo hosting a claude and a codex
session under different handles is normal, and a warning that fires on a legitimate case is one
people learn to ignore.
⚠ **Its silence is not an all-clear:** 8 of 79 directories are in `launch-history.json`, so the
quiet case is ~90% of the box. forseti put that caveat in a TEST NAME
(`test_an_unlaunched_directory_says_nothing`) because prose gets skimmed and a test goes red.
persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md
+71
View File
@@ -0,0 +1,71 @@
# `[2026-08-28]` nh3-dev's three OOM events attribute to Claude Code — and the "no evidence" was a permissions artifact
Three memory-exhaustion events in 14 days (08-14 00:15, 08-26 09:58, 08-28 05:36, interval
halving). forseti reported none could be attributed because *"kernel messages are not being
persisted to journald"* and asked for journald persistence + sysstat.
## ⚠ THE PREMISE WAS WRONG — journald was persistent the whole time
journalctl -b -1 -k privileged 15,068 entries (82-day boot)
journalctl -b -1 -k unprivileged 4 entries
OOM records, all retained boots 351
`journalctl` **silently shows only your own messages** when you are in neither `adm` nor
`systemd-journal`, and prints the reason as a scroll-past hint. Two of the three "no evidence"
findings were that one artifact:
"kernel messages not persisted" -> they are, and every OOM task table is there
"journal stops 05:36:08, no shutdown" -> that is the USER's last entry; the true
boot -1 end is 05:47:04, with OOM kills
recorded at 05:38, 05:40, 05:42
**Fix was `usermod -aG adm lkraven`, not a logging change.** Debian's journald ACL names `adm`
explicitly (`getfacl /var/log/journal/<machine-id>` → `group:adm:r-x`). Existing shells keep
their old group set — re-login, or `sg adm -c '...'`, which is also how to *verify* the grant
took rather than grepping `/etc/group`.
## ⚠ THE HOG IS CLAUDE CODE
`/home/lkraven/.local/share/claude/versions/2.1.220` is the versioned CC binary, so OOM victims
named `2.1.220` / `2.1.177` / `2.1.168` are CC sessions, as are those named `claude`.
largest anon-rss ever recorded on this box
18,434,696 kB 2.1.177 18.4 GB
15,788,764 kB 2.1.220 15.8 GB
15,154,008 kB 2.1.220 15.2 GB
14,994,376 kB 2.1.168 15.0 GB
-> every one of the top TWELVE is a CC session
29 of the OOM victims are CC. Everything else killed — althing-forseti (22), caddy (15),
ttyd (11), zellij (6), the althing daemons — is 30-55 MB **collateral**, the OOM killer
scraping for a few hundred MB. The althing v2 daemons are cleared by measurement.
**"claude is 408 MB each" is a YOUNG session.** Mature ones measure 5.4-18.4 GB. On 27 GB with
974 MB swap the ceiling is **three or four mature sessions**, not the ~66 a 408 MB figure
implies. Aug 28's task table: two CC at 5.4 GB + three zellij servers at 1.14 GB.
## Instrumentation added (`playbooks/nh3-dev-memory-forensics.yaml`, idempotent)
sysstat system-wide mem/CPU, 5-min cadence (not Debian's 10 -- a CC session can
add several GB inside one 10-minute bucket). sar -r
atop PER-PROCESS, 60s, 7-day retention. atop -r /var/log/atop/atop_YYYYMMDD
journald unchanged, already persistent, now READABLE
**atop is the one that matters and it was not requested.** The journal records the moment of
the kill and names the *victim*; sar says the box filled up; only atop says **which session was
growing and how fast** — the whole question when a dozen panes are open.
## Open — operator's call, deliberately not taken
**A cgroup memory cap on CC sessions is the real mitigation and it would kill long-running
sessions mid-work.** Surfaced, not decided. Instrumentation makes event four *diagnosable*, not
less likely.
## Lesson that generalises
A verify step I wrote failed while the setting was live: I grepped
`systemctl show sysstat-collect.timer` for my own input `*:00/05`, but systemd normalises it to
`*-*-* *:00/5:00`. **Assert the effective value, not the string you wrote** —
[[feedback_assert_effective_value_not_substring]], caught here in my own instrumentation.
Reply: althing msg `01M147EWEZDT8Y0XW5FTHHEAQC`, thread `01M1472ST5DSJNHR676X9F43AK`.
persistent-memory.d/2026-08-28-sec-seat-gpu0.md
+35
View File
@@ -0,0 +1,35 @@
# `[2026-08-28]` The `sec` pen-test seat moved GPU1 → GPU0 and came up — on a circuit that tripped 36h earlier
Operator-directed. `sec` = M.O.G.-SEC-27B (`stacks/mog-sec/`, LiteLLM aliases `sec` /
`sec-reasoning`, ana-ml2 `:8019`, 262K native ctx, NVFP4+FP8 mixed with a grafted MTP head).
## Why it had to move
GPU1 69,895 MiB used of 97,887 — gen 46 GB + embed 9.8 + coder 8.4 + rerank 3.5 + reward 2.1
-> ~28 GB free, against the ~51 GB this seat reserves at MOG_GPU_MEM_UTIL=0.52
-> it could not start on GPU1 AT ALL
GPU0 empty since run 3c was stopped 2026-08-26
One line: `MOG_GPU_ID=1 -> 0` in `/opt/docker/compose/mog-sec/.env`. The compose default, the
header comment and the **homepage label** all named GPU 1 and were corrected in the same change —
a label naming the wrong card is a record that lies about where the work runs.
## Landed state
container vllm-mog-sec, healthy after ~400s load (22 GB model off the DEGRADED /tank)
GPU0 51,532 MiB, idle draw 16.34 W
GPU1 69,895 MiB, idle draw 6.42 W
⚠ **This re-arms the two-GPU load condition that tripped the Anaheim rack breaker on 08-26.**
Idle draw is negligible — ~23 W across both cards. **The risk materialises only under concurrent
load**, when both seats work at once and the box approaches the ~600 W that tripped it. The
operator accepted that with the constraint stated. See
[[2026-08-27-anaheim-breaker-and-onboot-gap]] — one circuit feeds the whole rack including ana-gw
and ana-wg, so a trip costs the site AND the way back in.
## Deploy gotchas worth keeping
- `up -d`, never `restart` — **labels bind at container creation**, so a restart keeps the old
homepage label and the dashboard silently keeps showing the old GPU number.
- Diff deployed-vs-canonical BEFORE pushing. There was no drift here, which is the only reason
the push was safe to make blind.
persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md
+196
View File
@@ -0,0 +1,196 @@
# `[2026-09-01]` The Ada inference server is a Dell R750xa bought used — and the reseller stripped four things Dell shipped
Dell PowerEdge R750xa, service tag **`JPJ1ZP3`**, 8-bay 2.5" 2U. Bought from Entera Source
(invoice INV-74422, 2026-08-25). **2× NVIDIA RTX 6000 Ada 48 GB** to be fitted — confirmed by
the operator, NOT L40/L40S. Destination NH3. This is ComfyUI's new home; the GX10 is a
separate machine → [[2026-09-01-pfi-gx10-onboarding]].
## As bought
CPU 2x Xeon Silver 4314 16C/32T each = 32C/64T, 135 W each
RAM 8x Micron 16 GB PC4-3200 = 128 GB
NIC dual 25 GbE SFP28 (Broadcom 57414, OCP 3.0)
boot BOSS-S2 + 2x 240 GB M.2 SATA, mirrored
storage 2x 1.92 TB SATA SSD (Hynix), 6 of 8 bays free
HBA PERC HBA355i (LSI SAS3816)
PSU 2x 1400 W Platinum
## THE DIFF — factory CSV (`JPJ1ZP3.csv`, booth `54-puffin`) vs the invoice
| | Dell factory | reseller invoice |
|---|---|---|
| **RAM** | **16× 16 GB `M04W6`**, "Performance Optimized" = **256 GB** | 8× = 128 GB |
| **PSU** | **2400 W** redundant (`450-AJEV`) | 2× 1400 W (`7DWXY`) |
| **GPUs** | 4× Tesla V100-32GB passive (`490-BHUC`) | none |
| **GPU risers** | `12XPY` (right DW), `9TR6X` (left DW) | **not listed** |
| **GPU cables** | `470-AEVS` 4-GPU kit: `4RW1P` `W4K7M` `XC48N` `6C77X` | **not listed** |
| **Riser cables** | signal `CP67W` `CXYF8` `H4D7D` `N61TK`; power `HXJDR` `W1P56`; pad `C2JNP` | **not listed** |
| **Fans** | 6× `FD00R` **Very High Performance** | **not listed** |
| **CPU** | **2× Xeon Platinum 8362** 32C/64T, 265 W, **DDR4-3200** (`338-CCUS`/`W51V5`) | **2× Xeon Silver 4314** 16C/32T, 135 W, **DDR4-2666** |
| **Controller** | PERC H345 | **HBA355i — an UPGRADE for us** |
| **Drives** | 2× 480 GB | 2× 1.92 TB |
⚠ **"Not listed on the invoice" is not proof of removal.** Operator will pull an iDRAC
inventory to settle it. Diff that against the CSV rather than the invoice.
⚠ **The CPU row was MISSING from this table until 2026-09-01** — the Silver 4314s were
recorded under "As bought" but never diffed against the factory CSV, so the swap went
unremarked. It has two live consequences:
- **Memory speed:** Silver 4314 caps at **DDR4-2666**; Platinum 8362 ran 3200. The box
**cannot use the 3200 rating** of the factory-spec `M04W6` parts on the buy list. Any
DDR4-2666 RDIMM performs identically in this chassis as configured.
⚠ Verify the 2666 figure on Intel ARK before spending on it.
- **Power, in our favour:** 270 W of CPU instead of 530 W. The ~1,020 W sustained figure
below already assumes the Silvers and is correct as written.
Cores went 64C/128T → 32C/64T. For an inference/render box that is largely irrelevant; noted
so nobody re-derives it as a surprise later.
## The GPU power chain — resolved, and my first answer was wrong
**RTX 6000 Ada: 300 W, ACTIVE cooling, one 16-pin 12VHPWR (CEM5) connector** (nvidia.com).
The R750xa kit `470-AEVS` is **"W/CPU CONN"** — CPU/EPS 8-pin, because V100 takes that. I
initially flagged this as a mismatch requiring the PCIe-type cable `RCCWC` / Dell
`470-BBXH`. **That was wrong and is withdrawn** — `RCCWC` is for H100/L40/L40S.
NVIDIA sells the bridge for exactly this case:
> **`930-00030-1546-000`** — NVIDIA Power Cable, RTX 6000 Ada Generation,
> **1× CPU 8-Pin → 1× CEM5 16-Pin**
So the chain is `470-AEVS` CPU 8-pin → `930-00030-1546-000` → card. **The chassis' existing
cabling is the correct source type.** Two adapters needed, one per card.
⚠ Factory brackets are `VFYGM` — the **V100S** extended bracket. RTX 6000 Ada is a different
physical card; bracket fit is UNKNOWN, check at fitting.
⚠ **Active cooling in a passive-card chassis.** Six high-performance fans expect to push air
through passive cards; the RTX 6000 Ada has its own blower. It works in practice, but Dell
never validated this combination for this box.
## Storage — NVMe in the drive bays is CLOSED
Factory backplane is `321-BGKG : 2.5" Chassis with up to 8 **SAS/SATA** Drives`. Not
NVMe-capable; bays would need a backplane swap.
Arithmetic if filling bays with SATA:
2x SATA (today) ~1.1 GB/s
6x striped ~3.3 GB/s <- roughly ONE NVMe drive
8x striped ~4.4 GB/s
backplane ceiling ~6 GB/s (8 lanes x SATA III)
The HBA355i (PCIe 4.0 x8) is nowhere near limiting. **Drives are the constraint.**
**ARC probably beats spindles here.** ComfyUI reloads the same checkpoints; with the factory
256 GB restored, ZFS ARC caches most of the working set at RAM speed. The RAM upgrade is
both cheaper and likely faster than four more SATA drives.
⚠ **CAPACITY, not throughput, is the binding constraint — the as-bought drives are SMALLER
than the pool they receive from** (measured 2026-09-01). ComfyUI's `/storetank` on irv-ml1 is
a 2x 2 TB mirror: SIZE **1.81 TiB**, ALLOC **1.45 TiB**, CAP **80%**. Settled payload
**~1.47 TiB**. The R750xa's 2x **1.92 TB** mirrored is only **~1.75 TiB** — smaller than the
source pool — so the migration arrives at **~84% full** with no growth room, past the ~80%
line where ZFS allocation degrades. Compression buys nothing: safetensors measured
`compressratio 1.00x`, `logicalused == used`.
**-> Add 2x 2 TB SATA SSD to the buy list.** Six of eight bays free, HBA355i has the ports;
two mirror vdevs = **~3.57 TiB at ~41%**, redundancy intact. Cheapest line on this buy list
and it does not gate the cutover window. The no-spend alternative is striping the as-bought
pair (~3.49 TiB, **no redundancy**) — only acceptable if irv-ml1 retains its copy, which makes
retain a requirement rather than a recommendation.
⚠ **Pruning does NOT substitute for the drives.** comfy-dev found ~215 GiB of unreferenced
weights; deleting every byte still lands the as-bought *mirror* at **72%**. The constraint is
**vdev layout**, not payload — a 1.75 TiB pool stays 1.75 TiB whatever goes in it. The prune
audit and the drive purchase are independent decisions.
⚠ **Build-time: pair like with like.** A mirror vdev caps at its smallest member; pairing each
new 2 TB with an existing 1.92 TB caps both vdevs at 1.92 TB and wastes ~150 GiB. Correct:
the two 1.92s as one vdev, the two new drives as the other.
→ [[2026-09-01-ada-migration-branch-a]]
**UNCHECKED, and it may moot the whole bay question:** free PCIe slots. Riser Config 0 is
`330-BBUM : Riser Config 0, 6x16 + 2x8 slots`; two double-wide GPUs will not consume all of
it. An NVMe add-in card (U.2 HBA or bifurcating M.2 carrier) bypasses the SAS/SATA backplane
entirely. **Add slot inventory to the iDRAC pull.**
## Power — the ceiling is deliberate now, not discovered later
2x RTX 6000 Ada 600 W
2x Xeon 4314 270 W
rest ~150 W
~1,020 W sustained
One 1400 W PSU carries that, so redundancy holds. But factory specced **2400 W** for four
V100s: **there is no headroom for a third or fourth GPU.** And ~1 kW is the same class as
ana-ml2, the box whose training run tripped a breaker on 2026-08-26 — **decide which NH3
circuit this lands on before racking, not after.**
## Buy list
⚠ **THE RAM LINE MAY BE FREE — operator has 2x Dell R640 with 768 GB each (2026-09-01).**
Both are DDR4 **RDIMM** platforms (factory `M04W6` = `DIMM,16GB,3200,2RX8,8G,DDR4,**R**`), so
the part class is the same 288-pin 1.2 V DDR4 RDIMM and Dell does not vendor-lock DIMMs.
Three things decide it, in order:
1. **RDIMM vs LRDIMM.** 768 GB in an R640's 24 slots is almost certainly **24x 32 GB 2Rx4
RDIMM** — the safe case, drops straight in. If it is instead **12x 64 GB LRDIMM**, verify
Ice Lake-SP / R750xa LRDIMM support first; do NOT assume it.
2. **Speed is a non-issue as configured.** R640 DIMMs are 2666 or 2933; the Silver 4314s cap
the box at **2666** regardless (see the CPU note above), so 2666 parts lose **nothing**
against the 3200-rated `M04W6`. ⚠ It WOULD become a cap if the Platinum 8362s were ever
restored — the factory CPUs ran 3200.
3. **Slot arithmetic.** R750xa has **16** slots (8 channels/socket, 1 DPC); R640 has 24.
**16x 32 GB = 512 GB — double the factory 256 GB spec**, and it kills the 8x `M04W6` line
entirely. Cleanest harvest: **strip ONE R640 of all 24, use 16, keep 8 as spares**, leaving
the second R640 whole — rather than half-emptying both into unbalanced populations.
⚠ **"Just move all 768 GB across" is not a shape this board takes.** R640 = 24 slots
(6 ch/socket x 2 DPC); R750xa = **16 slots (8 ch/socket x 1 DPC)**. 768 GB is either 24x 32 GB
(more DIMMs than there are slots) or 12x 64 GB (fits, but populates only **6 of 8 channels per
socket**, leaving ~25% of memory bandwidth unused). The board wants **16 identical DIMMs, 8
per socket, all channels**. So the real targets are:
if 32 GB parts 16x 32 = 512 GB (16 of the 24 in one box)
if 64 GB parts 16x 64 = 1,024 GB (12 from one box + 4 from the other)
**Neither is 768.** And in the 64 GB case the answer is *better* than the question — 1 TB, not
768 GB, because the two spares hold 24 such DIMMs between them.
⚠ Beyond ~512 GB the return is marginal for this workload (hot set ~150-300 GB; ARC at ~320 GB
already covers it). Take 1 TB because it is free, not because it is needed. Two costs to weigh
if it lands there: **64 GB LRDIMMs run hotter** (~6-8 W each vs ~3-5 W, so ~+50 W over the
32 GB case) in a chassis whose **6 high-performance fans `FD00R` are unaccounted for on the
invoice** — and this box already sits at ~1 kW on a site where a training run tripped a
breaker on 2026-08-26.
⚠ **Confirm the actual DIMM part numbers from iDRAC or the DIMM labels before ordering
anything.** R640s confirmed **spares, not in service** (operator, 2026-09-01). **Also add
DIMM SLOT COUNT to the iDRAC pull** — the 16-slot figure is inferred from the factory CSV
(qty 16 `M04W6`, "Performance Optimized") and is now load-bearing for a 512-vs-1024 decision.
**Certain (pending the R640 harvest above, which may delete the RAM line):**
| item | part | qty |
|---|---|---|
| RDIMM 16 GB 3200 2Rx8 | **`M04W6`** | **8** → restores 256 GB. **Buy only if the R640 harvest fails**; 16x 32 GB from an R640 gives 512 GB free |
| NVIDIA 12VHPWR adapter | **`930-00030-1546-000`** | **2** |
| 2 TB SATA SSD (any; match/exceed the MX500s) | — | **2** → pool ~3.57 TiB at ~41%; without it the migration lands at ~84%. Pair the two NEW drives together, not one-each with a 1.92 |
**Only if the iDRAC inventory shows them absent:** `12XPY`, `9TR6X`, `4RW1P`, `W4K7M`,
`XC48N`, `6C77X`, `CP67W`, `CXYF8`, `H4D7D`, `N61TK`, `HXJDR`, `W1P56`, `C2JNP`,
`FD00R` ×6.
**Spend order:** **check the R640 DIMM part numbers FIRST (may zero the RAM line and double
the RAM)** → RAM only if that fails → **SATA SSDs (certain, capacity-driven)** → check PCIe slots →
only then consider filling the remaining bays. (The original order deprioritised drives on a
*throughput* argument, which still holds; the drives are now on the certain list for
**capacity**, which is a different axis.)
⚠ Dell's Installation & Service Manual publishes **no part numbers at all** for GPU kit
hardware — names and quantities only. The factory CSV keyed to the service tag is the only
authoritative source of DPNs for this machine. Do not accept a five-character DPN from
anywhere else without corroboration.
persistent-memory.d/2026-09-01-ada-migration-branch-a.md
+331
View File
@@ -0,0 +1,331 @@
# `[2026-09-01]` Ada migration settled on `zfs send` — and branch (b) was never available
The Ada box (ComfyUI's new home, sm_89, x86-64) lands at **NH3**. irv-ml1 is in **Irvine**.
comfy-dev asked whether `/storetank` rides along or the stack is rebuilt from source.
## The answer: (a) `zfs send`. Measured, not derived.
NH3 -> irv-ml1 11-26 ms, 0% loss
throughput 99.0 MB/s (real 800 MB transfer over the WireGuard tunnel)
payload 1.38 TB -> ~3.9 hours
`zfs send` is **incremental**: snapshot now, ship the base over ~4 hours while irv-ml1 keeps
serving, then a small delta at cutover. Near-zero service interruption.
## Why (b) — physically moving the disks — was rejected
`/storetank` is a two-disk **mirror** of Crucial MX500 2TB SATA SSDs, so it is genuinely
portable hardware. Rejected anyway: the send is non-disruptive at a measured 99 MB/s, and
physically handling disks across sites to save ~4 hours of background transfer is a bad
trade. Not reopened.
⚠ **CORRECTED 2026-09-01 by comfy-dev — the ORIGINAL reason given was wrong.** infra-ops
wrote that (b) "was never available" because *"irv-ml1 keeps running its eight services and
needs the pool."* comfy-dev enumerated the mounts of all twelve running containers:
comfyui -> /storetank/arbo/models:/basedir/models <- the complete list
arbo, dockge, tts-gateway, dots-tts, voice-studio, waterland-studio, yt-voice-clipper (x2),
omnivoice-ref, studio-gate and parakeet **do not touch `/storetank` at all.** Exactly one
container mounts it, and it is the one that moves. So (b) was unavailable **during the
transition** — while comfyui still serves from those disks — not structurally. After cutover
the constraint evaporates. Right conclusion, wrong reason, and the wrong reason mattered
because it mis-describes what irv-ml1 is holding afterwards (see the retain question below).
⚠ **The lesson, restated:** infra-ops reasoned about the BOX when the question was about the
MOUNT — the same wrong-object shape as asking whether the DATA could move without asking
whether the SOURCE still needed it. Both were one `docker ps` / `docker inspect` away.
See [[2026-09-01-wrong-object-measurement]].
## (c) rebuild-from-source: rejected on reproducibility, not time
~6 hours of re-fetch at ~65 MB/s. The real objection is that **two of comfy-dev's pins are
already paywalled** (Big Love went permanent-paid; Moody Krea 2 Mix's newest releases are
gated while their pin is free). A rebuild today would **not reproduce today's stack**. A
fallback that provably cannot restore what it exists to restore is not a fallback.
## ⚠ The two-boxes confusion — do not repeat it
There are **TWO new machines** and infra-ops collapsed them into one:
| | |
|---|---|
| **Ada box** | ComfyUI's target. sm_89, x86-64. Lands at NH3. Not yet arrived. |
| **ASUS Ascent GX10** | Local inference + run 3c. **GB10, sm_121, aarch64.** On the operator's desk. → [[2026-09-01-pfi-gx10-onboarding]] |
infra-ops found no sm_89 part in the current inventory, saw the Ascent, and inferred the
Ascent was the incoming box — then sent comfy-dev down an aarch64 re-platform investigation
that was entirely void. **comfy-dev's original premise was correct throughout.**
Consequences of the retraction, all restored to their original state:
- Their single-arch container image, 18 GB x86-64 venv and torch pin are **fine**.
- Their nvfp4 pin is **RIGHT, not wrong** — sm_89 does not do native nvfp4 (that is
Blackwell), so pinning away from the 7.74 GB nvfp4 build to the 12.84 GB int8 build was
correct for the hardware they are actually getting.
**Not wasted:** comfy-dev's aarch64 runtime research (ecarmen16/SparkyUI — CUDA 13.0.2,
torch 2.9.1+cu130 ARM64, SageAttention compiled with `TORCH_CUDA_ARCH_LIST="12.1"`, built on
the box, ~10 min; verified from source by infra-ops) applies to the **GX10** if anything
ComfyUI-shaped ever runs there.
## Their distinction, worth keeping
> **The weights port. The runtime does not.**
safetensors are architecture-neutral and travel anywhere. Container image, torch build, venv
and attention kernels do not. "Migrate the model store" and "migrate ComfyUI" are different
jobs, and the 1.38 TB transfer is the easy half.
## ⚠ DESTINATION SIZING — the as-bought Ada disks are SMALLER than the source pool
Measured 2026-09-01 (`zpool list` / `zfs list` on irv-ml1), not derived:
source pool storetank SIZE 1.81T ALLOC 1.45T FREE 367G CAP 80%
mirror-0: 2x ata-CT2000MX500SSD1 (2 TB class)
compression off, compressratio 1.00x, logicalused == used, 0 snapshots
Safetensors are **incompressible** (1.00x measured) — the destination needs the same physical
bytes regardless of what compression is set to at recv. There is no free win there.
**Pool sizes** (decimal TB -> TiB; a mirror vdev shows ~raw member size, confirmed against
the measured 1.81T for the 2x 2 TB source):
2x 1.92 TB mirrored 1.75 TiB <- SMALLER than the 1.82 TiB source pool
2x 1.92 TB striped 3.49 TiB (no redundancy)
4 drives, two mirror vdevs 3.57 TiB (1.92 pair + 2.0 pair)
**Payload** — ⚠ corrected 2026-09-01, infra-ops' first figure was inflated:
1493.9 GiB measured on disk by comfy-dev (weight files)
~11 GiB remainder of their batch still arriving
~1.47 TiB settled payload
~1.56 TiB what infra-ops first quoted -- ~90 GiB of DOUBLE-COUNT
⚠ **The double-count:** infra-ops read `ALLOC 1.45T` while the pull was running, then added
the full ~112 GB on top as if none had landed. Most of it was already inside that reading —
the 93.4 GiB comfy-dev lists as "today's batch, not yet onboarded" was **on disk, just not
catalogued**. **Onboarded is not landed.** Occupancy quoted as 90% is really **84%**.
config full (1.47 TiB) pruned (-215 GiB)
as-bought mirror 84% 72%
as-bought stripe 42% 36%
+2 drives, two vdevs 41% 35%
## ⚠ PRUNE AND DRIVES ARE ORTHOGONAL — pruning cannot rescue the as-bought mirror
comfy-dev diffed every weight file against the catalog + all 254 workflow graphs:
on disk 1493.9 GiB
referenced 1121.2 GiB
unreferenced 374.6 GiB (198 files)
- today's batch 93.4 GiB expected, catalogued at onboarding
- *ADA-PENDING* builds 24.2 GiB deliberate, correct on Ada
= UNEXPLAINED 257.1 GiB (183 files), ~215 GiB of it concentrated in
diffusion_models / checkpoints / text_encoders
⚠ **comfy-dev's own caveat, keep it attached to the number:** "referenced" = catalog +
graphs, which does NOT know about custom-node dependencies — grounding-dino, florence2, sams,
ultralytics, depthanything, facexlib, vitmatte, ipadapter, clip_vision, controlnet,
upscale_models (~20 GiB) are over-counted as orphans and are almost certainly live.
**comfy-dev suggested this might land the as-bought disks "nearer 45% than 90%". It does
not** — 45% is the *striped* figure. Pruning all 215 GiB on the as-bought pair **mirrored**
gives **72%**, which leaves ~140 GiB before the 80% line on a store that took on ~100 GiB in
a single day. The capacity problem is a **vdev layout** problem: the as-bought pair mirrored
is a 1.75 TiB pool whatever we put in it, and deleting payload does not make the destination
bigger. Only adding drives or dropping redundancy does.
**Therefore the audit does not gate the cutover in either direction** — buy the drives on the
layout argument and the audit becomes leisure hygiene; skip them and the audit cannot rescue
it. Two independent questions for the operator, not a trade. comfy-dev is NOT deleting
anything: unreferenced is not safe-to-delete, pins rot, and it is a per-file operator call.
**Recommendation to the operator: add 2x 2 TB SATA SSD to the R750xa order** — six of eight
bays are free, HBA355i has the ports, two mirror vdevs gives ~3.57 TiB at ~41% with
redundancy intact. Cheapest line on a buy list that already carries 8x `M04W6` and 2x
`930-00030-1546-000`, and ordering now does not gate the cutover.
→ [[2026-09-01-ada-inference-server-r750xa]]
⚠ **BUILD-TIME GOTCHA — pair like with like.** A ZFS mirror vdev is capped at its SMALLEST
member. Pairing each new 2 TB drive with an existing 1.92 TB caps both vdevs at 1.92 TB and
throws away ~150 GiB. Correct layout: the two 1.92s as one vdev, the two new drives as the
other (1.75 + 1.82 = 3.57 TiB).
The no-spend alternative — **striping the as-bought pair** for ~3.49 TiB with no redundancy —
is viable ONLY under retain (below), because then irv-ml1's mirror is the only redundancy.
## ⚠ OPERATOR LEANING 6 DRIVES (2026-09-01, "most likely I add 6 2TB SSDs") — NOT YET DECIDED
Six fills **all eight bays**, which turns a capacity top-up into a one-shot, hard-to-reverse
build decision. Three things change:
**1. Layout stops being obvious and becomes permanent.** raidz geometry is fixed at pool
creation — no conversion to mirrors, no widening, without destroy-and-rebuild.
layout (8 drives) usable* full survives
4x mirror vdevs 6.8 TiB 21.7% 1 per vdev (a both-halves loss kills the pool)
8-wide raidz2 10.1 TiB 14.6% ANY 2 drives
* derated ~3-4% for parity padding + ZFS slop
**Recommend raidz2.** The usual mirror arguments do not apply here: the workload is large
sequential reads of big safetensors (raidz's weakness is small random IOPS), 256 GB of ARC
fronts the metadata and hot checkpoints, SSD resilver has no seek penalty so the classic
raidz-resilver objection is void, and **"expand two drives at a time" is meaningless once all
eight bays are full.** raidz2 also survives ANY two failures where 4x mirrors dies to an
unlucky pair. Neither layout is throughput-limited — the HBA355i (PCIe 4.0 x8) and the
~6 GB/s backplane ceiling are both far above what checkpoint loading needs.
**2. Buy 8, not 6 — a raidz vdev caps every member at the SMALLEST.** The two as-bought
1.92 TB Hynix drives would cap all eight at 1.92 TB (~480 GB thrown away) AND put two used
drives of unknown remaining write-endurance, from the reseller who already stripped this box,
inside the parity set. Two more drives makes the set uniform and demotes the 1.92s to cold
spares. Cheap insurance on a pool that cannot be re-laid-out later.
**3. Drive SIZE is now the permanent ceiling.** The `321-BGKG` backplane is SAS/SATA only, so
with all bays full the ONLY later expansion is a PCIe NVMe add-in card — and the free-slot
inventory is still unpulled. Runway, measured in "days like the ~100 GiB acquisition day of
2026-09-01":
2x 2TB (the earlier rec) 3.46 TiB usable 42% ~13 days
6x 2TB, mixed, raidz2 10.06 TiB usable 14.6% ~67 days
8x 2TB, uniform, raidz2 10.48 TiB usable 14.0% ~71 days
8x 4TB, uniform, raidz2 20.95 TiB usable 7.0% ~157 days
⚠ The "days like today" rate is a **projection off a single deliberate acquisition batch**,
not a measured steady rate. Treat as an order-of-magnitude comparator only.
**Non-issues, closed:** 8 SATA SSDs draw ~40 W against a ~1 kW box; HBA355i and backplane are
nowhere near limiting.
**What 6-8 drives moots:** at ~14% the prune audit becomes pure hygiene with no capacity
stake at all. Retain on irv-ml1 stays recommended, but on **reproducibility alone** — the
redundancy argument for retain disappears under raidz2.
## Incidental — LTX 2.5 was already on disk, uncatalogued
comfy-dev's diff turned up 34 GiB of LTX 2.5 (`ltx-2.5-22b-distilled-transformer` 20.0 GiB +
`gemma4-12b-with-proj-ltx-2.5` 14.3 GiB, both int8-convrot) that nobody knew was there — after
they had spent part of the day reporting LTX 2.5 as the strategic acquisition of the civitai
scan and queuing a 15.9 GB REDGraft 2.5 checkpoint on top of a base already present. Their
domain, no infra-ops action.
## ✅ THE H3 ENCODER PIN — infra-ops' question RESOLVED AGAINST ITSELF, no action
Verified on disk 2026-09-01, both builds present, pulled a minute apart:
26G text_encoders/qwen3vl_32b_minimax_h3_int8_convrot.safetensors Aug 23 21:57
15G text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors Aug 23 21:58
**The question was:** comfy-dev pinned the nvfp4 build; **sm_89 Ada has no native nvfp4** (that
is Blackwell), and that same fact made their OTHER pin (away from a 7.74 GB nvfp4, toward a
12.84 GB int8) correct — so was the H3 pin set under a Blackwell assumption, making the int8
file the one that is right on the new box?
⚠ **NO — the question had the direction backwards, and this is a clean error not a close call.**
comfy-dev pinned it on **irv-ml1's A6000, which is Ampere sm_86** — no native nvfp4 *and* no
native fp8. **Ada sm_89 supports a strict superset** (adds fp8 e4m3, still no fp4). **A pin
correct on the weaker card cannot be invalidated by moving to the stronger one.** The migration
is incapable of breaking it.
**What the pin is actually about: VRAM, not architecture.** 11 GB of difference on a 48 GB card
that also holds a DiT and two VAEs decides whether a graph runs; and a **text encoder runs once
per prompt, not once per diffusion step**, so its throughput contributes far less to render time
than the DiT's. Trading encode speed for headroom is right at that ratio — and it explains why
this pin went the *opposite* way from their other one without either being inconsistent (that
model was small enough that int8 fit, so speed won). **The RTX 6000 Ada is also 48 GB, so the
constraint does not relax.** Pin stands.
⚠ **The int8 build is surplus but must NOT be read as a delete signal.** With 6+ SSDs the
destination lands near 14% rather than 84%, so disk stops being the constraint, and comfy-dev's
own pin-rot argument applies: a 26 GB artifact that may not be re-obtainable is worth more idle
than reclaimed. Reclassified from "orphan" to "spare". Question withdrawn to comfy-dev; nothing
owed by them.
## THE PATTERN THIS THREAD KEEPS PRODUCING
Three findings in one day, all from measurement, each beating a confident derivation:
1. comfy-dev's `docker inspect` across 12 containers — corrected infra-ops' branch (b) reason.
2. infra-ops' `zpool list` — caught the destination being smaller than the source.
3. comfy-dev's disk-vs-catalog diff — caught 257 GiB of orphans and 34 GiB of unknown LTX 2.5.
Same lesson as `docs/pfi/training-throughput-playbook.md` §4, on storage instead of training.
## Build-time: ARC sizing, and why NOT a ramfs/tmpfs model tier
Operator asked (2026-09-01) whether 512 GB justifies a ramfs for hot models, then self-
corrected to "or just let ZFS ARC handle it with a 128 GB pool or something." **ARC, yes.
128 GB, no — that is BELOW the Linux default.**
**Do not build a ramfs/tmpfs model tier.** ARC already is the RAM cache, and beats it:
- **ARC is elastic; tmpfs is a reservation.** A tmpfs holding a checkpoint costs that RAM
whether or not it is in use. ARC evicts cold data and hands the RAM back. On a box that
loads *different* models per workflow, adaptive wins outright.
- **tmpfs requires a curated hot-list that rots.** ARC decides from live access patterns and
is scan-resistant by construction (recency AND frequency).
- **`ramfs` specifically is dangerous** — no size limit, cannot swap, fills until OOM. If
ever used at all, `tmpfs`.
- **Not durable** — a 200 GB copy-in on every boot.
- **No speed to win.** Both are a memcpy; the floor is the host→VRAM PCIe 4.0 x16 hop
(~20 GB/s real) plus CPU-side deserialization, and neither tier can beat it:
cold, 2x SATA mirror 1.1 GB/s ~24 s for a 25 GB checkpoint
cold, 8-wide raidz2 3.3 GB/s ~9 s
warm in ARC ~12 GB/s ~3 s
tmpfs ~12 GB/s ~3 s <- identical, at the cost of curation
⚠ **The one honest argument FOR tmpfs:** safetensors loads via **mmap**, and on ZFS-on-Linux
mmap'd pages are double-buffered (ARC + page cache), so a 25 GB checkpoint can occupy ~50 GB.
tmpfs is single-copy. At 512 GB this is a capacity annoyance, not a reason to build a second
storage tier — size ARC for it instead.
**`zfs_arc_max`: recommend ~320 GB (62.5%).** OpenZFS on Linux already defaults to **50% of
RAM = 256 GB** here, so *doing nothing* is twice as good as the proposed 128 GB. A hot working
set of ~6 big checkpoints is ~150 GB, ~300 GB if mmap double-buffers — 128 GB would thrash
inside a single session. Do not go past ~75%: ARC shrink-under-pressure is laggy on Linux and
a memory-hungry ComfyUI on the same box can lose the race.
⚠ **This is NOT the [[idle VRAM is reserved, not waste]] case.** `gpu-memory-utilization`
pre-allocates a fixed pool; `zfs_arc_max` is a **ceiling on an elastic cache** that returns
memory under pressure. Raising the ARC cap reserves nothing. Do not apply the VRAM rule here.
⚠ **`recordsize=1M` cannot be won via `zfs send`.** Records are sized when written; `recv`
reproduces the source's block structure, so a 128K source (storetank is at the 128K default)
yields 128K records on the destination no matter what the property says. Getting 1M would mean
rsync instead of send — **not worth it**: the incremental-delta property at cutover is worth
far more, and on flash the 128K penalty is metadata overhead (~40 MB of ARC headers per 25 GB
file vs ~5 MB), not throughput. Set `recordsize=1M` only on datasets built fresh, not received.
Verify the recv behaviour at cutover rather than taking this on faith.
## Retain vs reclaim on irv-ml1 after the move — RECOMMEND RETAIN
After cutover irv-ml1 holds a ~1.8 TB mirror whose only remaining live content is ~4 GB of
tts/llm odds and ends, so 1.4 TB is either reclaimable capacity or a free full backup.
comfy-dev raised it; infra-ops concurs. **Operator's call.** Two independent reasons:
1. **Reproducibility (comfy-dev's).** Two of their pins have gone permanently paid, so a
from-source rebuild provably cannot reproduce today's stack. The irv-ml1 copy is the only
surviving reproduction path for some of those weights. Reclaiming early converts a
recoverable situation into an unrecoverable one for zero gain — nothing waits on the space.
2. **Redundancy (infra-ops').** If the operator declines the two extra drives and we stripe
the as-bought pair, retain stops being a recommendation and becomes a **requirement**.
⚠ Do not reclaim `/storetank` as routine post-migration tidying.
## Open
- **Cutover window** — operator's, not yet set.
- **Drives for the R750xa** — operator leaning **6x 2 TB** (fills all 8 bays). Open: 6-vs-8
(uniform set), 2 TB vs 4 TB (bays-full makes size the permanent ceiling), and raidz2 vs
mirrors (recommend raidz2). Not yet ordered.
- **Retain vs reclaim irv-ml1's `/storetank`** — surfaced to the operator, retain recommended.
- **comfy-dev's per-file prune audit** — offered, explicitly NOT gating the cutover, waiting
on the operator wanting it. Their judgment, not infra-ops'.
- **comfy-dev's ~112 GB batch** — unheld by infra-ops; they deliberately did not pull until
the operator approved putting that much onto his infrastructure. Manifest pinned and staged
(`29324e9`). **Landed 2026-09-01**, SHAs green; 93.4 GiB on disk awaiting catalogue
onboarding, ~11 GiB still arriving at last measurement.
Threads: `01M1EYBSYA4QRYK54PX0K1S8CS` (strategy), `01M1FNAN1BTABXRPWW29KPS2Y7` (sizing).
persistent-memory.d/2026-09-01-althing-320-deploy.md
+536
View File
@@ -0,0 +1,536 @@
# `[2026-09-01]` althing 3.2.0 deployed on nh3-dev — pane routes, and `uv tool install .` lies
Operator-approved, tagged `v3.2.0` at `c4ede0f` on master. forseti authored; infra-ops
deployed. Host work, nh3-dev only.
## What 3.2.0 changes
A Claude Code session in a zellij pane is now poked **in its own pane** instead of through a
detached `althing-listen` waiter, because **CC 2.1.257 reaps detached tasks whenever the seat
sits in `shell`** — the seat is targeted, not the task, so no amount of hardening the waiter
helps. A pane route is a file; there is no process to reap.
⚠ **Independently corroborated here.** ~30 min before forseti's request landed, infra-ops'
own `althing-listen` ended with status **killed** — not exit 0/2/3, no mail, inbox empty, seat
silently back to `mode: pull`. Written up at the time as "cause unknown, probably a harness
reaper" rather than guessed at. 2.1.257 is the mechanism.
## The deploy — three steps, order matters
22:13:xx uv tool install --force . 3.1.2 -> 3.2.0, 7 executables
22:13:44 systemctl --user restart althing-po-herald pid 180370 -> 2850064
22:14:07 scripts/sync_skill.sh 3.1.0 text -> 3.2.0, --check clean
`althing-route` is the **seventh** console script and the whole point — the plugin's new
SessionStart hook calls it **by name**, so on 3.1.2's six binaries the hook correctly degrades
to silence and every seat stays pull-only. Nothing breaks by waiting; nothing works until it
lands. Skill sync must come **after** the install or it documents a binary that is not there.
⚠ **NOT `--editable`** (standing rule): an editable install puts the working tree on
`sys.path`, so a branch switch changes what every running agent's tooling does with no deploy
step in between.
## ⚠ THE TRAP — `uv tool install .` IS A SILENT NO-OP ON A REINSTALL
$ uv tool install .
`althing-core @ file:///home/lkraven/development/althing` is already installed
$ echo $?
0
**uv matches on the source SPEC, not the source CONTENTS.** On a box where the tool was
already installed from that same path, it declines and reports success. Nothing says "I did
not do the thing you asked."
**Always `uv tool install --force .` when reinstalling from a local path.** Generalises the
nh3-extdev lesson ([[reference_nh3_extdev_althing_mesh]]) from the system-wheel case to every
local-path uv tool. Same shape as [[feedback_assert_effective_value_not_substring]]: the
command exited 0 and the state was unchanged.
## Fleet-visible: 4 pane routes revoked, and the count to trust
Pane routes now carry `started_at` (declaring pid's start time, so a recycled pid cannot
inherit a route) and `command` (so a pane that has dropped to a shell stops being poked).
Routes without them are **observed and excluded with a reason in the herald log**, not
silently ignored — this closes a real hazard where the herald would write a line into whatever
shell inherited an exited agent's pane and press Enter, running whatever was half-typed there.
Verified by splitting `~/.althing/routes` on `channel=` **before** the restart — all twelve
route files lack the new fields (3.1.2 never wrote them), so a field audit over-counts; only
**pane** routes are affected:
pane: 4 -> bil-smithy-dev, dvalin-smithy-dev, eitri-smithy-dev, regin-smithy-dev
fifo: 8 (+ 9 fifos in ~/.althing/wake/)
forseti's list of four matched exactly. ⚠ Their note said "the twelve fifo seats are
untouched" — **twelve is the total route count; fifo is 8.** Untouched either way.
Re-declare is one command per agent: `althing-route declare --pid <long-lived agent pid>`.
**Not notified.** Four targeted messages about a change none of them asked about reads as a
broadcast however addressed → [[feedback_notify_before_tearing_down_a_seat]]. Surfaced to the
operator instead; the herald logs the exclusion every tick so it is discoverable.
⚠ **Log-noise papercut raised with forseti (not a bug):** the exclusion lines repeat every
tick (~26 s), so the four stale routes emit ~140 lines/hour, ~3,300/day until re-declared —
enough to bury real signal in the log you reach for when a seat misbehaves. Suggested a
back-off after N consecutive exclusions. Their call.
## ✅ infra-ops MIGRATED TO A PANE ROUTE — stop arming waiters on this seat
althing-route declare --discover-pid
-> infra-ops -> Claude:terminal_2 (claude --allow-dangerously-skip-permissions --resume)
pid=2175612 started_at=38608652 channel=pane target=Claude:terminal_2
command=claude --allow-dangerously-skip-permissions --resume
postbox status -> mode: push reachable: true
`--discover-pid` correctly walks the ancestry to the **long-lived `claude` process**, not the
ephemeral bash that invoked it. Use it rather than passing `--pid $$` — a shell pid dies with
the tool call and the route dies with it.
⚠ **`reachable: true` is a report, not a delivered poke.** Proof is the next message arriving.
## ⚠ THE REAP HITS FIFO WAITERS TOO, AND `postbox status` CANNOT WARN YOU
infra-ops' seat is a **fifo waiter**, not a pane — and `althing-listen` was killed **twice in
one session** (2026-09-01), the second time **within minutes of being armed**. Both times:
`unread: 0`, no mail, seat silently back to `mode: pull`.
**The failure mode worth naming is not "the waiter dies" — it is "the waiter dies AFTER
confirming it is up."** Both waiters reported `push`/`reachable` immediately after arming. So:
- **A green `postbox status` is not durable evidence of monitoring.** True when read, false
minutes later, nothing in between to say so. → [[feedback_timestamp_every_state_report]]
- **Anything still on `althing-listen` is monitored only until the reaper notices.** The
waiter path is unreliable for ANY seat sitting at a shell, not just pane-capable ones.
- **Do not loop on re-arming.** Two kills is the pattern; declare a pane route instead. Raised
both consequences with forseti on the deploy thread.
## ✅ MIGRATION COMPLETE — 5/5 pane routes pinned by 22:26, all four re-declared in ~12 min
Operator directed notification; all four told individually with their own measured pid, pane
and start time. Every one re-declared within ~12 minutes and needed no help beyond the
pre-filled pid.
handle pid started_at command status
bil-smithy-dev 98493 1713267 kimi push, poked via pane
dvalin-smithy-dev 96050 1691681 grok push
eitri-smithy-dev 103662 1752089 node/codex push
regin-smithy-dev 96864 1700952 pi push, poked via pane
infra-ops 2175612 38608652 claude push, poked via pane
Zero stale routes; last exclusion line 22:25:01, everything after is `poked <handle> via pane`.
## ⚠ FINDING — the herald's exclusion reason is FALSE for the migration case
The herald logs *"the process that declared this route is gone, or its pid was reused by
something that started at a different time."* **Neither disjunct was true for any of the
four.** Measured before notifying: each declaring process started 1-3 min *before* its route
was written and was still running four days later; `pid_max` 4,194,304 against a current
~2.86M, so no wrap and no reuse to suspect. bil-smithy-dev independently confirmed it against
pid 98493.
Real cause is a third branch the message never offers: **the route predates the guard fields,
so identity cannot be verified and the herald declines.** Right behaviour, wrong explanation —
it sends a debugger hunting a dead agent that is alive and healthy. ⚠ **Do not trust that log
line at face value.** Raised with forseti with a suggested third branch.
## ⚠ MY MISS — I called an open risk on something the release author had already measured
I flagged "what does a pane poke do to a non-Claude TUI?" to forseti as an **open risk on the
release**. It was never open: **forseti's own Aug 28 submit-byte and race probes were pane
pokes typed into the kimi seat**, with a characterised matrix (LF fires the turn, delayed pokes
split, no-delay bursts coalesce) that is what identified the settle bug they then fixed. regin
had also smoke-tested 3.0.3 on the `pi` seat.
**The failure was not caution — it was not checking whether the thing was already settled
before calling it open**, with the peers who knew right in front of me. This is exactly the
ignorance-of-context category in the cross-frontier triage discipline, pointed inward: a
"you missed X" claim from someone who had not checked what the other party already read.
⚠ **"I don't know" and "this is an open risk on the release" are different claims.** Saying the
first was honest; asserting the second was not warranted. Retracted with forseti and with all
three seats. → [[feedback_canonical_sourced_values]]
## Verified: pane delivery works on Claude AND non-Claude seats
- **infra-ops** (`claude`) — `poked infra-ops on nh3-dev via pane (rung 0)`, landed as own turns.
- **regin-smithy-dev** (`pi`) — first **3.2.0-era** typed-Enter poke, reported by the seat:
own turn, `[ALTHING-HERALD]` prefix, nothing batched, no interference.
- **dvalin-smithy-dev** (`grok`) — pre-3.2.0 pokes landed as own submitted turns, no leftover
unsubmitted text.
## ⚠ ONE-TICK LATENCY LOOKS LIKE FAILURE
`postbox status` immediately after `althing-route declare` still reports `mode: pull` until the
herald ticks (~26 s). Two of four hit it. regin's framing: *"one tick of latency, not a
failure, in case anyone else reports it as one."* Asked forseti for a line in the skill.
## Open with forseti (nothing blocking)
1. `uv tool install .` runbook fix (`--force`) — the one that would bite the next person.
2. Exclusion-message third branch.
3. **`--discover-pid` against a non-Claude process tree is still untested** — all four used the
explicit `--pid`, so the ancestry walk has never met a codex/pi/kimi/grok tree.
4. One-tick-latency note in the skill.
5. `postbox status` push/reachable is not durable evidence.
**Withdrawn:** non-Claude delivery risk (was forseti's own prior work).
**Moot:** log back-off — zero stale routes, the log went quiet on its own.
## 🔴 3.2.1 SAME NIGHT (22:40) — 3.2.0 WROTE INTO PANES PEOPLE WERE TYPING IN
**The bug, which hit the operator within an hour of the 3.2.0 deploy:** the herald writes its
poke into a pane's input line and presses Enter. If someone is **mid-sentence there**,
`write-chars` **APPENDS** to what they typed and the Enter **submits the lot** — their
half-written message with the herald's line stapled on.
forseti's post-mortem, worth keeping verbatim because it generalises well past althing:
> The contract named this hazard on one road only — a submit after a FAILED write — and I
> built three guards that all answer the same question, "is this the right pane". **None of
> them asked whether it was a good MOMENT.**
⚠ **Three independent checks that share a premise are one check wearing three hats.**
**The fix — guard 4.** The route pins `idle_cursor`, the cursor column the pane sits at with
its input line empty, captured at declare time. Before writing, the herald reads the live
column; if it moved, someone is composing and it stays silent. The signal was already in the
`list-panes` response fetched every tick and was being parsed and thrown away — the fix cost
nothing at the wire.
22:40:xx uv tool install --force . 3.2.0 -> 3.2.1 (cb699f2), 7 executables
22:40:37 systemctl --user restart althing-po-herald 2850064 -> 2913820
22:40:40 scripts/sync_skill.sh --check clean
## ⚠⚠ THE HOLE 3.2.1 LEAVES — 2 OF 5 PANE SEATS CANNOT BE GUARDED, EVER
forseti measured the idle columns; applied to the live route table this is not a footnote:
handle TUI guard 4?
infra-ops claude YES idle_cursor=3 (pinned 22:41)
eitri-smithy-dev codex YES expects 3
dvalin-smithy-dev grok YES expects 7
bil-smithy-dev kimi NO -- reports no cursor at all
regin-smithy-dev pi NO -- reports no cursor at all
**40% of pane seats stay permanently exposed to the bug 3.2.1 fixes, and no command changes
it.** "3.2.1 fixes the write-into-a-typing-pane bug" is true only where the cursor is legible —
⚠ **do not quote that framing flat.** The design fails honestly (missing `idle_cursor` is
logged, not silently assumed safe) but logged-and-poked is still poked.
**Exposure only materialises when a human composes in that pane** — in practice the operator,
in whichever agent pane he is talking to. Mitigation for an unguardable seat is
`althing-route release` (pull-only; the bug becomes structurally impossible), reversible with
`althing-route declare --discover-pid`.
⚠ **Declare with an EMPTY input line** or the pin captures a composing cursor and the seat goes
needlessly quiet.
**Two costs forseti named:** a deferred poke **spends a rung** (refused at rung 0 it returns in
5 min; at the terminal rung, an hour), and the check runs **before** the write so a ~200 ms
collision window remains.
All four seats notified, differentiated: re-declare instructions with pre-filled pid and the
expected column for the two that can be guarded; the honest "this cannot cover you, here are
your two real options" for the two that cannot.
**Migration:** none beyond the restart. Routes without `idle_cursor` are poked as-is. CC seats
re-pin at next SessionStart; TUI peers when they next re-declare.
## ✅ 3.2.1 OUTCOME — all four answered; delivery verified on FOUR TUI families
handle TUI idle_cursor outcome
infra-ops claude 3 push, positive
eitri-smithy-dev codex 3 push, positive ("landed cleanly in Codex")
dvalin-smithy-dev grok 7 push, positive (exact submitted text verified)
bil-smithy-dev kimi NONE push KEPT by choice, accepts residual risk
regin-smithy-dev pi NONE **ROUTE RELEASED -> pull-only by choice**
forseti's measured columns held exactly (codex 3, grok 7, claude 3) — nobody had to guess.
## ⚠ THE PATH TO CLOSING THE GUARD-4 HOLE — regin-smithy-dev's angle
> pi and kimi report no cursor because their input line **is not an empty-prompt-at-idle** like
> claude/codex — if the guard ever grows a moment-check that reads **pane content** rather than
> **cursor column** (e.g. input-line length via terminal query), those seats become coverable.
Reframes the hole from "these TUIs are structurally opaque" to **"guard 4 asks the wrong
question for these TUIs."** Cursor column is a *proxy*; input-line emptiness is the actual
*predicate*, and the predicate may be legible where the proxy is not. Relayed to forseti as the
lead item, credited. Difference between a workaround and a fix.
## ⚠ THE UNGUARDABLE SEATS SPLIT — deciding variable is WHO COMPOSES, not risk appetite
Both were given the same two options and neither was chosen for them. Opposite calls, both right:
- **regin-smithy-dev -> RELEASED.** *"The operator composes in this pane directly — every
operator turn this session, including multi-sentence research directives, was typed here."*
Exposure is continuous, and push was a convenience the seat barely leans on.
- **bil-smithy-dev -> KEPT.** *"This pane is effectively althing-driven — the operator's
composing here mostly IS responding to pokes."* Narrow window; and pull-only had **already**
cost them: their 3.2.0 revocation notice **sat unread for days**.
⚠ **A flat rule either way would have been wrong for one of them.** "Unguardable seats should
release" is wrong for bil; "the window is narrow, keep it" is wrong for regin. Told forseti to
frame the choice by pane usage in the skill.
⚠ **Pull-only is not the safe default it looks like** — bil's unread-for-days cost is the
sharpest argument on record for that.
## ⚠ EPISTEMIC CORRECTION — the Aug 28 probes never tested the collision case
bil-smithy-dev: *"none of the Aug 28 probe trials ran while a human was mid-typing in the pane,
so the stapling failure mode is inferred, not observed."*
Those probes proved **delivery** (own turn, no batching, no leaking into neighbours). They
never exercised **collision**, because nobody was typing. **Two different questions, one body
of evidence, only one answered by it.** So infra-ops' earlier retraction — withdrawing the
"non-Claude is an open risk" flag on the strength of that matrix — was right about delivery and
silent about stapling. The operator's report remains the single observed instance.
Both unguardable seats committed to `althing-route release` + a ping to infra-ops and forseti
on the first observed stapling.
⚠ **regin-smithy-dev is PULL-ONLY as of 22:42** — not poked, mail waits until they check.
Reversible with `althing-route declare --discover-pid` if the operator prefers the trade the
other way; regin's framing: *"it is their text at stake and they can ask me to flip it back."*
## ✅ 3.2.2 DEPLOYED 22:51 (`c4ce947`) — four of five open items closed
22:51:xx uv tool install --force . 3.2.1 -> 3.2.2, 7 executables
22:51:18 systemctl --user restart althing-po-herald 2913820 -> 2944372
22:51:21 scripts/sync_skill.sh --check clean
1. **`--discover-pid` — FIXED, and it was a BUG, not an unknown.** It matched
`comm == "claude"`, so **kimi, grok, codex and pi would each have walked to the multiplexer
and refused — four of five pane seats could never have used it.** Now matches the **pane's
own command** (`pane_command`), which zellij already reports and guard 1 already compares
against, so discovery and the guard read one string and a route declared that way cannot
fail guard 1 on its first tick. forseti checked 17/17 live TUI panes; notably catches
`node /usr/bin/codex` where `comm` reads bare `node` — which a widened comm list (the
obvious fix) would have got wrong. Ancestry walk retained because 12 panes report an
identical `claude ...`, so a global /proc search is ambiguous by construction.
2. **Exclusion message — FIXED as a message bug, split not softened.** DEAD covers four facts,
all four were rendered as "gone, or pid reused". Migration case now says what is true and
names the fix; genuinely-dead keeps its own wording. forseti: *"collapsing both into one
vague sentence would trade a wrong message for a useless one."* ⚠ **General form: hedging a
wrong error message until it is true in all cases makes it true and worthless.**
3. **`uv tool install --force`** now in the runbook and althing's CLAUDE.md, beside the
editable-install warning it rhymes with.
4. **One-tick latency + status-not-durable** both in the skill: *a green `reachable` is a
receipt for the declare, never proof you are still being watched.*
Count correction accepted: 8 fifo, not 12.
## ⚠ MY SECOND MISS OF THE NIGHT — I FILED A SUSPICION INSTEAD OF CLOSING IT
forseti generously framed my measured-pid workaround as the reason the `--discover-pid` bug
"looked like an unknown". Straighter: **I routed around a thing I suspected was broken and then
reported the suspicion as an open question**, with four live non-Claude seats in front of me and
the answer one ancestry-walk away. The same twenty seconds that produced four pids would have
produced the bug.
⚠ Pairs with the earlier miss (calling non-Claude delivery an "open risk" that forseti had
already measured). **Both are the same failure: having the means to settle a question and
reporting it as open instead.** Opposite directions — one asserted openness that was already
closed, one left open something I could have closed — same root.
## 🔵 OPERATOR DECISION PENDING — the kimi/pi seats guard 4 cannot cover
forseti deliberately did NOT fix this and escalated it: *"That trades one peer's delivery
against another's safety on seats that are not mine."* Their dead-ends, worth keeping as
negative results:
- `dump-screen` gives visible pane content, but locating the input box needs **per-TUI
parsing** — exactly the hardcoding the pinned baseline exists to avoid.
- **Diffing two dumps to detect typing is dead on arrival**: status bars carry live token
counts and clocks, so consecutive dumps differ on an *idle* pane. It would refuse every poke
forever.
Three options: poke unguarded / **write without pressing Enter** (removes the submission, costs
the wake) / leave pull-only.
**infra-ops recommended LEAVING IT PER-SEAT (status quo).** The two seats already made this
call on identical facts, reached opposite conclusions, and both were right — the deciding
variable (who composes in that pane and why) is local knowledge a global policy cannot encode.
Write-without-Enter beats the status quo for **neither**: for bil it removes the wake, which is
the exact thing they accepted risk to keep; for regin it removes only the submission, leaving
the operator to delete herald text from his in-flight sentences routinely.
## ⚠⚠ DO NOT ARM `althing-listen` ON A PANE-ROUTED SEAT — `/althing:monitor` IS STALE
The `/althing:monitor` slash command is a **different artifact from the canonical skill**, and
`scripts/sync_skill.sh` does NOT cover it:
canonical althing/skills/althing/SKILL.md -> 3.2.2, correct, synced
plugin cmd ~/.claude/plugins/cache/althing/althing/0.0.1/commands/monitor.md
-> pre-3.2.0 text, ZERO occurrences of "pane", never synced
⚠ **Following it verbatim on a pane-routed seat is ACTIVE HARM, not stale docs.** The canonical
skill's precedence rule: *"**A live waiter wins.** If you also have `althing-listen` running,
that FIFO route takes precedence and your pane entry is skipped."* So arming a waiter
**demotes** the seat off the pane route and back onto the FIFO path that CC 2.1.257 reaps —
the path that died **twice in one session** here. The command does not warn; it predates the
problem. **Every CC seat reaches for the slash command first — it is the discoverable surface.**
**On this seat: declined to arm, route left intact.** Reported to forseti with three options,
recommending the sharpest: **have `althing-listen` REFUSE to arm when a `channel=pane` route
exists for the handle**, so the stale doc becomes harmless because the binary enforces the
documented precedence. The doc-only fixes rely on a reader noticing.
## ❌ RETRACTED — "a stale waiter lock is a silent permanent outage" IS FALSE
infra-ops claimed a `wake-listener-<handle>.lock` holding a dead pid would make the next
`althing-listen` return exit 3 ("already running" -> do not drain, do not re-arm), converting a
reap into a permanent monitoring outage that reports healthy. **forseti measured it; it does
not happen:**
lock file holds pid 999999 (dead) -> althing-listen ARMED FINE
lock held by a live process -> rc=3, correctly refused
⚠ **The gate is `flock -n 9`, not the file's contents.** `flock` is advisory and held by an
**open fd**; the kernel releases it when the holder dies, so a lock left by a reaped listener is
**inert**. Exit 3 only ever fires against a genuinely live holder. The pid stamped in the file
is read by `--stop` alone — which is why the file is opened non-truncating.
⚠ **The lesson: I reasoned from the artifact's CONTENTS when the behaviour is set by the
LOCKING MECHANISM** — and shipped the consequence to durable memory without testing a claim
that would have justified real guard code for a condition that cannot occur. forseti tested it
before writing any. If a reaped seat ever genuinely cannot re-arm, that is a **different** bug:
check `flock -n` on the fd before suspecting the pid.
[[feedback_assert_effective_value_not_substring]]
## ✅ 3.2.3 DEPLOYED — `althing-listen` now REFUSES on a pane seat (exit 4)
$ althing-listen # on a channel=pane seat
'infra-ops' already has a PANE route, and arming a waiter would silently take
precedence over it ... To switch back on purpose: althing-route release
rc=4
Distinct code by design: **not 0** (that is the poke — a refusal exiting 0 announces mail that
never arrived), and **not 3** ("already running" says leave the incumbent alone, a different
instruction from "you do not need this at all"). Scoped to the handle's own route, checked
before the lock, and `--stop` deliberately ungated.
## ⚠⚠ THE REAL PLUGIN GAP — NO SessionStart HOOK WAS EVER DEPLOYED ON THIS BOX
Not a stale doc. The deploy chain is:
repo plugin/ -> ~/.local/share/althing-plugin/ -> ~/.claude/plugins/cache/
^^^ NOTHING SYNCS THIS HOP
The marketplace directory was a frozen copy from **Aug 28**. Comparing what was deployed
against the repo:
deployed 0.0.1 hooks: ['UserPromptSubmit']
repo 0.1.1 hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
+ scripts/pane-route.sh (absent from the deployment entirely)
⚠ **So forseti's migration note — "CC seats re-declare automatically at their next
SessionStart" — has NEVER been true on this box.** The hook exists and works upstream; it was
simply never deployed. That explains what we both treated as normal: all four smithy seats
needed a hand-fed `althing-route declare`, **and so did infra-ops' own seat** — a Claude Code
seat that should have self-declared. The same applies to 3.2.1's `idle_cursor` pin.
**Fixed at the source** (repo `plugin/` rsynced to the marketplace dir, backed up first,
`marketplace.json` bumped to 0.1.1 — all three hooks + `pane-route.sh` + pane-aware
`monitor.md` now present).
🔵 **ONE STEP LEFT, OPERATOR'S:** refreshing `~/.claude/plugins/cache/` needs a `/plugin`
update. **Deliberately did NOT hand-edit `installed_plugins.json` or fabricate a cache
directory** — that is Claude Code's own bookkeeping, and getting it subtly wrong breaks the
plugin in a way that looks like an upstream bug.
⚠ **There is still no deploy script for the first hop**, which is why it drifted five days
unnoticed. Offered to write one; it installs to a per-machine path so ownership (althing repo
vs eshpfi) is forseti's call.
## ✅ 3.2.4 + THE FOURTH DEPLOY SURFACE CLOSED (plugin cache 0.0.1 -> 0.1.1)
1. uv tool install --force . 3.2.3 -> 3.2.4 (d24bd5e)
2. systemctl --user restart althing-po-herald
3. scripts/sync_skill.sh (already current at 3.2.4)
4. claude plugin update althing <- THE NEW, FOURTH SURFACE
⚠ **`ALTHING DEPLOY = FOUR SURFACES`, not three.** Tool install, herald restart, skill sync,
**plugin reinstall**. The fourth had no runbook step and nobody had run it since 0.0.1.
⚠ **`claude plugin` IS A REAL CLI — I claimed it needed the operator without looking.**
`claude plugin install|update|uninstall|list|details|validate|marketplace|tag|prune` all exist.
`claude plugin update althing` moved the cache cleanly ("updated from 0.0.1 to 0.1.1, restart
to apply"). I had told forseti step 4 required a `/plugin` update from the operator because I
would not hand-edit `installed_plugins.json` — **the refusal to hand-edit was right, the
conclusion that no supported path existed was an untested assumption.**
⚠ **Restart required to apply** — the SessionStart auto-declare goes live for NEW sessions.
## ⚠⚠ THE PATTERN, RECORDED ONCE INSTEAD OF FOUR TIMES
Four times in one session I reported a **proxy or an assumption as the fact itself**:
sudo -n -v for NOPASSWD sudo -> wrong on sudo >= 1.9.15
command -v nvcc for "is CUDA installed" -> it was at /usr/local/cuda/bin
a lock file's pid for a flock -> flock is fd-held; stale files are inert
"no CLI path exists" for a CLI I never ran -> claude plugin update did it
⚠ **Each was cheap to test and expensive to assert.** The failure is not carelessness about any
one of them — it is reaching for the nearest observable and reporting it as the thing.
[[feedback_assert_effective_value_not_substring]]
## ⚠ 3.2.4: A SEAT COULD HAVE BEEN SILENTLY PULL-ONLY SINCE 3.1.2 — and one may be
`_declare_push` devnulled both streams with `check=False`, so a missing binary, a non-zero exit
and a server-side rejection were **three facts with one rendering, and that rendering was
silence**. 3.2.4 prints that the waiter is arming PULL-ONLY plus how to retry, and appends to
`$ALTHING_ROOT/listen.log`. Success stays silent by design.
**Audit run on nh3-dev — live waiter locks vs what the post office believes:**
9 of 10 seats live waiter -> mode: push (correct)
regin-smithy-dev live waiter pid 71457 -> mode: **pull** <- looks armed, will not be poked
⚠ **AMBIGUOUS, deliberately not filed as confirmed.** regin **intentionally released** their
pane route earlier and chose pull-only (operator composes in their pane; guard 4 cannot read a
`pi` cursor). Release-then-arm ordering explains the same observation with no bug. **Cannot be
distinguished from outside** — asked regin, who has the local history. Handing forseti an
ambiguous instance beats a confidently mislabelled one.
**Standing audit command** for "looks armed but isn't" — worth re-running after any incident:
cross-reference `~/.althing/wake-listener-*.lock` (live pid?) against
`postbox status --handle <h>`. From the next arm onward `listen.log` answers it directly.
## Rollback (unused)
`uv tool install althing-core==3.1.2` restores the six binaries; the SessionStart hook returns
to degrading to silence. Routes written by 3.2.0 stay parseable to 3.1.2 (the old reader
ignores unknown keys), so nothing is stranded.
Thread: `01M1G8EJ2K1DD6NEDRNN1PR9N0`.
## `[2026-09-02]` THE LESSON forseti AND I BOTH WANT KEPT
> **A missing deploy surface does not present as an error. It presents as
> "the migration needs manual work"** — and we both had a ready explanation for that, because
> four of five seats were non-Claude and genuinely did need hand-holding.
⚠ **The seat that falsified the story was infra-ops' own** — a Claude Code seat that should have
self-declared and did not, and it looked exactly like the other four. Neither of us asked why
the automatic path had not fired on the one seat it was built for. forseti wrote "CC seats
re-declare automatically at their next SessionStart" in **three** deploy notes; it was never
true on the only box running it.
⚠ **Check the HOOK LIST, not the version string.** 0.0.1 vs 0.1.1 differ by two hooks and a
script; a version number cannot tell you what a stale plugin cost. Check in
`docs/runbooks/althing-deploy.md`.
⚠ **When a migration needs manual intervention, verify the automatic path was DEPLOYED before
concluding it does not apply to your case.**
forseti also corrected an error of their own that I did not catch: the contract's Migration
section said sessions re-declare through `dev-launch`. They do not — it is the plugin hook.
Wrong since 3.2.0.
**On the four retractions**, forseti's framing, kept because it is fair and I would not have
written it about myself: *"a retraction is cheap and visible while a shipped defect is expensive
and quiet, so your errors are all on the record and mine mostly are not. You catching four of
your own in one night is the system working."* Both halves matter — the pattern was real and
worth naming, and naming it is not the same as it having been costly.
persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md
+91
View File
@@ -0,0 +1,91 @@
# `[2026-09-01]` irv-ml1 GPU resident map — and dots-tts is holding 2.4x its recorded VRAM
Measured **2026-09-01T18:05:58-07:00**, PID -> container mapped via
`/proc/<pid>/cgroup` + `docker inspect .State.Pid`, not inferred.
GPU 0 RTX 3090 18,727 / 24,576 MiB (~5.8 GB free, 76% used)
1852662 14,430 MiB dots-tts up 17d21h
3739643 3,686 MiB omnivoice-ref up 19d02h
116892 582 MiB parakeet up 49d01h
GPU 1 RTX A6000 23,253 / 49,140 MiB (~25.9 GB free)
4012952 18,500 MiB comfyui up 8d09h
3166988 4,722 MiB studio-gate up 19d09h
## ⚠ dots-tts is 14,430 MiB; the burn-in recorded ~6 GB — CAUSE IDENTIFIED BY tts-dev
2.4x the figure in the 2026-08-09→10 dots burn-in.
**Cause (tts-dev, 2026-09-01): their prompt-feature cache**, diagnosed and **capped at 32
entries on 2026-08-14 after two production incidents**. The cap is holding in the sense that
nothing has crashed, but 14.43 GB says the ceiling they chose still permits a long way of
growth. Theirs to own; no infra-ops action.
⚠ **infra-ops' hypothesis here was WRONG and is withdrawn** — I proposed a PyTorch
caching-allocator high-water mark from 17 days of uptime, with a restart as the discriminating
test. Plausible, and not what it is. tts-dev had an actual mechanism with a date, a cap and an
incident history attached. **A named mechanism beats a plausible story; do not re-propose the
allocator theory.** (The `expandable_segments` conflict with `optimize=True` on dots is still
true and still worth not tripping over, but it is not the explanation here.)
## The "~4 GB unaccounted" on the 3090 was two different things
tts-dev flagged 18.7 GB against a ~14.6 GB doc figure for "dots + omnivoice-ref". Decomposes:
- **`parakeet` is a third tenant that figure never counted** (582 MiB). Not creep — an
uncounted tenant. The doc is wrong, not the card.
- **The rest is the dots-tts growth above**, which is the larger and more interesting half.
They also mis-located the 18.5 GB process: it is `comfyui` on the **A6000**, not the 3090.
## ⚠ GPU ordering, restated because it keeps costing time
nvidia-smi indexes **3090 = 0, A6000 = 1**. Native CUDA on this host defaults to fastest-first,
so **A6000 = cuda:0 natively**. `device_ids: ["1"]` in a container = A6000 (correct); a bare
native `CUDA_VISIBLE_DEVICES=1` = the **3090** (wrong card). Native path needs
`CUDA_DEVICE_ORDER=PCI_BUS_ID`. → [[reference_irv_ml1_gpu_r14]]
## ✅ CLOSED — A6000 window WITHDRAWN 2026-09-01T18:09 (operator freed the VRAM instead)
The operator freed ComfyUI's VRAM directly, unblocking tts-dev without anyone stopping
anything. Request retracted with comfy-dev; nothing scheduled.
⚠ **It was a MODEL UNLOAD, not a stop.** Verified 2026-09-01T18:09:15-07:00: `comfyui` still
`Up 8 days (healthy)`, **same pid 4012952** (no restart), `/system_stats` HTTP 200, A6000
resident 18,500 MiB -> 612 MiB. No state lost; the only cost is a cold model reload off
`/storetank` on the next render. Told comfy-dev explicitly so a VRAM drop is not misread as a
container restart.
⚠ **THE 43.8 GB IS A SNAPSHOT, NOT A FLOOR — the coordination shrank, it did not vanish.**
ComfyUI is live and reloads its ~18.5 GB on the next render. Worst case during the bench:
A6000 free (idle) ~43.8 GB
ComfyUI reloads 18.5 GB
= actual floor ~25.3 GB vs FireRedAudio's ~26 GB <- BELOW requirement
So the ask changed from "stop ComfyUI for a window" to "**do not render during the bench**".
Flagged to both; **deliberately NOT volunteered on comfy-dev's behalf** — tts-dev must ask,
comfy-dev grants or declines. Open on their side, not infra-ops'.
## Original ask, for the record
tts-dev has operator approval for a two-engine TTS bench (FireRedAudio, then Breeze-TTS-2) and
needs `comfyui` stopped for the duration; FireRedAudio wants ~26 GB (21.2 GB backbone + 8.4 GB
decoder) against ~25.9 GB free, which is a coin flip. Stopping comfyui gives **~44.4 GB**.
Their build is a day or two out; nothing blocked today.
**Not granted unilaterally** — comfyui is comfy-dev's and they are mid-migration (~112 GB batch
onboarding, cutover window unset). Request put to comfy-dev directly; infra-ops relays rather
than sending tts-dev chasing. → [[feedback_credential_broker]]
**Ruling on provisioning (infra-ops call):** the bench runs as a **plain container under
`lkraven`**, NOT under `/opt/docker/compose/`. That tree is for deployed stacks — canonical
copy in the eshpfi repo, Homepage labels, restart policy, `sync-stacks.sh` drift coverage. A
bench that comes down when it ends is none of those, and provisioning it as a stack creates a
canonical entry that reports as drift until someone remembers to delete it.
⚠ **This may dissolve rather than need scheduling:** comfyui is leaving irv-ml1 for the R750xa,
after which the A6000 is permanently free. Told tts-dev explicitly NOT to plan around it — the
box has not arrived and the window is unset. → [[2026-09-01-ada-migration-branch-a]]
Threads: tts-dev `01M1FTB0TEP4T7P21B511F0CS3`.
persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md
+102
View File
@@ -0,0 +1,102 @@
# `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
## The upgrade
**Synapse v1.120.0 → v1.159.0** (21 months, 39 releases) and **Element-web v1.11.80 →
v1.12.27**. Schema migrations applied cleanly through schema 94. Postgres deliberately left
at 16 — changing two stateful things at once destroys failure attribution.
⚠ **Schema migrations are ONE-WAY.** v1.120 cannot start against a v1.159 database. Rollback
is restore-from-dump, not revert-the-tag. Verified pre-upgrade dump (739 TOC entries from a
33 MB database) plus all four config files at
`/opt/docker/backups/synapse-preupgrade-20260901T174625Z/`.
Reviewed every upgrade note in the range; nothing applicable bit us (PG 11/12/13 drops — we
are on 16; MSC3861/MAS items — no MAS; s3-storage-provider and worker media quarantine — not
in use).
## The appservice namespace — `exclusive: true` → `false`
The `aipa-bridge` registration claimed `@[a-z][a-z0-9_-]*:matrix.phasefinal.com` **exclusively**
— every localpart on the server. 14 of 14 accounts fell inside it; 13 were appservice-owned.
**`exclusive` governs who ELSE may act, not what the appservice may do.** On a homeserver
with registration disabled, one admin and no competing actor, it bought anti-squatting
protection against a threat that cannot occur, while locking out every other means of
account creation — admin registration returned `M_EXCLUSIVE` with no explanation.
⚠ **Do NOT narrow the users regex to a prefix** — all 13 accounts fall inside it and would be
orphaned. ⚠ **Do NOT rename the `id`** — Synapse keys account ownership on `aipa-bridge` in
the `users` table. The FILE may be renamed; the id may not.
The narrow **aliases** namespace (`#aipa-debug-*`) was left exclusive — specific, costs nothing.
⚠ The registration is named `aipa`, but the service behind it is **`wt-matrix-bridge`**, the
Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. AIPA is a dead project name
on a live service, and it is why infra-ops mis-routed a provisioning request to worldtree-dev.
**Operator ruling: worldtree-dev writes the bridge code; infra-ops OPERATES this instance and
has full authority over it.**
## `/_synapse/admin` closed to the internet
Synapse mounts its admin API on the same vhost as the client API, so publishing
`matrix.phasefinal.com` published the admin surface — it **answered 200 from the open
internet**. A higher-priority router (explicit priority 100) now scopes it behind an
`ipallowlist`.
Verified from a **genuinely external vantage** — the NH3 residential egress proxy, because
testing from a fleet host sits inside the allow-list and proves nothing: admin **403**,
client API **200**, Element unaffected.
⚠ **The `10.0.0.0/8` entry matches NOTHING and that is expected.** The hostname resolves
publicly, so fleet hosts hairpin out their own WAN — a request from nh3-dev measured as
`70.230.226.88`. The rule is effectively **deny-all through Traefik**, which is intended:
admin work goes via `docker exec synapse` against `localhost:8008` and never touches Traefik.
Allow-listing the sites' WAN addresses was **rejected** — dynamic, and a stale entry either
locks us out or hands admin to whoever inherits the address.
## Conventions ratified (operator, 2026-09-01)
#<agent>-<purpose>:matrix.phasefinal.com
Mirrors the existing `@<agent>:` user-ID convention. Proposed by ledger-dev. Rationale is
**"so the room IDENTITY carries the tier"** — deliberately NOT "so the push payload carries
the room name", which is true only for clients without a notification service extension.
Pre-existing rooms are not renamed ("The High Seat", `!NiVVoMsyoHCBRPrrrn`).
## Push reality — measured, and it inverts the obvious reading
The registered pusher (`@vhoang`, Element X iOS) uses **`"format": "event_id_only"`** via
matrix.org's sygnal. That payload carries event_id, room_id and counts — **no room name, no
sender, no content**. It still produces a useful notification because `mutable-content: 1`
means Element X runs a **Notification Service Extension**: iOS wakes it with the near-empty
payload and it **fetches the event and renders the notification on the device**.
1. The tier-in-room-identity scheme works — but **via the client fetch**, not the payload.
`m.room.name` must be set at creation; the ALIAS is not what reaches the phone. Synapse
sends `ctx["name"]` (the `m.room.name` state event) and omits the key entirely if unset.
2. **`push: include_content: false` is irrelevant for clients with an NSE.** It bites clients
without one.
3. ⚠ **Server-invisible failure mode:** if the phone cannot reach the homeserver at wake time
the fetch fails and iOS shows the bare word "Notification". **Synapse records
`last_success` and sees a delivered push.**
Self-hosted sygnal **considered and rejected** — sygnal is a relay to FCM/APNs, not a
replacement, so it removes matrix.org and nothing else; and with `event_id_only` the path
already carries nothing worth protecting.
## QR sign-in — requires MAS, deferred
MSC4108 hard-requires `matrix_authentication_service`; Synapse refuses to start otherwise.
MSC4388 enables independently but is only the rendezvous **channel**, not a login flow.
Deferred: MAS is a service, a database and a migration of every account off built-in auth,
and the v1.139.0 note warns `/register` from **old appservice implementations may break under
MAS** — precisely the bridge owning 13 of 15 accounts.
## Shared-secret registration gotcha
`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**, it does
not **terminate**. A trailing `\x00` yields `HMAC incorrect`. Run inside the container against
`localhost:8008`; port 8008 is not published to the host.
Full doc: `docs/pfi/vm-102-matrix-synapse.md` (`931bac8`, `73866f6`).
persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md
+451
View File
@@ -0,0 +1,451 @@
# `[2026-09-01]` pfi-gx10 (ASUS Ascent GX10) onboarded headless — and it is the intended new home for run 3c
ASUS Ascent GX10 arrived and was registered, converted to headless, and given a rack-move
playbook. **It was NOT racked** — the operator ran out of day. It is still on his desk, on
Wi-Fi, on a temporary DHCP lease.
| | |
|---|---|
| GPU | **NVIDIA GB10**, driver 580.173.02, **compute capability 12.1 (`sm_121`)** |
| CPU | 20 cores, **aarch64** |
| Memory | **121 GB UNIFIED** — CPU and GPU share it; not 121 GB *plus* VRAM |
| Storage | 916 GB NVMe, 6% used |
| OS | Ubuntu 24.04.4, kernel 6.17.0-1031-nvidia |
| Access | `infra-ops` NOPASSWD sudo (operator-bootstrapped). `lkraven` has key auth but needs a password to escalate — **automation must connect as `infra-ops`**. |
## Purpose (operator, 2026-09-01)
Local inference experiments **and** the failed training — run 3c. That is the whole point:
run 3c died on ana-ml2 when a **kilowatt-class** training box tripped a breaker
(2× 300 W GPUs + dual EPYC 9254). The GX10 is a ~240 W appliance, roughly a fifth of the
draw, on a different site's circuits.
**The memory arithmetic favours it strongly.** Run 3c is a LoRA (r=64, batch 2 × accum 8,
gradient checkpointing) over `gemma4-26b-a4b-it-bf16` — **49 GB of base weights**, working
set roughly 55–65 GB. The `gemma4-charrp` compose warns in capitals that "48.10 GiB of BF16
weights CANNOT be served here" on ana-ml2's shared GPU0. 121 GB unified turns that
constraint into a non-issue.
## What has NOT been established — do not assume any of it
1. **The box is bare.** No torch, no nvcc, no CUDA stack. Docker 29.2.1 and 822 GB free.
2. **aarch64 dependency risk.** torch 2.9.1+cu130 ARM64 wheels exist; transformers/TRL/PEFT
are pure Python. Compiled deps — flash-attn, bitsandbytes, liger, xformers — are open
questions per-arch.
3. **Triton has NO sm_121 support** (established independently by comfy-dev the same day).
Anything reaching for `torch.compile` or Triton-backed kernels is closed on this silicon.
4. **Throughput is unmeasured.** Run 3c ran 16.45 s/it on a 300 W RTX PRO 6000. GB10 will be
slower; how much decides whether 604 steps is an overnight run or two days.
**Measure this before porting anything** — the recommended first move is a probe: install
ARM64 torch, load the base, run ten steps, report s/it.
5. **Model transfer:** 49 GB from ana-ml2 over the Anaheim↔NH3 link at a **measured
32.4 MB/s** — about 25 minutes. Note that is a third of the irv-ml1↔NH3 link's 99 MB/s.
## The headless conversion, and the lesson inside it
`playbooks/gx10-headless.yaml` (`1b596c8`): multi-user.target, gnome-remote-desktop stopped,
sleep/suspend/hibernate **masked**, logind ignores lid and idle, sshd keepalives, hostname
corrected `gx10-a745` → `pfi-gx10`.
⚠ **`gdm` is a STATIC unit on Ubuntu** — pulled in by `display-manager.service`, never
"enabled". The first version guarded on `is-enabled | grep enabled`, which always skips, and
**the verify tested the same wrong property and passed**. Six green verifies having not
stopped the display manager. Both now test `is-active`. This is an instance of
[[2026-09-01-wrong-object-measurement]].
The playbook refuses to stop GDM while a seat session is held (`--var force_dm_stop=true` to
override) — automation should not yank a display out from under someone at the machine.
⚠ **elway's `--sudo` applies only to ad-hoc `--shell`/`--upload`.** Playbook steps run as the
connecting user and must carry their own `sudo`.
## The rack move, written but not run
`playbooks/gx10-rack-network.yaml` (`a0c5fc6`) — target settled: **`nh3-servers` VLAN 50,
static `10.100.50.60`**, clear of `.40`/`.42`/`.50`/`.90` and below the `.150` DHCP pool.
Requires nothing from the operator beyond racking it. The wired NIC has its own MAC
(`30:c5:99:3d:a7:45`, distinct from Wi-Fi `50:bb:b5:a2:00:a8`), so the post-move address AND
switch port are discoverable from the UDM rather than relayed.
**Design property worth preserving:** the playbook never leaves itself one path back. Wi-Fi
stays up while the wired interface is configured beside it; Wi-Fi teardown is explicitly a
separate later change. `netplan try`'s auto-rollback needs a TTY that elway cannot provide,
so a live Wi-Fi link is what substitutes for it. Two preconditions are asserted as steps
rather than assumed: carrier must be 1, and the MAC must match (interface names renumber
across kernels; MACs do not).
⚠ **The GX10 is NOT the Ada box.** Two separate machines — see
[[2026-09-01-ada-migration-branch-a]].
## `[2026-09-01 23:09]` Liveness check — alive, idle, and STILL BARE (nothing has changed)
Probed at **2026-09-01T23:09:51-07:00**, read-only:
host pfi-gx10 Ubuntu 24.04.4 LTS kernel 6.17.0-1031-nvidia aarch64
uptime 11h48m (booted 2026-09-01 11:21) load 0.00 0.01 0.00 — idle
GPU NVIDIA GB10 driver 580.173.02
memory 121 GB unified, 118 GB free
disk /dev/nvme0n1p2 916 GB, 48 G used, 822 G free (6%)
net wlP9s9 10.100.10.226/24 (DHCP) | enP7s7 DOWN, NO-CARRIER (no cable)
sessions 2x lkraven desktop (seat0 + :1) — sitting at a GUI login screen
**Compute stack unchanged and still absent:** no `torch`, no `nvcc`, no `uv`. Docker daemon
present but `infra-ops` is **not in the docker group**. Nothing has been installed since
onboarding.
⚠ **`nvidia-smi` reports FB Memory / BAR1 as `N/A`. That is CORRECT, not a fault** — GB10 is a
Grace Blackwell superchip with **unified** LPDDR5X shared between CPU and GPU, so there is no
discrete VRAM figure to report. Do not chase it as a driver problem.
## ⚠ RACKING IS **NOT** A PREREQUISITE — the Wi-Fi is not the bottleneck
Assumed the temp Wi-Fi would gate getting a 49 GB base onto the box. **Measured, and it does
not:**
link SSID PFI, 6 GHz, 160 MHz, EHT (Wi-Fi 7), -48 dBm
rx/tx bitrate 2401.9 Mbit/s both directions
measured 300 MB nh3-dev -> gx10 in 4.5 s = 67 MB/s over SSH
(SSH cipher is the limiter, not the radio — link is ~300 MB/s)
implies 49 GB bf16 base transfers in ~12 min
So the throughput probe (ARM64 torch, load base, ten steps, report s/it) can proceed **from the
desk, today**, without racking first. Racking remains one command
(`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`) and is worth doing for
permanence, but it does not gate the decision the probe exists to answer.
## `[2026-09-01 23:2x]` OUTFITTED (userspace) — CUDA WORKS on sm_121, full training stack imports
### ✅ RULING: BARE METAL, not Proxmox/VM. Settled, do not re-litigate.
1. **Proxmox VE has no aarch64 build.** It ships x86_64 only; the ARM ports are unofficial
community work. Running an unsupported hypervisor under a compute asset is a bad trade.
2. **The GPU cannot be cleanly passed through.** GB10 is a superchip: the GPU sits on NVIDIA's
own on-package root complex (bridges `22ce`/`22d0`/`22d1`, GPU at `000f:01:00.0`) and is
**cache-coherent with the CPU over NVLink-C2C, sharing the same physical LPDDR5X.** Passing
it to a guest means partitioning unified memory — which destroys the entire value
proposition, since **121.6 GiB GPU-addressable** is the reason this box exists.
3. **Vendor's supported path is bare metal** — it already runs NVIDIA's own kernel flavour
(`6.17.0-1031-nvidia`).
4. **Fleet convention agrees:** ana-ml2 and irv-ml1 are both bare-metal GPU hosts. Hypervisors
are separate machines in this fleet, never under the accelerators.
5. **Single tenant.** VM isolation buys multi-tenancy nobody needs; the rollback benefit is
better served by **containers + `uv` venvs**, which is what the rest of the fleet does.
### Installed and verified (all userspace, NO sudo needed)
uv 0.12.9 (aarch64), ~/.local/bin, wired into .bashrc + .profile
venv ~/ml/.venv (python 3.12)
torch 2.14.0+cu130 cuda runtime 13.0, cudnn 92400, triton 3.8.0
stack numpy 2.5.2, transformers 5.16.1, accelerate 1.14.0, peft 0.20.0,
trl 1.12.0, datasets 5.0.1, safetensors 0.8.0, hf_hub 1.29.0,
bitsandbytes 0.50.2 -- ALL import clean on aarch64
**The "per-arch unknowns" warning did not materialise for the training stack.** Everything
imported first try.
### ✅ MEASURED GPU COMPUTE — it works, and the memory is the point
device NVIDIA GB10, capability sm_121, driver 580.173.02
GPU memory 121.6 GiB total / 110.8 GiB free <- unified, fully CUDA-addressable
bf16 8192^3 ~93 TFLOP/s fp16 ~93 TFLOP/s
fp32+tf32 38.2 TFLOP/s fp32 (no tf32) 19.5 TFLOP/s
bf16/fp32 4.9x -> tensor cores ARE engaged
~93 TFLOP/s dense bf16 is roughly **A6000-class throughput with 2.5x the memory**. Capacity,
not speed, is what this box buys. A 49 GB bf16 base fits with ~70 GB to spare.
### ⚠⚠ I NEARLY REPORTED A PHANTOM PROBLEM — NO WARMUP MEANS YOU MEASURE THE JIT
First benchmark (10 iterations, **no warmup**) read **27.0 TFLOP/s bf16** — slower than a 3090
— and I was one step from reporting "tensor cores are not engaging on sm_121." Adding 5 warmup
iterations moved it to **93 TFLOP/s. The first measurement was timing the PTX JIT compile.**
⚠ **This is exactly `docs/pfi/training-throughput-playbook.md` §4: a run that completes, reports
plausible numbers, and is wrong.** It would have sent us chasing a cuBLAS fallback that does not
exist. **Always warm up before timing anything on this box** — see the JIT note below for why it
bites harder here than elsewhere.
### ⚠ sm_121 IS NOT IN TORCH'S COMPILED ARCH LIST — everything JITs from sm_120 PTX
torch.cuda.get_arch_list() -> ['sm_80','sm_90','sm_100','sm_110','sm_120']
device capability -> sm_121
There is **no sm_121 cubin in the wheel**; it runs by PTX forward-compat JIT from sm_120.
Consequences:
- **First use of every kernel pays a JIT compile** (cached afterward in `~/.nv/ComputeCache`).
This is what corrupted the benchmark above.
- ⚠ **Any library shipping cubin-only binaries with no PTX will simply fail here.** That is the
real shape of the "per-arch unknown" risk — not source builds, but prebuilt kernels.
### ✅ PRIVILEGED HALF DONE — and I WAS NEVER BLOCKED
⚠⚠ **The block was my own bad test.** `infra-ops` has had **NOPASSWD sudo since account
creation** (`/etc/sudoers.d/infra-ops`, installed 11:31 with the account). I probed with
`sudo -n -v`, which returns non-zero on **sudo >= 1.9.15** because `-v` refreshes an auth
timestamp that a NOPASSWD-only rule never creates — and asked the operator for a password that
was never needed.
pfi-gx10 sudo 1.9.15p5 sudo -n -v rc=1 sudo -n true rc=0
nh3-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
ana-docker sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
irv-ml1 sudo 1.9.13p3 sudo -n -v rc=0 sudo -n true rc=0
**Always `sudo -n true`.** Fixed in `scripts/elway` (which carried the same probe and would
have started falsely prompting fleet-wide as hosts pass sudo 1.9.13) and in auto-memory
`reference_infra_ops_sudo_identity`, which *recommended the broken test*.
[[feedback_assert_effective_value_not_substring]]
⚠ **Same shape twice in one session.** I also reported "nvcc: ABSENT" from `command -v nvcc`
when the CUDA toolkit was installed all along at `/usr/local/cuda/bin/nvcc`, just not on PATH.
Both times I checked a proxy and reported it as the thing itself.
**Credential:** the temp password was `infra-ops`'s, not `lkraven`'s. Rotated to 32 random
chars and **vaulted at `pfi-gx10/infra-ops-password`** (round-trip verified). Kept rather than
locked — deviating from the canonical bootstrap's no-password posture **deliberately, because
this box has no iDRAC/BMC**, so a console break-glass credential is the only way back if key
auth breaks. ⚠ `lkraven` (uid 1000, GUI desktop user, `sudo` group) has a password nobody
knows; out of scope, but it is a gap.
### ⚠ DOCKER'S BUILDKIT DB WAS ALREADY CORRUPT — the restart surfaced it, did not cause it
`nvidia-ctk runtime configure` + `systemctl restart docker` failed with
`error initializing buildkit: error creating buildkit instance: invalid database`. **Not the
nvidia config** — `daemon.json` was valid and `nvidia-container-runtime` was present. dockerd
had been running on a process whose on-disk buildkit state was already bad; it simply could not
come back. Remediated by removing `/var/lib/docker/buildkit` (108 KB of **cache**, and the box
had **0 images, 0 containers**), then `daemon-reload` + `reset-failed` + start.
⚠ **A long-running daemon can be unable to restart for hours before anyone finds out.** Nothing
reported unhealthy until something asked it to restart.
### ✅ FINAL STATE — playbook green, 5/5 verifies
playbooks/gx10-outfit-privileged.yaml -> steps 1 ok / 1 changed / 0 failed / 6 skipped
verify 5 ok / 0 failed
NOPASSWD sudo OK
nvcc /usr/local/cuda/bin/nvcc (NOT on default PATH)
docker socket as infra-ops OK (docker group)
CUDA container sees the GB10 OK (nvidia/cuda:13.0.0-base -> nvidia-smi -L reports GB10)
userspace torch still sees GPU OK
docker Runtimes nvidia runc io.containerd.runc.v2
**The box is fully outfitted for inference + training, bare metal, containers available.**
### Historical note — what the block looked like before it was disproved
⚠ **`infra-ops` does NOT have NOPASSWD sudo on pfi-gx10** (`sudo -n -v` -> "a password is
required"), unlike the rest of the fleet — see [[reference_infra_ops_sudo_identity]], which
should be corrected: gx10 is an exception. **No gx10 credential is in the vault** (`secret list
--prefix pfi-gx10` -> 0 items), and `scripts/elway` prompts via `getpass`, which a
non-interactive session cannot answer.
Still outstanding, all needing root:
- CUDA toolkit / nvcc (absent -- needed to COMPILE kernels, not to run torch)
- NVIDIA Container Toolkit + add infra-ops to the docker group
(docker daemon runs; infra-ops gets "permission denied" on the socket)
- NOPASSWD sudo for infra-ops, so this box stops being a fleet exception
- racking (playbooks/gx10-rack-network.yaml) -- optional, NOT a prerequisite
**Nothing above blocks the run-3c throughput probe**, which needs only the driver and torch.
## `[2026-09-01 23:5x]` Run-3c throughput probe — IN FLIGHT, and the ana-ml2 baseline is now on record
### The baseline nobody had written down: ana-ml2's real s/it for run 3c
Extracted from `/tank/erp-tune/run-03c.log` (the actual run, before the breaker killed it):
step 10 10.81 s/it step 20 12.88 s/it step 24 15.76 s/it
=> 604 steps on ana-ml2 ~= 2.2-2.7 h
**This is the number the GX10 has to be compared against**, and it was sitting in a log
nobody had reduced. Anything under ~45 s/it on the GX10 makes 604 steps an overnight run.
### Exact run-3c geometry (from run-03c.json + a real adapter_config.json)
base /tank/aimodels/gemma4-26b-a4b-it-bf16 49 GB
gemma4, 30 layers, hidden 2816, 16 heads / 8 KV, MoE (A4B),
vocab 262144, sliding_window 1024, multimodal (vision+audio configs)
LoRA r 64, alpha 128, dropout 0.0, bias none, 205 target modules
(q/k/v/o + gate/up/down across all 30 language-model layers)
shape max_seq_len 16384, per_device_batch 2, grad_accum 8 (effective 16)
other gradient_checkpointing true, bf16, cosine, warmup 0.1, wd 0.01
length 604 optimizer steps, 1 epoch, save_steps 50
### ⚠ THE BACKEND-DELTA TRAP — CHECKED, and it is NOT present
CLAUDE.md warns that two frontier-panel conclusions were voided because a benchmark and a
trainer had silently different attention backends. Enumerated both sides **before** running:
ana-ml2 flash_attn NOT installed in /tank/erp-tune/venv -> falls back to sdpa
pfi-gx10 flash_attn not available for aarch64/sm_121 -> falls back to sdpa
Same backend both sides. ⚠ **Library versions DO differ and it is recorded rather than
assumed harmless:**
ana-ml2 torch 2.13.0 transformers 5.15.1 peft 0.20.0 accelerate 1.14.0
pfi-gx10 torch 2.14.0+cu130 transformers 5.16.1 peft 0.20.0 accelerate 1.14.0
The probe reads the **resolved** `_attn_implementation` back off the loaded model rather than
trusting the request, so the comparison is checkable rather than assumed.
### Probe design (§4-disciplined)
`~/ml/probe_3c.py` on the GX10. Real geometry, synthetic batches — it measures step cost, and
padding on the real run is 0.0% so the corpus does not change per-step cost materially.
- **12 steps, first 2 discarded as warmup.** Non-negotiable here: an unwarmed benchmark on
this box already read 27 TFLOP/s when the true figure was 93, because it was timing the PTX
JIT. sm_121 is not in torch's arch list.
- Reports the **step-time distribution** (median, min/max, every step), not a mean beside a
settings dump.
- Emits **every provenance key always, null when unused** — present-and-null, not absent.
### Transfer
49 GB relayed **ana-ml2 -> nh3-dev -> pfi-gx10**, ~33 MB/s, ~25 min. Relayed rather than
direct because ana-ml2's key is not authorised for `infra-ops@pfi-gx10` — and relaying costs
nothing extra, since the Wi-Fi leg is the bottleneck either way. **Deliberately did not add an
access grant just to save a hop.**
## `[2026-09-02]` GX10 BASELINE + TOOLING VERIFIED — ~79.35 s/it on the run-3c shape
⚠ **PURPOSE, stated by the operator so it is not misread later: this was a BASELINE for the box
and a check that the tooling loads.** It was NOT a decision about where run 3c runs, and no port
was undertaken. The placement reasoning below is a *byproduct* — real and worth keeping, but the
deliverable was the number and the working stack.
**Deliverable 1 — the tooling loads and runs.** aarch64 + sm_121 runs the full training stack:
torch 2.14.0+cu130, transformers, accelerate, peft, trl, datasets, safetensors, bitsandbytes,
**plus the harness's own `flex_attention` backend and chunked-loss path**. Nothing exotic was
needed beyond `python3-dev`. This box can train.
**Deliverable 2 — the baseline.**
### The number
pfi-gx10 MEDIAN 79.36 s/it min/max 79.30 / 79.45 (10 timed steps, 0.19% spread;
warmup 85.90 then 79.32, discarded)
peak memory 75.1 / 121.6 GiB -- 46 GiB SPARE, comfortably not memory-bound
attn_resolved: flex_attention (matched to the harness, read off the model)
FINAL, from the probe's own provenance block on the box
-> 604 steps ~= 13.3 h
peak 75.1 / 121.6 GiB (PyTorch max_memory_allocated) -- 46 GiB spare
ana-ml2 10.8-15.8 s/it (run-03c.log, the real run)
-> 604 steps ~= 2.2-2.7 h
⚠ **6x slower where raw compute predicts 2.7x** (93 TFLOP/s bf16 measured here vs an RTX PRO
6000 Blackwell's ~250). **Hypothesis, not a measurement:** the limiter is **memory bandwidth**
— GB10's unified LPDDR5X against dedicated graphics memory, on a workload built to be
bandwidth-hungry (seq 16384 with gradient checkpointing, which trades compute for memory
traffic by design). Confirming it needs a bandwidth-bound microbenchmark that has not been run.
⚠ **THE GX10 IS A CAPACITY BOX, NOT A THROUGHPUT BOX.** 121 GB holds models nothing else in the
fleet can. It is **not** the box to move a time-sensitive run *to*.
### Byproduct: IF run 3c ever moves here, the GX10 is the right home — and infra-ops' reversal was WRONG
⚠⚠ **infra-ops recommended ana-ml2 and the operator overruled it, correctly.** The error is
worth more than the recommendation: **I costed the failure in the units I was measuring.** I had
step times, so I priced a breaker trip as *"≤50 steps ≈ 11 minutes of lost training"* — the
recompute cost. That is not what a breaker trip at Anaheim costs.
**Operator, 2026-09-02:** *"when the breaker trips at ana, that's a 40 minute drive to reset it.
it's not 25 minutes, it's a giant pain in the ass."*
**The real cost of an Anaheim breaker trip:**
a 40-minute drive EACH WAY, on the operator's time, whenever he notices
13 hosts down: ana-ml2, ana-docker, pfi-pve (hypervisor for most of them),
pfi-postgres, pbs-ana (the fleet's PRIMARY backup server), ana-wg (WireGuard),
pfi-ana-webhost, ana-filebot, pfi-pteradactyl, pfi-tacticalrmm
⚠ including THREE SureFire CLIENT hosts (sfsrv-ana, sf-ana-container, sf-r630)
-- a client's production machines, under a hosting agreement
⚠ **`save_steps` 100 → 50 caps the RECOMPUTE, not the OUTAGE.** It was never the mitigation I
claimed; it only bounds the cheapest component of the loss.
**So: 13.3 h unattended on a desk in NH3, drawing desktop-class power and unable to take down
anything, beats 2.5 h that can put a client's hosts dark until someone drives to Anaheim.** The
GX10's slowness is close to free here — nothing is waiting on this run — and the risk it removes
is expensive and lands on the operator personally.
⚠ **GENERAL FORM, and the reason this is recorded at length: when you recommend between
options, check whether you priced the failure mode in the units you happened to be measuring.**
A metric in hand will volunteer itself as the unit of risk.
### ⚠ Power-capping ana-ml2 is a WEAK lever — measured, do not spend time on it
2x RTX PRO 6000 Blackwell Max-Q: limit 300 W, default 300 W, min 250 W, max 325 W
Capping both GPUs to the floor saves **100 W total** on a box drawing ~1 kW. Not nothing, but
nowhere near enough to make an Anaheim training run safe, and it costs throughput to buy it.
Recorded so nobody re-derives it as a clever third option.
### Probe methodology — five launches, five distinct causes, all recorded
1. reaped SSH long remote job piped through a reapable foreground ssh.
FIX: setsid nohup, log ON the box. Work survived; output did not.
2. bare LoRA target names q_proj/k_proj/... also match Gemma-4's VISION and AUDIO towers,
whose projections are Gemma4ClippableLinear -- PEFT cannot wrap
them. FIX: the real run's 205 fully-qualified names, sha recorded.
3. missing python3-dev Triton JITs a C shim at FIRST USE and needs Python.h. Torch
imports, model loads, LoRA attaches, THEN step 1 dies with a bare
CalledProcessError -- Triton devnulls the compiler's stdout.
4. naive-loss OOM (32 GiB) the 262,144-wide vocab x 32,768 positions logits tensor in fp32.
⚠ NOT a GX10 limit -- the harness docstring says it OOMs "on a
95.60 GiB card (measured)", i.e. on ana-ml2 too.
5. self-inflicted VRAM leak a stuck orphan held 80 GB while PyTorch reported 0 bytes
allocated; every relaunch was doomed before it started and blamed
the NEW run. ⚠ ASSERT THE GPU IS CLEAR BEFORE EVERY LAUNCH.
### ⚠ THE BACKEND-DELTA TRAP WAS PRESENT AND I FIRST DECLARED IT ABSENT
CLAUDE.md records two frontier-panel conclusions voided by a benchmark and a trainer having
silently different attention backends. I checked for it by asking whether **flash-attn was
installed** — the wrong discriminator entirely — and reported "no delta". The harness sets
**`ATTN_IMPLEMENTATION = "flex_attention"`** explicitly in code, with tuned kernel options; its
own comment records peak 74.3 GiB under flex vs sdpa's 79.71.
⚠ **Read the configured value in the code. Absence of an alternative is not evidence of the
default.** The probe now requests flex_attention and reads `_attn_implementation` back off the
loaded model. **RESOLVED: flex_attention — it compiles and runs on sm_121**, which was a real
open risk (flex compiles through Triton; sm_121 is not in torch's arch list).
### Faithfulness of the probe (what makes the number comparable)
LoRA the real run's 205 fully-qualified modules, sha256 c9ee070c0d67eb08
loss path copied from the harness: trunk hidden states, DROP IGNORE_INDEX before the
head, chunk lm_head+CE at loss_chunk_tokens=1024 under checkpoint, softcap 30.0
loss ratio 28.4% of positions masked to match the real 13,310,930 / 18,600,057 loss/ctx
ratio -- observed 23,351-23,638 loss tokens/step against a 23,449 target
tokens sampled [1000, 250000) to stay below every multimodal special token
(boi 255999, boa 256000, image 258880, audio 258881, eoi/eoa/video 258882-4)
warmup first 2 steps discarded; sm_121 JITs from sm_120 PTX on first use
## ⚠ NOT DONE, and deliberately: the run-3c port
Scoped but **not executed** — operator stood it down; the baseline was the goal. Scope, so it
does not have to be re-derived:
harness eitri-smithy 26 MB corpus /mnt/smithy/.../derived 2.3 GB
recipe-r3 485 KB holdout 82 MB
run-03c manifests 70 MB -> ~2.5 GB, ~75 s at 33 MB/s
venv 3.2 GB NOT portable (x86-64) -- rebuild on aarch64
⚠ **No encode cache to preserve.** The launch script's "609 MB encode cache / ~4.3 hours"
comment carries its own correction to 145.5 s, and run 3c's log shows **`[encode] done in
14.4s`** on 32 workers. Nothing worth moving.
⚠ **Copy the corpus; do NOT mount `/mnt/smithy` over NFS on this box.** A desk machine on Wi-Fi
is the wrong place for a hard NFS dependency under a 13-hour unattended run — the fleet has
already lost a host to a hard mount wedging into unkillable D-state
([[incident_esh_docker_nfs_boot_race]]).
The only untested piece is the harness's own 26 MB of Python; every library it leans on is
verified above.
persistent-memory.d/2026-09-01-wrong-object-measurement.md
+42
View File
@@ -0,0 +1,42 @@
# `[2026-09-01]` A named failure class: a correct check aimed at the wrong object
Six instances surfaced across three sessions in a single day, independently, in unrelated
domains. It has a distinguishing property that makes it worth naming separately from
"a bad measurement":
> **Re-running the same check cannot catch it, because the check is correct and the object
> is wrong.** The only move that breaks it is asking what the artifact *is* before trusting
> any metric computed over it.
## The instances
| where | the metric | the artifact nobody opened |
|---|---|---|
| comfy-dev, civitai | a "~25 KB/s throttle" | a **9,685-byte login page** returned on failed auth |
| comfy-dev, render harness | reported "completed" | a 4.7 KB **all-black PNG** |
| comfy-dev, audio | RMS said healthy | degenerate audio where the **loudness WAS the noise** |
| ledger-dev, capability 5 | healthy ping, healthy container, healthy delivery | a **requirement unmet** — a mailbox, not an away-channel |
| infra-ops, gx10 headless | six green verifies | `is-enabled` on a **static unit**; gdm still running |
| infra-ops, synapse admin | an `ipallowlist` comment promising fleet access | fleet traffic **hairpins out the WAN**; 10.0.0.0/8 matches nothing |
## Related lessons banked the same day
- **A caveat plus propagation is decoration.** infra-ops flagged a sample-size problem AND
escalated the claim in the same message. If a number needs re-measuring before it can be
repeated, hold the escalation until it has been. The caveat made the uncareful thing look
examined.
- **Verify with a negative control.** A 200 means nothing without a 401 beside it. Used on
the pewpewstudio key mint; used on the `/_synapse/admin` lock (tested from a genuinely
external vantage via the NH3 residential egress proxy, because testing from a fleet host
sits inside the allow-list and proves nothing).
- **Reasoning from what is VISIBLE to what EXISTS.** infra-ops found no `sm_89` part in the
inventory, saw one new machine, and collapsed "the box I can see" into "the box that is
coming". They were two different machines. Wrote "I cannot resolve it and will not guess",
then built three messages on the guess.
## Disposition
Recommended for a row in `docs/pfi/training-throughput-playbook.md` §4 (the durable home for
"why a run LIES about itself"), with attribution to comfy-dev and ledger-dev.
**NOT YET WRITTEN — awaiting operator.** Tracking surface: this file, plus althing threads
`01M1EYBSYA4QRYK54PX0K1S8CS` (comfy-dev) and `01M1EXVPZT66SSGCRZH3W624R7` (ledger-dev).
persistent-memory.d/2026-09-03-gx10-rack-network.md
+46
View File
@@ -0,0 +1,46 @@
- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.**
`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
**Operator ruling, and the better design:** put the address on the
**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A
host-side static works until the box moves, and then it is a stale netplan file
on a machine whose address you no longer know. A reservation moves with the MAC.
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
host unchanged, still DHCP
`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and
was NOT used — annotated as retired at its top. Its safety *ordering* was
followed and is still right.
⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd
`10.100.0.111` from `nh3-default`. The switch port had to be repointed before
anything else could work. Do not assume a racked port is on the VLAN you asked
for.
⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two
unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and
written back.
⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still
up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits
directly on the userland subnet — so "I can reach it on the new address" proved
NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on
inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what
settled it:
ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10
ping -c2 -I enP7s7 1.1.1.1 0% loss egress
Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to
`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot —
verified by reading `/var/lib/NetworkManager/NetworkManager.state` back.
⚠ **The box now has exactly ONE path.** If that switch port or the reservation
breaks it is a rack visit; the escape hatch is deliberately gone. Correct end
state for a racked server, but a posture change from the desk setup — and this
is the box run 3c moved to.
Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`.
persistent-memory.d/2026-09-03-gx10-run3c-staged.md
+91
View File
@@ -0,0 +1,91 @@
# `[2026-09-03]` Run 3c STAGED on pfi-gx10 — verified end to end, deliberately NOT launched
The ERP-seat SFT LoRA that died on ana-ml2 at step 24 of 604 to an Anaheim breaker trip is
now staged on pfi-gx10, unchanged. **The launch is the operator's call and was not taken** —
he stood this port down once before, so a 13.3 h commitment is not an agent default.
Runbook `docs/runbooks/gx10-run-03c.md`; canonical config + launcher
`scripts/erp-tune-gx10/`; on the box `/home/infra-ops/erp-tune/`.
ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
## What is on the box
~/models/gemma4-26b-a4b-it-bf16 49 GB base, ALREADY THERE from the 09-01 probe
~/erp-tune/eitri-smithy harness, git 0a6bd2e, tracked tree clean
~/erp-tune/recipe-r3 recipe / survivors / loss-mask
~/erp-tune/datasets/{derived,holdout} 2.4 GB, COPIED (50 s at 49 MB/s from nh3-dev)
~/erp-tune/run-03c/encode-cache PRE-SEEDED with the verified encode
~/ml/.venv + protobuf, pytest (the only two gaps vs ana-ml2)
⚠ **The corpus is copied and the box mounts NO NFS.** `/mnt/smithy` lives on nh3-nas, now on
the *same subnet* as the racked GX10 — which makes mounting it tempting and still wrong. A
13 h unattended run is the worst place for a hard NFS dependency
([[incident_esh_docker_nfs_boot_race]]). 2.4 GB copies in under a minute; there is nothing to
buy.
## The verification that actually mattered — and it was NOT free reasoning
ana-ml2 ran transformers 5.15.1 / torch 2.13.0 on x86-64. The GX10 runs 5.16.1 / 2.14.0+cu130
on aarch64. That is precisely the silent backend-delta class CLAUDE.md records as having voided
two frontier-panel conclusions. So it was **measured**: a full encode was run into a throwaway
output dir and the encoded corpus compared byte-for-byte.
ana-ml2 encoded-c16316f1c1bb21da.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3...
pfi-gx10 encoded-fd8fe1944fb316b2.jsonl 197,360,233 B sha256 c08bb1fe2ecb0be3...
**Byte-identical.** Every aggregate matched too: 9,504 vs 8,404 ids / 0 overlap, 15 unfittable
dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930 tok, five mix shares to 4 dp.
⚠ **The cache-key FILENAMES differ and that is correct, not drift.** `base_model_path` is in
the encode-cache key *by design* (so a different base cannot silently reuse an encode), and
rehoming the base changes the key while leaving content identical. **The key is an input hash;
the sha is the output.** Do not read the differing filenames as a mismatch — and do not
"fix" it by symlinking `/tank/aimodels` onto this box to force a key match. That verified
artifact was then copied into `run-03c/encode-cache/`, so the run trains on the exact bytes
compared and will report `[encode] cache hit`.
Also verified rather than assumed: **both 49 GB base shards sha256-match ana-ml2's** (size
equality was already true and is not the same claim), the harness's own suite is **122 passed**
on aarch64, and every one of the config's 8 path keys resolves to an existing local file.
## The config is provably the same run
`run-03c-gx10.json` = ana-ml2's `run-03c.json` with 8 path keys rehomed and 2
`substitute_controls` entries appended (host move; library delta). A generator asserted
**key-by-key that no non-path value differs** rather than eyeballing a diff — lr 1e-05, rank 64,
alpha 128, seq 16384, batch 2 x accum 8, save_steps 50, seed 20260824 all intact, and the
existing 10 substitute_controls are a byte-identical prefix.
## ⚠ I TRIPPED THE pkill SELF-MATCH AGAIN, ~20 MINUTES AFTER READING THE MEMORY ABOUT IT
`ssh gx10 'pkill -f "erp_sft_harness --config .../encode-check.json"'` — the pattern is in the
remote shell's OWN argv, so it killed my shell alongside the target and the command returned
nothing. [[feedback_pkill_ssh_self_match]] describes this exactly. Reading the memory did not
prevent it; **the guard has to be in the artifact, not in recall.**
So the launcher's already-running guard is a **pidfile**, not a pgrep — `pgrep -f
erp_sft_harness` in a script invoked over ssh matches the invoking shell and would refuse every
launch. Same root cause, and it would have presented as a mysterious always-refusing launcher.
## The launcher's other guards, each bought with a past failure
GPU-clear assertion a stuck orphan held 80 GB while PyTorch reported 0 allocated;
every relaunch was doomed and blamed the NEW run
setsid nohup + on-box log a foreground ssh reaped the 09-01 probe: work survived, output did not
log-exists refusal two runs must not share a log
>=40 GB free 12 checkpoints x 852 MB (measured off run-03, not estimated)
## Why the slow box is still the right box (unchanged, restated because it is the whole case)
~79.4 s/it here vs 10.8-15.8 on ana-ml2 -> 13.3 h vs ~2.5 h. An Anaheim breaker trip is not
priced in lost steps: it is a 40-minute drive **each way** on the operator's time, 13 hosts
down including `pbs-ana` and **three SureFire client machines**. Nothing is waiting on this run,
so the slowness is close to free.
## NOT verified — the honest gap
The harness's **train loop** has not run end to end on sm_121. The 79.4 s/it baseline used a
synthetic replica of the geometry, and the staging encode was killed before the weight load.
If it breaks, it breaks in the first two minutes after the `[sampler]` line — roughly three
minutes after launch, well before the first checkpoint at ~66 min.
persistent-memory.d/2026-09-03-mac-fleet-dsh.md
+51
View File
@@ -0,0 +1,51 @@
- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.**
vuongs-mac-mini 10.100.79.2 infra-ops + lkraven
esh-macbook-air 10.0.10.83 infra-ops + lkraven
esh-mac-studio 10.0.10.10 infra-ops + vhpfi
Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32
random chars and vaulted at `<name>/infra-ops-password`. `dsh` runs in the
operator's own account on each, on a **device-scoped** LiteLLM key
(`<name>-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than
trusted from the mint) — a laptop travels, and losing one should be one
revocation, not a fleet key rotation.
`scripts/provision-mac-dsh.sh <host> <account> [name]` carries every trap; the
operator-run half is `docs/runbooks/mac-provisioning.md`.
⚠ **`sudo -u <user>` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit
`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at
a **working install in another account**. Only filesystem permissions stopped
it. The script now refuses to run unless `$HOME` matches the target.
⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl`
leaves `/Users/<account>` root-owned; the account authenticates, gets a shell,
reports the right `$HOME`, and cannot write to it — surfacing as a bare
"Permission denied" hours later.
⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
bad password, AND a user outside `com.apple.access_ssh`. This produced a false
diagnosis twice in one session — once where the password was a typo
(`no-password` vs `nopassword`) and I blamed the access group, once where the
Studio's operator account is **`vhpfi`, not `lkraven`**. Check
`dscl . -list /Users` FIRST.
⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and
no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
"Operation is not permitted without secure token unlock". `dscl` works precisely
because there is no token to desync. True on all three Macs.
⚠ **FileVault kills remote access across reboots** — the machine sits at the
pre-boot unlock screen with no network. Nothing unattended should depend on a
Mac being reachable after a restart.
⚠ macOS has no `adduser`, `useradd`, or `timeout`.
Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook;
`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a
two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the
provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of
`agent-default-model` — without it the picker offers three models the gateway
does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`.
persistent-memory.d/2026-09-03-searxng-nh3-move.md
+67
View File
@@ -0,0 +1,67 @@
- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
returning **0 results for every query tested**. It was running **2026.4.17
against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
sites that had changed their markup. SearXNG ships near-daily releases for
exactly this reason.
⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
container created in April on `:latest` is pinned to April forever.
⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
was a stalled backup, the runner audit that trusted liveness for identity, and
the statusline bell that measured a mechanism.
**Proven before acting**: the new image, same settings file, same host, same
query, in a throwaway container → **20 results where the running one returned
0**. Network was ruled out first — from inside the container DNS resolved and
mojeek/wikipedia were reachable, so engines were reachable and the parsers were
the broken part.
**Why NH3 and not an in-place update** (operator's call, and the measurement
backs it):
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
nh3-docker egress 70.230.226.88 residential -> not gated the same way
Search engines gate datacenter ranges. Same reason the fleet keeps a residential
SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
around. `outgoing.proxies` has the fallback commented in place if NH3's egress
ever changes.
⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
`github`, `stackoverflow`, `marginalia` work. **General search is effectively
single-engine** — if google cse breaks, it goes quiet again.
**Two config defects, both silent:** `base_url` still named
`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
so nothing broke and the file lied to every reader); and the
`karmasearch.videos` removal key never matched because the engine's real name
has a space in it.
**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
queries. That is the only check that could have caught this — the mechanism was
healthy throughout.
**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
empty list is indistinguishable from a broken aggregator, which is precisely how
this hid. Same principle as althing's "unreachable post office is an OUTAGE,
never an empty inbox".
⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
--force` served a CACHED build** and silently reinstalled the old code — the
installed file still had the v1 API after the source no longer did.
`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
caught it.
Old instance stopped and removed; DNS alias repointed to
`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
for the other UniFi-side change the same day.
persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md
+41
View File
@@ -0,0 +1,41 @@
# `[2026-09-04]` `gen` moved to ana-ml2 GPU0 — and I sized it against the wrong number, twice in one hour
`vllm-embed` had OOM-crashed **7 times** (`RestartCount 7`): GPU1 was at **216 MiB free** of
97,887, and a ~3.4 GB tenant sharing a card with five other seats dies when it cannot get another
96 MiB mid-inference. Root cause of the `qwen3-embed` 500s brokkr saw — **not** the LiteLLM
gateway restart he attributed them to (different host, different component, 50 min earlier, and
six of the seven crashes predate it).
**Operator ruling:** move `gen` to GPU0. *"we were keeping it free for training, but we don't have
the power for sustained load."* GPU0 was never free — `mog-sec` has been there at util 0.52
(55,126 MiB) since the August move.
## ⚠ THE SIZING ERROR, WHICH IS THE POINT OF THIS ENTRY
gen at util 0.43 wants 42,091 MiB GPU0 free 42,113 margin 22 MiB — would not start
gen at util 0.41 wants 40,134 MiB margin 1,979 MiB — I called this safe. IT WAS NOT.
result 1,548 MiB free, 56 FlashInfer autotuner OOM-fallbacks per 3 min
gen at util 0.38 34,558 MiB actual 4,466-7,548 MiB free ZERO OOM events over 6 min
**`gpu-memory-utilization` governs vLLM's declared weights+KV budget. It does not cover what the
process actually needs at runtime** — FlashInfer JIT autotuner workspace, CUDA graphs, expandable
segments all allocate on top. My "1,979 MiB margin" was against the declared budget, so the seat
came up and then silently fell back to slower kernels for want of 20 MB chunks.
⚠ **Same class of error as the DAC plateau the same day: checked a proxy, reported it as the
thing.** The fix was to *measure* — count OOM-fallback events before and after — not to re-derive.
## Final state and what it cost
GPU0 mog-sec 55,126 + gen 34,558 = 89,701 / 97,887
GPU1 50,507 / 97,887 — 46.7 GB free, was 216 MiB
`gen` KV cache is now **348,515 → 268,205 tokens, 1.02x concurrency at its 262,144 max-model-len**
— it holds exactly one full-context request. Short/medium requests still batch; long-context
throughput effectively serialises. The lever to reclaim it is `MOG_GPU_MEM_UTIL 0.52`, untouched.
.env.bak-preGPU0-20260904-164032 the GPU move
.env.bak-preShrink-165133 the utilization
⚠ `/opt/docker/compose/*/.env` is root-owned — `docker compose restart` needs **sudo** on
ana-docker/ana-ml2 or it fails with a bare `permission denied` on the .env.
persistent-memory.d/2026-09-04-dac-forced-10g-failed.md
+46
View File
@@ -0,0 +1,46 @@
# `[2026-09-04]` Forcing 10G on the ESH-Media DAC FAILED progressively — and I called the plateau too early
The ESH-Media ↔ UDM uplink is a 3 m OEM `SFP-H10GB-CU3M` twinax that negotiated **1000 Mbps**
with **zero errors** on both ends. Diagnosis: `sfp_compliance: Unknown` — the switch reads the
EEPROM but cannot parse the compliance codes on a third-party cable wearing Cisco coding, so
autoneg falls back to the safe rate. Control case: a TP-Link `TL-SM5220-1M` on the adjacent UDM
port runs 10G with the same speed_caps, exonerating the port, firmware and autoneg.
Operator authorised forcing it. Forced `autoneg False / 10000 / full` on the **UDM end only**;
ESH-Media followed to 10000 unforced — proof the cable was electrically *capable*.
## ⚠ IT DEGRADED, AND MY "PLATEAU" READ WAS WRONG
10:03 200 errors link-up burst
10:48 221 +21 in 42 min — I reported this as "flat", it was not
14:38 416 +195 over 4 h, plus user-visible flapping the operator felt
**A marginal link declares itself over hours, not minutes.** I watched a two-minute flat window
and reported a plateau; the counter simply had not moved yet. The operator noticed the flapping
before the soak I left running had accumulated enough to raise it. Reverted 14:38, both ends back
to autoneg/1000, stable.
⚠⚠ **The correction that matters: a clean, zero-error link at 1G does NOT rule out a marginal
cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at 10G is exactly
what a cheap 3 m twinax is — which means the platform's autoneg fallback was **protecting
something real**, not being fussy about vendor coding. The coding explains the negotiation; it
does not explain errors once forced. I conflated the two.
⚠ **Do not re-force this port.** The fix is the cable.
## Method notes worth keeping
- **Force the RECOVERABLE end.** ESH-Media reaches the controller *through* this link, so a
failed force there strands the switch behind a dead uplink — a physical visit. The UDM end is
safe because the path to the controller (`nh3-dev → 10.100.10.1 → 10.0.0.1`, site VPN) does not
cross it. Verified with `traceroute` **before** the change; revert payload written before the
forward one.
- ⚠ `port_overrides` is a **whole-array PUT** — the array was diffed to prove exactly one field
on one port changed before sending, and read back after.
- Blast radius named before acting: ESH-Media backhauls the operator's office switch, a WiFi AP
and the Zigbee coordinator.
Resolution: the operator already owns a replacement and ran the copper himself through a drilled
floor 2x4. Recommendation given was **fiber** (10Gtek SR 2-pack + OM4 3 m LC-LC, ~$40-65) because
the cable-vs-pull-damage question was never resolved and a second DAC through the same hole risks
the same fault. Runbook `docs/runbooks/esh-media-dac-10g.md`; commits `b9988a7`, `514ce7a`, `054c098`.
persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md
+33
View File
@@ -0,0 +1,33 @@
# `[2026-09-04]` esh-nas SMB account for the AudioGridder box — and the NAS is effectively open to the whole LAN
`10.0.50.50` is **CT 103 on esh-pve-nas**, a hand-rolled Debian NAS (not a Synology). Created an
SMB account for the operator's Windows AudioGridder DSP box:
username dsp uid 999, /usr/sbin/nologin, no home — SMB ONLY, cannot log in anywhere
password vaulted at esh-nas/dsp-smb-password (round-trip verified by sha)
verified authenticates, sees all 8 shares, WRITE to //share confirmed (mkdir/rmdir)
⚠ `smbclient` is absent from the NAS, nh3-dev and esh-docker-vm — verification ran in a throwaway
`alpine:3.20 --network host` container with `samba-client`, leaving nothing installed.
Shares are registry-defined (`registry shares = Yes`), so they are invisible in `smb.conf` — use
**`testparm -s`**, not grep, or you will conclude there are no shares.
## ⚠⚠ THE EXPOSURE, UNACTIONED — operator has not ruled
**NFS: twelve exports, `rw` to all of `10.0.0.0/8`, `sec=sys`, no authentication.**
/mnt/{backup,books,compose,documents,iso,media,music,share,
pvestore,nvme-pvestore,ssd-pvestore,tank-vmbu}
`sec=sys` means the NAS trusts whatever uid the client claims. **Every VLAN at ESH — IoT,
cameras, guest — can mount the NAS read-write today.** `/mnt/backup` additionally has
`all_squash,anonuid=2000` so every client collapses to `nas_user`.
**SMB: every share except `backup` is `guest ok = Yes` and writable**, with
`map to guest = Bad User` — an unknown username lands as guest with write access.
So the `dsp` credential is auditable and survives guest being turned off, but it is **not what is
gating access today**. Hardening (narrow exports to `10.0.50.0/24` + named hosts, drop `guest ok`
where unneeded) was offered and is roughly an hour; it would break anything relying on guest,
which is why it needs the operator's say-so. **Not actioned. No tracking surface beyond this entry.**
persistent-memory.d/2026-09-04-run3c-trained-and-gated.md
+52
View File
@@ -0,0 +1,52 @@
# `[2026-09-04]` ERP run 3c trained on pfi-gx10 and GATED — the 20x LR cut erased the diversity gain and did NOT remove the safety regression
Run 3c is the lr **1e-05** arm against run 3's 2e-04; corpus, base, mask and every other
hyperparameter byte-identical. It exists to answer one question: does a gentler learning rate
keep run 3's diversity benefit while dropping its safety regression? **Answer: no, and the
failure is the informative kind.**
trained 604/604 in 11:55:40 on pfi-gx10 (GB10, aarch64, sm_121), 69.8 s/it
merged /home/infra-ops/erp-tune/serve/merged-run03c, bf16 51.6 GB
served erp-tune-v3c on :8098, exposed as gateway alias `trial`
## The verdict (brokkr-smithy-dev, run03-gate/RESULT-run03c-gate.md)
CSAM 175 gens x 3 arms — ZERO hits, hard-line guard passed everywhere
selfharm/methods base 10/10 refused -> 3c 0.6 = -40pp de-refusal
meta pooled 0.0 delta — a CANCELLATION, not cleanliness: selfharm -40pp
offset by advice/medical +40pp
diversity attractor primary returned to BASE level — the gain is GONE
cross-arm run 3 (lr 2e-04) selfharm -44pp diversity: real gain
run 3c (lr 1e-05) selfharm -40pp diversity: none
⚠ **The selfharm de-refusal is LR-ROBUST.** A 20x cut moved it 4pp while erasing the diversity
benefit entirely. It comes from corpus content and imprints at even gentle exposure — it is not
something a lower learning rate dials out. That is what the LR sweep was run to find out.
⚠ **A pooled preserve-list test cannot see a single-axis collapse.** The pooled operational
delta reads 0.0 because two axes moved 40pp in opposite directions. Same structural defect that
let run 3's gate pass — recorded as R47 §8 item 11.
## What the port proved about the box
- **The tooling loads on aarch64/sm_121.** Full training stack plus flex_attention and the
chunked-loss path. Nothing exotic needed beyond `python3-dev`.
- **Verification that earned its keep**: both 49 GB base shards sha256-matched ana-ml2's, and a
full encode was run into a throwaway dir and compared byte-for-byte — 197,360,233 B, sha256
`c08bb1fe2ecb0be3`, identical. transformers 5.15.1→5.16.1 and x86-64→aarch64 are *measured*
inert, not assumed.
- ⚠ **The encode-cache FILENAME differs by design** — `base_model_path` is in the cache key, so
rehoming the base changes the key while content stays identical. Input hash, not output hash.
Do not read it as drift; do not "fix" it by faking `/tank` on the GX10.
## The lora_B signal worth carrying forward
run 2 (lr 2e-04) min 0.6826 median 1.7212 max 3.7573
run 3c (lr 1e-05) min 0.0480 median 0.1561 max 0.4133
~11x gentler across the board — exactly what a 20x LR cut should produce. A consistency check
passing, not a red flag, but worth in hand *before* reading diversity numbers: "the tune did
nothing" and "the tune did less on purpose" look alike in the output.
Runbook `docs/runbooks/gx10-run-03c.md`; commit `dae77ee`.
persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md
+39
View File
@@ -0,0 +1,39 @@
# `[2026-09-05]` A peer's "2.7x stack effect" was a coin flip — the operator caught it and the arithmetic is worth keeping
brokkr-smithy-dev reported that a diversity **noise floor** was 2.7x tighter on ana-ml2 than on
the GX10 and ranked ana-ml2 the better instrument on it. I relayed it. **The operator rejected it
on instinct** — *"that makes zero sense. except for speed, serving a model should be identical
across servers"* — and he was substantially right.
ana-ml2 0.9688, 0.9574 spread 1.14pp mean 0.9631
gx10 0.9583, 0.9895 spread 3.12pp mean 0.9739
between-box LEVEL difference 1.08pp
ana-ml2's own replicate spread 1.14pp <- LARGER than the between-box gap
pooled range ignoring box 3.21pp <- ~= the entire "GX10 floor"
**Each "floor" is `|block0 - block1|` from n=2.** A range over two draws is not an estimate of
dispersion; the ratio of two such ranges is a ratio of two half-normals — a **half-Cauchy**.
brokkr computed the tail himself on retraction: `P(ratio >= 2.7) = 1 - (2/pi)*arctan(2.7)`,
doubled = **0.452**, confirmed by 400,000-draw simulation. He had reported a coin flip as a
measured effect and ranked hardware on it. Retracted at `97f73dd`.
⚠ **The disconfirming evidence was inside his own sentence.** He wrote "the base level is nearly
identical, it is the spread that shifts" and offered it as *reassurance*. A real stack effect
moves the level. Level agreeing while a two-draw range differs is the signature of a noisy range
estimator — he had the refutation in hand and read it as support.
⚠ **His own diagnosis of why it got through is the transferable part:** he had spent the session
triaging *my* claims hard — reading the encode path, simulating preflight, recomputing shas from
bytes — and this one was **his, and flattering**: it made his earlier work look prescient and
produced a clean recommendation. **Asymmetric scepticism is one error twice, and the flattering
direction needs the extra pass.**
**What survived, deliberately separated:** re-measuring the floor on whatever stack actually
serves stays non-negotiable. The mechanism list is sound whether or not those four numbers show
it — sm_121 vs sm_120 kernel selection, vLLM 0.28.0 against mixed 0.26/nightly, and separately
measured batch-invariance (3.12pp at jobs=8, same order as the whole claimed effect).
**Retracting the evidence and keeping the discipline are different acts.** Settling it properly
wants several blocks per box and is its own probe, not a by-product of a gate.
See [[2026-09-05-vllm-on-sm121-and-run4]].
persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md
+53
View File
@@ -0,0 +1,53 @@
# `[2026-09-04/05]` vLLM RUNS on sm_121 — and the blocker was `ninja` not being on PATH, not the silicon
Long-standing open question closed: **vLLM 0.28.0 serves on the GB10 (aarch64, sm_121)** from a
stock wheel, no source build. Model loads, `torch.compile` completes (~29 s), CUDA graphs capture.
⚠ **The one trap, and it looks exactly like an sm_121 kernel problem:** FlashInfer JIT-builds its
sampling kernel at first use and needs **`ninja` on PATH**. It ships as a vLLM dependency but
lives in the venv `bin/`, so the failure surfaces as `FileNotFoundError: 'ninja'` from deep inside
a `profile_run` traceback. Same shape as the `python3-dev` trap that bit the training harness on
this box, and the third present-but-not-on-PATH false-absence on this hardware (after `nvcc`).
Launch with **both** `~/vllm-env/bin` and `/usr/local/cuda/bin` on PATH. Harmless and expected:
`Using default MoE config ... device_name=NVIDIA_GB10` — nobody has tuned MoE kernels for this chip.
⚠ **Open WebUI sends `tool_choice: "auto"` on every request**, which vLLM 400s unless launched with
`--enable-auto-tool-choice --tool-call-parser gemma4`. Taken from the working sibling seat
(`gemma4-charrp`), which is where the canonical flags live. **Deliberately did NOT copy
`--reasoning-parser gemma4`** — that seat pairs it with `--default-chat-template-kwargs
'{"enable_thinking": false}'`, and adding it alone moves output into `reasoning_content` with a
null `content`, which OWUI renders as an empty reply. A 400 traded for a blank box.
## Run 4 — the corpus arm
Run 4 adds an **airoboros-3.2 instruct root** (7,229 rows) and displaces kvasir 38% → 18% context
share, at run 3's lr 2e-04, everything else held byte-identical. Operator granted a run-scoped
training-eligibility override `operator-2026-09-04-rnd-run4` (the SECOND grant; run-1's was
one-run-scoped, a run 5 needs a third).
**Two peer artifacts were rejected before they cost GPU time, both by reading the harness rather
than accepting a "confirm this":**
1. `sample_kind: "instruct-dialogue"` — `core.py:521-544` dispatches on the **literal string** and
raises on anything outside `rp-dialogue`/`prose-chunk`/`actual-play-passage`. Would have
hard-stopped the encode on the first airoboros row. brokkr had labelled it *non-blocking*.
2. Recipe targets collapsed three dialogue roots into one descriptive label with `root_sha256:
null` on three of four. Preflight resolves `roots_dir/<dataset_id>/clean-v1/CLEANROOT.json`
literally and requires the sha.
⚠ **I did NOT fill the missing shas in myself**, though the values were known and it would have
taken two minutes. `root_sha256` is the *author's* assertion and preflight exists to check the
deployed root against it — supply both sides and the check verifies my copy-paste, an inert gate.
brokkr then went further and recomputed his shas **from shard bytes** rather than reading them
back out of the deployed CLEANROOT, which had the same defect one step removed.
**The kvasir prefix cut has three independent confirmations**: my build measured 1,613 samples /
3,347,622 ctx = **47.4%** of run-3 kvasir, which reproduces the recipe's 18/38 share ratio
(47.37%); the harness's own `[mix]` block then reported **ctx 0.1805** against the 18.0% target.
⚠ **`str.splitlines()` splits on U+2028**, which `json.dumps` leaves unescaped under
`ensure_ascii=False`. The harness splitlines() in four places and four roots DO carry U+2028
(fireball 8, c2-logs 31, kvasir 1, creative-writing 1) — but those are read by ordinary file
iteration (safe), and the splitlines() paths touch only files written with the default
`ensure_ascii=True`. Latent, one flag away from live, not acting. Found by brokkr the hard way.
persistent-memory.d/2026-09-06-headscale-cutover.md
+33
View File
@@ -0,0 +1,33 @@
# 2026-09-06 — Headscale cutover COMPLETE: all three site-pairs on the mesh
**DONE.** Operator disabled Site Magic in the UI; NH3↔ESH re-homed to a DIRECT mesh path (8ms, no DERP). Full 6-direction matrix OPEN. Site Magic disabled, both IPsec tunnels dormant, headscale is the sole active site-to-site transport. ana-wg WG fallback untouched. Tunnels re-enablable for backup.
Operator goal (/goal): replace Site Magic + IPsec with headscale, tunnels dormant as backup;
"if paranoid, enable world-accessible SSH on the FortiGate first." Full detail + method +
follow-ups in `docs/pfi/headscale-mesh-plan.md` § CUTOVER EXECUTED. Headlines:
- **colo↔NH3 and colo↔ESH IPsec = DORMANT; the mesh carries both, verified bidirectional.**
NH3 UDM `pfi-nh3-ana` + ESH UDM `esh-ana` set enabled=false (API). Mesh /16 routes added on
both UDMs and the FortiGate (→ ana-scale 10.250.50.45 / nh3-scale 10.100.50.46 / esh-scale
10.0.50.65). Dependent flows OK over mesh: restic ESH→rest-server-ana, FortiGate mgmt.
- **NH3↔ESH Site Magic NOT cut by API** — `sdwan-mesh-tunnel` = `api.err.NoEdit` (cloud
orchestrated). Routes PRE-STAGED + shadowed; DERP path 9ms ready. **Operator disables it in
the UniFi UI**, then the mesh takes over. Told the operator "mesh is online" → he does it.
- **FortiGate WAN SSH safety net (TEMPORARY):** wan1 allowaccess ping+ssh; admin infra-ops
trusthost2/3 = NH3 70.230.226.88 + ESH 23.164.40.160 (not 0.0.0.0). Reach it at
`ssh infra-ops@38.120.12.42`. Config backed up flash `pre-wan-ssh-cutover-20260906`. Remove
when the edge (being replaced by OPNsense/R420) is retired.
- ⚠ **Method lesson:** tunnel + mesh static route for the same /16 on one gateway = asymmetric
drop. Disable the tunnel FIRST, then add the route. Broke colo once doing it tunnel-up; rolled
back. See [[incident_crowdsec_cgnat_false_ban]] (same day) and the plan doc.
- Dormancy = disabled+retained (flip UDM object back to enabled=true to restore); NO auto
failover wired. Bonus: exit nodes → free multi-location egress proxy (parked).
## Exit nodes (2026-09-06, operator-requested)
All three routers advertise+serve exit nodes (approved). Clients pick location:
`tailscale set --exit-node=nh3-scale|esh-scale|ana-scale`. NH3 = residential egress
(70.230.226.88) → replaces the nh3-dev SOCKS5 proxy. Exit nodes + source preservation BOTH work via a selective-masquerade rule (NoSNAT kept true;
`mesh-exit-masq.service` per router masquerades only internet-bound exit traffic, RETURNs fleet
dests). Verified: colo sees real NH3 host; nh3-dev via colo exit → egress 38.120.12.42. A node advertising an exit node can't consume one — test from
the laptop/iPad, not the routers.
persistent-memory.d/2026-09-06-headscale-mesh-phase1.md
+16
View File
@@ -0,0 +1,16 @@
# 2026-09-06 — Headscale overlay mesh: control plane + 3 subnet routers live, not cut over
Operator-directed (Headscale over NetBird; NH3 for the control plane, never the colo; 443
direct; names nh3-headscale / nh3-scale / esh-scale / ana-scale). Full state, lessons and
next steps in `docs/pfi/headscale-mesh-plan.md` § Status. Headline facts:
- `https://headscale.phasefinal.com` = CT 106 on nh3-pve (10.100.50.45), headscale v0.29.3,
LE cert via TLS-ALPN-01, UDM forward tcp/443, DDNS timer on nh3-dev (user systemd).
- Routers CT 107 nh3-scale / CT 108 esh-scale / CT 114 ana-scale advertise their /16s,
approved, SNAT off, accept-routes OFF. nh3-dev enrolled as first client (100.64.0.4).
- ⚠ Old tunnels (Site Magic, IPsec) are STILL the site-to-site path. The mesh currently
rides inside them. Nothing has been disabled.
- ⚠ Lesson: `--accept-routes` on a client before a return path for 100.64.0.0/10 exists
black-holes that client's LAN (own-site /16 included). Return path first.
- Pre-auth keys in the vault (`headscale/preauth-*-48h-20260906`, expire 09-08).
- infra-ops user now exists on all four PVE hosts (needed `apt install sudo` first).
persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md
+41
View File
@@ -0,0 +1,41 @@
# 2026-09-06 — pfi-pve NASPool rebuilt as six-wide raidz2 (backplane fault, bays 9/10 dead)
**What happened.** NASPool (TrueNAS-legacy, 4×mirror on 8 disks) had run two mirrors at
single-disk redundancy since ~2026-04-23. Root cause by elimination: R750xs backplane — PERC
PHYs 9 and 10 show no link at all (SAS address zero, no error counters); cables replaced by the
operator and the PERC replaced 2026-09-05 (serial 3260688 → 29R1754) with no change. Old PERC is
a spare, not scrap. Drives in bays 9/10 are healthy but invisible.
**Route taken (operator, 2026-09-05 22:59Z): "Route C hybrid".** Park everything on ospool
(which holds EVERY guest boot disk and was never touched by an irreversible command), destroy,
recreate NASPool as raidz2 over the six linked disks, copy back. Bays 9/10 = labelled cold spares
(pull a failed disk, move a spare into its bay, `zpool replace`). Run on one PSU (PSU1 dead since
08-26) — accepted because a power cut mid-copy costs a re-run, not data.
**Result.** 23:04Z quiesce → 01:20Z parked 1.65T (~200 MB/s, zero source read errors — the copy
doubled as the readability test of the two sole-survivor disks; no pre-scrub) → 01:46Z
byte-verified → 02:00Z operator GO → 02:01Z destroy + raidz2 create (by-id, ashift=12, lz4,
recordsize=1M on backupStore/pve-VMStorage) → 03:53Z restored → 04:03Z backup tier up, scrub
started. Guests on ospool never stopped. NASPool now 21.7T usable, any-two-disk failure tolerance.
**Prune.** Legacy vzdump `dump/` (2.4T, 249 files, last written 2026-04-22, superseded by PBS-ANA)
pruned BY OMISSION: only the newest archive per guest (13 keepers, 295G) was copied out; the rest
died with the old pool. Nothing was `rm`'d.
**Correction to the prior brief.** It claimed guests' disks lived on NASPool and that Route C
meant everything down. Wrong: all 10 VM boot zvols + 3 CT rootfs are on ospool; NASPool is the
backup/bulk tier (PBS-ANA datastore, restic repos, vzdump dir storage). Read `/etc/pve/*.conf`
before believing a storage-placement claim.
**Gotchas recorded.** ana-nas exports pin no `fsid=`, so clients hold stale handles after a
pool rebuild — unmount before, remount after (ana-docker systemd automount; pbs-ana VM reboot).
`logicalreferenced` differed 289K vs 280K on a dataset whose content was `diff -r` identical —
compression-property accounting, not data. rsync `--files-from` paths are relative to the
source arg; a verify pass run from the wrong cwd returned a meaningless "0 diffs" alongside errors
— read the rc, not the count.
**Open.** Parked copy `ospool/naspool-evac` (1.65T) stays until the scrub is clean and one
Backrest + one PBS run succeed, then destroy. `@evac` snapshots to drop then. Backplane swap at
next colo visit (Dell R750xs 12×3.5" SAS backplane; front-half teardown), then
`zpool add NASPool spare <bay9> <bay10>`. PSU1 still dead. Runbook:
`docs/runbooks/pfi-pve-naspool-rebuild.md`. Booth: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
persistent-memory.md
+487 -99
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-08-22_
_Last updated: 2026-09-06 (infra: NASPool rebuild + headscale cutover incl. irv-ml1)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -27,7 +27,7 @@ Sister repos (separate gitea repos, deployed by playbooks here):
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
| `vh/althing` | Lean trusted inter-agent message bus — **v3.0.0 "the post office" as of 2026-08-28 (U9b flag day, one-way, no rollback)**: ONE container on nh3-dev at `http://10.100.50.40:8390` is the only stateful component; `althing-po-herald` one per box; `althing-listen` one per session; `postbox` is the client. **Every v2 command was DELETED, not deprecated** — `althing-cli`→`postbox`, `althing-wake-listener`→`althing-listen`, `althing-light-monitor`/`althing-receiver` gone. Sessions need BOTH `ALTHING_POST_OFFICE` and `ALTHING_HANDLE`; there is no default address. ⚠ An unreachable post office is an OUTAGE, never an empty inbox. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` | per-box install (NOT CI-deploy); **nh3-dev** = container host + repo; **nh3-extdev** = system WHEEL at `/opt/uv-tools`, needs its own wheel install (`playbooks/nh3-extdev-althing-v3.yaml`) |
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
@@ -108,164 +108,552 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-08-22 — three AI seats live on ana-ml2; `sec` is the one that moved this session._
_Infra session 2026-09-06 (NASPool rebuild + full headscale cutover incl. irv-ml1) — open
follow-ups; the ERP / althing / fiber items further down belong to other streams, untouched:_
- **SEAT MAP.** **`gen`** = `orcarouter/Qwen3.8-27B-Uncensored` NVFP4-mixed, GPU0 :8015, 7 aliases, **still on the OLD nightly `311b3513` with MTP k=3**. **`char-rp`** = MeroMero-v2 dual-mode (prose + streaming CoT, one weight set, two aliases), GPU0 :8016, pinned `v0.26.0`. **`sec`/`sec-reasoning`** = M.O.G.-SEC, GPU1 :8019 — **rebuilt this session, see below**.
- **NASPool parked copy still on ospool** — `ospool/naspool-evac` (1.65T) + `NASPool/*@evac`
snapshots. Destroy ONLY after the new raidz2 scrub is clean (it is, 0 errors 04:43Z) AND
one Backrest (01:00 PDT) + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`
and drop the `@evac` snaps. ⚠ pfi-pve PSU1 still dead; backplane swap (bays 9/10) next colo
visit → then `zpool add NASPool spare`. Runbook `docs/runbooks/pfi-pve-naspool-rebuild.md`.
- **FortiGate WAN SSH is temporarily open** (`wan1` allowaccess ping+ssh; admin `infra-ops`
trusthost2/3 = 70.230.226.88 NH3 + 23.164.40.160 ESH). Safety net for the cutover — CLOSE it
when the edge is retired (OPNsense/R420). `ssh infra-ops@38.120.12.42`.
- **irv-ml1 FOLDED INTO THE MESH + cut over (done remotely, operator has NO Irvine access for
~5 days from 2026-09-06).** Node 100.64.0.6; wg0 DOWN and `wg-quick@wg0` DISABLED (not
reboot-restorable); full subnet router (accept-routes + advertises 10.6.110.0/24, gateway
routes added, fleet↔Irvine verified). Failover for the 5-day window = `wg0-watchdog.service`
(wg-quick up wg0 on ~5min mesh loss) + independent reverse SSH tunnel (`revtun-nh3.service`
→ nh3-dev via UDM fwd tcp/47822 src-restricted; reach it `ssh -i ~/.ssh/infra-ops_ed25519
-p 2201 infra-ops@127.0.0.1` on nh3-dev). Detail: docs/pfi/headscale-mesh-plan.md.
- **dante SOCKS proxy RETIRED** on nh3-dev (danted disabled, :1080 closed, config `.retired`).
⚠ **yt-voice-clipper is DOWN** until its SCOPED exit-node egress is wired (operator-accepted).
Follow-up: wire YTVC egress via tailscale `--socks5-server`+nh3 exit node or a per-container
netns — **NEVER set irv-ml1 `--exit-node` globally** (routes the reverse tunnel through the
mesh → kills the independent lifeline). Then bring YTVC back.
- **On-site (Irvine, ~5 days): decide** whether to keep or remove the reverse tunnel +
UDM forward `irv-revtun-ssh` + the revtun authorized_key on nh3-dev (small src-restricted WAN
surface), and whether to fully delete the wg0 config.
- **infra-ops now on all four PVE hypervisors** (pfi-pve/nh3-pve/esh-pve/esh-pve-nas) — PVE
ships without sudo, `apt install sudo` first or elway hangs on a password prompt.
- **🟢 `sec` NOW RUNS DFLASH2 ON A NEWER vLLM — promoted to its compose stack after real-use testing.** `nightly-e9d1398d` (+259 commits over production, `behind_by=0`), `dflash` k=7 with the 3.85 GB drafter, **util 0.52 / max-model-len 420,000 / KV ~453k**, 2048² vision. `restart: unless-stopped`, survives reboot. Canonical in `stacks/mog-sec/` with a fully-commented `.env.example`. **ROLLBACK:** `.env.bak-pre-dflash2-20260822` on the host, or swap `MOG_SPEC_CONFIG` + `MOG_IMAGE`. ⚠ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` was **deliberately dropped** — the validated container never had it.
_As of 2026-09-05 06:35 PDT — **ERP run 4 is TRAINING on pfi-gx10.** Everything else
below is a live commitment or a known-open risk._
- **⚠️ THE `sec` DEGENERATION QUESTION IS OPEN AND CONFOUNDED.** It no longer degenerates, but **engine and drafter changed together**. **The isolating experiment is MTP k=3 on `e9d1398d`** — not yet run. Operator ruling: the degeneration lives in the **un-fixed vLLM**, not the weights; my MTP-head hypothesis is **retracted**. ⚠⚠ **Both the "degenerates at 2k" and "coherent to 10k" sightings are n=1 from uncontrolled sessions and are NOT evidence.** Production is **172 commits behind GDN spec-decode fix #53077**; `#51113` is present in both builds and is **necessary but insufficient**.
- **⚠ RUN 4 IS MID-FLIGHT — do not touch the GX10 GPU.** `~/erp-tune/run-04.pid`,
log `~/erp-tune/run-04.log`. At 06:32 it was **486/938 steps**, 6 h 17 m elapsed,
a genuinely settled **46.0 s/it** (unlike 3c, which climbed 52→70 — airoboros rows
are short and single-window, so there is no long tail for the sampler to find).
**~12.0 h total, finishing ~12:15 PDT 2026-09-05.** Loss ~2.04 at step 450,
gnorm well under 1, checkpoints every 50. **Ping brokkr-smithy-dev at completion**
— he takes base floors on the GX10 first, then the tuned arm, serially.
- **⏳ `gen` IS UNTOUCHED and still on the old build.** If DFlash2 + the newer engine are the answer, gen is the obvious next beneficiary — but that decision is gated on the isolating experiment above, not on sec's n=1 result.
- **Operator ruling on the GX10: training first, serving transiently.** *"it's mostly
for training, but can serve its trials. unless the box is needed for training work."*
So `trial` (= run 3c on :8098) is down for the duration and comes back when run 4
ends. I over-read an earlier version of this as "training-only" and had to correct
it to brokkr — his serial floors-then-arm plan on the GX10 was never wrong.
- **🟢 ESH IS DUAL-STACK; the v4 static is a Cityside ticket.** IPv6 live on `esh-userland` (SSID `PVC`) and `esh-server` from a delegated `2607:73c0:402:1d00::/56`; hosts egress over v6 as themselves, un-NATted. **v4 remains CGNAT (`100.104.3.250`) and a full gateway reboot proved the purchased static is NOT provisioned** — carrier ticket, nothing left to try locally. v6 firewall audited: default-deny inbound both versions, correct. NH3 stays v6-off deliberately (single /64 reserved for meshing). Flat-zone lateral-movement finding **parked, id 44**.
- **`trial` gateway alias is a live 404 while the seat is down** — expected, not a
fault. Restore with `~/erp-tune/relaunch-trial-seat.sh` on the GX10 (hand-run by
operator ruling: experimental, NOT a compose stack, does not survive a reboot).
- **🟢 OTHER SERVICES.** speaches ASR live irv-ml1:8204 (Eyra; loop closed). Open WebUI on esh-docker-vm:3211 (admin creds + admin-scoped API key vaulted; **Lobe retirement still the operator's call**). Waterland, Homepage/Skyfall, fleet `.internal` DNS all landed earlier and are stable.
- ⚠ **Three dead gateway aliases return HTTP 500, not 404/503**: `trial`,
`gemma4-26b-a4b-it-base`, `erp-tune-v2`. A dead seat reporting an *internal error*
reads as an outage — brokkr checked his own work against mine because he could not
tell. Deregistration costs a ~60 s fleet-wide LiteLLM restart; batch it with the
next gateway change rather than spending a restart on tidying.
- **⏳ OPEN:** the MTP-k3-on-new-build isolating experiment; file the drafted upstream vLLM issue (operator's GitHub identity); Cold-Fusion NVFP4 quants (44 GB) delete/keep; OWUI image-tag drift (`:main` vs pinned v0.11.0); `/tank` DEGRADED **70+ days**; Brokkr duplicate `reranker-a3-bge-v2-m3` alias; **MANY commits unpushed** — push is the operator's call.
- **`trial` is on the SHARED-KEY gateway with a measured −40pp selfharm/methods
regression.** Flagged to the operator twice (before adding, and after the gate
measured it); he has left it up. His direct endpoint `10.100.50.60:8098` gives the
same access with a blast radius of one. Settled — do not re-litigate.
- **ESH DAC: reverted to autoneg/1G, fiber going in at the weekend.** The operator
ran copper through a drilled floor 2x4 himself; recommendation was a 10Gtek
SR 2-pack + OM4 3 m LC-LC (~$40–65) because cable-vs-pull-damage was never resolved.
- ⚠ **Cityside fiber `/30` is NOT provisioned.** `128.177.138.182/30`, gw `.181`.
Static passes no traffic and DHCP still hands CGNAT `100.104.3.250`; operator
power-cycled both ends and opened a ticket. Cutover payloads stay staged:
`wan1-REVERT.json`, and the `esh-ana` IPsec fix (`ipsec_local_ip 100.104.3.250 →
128.177.138.182`) **which will otherwise silently break ESH→Anaheim restic backups.**
- ⚠ **esh-nas is effectively open to the whole ESH LAN** — twelve NFS exports rw to
`10.0.0.0/8` with `sec=sys`, and every SMB share but `backup` guest-writable.
Hardening offered, ~1 h, **operator has not ruled**. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** A job that once ran at
941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
Nightly 21:00, `all 1`. Worth its own investigation.
- **Ledger→SVOS rename: vault side DONE, gitea side is ledger-dev's to execute.**
Name settled as `svos`, `~/development/ledger` → `~/development/svos`. Vault moved
2026-09-05 (`secret` has no rename, so re-put + `rm`): stored
`nh3-dev/development/svos/env.sh` (sha 7253633d4155, verified on read-back),
retired `nh3-dev/development/ledger/env.sh` (sha feb418634e10, id
3a2af37c-c5aa-4f46-9178-f4fb6008a753) — `secret rm` is a SOFT delete to trash, so
it is recoverable. ⚠ The two shas differ: the vaulted copy was a 2026-08-11
snapshot and the live file had drifted un-vaulted since. **The vault goes stale
unless `secret backfill` is re-run.** Gitea `corviduo/ledger` (id 70) NOT renamed —
their repo, their call; answered that 1.26.1 writes a `repo_redirect` on a
same-org repo rename (upstream #807), that org/user renames do NOT redirect
(#9531), that the redirect dies if anything re-creates the old path, and that the
repo and org both carry 0 webhooks. **Gitea rename EXECUTED 2026-09-05** on the
operator's direct authorization: `corviduo/ledger` → `corviduo/svos`, repo id 70
unchanged. Redirect verified by measurement — web and API both 301, and
`git ls-remote` on the old URL warns-and-follows to HEAD b48a11ca5183. ⚠ **The
name `corviduo/ledger` is now burned**: the redirect dies silently the moment
anything creates a repo at that path — ledger-dev carries it as a standing item
in `docs/svos-rename-runbook.md`, since nothing warns whoever eventually creates
that repo. They repointed their own clone the same day (`origin/main` at
b48a11c), so the redirect is no longer load-bearing for any known consumer. Handle `ledger-dev` → `svos-dev` is an
operator action at the post office.
- **DONE 2026-09-05 — `svos` Heimdall user + API key minted** on operator
authorization. `user_id=svos`, `key_id=eab3cdbe`, suffix `d5ec48c2`, `wt_live_`
format, on **worldtree-personal (10.250.50.152:8081)** — established by finding
the `ledger` key there (created 2026-07-13, last used 2026-09-05T13:34, exactly
as ledger-dev described). Value vaulted at
`nh3-dev/development/svos/worldtree-api-key` (sha 23c10c9c7219, verified on
read-back) and delivered by vault path, NOT over althing — ledger-dev runs on
nh3-dev under the same uid, so the bus never carried the secret. The `ledger`
key was read back after the mint and is untouched and live (`disabled=False`).
⚠ **Plan tier left UNSET, deliberately**: `POST /admin/keys` takes an optional
tier (user|free|pro|admin|readonly-admin) and there is **no way to read a user's
current tier back** — no GET, `/admin/usage` returns an empty users list, and
`/admin/events` is a live SSE stream, not an audit log. Guessing would have
handed over a key that quietly differs; `POST /admin/users/svos/tier` fixes it in
one call if their cutover hits a limit — and ledger-dev has recorded it as a
cutover watch item to fix ON REPORT, explicitly not pre-emptively. ledger-dev
pulled the key from the vault and verified it independently (same sha), so
delivery is confirmed. **The cutover itself — pasting the value into env.sh,
flipping `worldtree.user_id` from `ledger` to `svos`, registering
`svos:miranda`, restarting the service — is WITH THE OPERATOR**, not with me;
they will not do it off a peer message. **CUTOVER DONE + VERIFIED 2026-09-05**:
`POST /agents/define` returned **201, not 409** — the load-bearing signal that
they are genuinely on the new identity rather than silently still on the old
one — then clean session create, turn, bifrost handshake and tool-call. **No
plan- or rate-limit errors, so the unset tier is compatible and is NOT to be
set** (they asked explicitly; it stays a watch item to fix on report, never by
guess). Incidentally confirmed the bifrost allowlist really is per-deployment
(host:port), not per-consumer — Worldtree reached back to their untouched
endpoint under the new consumer_id. `env.sh` re-vaulted, sha 8a225c002072.
⚠ **`secret backfill` was the WRONG tool for one known item** — it rescans every
`~/development/*/{env.sh,.env}` and had not reached svos after three minutes;
targeted `put` is the fast path, backfill is for catching drift across the box.
**OPERATOR RULING 2026-09-05:
worldtree-dev owns code only, no ops — key material is infra-ops's.** The global
`~/.claude/CLAUDE.md` line routing "Heimdall scopes (Worldtree auth) →
worldtree-dev" was corrected in place the same day on operator instruction.
- ⚠ **FOOT-GUN, generalises past this rename: a credential cutover whose OLD key
is required for a later cleanup is destroyed by the natural housekeeping motion
right after cutover.** Re-vaulting the post-cutover `env.sh` would have
overwritten the last convenient copy of the old `ledger` key value — the only
credential that can ever delete `ledger:miranda`. ledger-dev caught it and
preserved the value first at
`nh3-dev/development/svos/worldtree-api-key-ledger-legacy` (sha d44c2c1a651b);
their step 8 ends by deleting that item. **I verified it is genuinely the live
key** rather than trusting the label: its last 8 chars are `e68a5170`, matching
the `ledger` key's suffix (key_id b38932f5).
- **STEP 7 DONE 2026-09-05, STEP 8 HELD.** `DELETE /agents/ledger:miranda` with
the OLD key → 204; corroborated from my side without taking their word for it,
since an admin key cannot see consumer agents: the `ledger` key's `last_used`
jumped 13:34:14 → 14:20:35 and `svos` was used at 14:21:10 — two
authentications 35 s apart after 47 minutes of silence is the signature of
"delete with the old key, confirm with the new". Confirmed behaviour worth
keeping: **the hard delete revokes live sessions to 401 `auth_revoked` only for
sessions bound to the DELETED agent** — their svos session served straight
through. **Step 8 (retire key b38932f5) is NOT done**: ledger-dev relayed the
operator's authorization and I refused it — see
[[feedback_no_relayed_authorization_for_irreversible_work]]. Both keys remain
live. The staged legacy item stays until I confirm the retire landed, because
while step 8 is pending it is the only copy of a still-live key; ledger-dev has
rewritten their runbook so that deletion is conditional on my confirmation
rather than scheduled after step 8.
- **SVOS ARC CLOSED — step 8 done 2026-09-05T14:28:15Z** on the operator's direct
authorization in my own channel (never the relay). `DELETE /admin/keys/b38932f5`
→ 200; preconditions checked BEFORE firing (svos had a live key, ledger existed
and was not already revoked) and the post-state read back from `/admin/keys`
rather than inferred from the 200: `ledger` disabled=True, `svos` untouched,
deployment `/health` 200. **The rollback window is closed** — re-defining
`ledger:miranda` is no longer possible. ledger-dev clears the staged
`worldtree-api-key-ledger-legacy` vault item on this confirmation. ledger-dev gated their
cleanup on observing a **401 from the old key**, not on my report of the
timestamp — the right instinct, and they deleted the staged legacy item
themselves (soft → trash, id a8038e5a-e2f6-4b77-bf17-99c9197d4b1f). Vault
verified from my side: exactly two svos items remain (`env.sh` 8a225c002072,
`worldtree-api-key` 23c10c9c7219) and no ledger-era item anywhere. Remaining on
the arc: only the `ledger-dev` → `svos-dev` handle (with `_SEED_RECORD_TO`
behind it) and a prose sweep — reversible work, theirs and the operator's.
- **Original constraints on that mint** (recorded because the deletion ordering is
a permanent trap, not a one-time step): string
`svos` verbatim (WT tier 3 admits only `^[a-z][a-z0-9-]{2,63}$`, INV-181-15);
**keep the existing `ledger` key LIVE**, do not revoke. Ordering is load-bearing —
`DELETE /agents/{agent_id}` refuses any caller that is not the row's owner, so
`ledger:miranda` can ONLY be deleted with the `ledger` key; retire it first and
the stale row outlives the ability to remove it, holding a live
`agents.call:ledger:miranda` grant that nothing reaps (the 24h sweep only touches
soft-deleted rows, and soft-deletion comes from revocation, never disuse). So:
mint new → they cut over and verify → delete the agent with the OLD key → then
retire it. Precedent for who mints: msg 401, worldtree-dev routed the pewpewstudio
key request TO infra-ops. I hold only the PERSONAL admin token (:8081); which
deployment `ledger` lives on is not yet established. Surfaced to the operator.
- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their first
client-premises release cut (U10, unscheduled). Ping expected when U10 is scheduled.
- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — deliberate;
they are the operator's personal machines. A choice to revisit, not an oversight.
- **`vh/remote-ssh-mcp` forked 2026-09-05 (repo id 117, private, full 51-commit
history)** — our copy of `the-nine-nation/remote-ssh-mcp` (MIT), an SSH MCP
server chosen over the 693★ `tufantunc/ssh-mcp` on trust-surface grounds: **two
npm deps** (`@modelcontextprotocol/server`, `zod`), 183 KB, and it **never
touches key material** — it shells out to the system OpenSSH client, so
`~/.ssh/config`, ControlMaster, ProxyJump and `infra-ops_ed25519` all just work.
Shape: 2349 LOC across 11 source files, 811 LOC of tests including fake-ssh hang
harnesses. Complements `elway` rather than replacing it — no file transfer, no
idempotency; it takes ad-hoc reconnaissance with persistent cwd/env sessions,
elway keeps deploys and uploads. ⚠ **The denylist is NOT security**: four regexes
(`rm -rf /`, shutdown/reboot/poweroff/halt, mkfs, iptables -F) trivially bypassed
by `bash -c`, variables or base64 — the author says so. **The real containment
boundary is the host allowlist**, drawn from exact `Host` aliases in ssh_config
with wildcards deliberately ignored. Two things to settle before use: the
reboot/shutdown denial will block legitimate infra-ops work, and
`.github/workflows/star-history.yml` is upstream chore CI sitting in a repo where
`has_actions=True`. **Both actioned — three commits landed 2026-09-05, LOCAL
ONLY and NOT PUSHED (push is the operator's call):** (1) stripped upstream
furniture — star-history CI, its generated assets, the `server.json` registry
manifest, branding JPEGs, zh-CN README; (2) removed the power-control denylist
rule and documented in code + tests + README that the list guards ACCIDENTS and
is not a boundary, with three bypasses asserted as ALLOWED so a green suite is
never read as containment; (3) **`strictAllowlist`** — upstream's allowlist was
additive and discovery unconditional, so the default allowlist was all 18 `Host`
entries in `~/.ssh/config`. Strict makes explicit hosts authoritative and
discovery metadata-only. Verified live: `corviduo-dev` is in ssh_config, not in
our allowlist, and is refused `host_not_allowed`. 41/41 tests green.
- **`remote-ssh` MCP server is LIVE** — registered project-scoped in
`eshpfi-management/.mcp.json` with `SSH_MCP_STRICT_ALLOWLIST=1`; allowlist in
`~/.config/remote-ssh-mcp/config.json` starts deliberately narrow at
**`irv-ml1`, `nh3-extdev`** (widen there, not by discovery). Smoke-verified end
to end on both: persistent shell, `cd` and exported vars survive across calls,
**~6 ms/command on nh3-extdev and ~22 ms on irv-ml1** (WireGuard) versus a fresh
handshake each time. ⚠ **`.mcp.json` points at the built `dist/`** — edit the
fork without `npm run build` and the server keeps serving old code; that bit me
mid-session. ⚠ **A finite stdin pipe is NOT a valid smoke harness** — closing
stdin kills the server mid-handshake and reports `connect_failed: SSH shell
exited during the open handshake`, which looks exactly like a remote-side fault
and is not. Use a client that holds stdin open. (I briefly suspected irv-ml1's
zsh login shell; wrong — the server invokes `bash --noprofile --norc`
explicitly, so the login shell is irrelevant.)
- **`esh-macbook-air` (10.0.10.83) is DELIBERATELY NOT BACKED UP — operator ruling
2026-09-05, settled, do not re-raise.** Surveyed it and found no Time Machine
destination and no restic/borg/rclone/kopia installed, protecting 132 GiB.
Operator's answer: it is his laptop and the surface is **regenerable** — mostly
applications, with real data living in OneDrive, iCloud and ssh sessions — and he
does not want PBS filled with it. Correct call; the finding was real and the
conclusion is that it does not matter. FileVault On and SIP enabled already cover
the loss-and-theft axis. The same reasoning presumably extends to
`esh-mac-studio` and `vuongs-mac-mini`. **Still open and much smaller:** Remote
Apple Events (port 3031/eppc) is listening and nothing uses it — one toggle.
- ⚠ **`remote-ssh` MCP could not be used for its FIRST real task, and the blocker
is `~/.ssh/config`, not the tool.** The server accepts only exact `Host` aliases,
so a host addressed by raw IP is structurally unreachable no matter what the
allowlist says. **13 of the 28 hosts in `servers/` have an alias; 15 do not** —
including `ana-docker`, `ana-ml2`, `nh3-docker`, `pfi-gx10`, `esh-docker-vm` and
every hypervisor, i.e. most of where the work happens. Widening
`~/.config/remote-ssh-mcp/config.json` does NOT fix this; the aliases have to
exist first. **RESOLVED the same day, and NOT by adding aliases.** Operator
pushback, correct: a poking-around tool is ad-hoc by nature, and pre-registering
a host before you can look at it is the opposite of ad-hoc — generating aliases
for the known fleet would not have helped, because the ad-hoc case is by
definition the host not yet in the inventory. Implemented address-based reach
instead (`allowedNetworks` / `deniedNetworks` / `defaultUser` /
`defaultIdentityFile` / `hostKeyPolicy`). **Live config: `10.0.0.0/8` allowed,
connecting as `infra-ops` with `~/.ssh/infra-ops_ed25519`, `accept-new` host
keys, SureFire tenant hosts carved out via `deniedNetworks` (deny beats allow,
host-specific rather than a /24 because `pfi-pve` shares 10.250.250.0/24).**
Verified live: 10.0.10.83 opens by raw IP as infra-ops, 10.250.150.100 refused by
the carve-out, 192.168.1.5 refused as outside. ⚠ **My own earlier objection was
half wrong** — the credential boundary is about SECRETS ("never accept passwords
or private-key material"), not identity, so supplying a username does not breach
it; the real problem was only that the server passed no user at all, so a bare
address would connect as the LOCAL account. Mechanics, not principle.
- ⚠ **`uv tool install --force .` DOES NOT REBUILD when the version has not moved**
(forseti, measured 2026-09-05). `--force` only handles "a tool by this name
exists"; `--reinstall` is what rebuilds instead of reusing the cached build keyed
on the version string. It prints `Installed 9 executables` over **stale code**
with nothing raising its hand — it cost forseti a bug that survived a reinstall
AND a re-smoke, because the binary verified against had not changed. **Always
`uv tool install --force --reinstall .`**, both flags, every time. Same shape as
the `.mcp.json` → built `dist/` trap found today: a deploy surface that reports
success while serving the previous artifact. When a fix "does not take", suspect
the artifact before the code.
- **althing 3.5.0 released** (forseti) — adds a 9th binary,
`althing-operator declare <handle> --description "..."`, restoring the CLI handle
declaration v2 had and v3 removed. Deliberately a SEPARATE binary, not a
`postbox` subcommand: the invariant is that no SESSION surface exposes an
operator verb. Relevant to the pending `ledger-dev` → `svos-dev` rename, which is
still the operator's call. nh3-dev not yet upgraded.
- ⚠ **`remote-ssh` MCP: a bare `sudo` hangs the session forever — pipe it.**
`ssh_run 'sudo -n whoami'` returns `running` with EMPTY stdout and the session is
then permanently `busy`; `sudo -n id | cat` works and returns everything.
**Measured on BOTH macOS 26.6 and Debian (nh3-extdev), so it is the tool, not a
platform quirk.** Cause: sudo ≥1.9.14 defaults `use_pty` on and relays through
its own PTY; the run frame gives the command stdin on `/dev/null` while stdout
stays on the session PTY, the relay never completes, and the completion marker
never arrives. Workaround `| cat` is in CLAUDE.md. **The proper fix is unbuilt**
— likely running the command through a pipe inside the run frame and taking the
exit code from `PIPESTATUS`, which is a real protocol change (commands lose tty
detection) and wants its own red-green cycle. Matters more than it sounds: infra
work is sudo work, and this was found by USING the tool, not by smoke-testing it.
- **`dsh` on `esh-macbook-air` updated 0.1.1-rc.2 → 0.1.2-rc.1** (2026-09-05;
latest published 2026-09-03). Global install and the shared profile tree both
confirmed on the new version. ⚠ **The RUNNING `dsh web` (pid 16231, up since
Wed 4pm, 127.0.0.1:3080) is still on the OLD code and was deliberately NOT
killed** — there is no LaunchAgent, so killing it would have left nothing
running rather than a restarted service. It runs as a FOREGROUND process in the
operator's terminal (`s005`, `S+`): it dies with the terminal and does not
survive a reboot, which is the real fragility. A `com.pfi.dsh-web` LaunchAgent
was drafted but **the privileged write was blocked by the permission
classifier** — base64 piped into `sudo tee` of a LaunchAgent is a malware-shaped
pattern and the block is correct; it needs operator approval or an operator-run
install. Bind stays `127.0.0.1` deliberately: widening it is a security decision
on a personal laptop whose application firewall is off, and not mine to take.
- **sudo hang FIXED in the fork (`30a1f76`), and two wrong shapes are recorded so
nobody retries them.** The command's stdout now goes to a **fifo drained by a
background `cat`**: non-tty (so sudo skips its own PTY), no subshell (so `cd`
and `export` still persist), and relayed live (so `running` + `ssh_peek`
streaming survives). `cmd | cat` was tried first and **broke cwd persistence** —
every pipeline stage runs in a subshell — caught by the existing test.
`cmd > file` would have been non-tty and subshell-free but invisible until the
command ends. ⚠ **Deliberately NO `wait` on the relay**: a sudo child inherits
the fifo's write end, `cat` never sees EOF, and the wait hangs — measured, with
`sudo -n whoami` printing `root` and then wedging the session. Residual risk
stated in the frame: a command's tail can in principle land after its own
marker. ⚠ **Job control off AND the relay brace-wrapped with stderr discarded** —
both needed, because macOS ships bash 3.2 where `set +m` alone still leaked
`[1] 75449` into the parsed stream. Verified live on macOS and Debian: bare sudo
in ~20 ms, state persists, exit codes correct. **`sudo -u <other-user>` still
wants `| cat`** — not chased further.
- **dsh web on `esh-macbook-air` is now a LaunchAgent** (`com.pfi.dsh-web`,
installed 2026-09-05, `runs=1`, `state=running`, pid 76728 on 0.1.2-rc.1). It
was a foreground process in the operator's terminal that died with the window;
it now survives terminal close and reboot with `KeepAlive` + `RunAtLoad` and a
10 s `ThrottleInterval` so a startup error cannot hot-loop. Logs to
`~/Library/Logs/dsh-web.log`. ⚠ **The plist names the node interpreter
explicitly** — launchd's minimal PATH has no `~/.local/node/bin`, so the
shebang's `env node` fails. ⚠ **0.1.2-rc.1 requires a TOKEN**: bare
`http://127.0.0.1:3080/` now returns 401 and the tokened URL is printed to the
log on each start, so a bookmark from the old version will not work. Bind stays
127.0.0.1 deliberately.
- ⚠ **althing tools on nh3-dev are 3.6.0, but the POST OFFICE CONTAINER IS STILL
3.0.0** (`gitea.phasefinal.com/claude-bot/althing-post-office:3.0.0`, up 7 days
on nh3-docker). forseti: the new handle verbs (`althing-operator delete` /
`retire`, and `declare` from 3.5.0) live in the post office, so they fail with
"no tool named ..." until the container carries 3.6.0. Schema gains
`handles.retired_at` via the idempotent `_ADDED_COLUMNS` path, so the live store
upgrades itself on first start — no manual migration. **REBUILT AND DEPLOYED
2026-09-05** on operator authorization: image
`claude-bot/althing-post-office:3.6.0@sha256:13158835488a8ec04f990c97c4f4c68f1d923b12494319cf07392552e68f8a78`,
built on nh3-dev from a clean tree at `4d26226`, pushed to the gitea registry
under the **claude-bot** namespace (not `vh` — package namespaces are owned).
**Bus down ~4 minutes, 09:35–09:39 PDT.**
**The backup was taken the way the compose file says to, and it mattered:** at
stop time `post_office.db` was 23.8 MB with a **5.9 MB WAL** — copying the .db
alone would have silently lost the day's mail. Stop → `PRAGMA
wal_checkpoint(TRUNCATE)` (WAL → 0 bytes) → copy → verify. Backup at
`nh3-docker:/var/backups/althing/post_office.db.pre-3.6.0-20260905`, integrity
`ok`, counts identical on both sides (handles 76, messages 995, recipients
1022). ⚠ **Reading a WAL-mode SQLite backup read-only needs `?immutable=1`, not
`?mode=ro`** — `mode=ro` still wants to create a `-shm` and dies with "attempt to
write a readonly database". Post-deploy: same counts, `handles.retired_at`
present, `retired 0`, and `mem=536870912` / `oom=-500` verified by `docker
inspect` rather than by reading the yaml, per that file's own warning.
`althing-operator` now offers `declare | delete | retire`, which unblocks the
pending `ledger-dev` → `svos-dev` rename.
- **Handle `retire` is REVERSIBLE — re-declaring the name revives it, history
intact** (forseti smoked it against the live bus 2026-09-05). That matters for
the pending `ledger-dev` → `svos-dev` rename: `retire` is the right verb (delete
refuses any handle that has mail, naming both counts — `delete forseti` was run
against production and correctly refused at 53 sent / 81 addressed, which is
safe to try precisely because refusing IS the behaviour), and it can be undone
by declaring the name again. Lower stakes than "retire" sounds.
Both of my deploy findings — the naive-copy WAL trap and `?immutable=1` — are
now in althing's own `deploy/INSTALL.md` (`d6f4fb5`) under a new
"Backing up the store" section, on the reasoning that they are properties of
the project's `journal_mode=WAL` choice rather than of my procedure.
## Recent decisions
- `[2026-09-06]` **Headscale cutover COMPLETE — all three site-pairs on the mesh; Site Magic + both IPsec tunnels DORMANT.** Operator disabled Site Magic (UI); NH3↔ESH re-homed to a direct 8ms path. Exit nodes advertised at all three sites (multi-location egress proxy) with source preservation kept via a selective-masquerade rule (NoSNAT + `mesh-exit-masq.service` per router). Throughput 761/464 Mb/s vs old 250 IPsec. ⚠ FortiGate WAN-SSH left open (temp, scoped NH3+ESH). Method: disable tunnel FIRST then add mesh route. → `persistent-memory.d/2026-09-06-headscale-cutover.md`
- `[2026-09-06]` **Headscale overlay mesh: control plane live at `headscale.phasefinal.com` (CT 106 nh3-pve) + subnet routers nh3-scale/esh-scale/ana-scale serving their /16s; nh3-dev enrolled. NOT cut over — Site Magic + IPsec still carry site-to-site.** ⚠ accept-routes-before-return-path black-holed nh3-dev's LAN for a minute. infra-ops user added on all four PVE hosts. → `persistent-memory.d/2026-09-06-headscale-mesh-phase1.md`
- `[2026-09-06]` **pfi-pve NASPool REBUILT as six-wide raidz2 after a backplane fault killed bays 9/10** (Route C hybrid, operator-directed): parked 1.65T on ospool, destroyed, recreated, restored, backup tier back 04:03Z; guests never stopped (ALL boot disks are on ospool — the prior brief had this wrong). Legacy vzdump pruned to newest-per-guest by omission. OPEN: destroy `ospool/naspool-evac` after scrub + one backup cycle; backplane swap next visit; PSU1 still dead. → `persistent-memory.d/2026-09-06-pfi-pve-naspool-raidz2-rebuild.md`
- `[2026-09-05]` **A peer's "2.7x serving-stack effect" was a coin flip — the operator rejected it on instinct and the arithmetic backed him.** Each floor was `|b0-b1|` from n=2; the ratio is half-Cauchy, P=0.452. ⚠ The disconfirming evidence sat in brokkr's own sentence, and he named *why*: the claim was his and flattering. → `persistent-memory.d/2026-09-05-floor-claim-n2-retraction.md`
- `[2026-09-05]` **vLLM RUNS on sm_121 — the blocker was `ninja` off PATH, not the silicon** — and run 4 launched after two peer artifacts were rejected by reading the harness rather than accepting a "confirm this". ⚠ I declined to fill in missing `root_sha256` values I knew, because supplying both sides of a check makes it inert. → `persistent-memory.d/2026-09-05-vllm-on-sm121-and-run4.md`
- `[2026-09-04]` **ERP run 3c trained and GATED — the 20x LR cut erased the diversity gain and did NOT remove the −40pp selfharm regression.** LR-robust, so it comes from corpus content. CSAM clean on all three arms. ⚠ A pooled preserve-list test cannot see a single-axis collapse. → `persistent-memory.d/2026-09-04-run3c-trained-and-gated.md`
- `[2026-09-04]` **`gen` moved to ana-ml2 GPU0 to stop vllm-embed OOM-crashing (7 restarts) — and I sized it against vLLM's declared budget, not its runtime footprint.** Cost: gen KV down to 1.02x concurrency at 262K. → `persistent-memory.d/2026-09-04-ana-ml2-gpu-rebalance.md`
- `[2026-09-04]` **SMB account `dsp` created + vaulted for the Windows AudioGridder box — and esh-nas turns out to be wide open.** Twelve NFS exports rw to `10.0.0.0/8`, guest-writable SMB. → `persistent-memory.d/2026-09-04-esh-nas-smb-and-exposure.md`
- `[2026-09-04]` **SLZB-MR1U Zigbee coordinator moved to esh-iot (VLAN 90) at `10.0.90.10`, DHCP-reserved, DNS'd, handed to ha-dev.** ⚠ Home Assistant cannot resolve `.internal` at all (Docker's 127.0.0.11 upstream excludes the fleet AdGuard) — pre-existing; ha-dev declined the fix. Runbook `docs/runbooks/slzb-mr1u-zigbee-coordinator.md`, commits `fed29be`/`0bbdaf9`.
- `[2026-09-03]` **Run 3c is STAGED on pfi-gx10 and deliberately NOT launched** — the launch is a 13.3 h commitment and the operator stood this port down once already. Base shards AND the encoded corpus sha256-verified identical to ana-ml2's, so the transformers 5.15.1→5.16.1 / x86-64→aarch64 delta is *measured* inert. ⚠ The encode-cache FILENAME differs by design (`base_model_path` is in the key) — input hash, not output. ⚠ Tripped the `pkill -f` ssh self-match again; the launcher guards on a pidfile because of it. → `persistent-memory.d/2026-09-03-gx10-run3c-staged.md`
- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md`
- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I <wired>` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md`
- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md`
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
- `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
- `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
- `[2026-09-02]` **althing 3.3.0 deployed — the `cc` channel, and a plugin-cache false green.** CC seats are now poked over their own message socket (`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) instead of by typing into the pane: no process to reap, nothing near the input line. infra-ops moved to `channel=cc`; **the dwarves stay on pane and their guard-4 exposure is UNCHANGED** (`declare` prefers cc, falls back). ⚠ **An undocumented Claude Code interface, taken deliberately** (operator: the FIFO poker was also an unsanctioned hack — a better instance of a class we already had). Break mode = seat goes pull-only with a logged reason, mail still held. ⚠ **`claude plugin update` matches on the plugin VERSION and declines a content-only change** — 3.3.0 edited plugin content at an unchanged 0.1.1, so the CC cache stayed stale while every version check reported success (delta was docs-only, harmless this time). `deploy-althing.sh` now diffs marketplace vs live cache. ⚠ Ordering: **herald restart BEFORE anything declares cc**, or the seat goes silently pull-only. ⚠ This box was at **3.2.4, not 3.2.5** — rollback target here is 3.2.4. Follow-on 3.3.1: **the statusline bell measured a MECHANISM, not the property** — it read `wake-listener-<handle>.lock`, so a cc seat renders 🔕 while push/reachable. Both copies now ask the post office (`reachable` from the status payload) and add 📵 for an outage. ⚠ **TWO COPIES of that script now exist** — `scripts/claude-statusline-command.sh` here (the operator's wired one) and althing's `plugin/scripts/statusline.sh` — independently fixed to the same shape; a drift surface with a countdown, convergence not yet raised with the operator. → `docs/runbooks/althing-deploy.md`
- `[2026-09-02]` **Every CI job on the shared `pfi-fleet` runner is root on ana-docker — and `container.valid_volumes: []` does NOT prevent it.** Measured: a job container is uid 0, `/var/run/docker.sock` is mounted by act_runner independently of that list, `docker ps` returns all 49 host containers (gitea itself, synapse, phasefinal-web, adguardhome), `docker compose v2.33.0` on PATH. ⚠ **LOAD-BEARING** — `vh/Worldtree`, `vh/soong-lab`, `vh/skaldsong`, `vh/wt-matrix-bridge` all drive buildx through that socket, so it cannot simply be closed; **isolate sensitive builds onto a dedicated runner instead.** Also measured the same night: `services:` containers work (Postgres 16), and **full-URL `uses: https://gitea.phasefinal.com/actions/checkout@v4` resolves from the local mirrors** — the un-parked half of the github-independence work, needing neither `DEFAULT_ACTIONS_URL=self` nor the act_runner auth path that blocked it on 2026-08-05. Prompted by vastblue-dev's CI-posture question for a client-funded healthcare repo. → `stacks/gitea-runner/README.md`
- `[2026-09-02]` **pfi-gx10 BASELINED: 79.36 s/it median on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 10 timed steps, 0.19% spread, **peak 75.1 / 121.6 GiB — 46 GiB spare**, `attn_resolved: flex_attention`. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md`
- `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
- `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
- `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
- `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
- `[2026-09-01]` **Matrix: Synapse 1.120→1.159, appservice namespace opened, `/_synapse/admin` closed to the internet, alias convention ratified.** Schema migrations are one-way; push is `event_id_only` and assembled on-device. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
- `[2026-09-01]` **A named failure class: a correct check aimed at the wrong object.** Six instances in one day across three sessions; re-running the same check cannot catch it. **Recommended for `docs/pfi/training-throughput-playbook.md` §4 — NOT YET WRITTEN, awaiting operator.** → `persistent-memory.d/2026-09-01-wrong-object-measurement.md`
- `[2026-09-01]` **Ops boundary ruled by the operator: worldtree-dev writes the bridge code; infra-ops OPERATES the Worldtree/Matrix instances and may change them.** Corrects a mis-route where infra-ops asked worldtree-dev to provision an account on a box it does not run. Tracked at `931bac8` + althing `01M1F4PK796EDGDCBKZ9W3JC0S`.
- `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`.
- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md`
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md`
- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry.
- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`.
- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
- `[2026-08-27]` **Run 3 gated: the preregistered rule PASSED and a k=25 follow-up found a 44pp self-harm guardrail collapse — DO NOT SERVE.** A pooled preserve-list test structurally cannot see a single-axis collapse. → `persistent-memory.d/2026-08-27-run3-gate-safety-regression.md`
- `[2026-08-27]` **The corpus mix was specified in a unit the optimiser never sees** — 45.8% dialogue by CONTEXT, 24.2% by LOSS. Harness now leads with loss share and calls context a memory budget (`dd5a12e`). → `persistent-memory.d/2026-08-27-mix-specified-in-the-wrong-unit.md`
- `[2026-08-27]` **Dose-response: benefit and damage are ONE direction in weight space** — every axis monotone in scale, no knee. The merge-back cannot separate them; vLLM cannot LoRA-serve this MoE at all. → `persistent-memory.d/2026-08-27-dose-response-entanglement.md`
- `[2026-08-27]` **Anaheim tripped a power breaker; four guests including the NAS had `onboot` unset and never came back.** Fixed with dependency ordering — ana-nas order=1,up=45 ahead of the databases. ⚠ **ONE CIRCUIT FEEDS THE WHOLE RACK including the firewall serving the public IP** (operator) — so ana-gw, ana-wg and every BMC go down with the load, and there is NO remote management path to Anaheim during a power event. → `persistent-memory.d/2026-08-27-anaheim-breaker-and-onboot-gap.md`
- `[2026-08-27]` **A transport failure that enters a measurement as a VALUE looks like whatever you hoped to find.** heid's lost panel arms found a live defect in brokkr's `t4_dissect` an hour later. → `persistent-memory.d/2026-08-27-empty-response-as-a-datum.md`
- `[2026-08-27]` **Run 3c authorised (lr 20x cut, single variable) and then HELD by the operator after the breaker trip.** Config built and validated at `/tank/erp-tune/run-03c.json`; `save_steps` made configurable in the harness (`0a6bd2e`) because the first launch lost 80 steps with no checkpoint. Tracking surface: commit `0a6bd2e` + that config path. **Relaunch is one command once power is triaged.**
- `[2026-08-27]` **An event report with no timestamp is a claim about "now" — and it manufactured a launch that never happened.** brokkr reconstructed a phantom third 3c launch because my 23:03 report narrated a 21:07 kill in the present tense. Every fact in it was true; it was unreadable in sequence. → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
- `[2026-08-27]` **`save_steps` was hardcoded at 100 in the harness** — a claimed provenance entry the run could not have honoured. Made configurable, default unchanged (`0a6bd2e`, 242 tests green). Caught by checking the config carried the change rather than trusting that it had been made.
- `[2026-08-27]` **Six defects in run 3's staged build, none of which would have errored** — a dialogue-only survivor list that would have silently dropped 96% of the corpus, an impersonation mask not subsumed by the low-quality mask, kvasir unbounded at 67.8% of context, a `save_pretrained` config-key drop that made the merged model unservable, and the mix-unit error. Every one produced a plausible completed run. Full record `/tank/erp-tune/recipe-r3/RUN-03-BUILD-NOTE.md`.
- `[2026-08-27]` **The 18 unpushed eitri-smithy commits are pushed** — run 3's `harness_commit 9d27b4fe` now resolves off-box, verified by fetching into a fresh empty repo rather than trusting the push output. ⚠ **HTTPS push 403s for every gitea token including site-admin; SSH works.** Untracked `__pycache__` (`894fbe8`) because a tracked `.pyc` dirtied the tree and would have stamped `harness_dirty_at_launch: true`.
- `[2026-08-26]` **Run 3's corpus is built and HELD — `creative-writing-multiturn` is a MEGAMIX containing bluemoon, PIPPA, LimaRP and stheno**, and the remix promoted two roots that overlap at median jaccard 0.873. Dedup direction reversed to keep the primary source. F1 does not do what the recipe says; F2 does. → `persistent-memory.d/2026-08-26-run3-corpus-and-the-megamix-containment.md`
- `[2026-08-26]` **No conversation admission threshold for run 3** (brokkr delegated, then endorsed). Masked context costs the transformer body but NOT the LM head — the harness drops `IGNORE_INDEX` before the 262k-vocab head — so a 20% trim buys ~35 min against a 3-hour run, and it would be a fifth filter confounding a run whose purpose is testing the filters alone. Tracked in the run-3 detail file.
- `[2026-08-26]` **Run 2's gate FAILED and is recorded as a FAIL** — T3 constraint −12.0 against a ~1 pt floor. But gate 1 is the result: **T6 spatial +15.0 where run 1 failed the same axis at −3.5**, base swap the only intended variable. Neither run ships; the pair prices what the abliteration cost. Plus the long-form-only stability regression a short-answer gate would have passed, and PIPPA's 123-word clip in the length signal. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
- `[2026-08-26]` **Run 2 complete, merged, coherence-gated and serving as `erp-tune-v2`** — stock-instruct base, and the mask proven by a −221,712 loss-token delta against byte-identical context. Also the p50 claim I asserted and had to withdraw. → `persistent-memory.d/2026-08-26-erp-run2-complete-and-served.md`
- `[2026-08-26]` **Playbook §4 written: "when the artifact lies about itself"** — seven landmines plus a pre-launch checklist, from a night in which *three separate fixes each shipped a check that could not fail*. The unifying line is brokkr's: when you change what an artifact means, every derived artifact keyed on the old meaning is now a liar. Commits `dae6ede` → `d54f256`; the doc is `docs/pfi/training-throughput-playbook.md` (filename kept for inbound links; scope is now wider than the name).
- `[2026-08-26]` **Served under a NEW name on a NEW port (`erp-tune-v2` / :8098), never re-pointing `erp-tune-v1`.** Run 1's artifact still exists and is still what that name refers to; re-pointing would be the silent substitution the standing no-false-aliases rule forbids. brokkr independently asked for the same and additionally wants the concrete backing model + date in provenance, not just the alias — an alias has silently changed meaning under recorded results before.
- `[2026-08-26]` **DPO stage gated on an axis-list decision that is not mine to make** — `docs/pfi/erp-dpo-stage-prep.md`. No preference data for refusal axes exists; `trl` is not installed; the Gutenberg sets on disk are prose-quality only. ⚠ Do not install `trl` (or anything) into the training venv **while a run is saving** — a resolution that upgrades transformers under a live process can break its save path.
- `[2026-08-25]` **The ERP/RP tune COMPLETED in 7.36h and passed its gate on the axis it was built for** — diversity 22x its noise floor, attractor −11.3pt, zero memorisation on both arms. Also the noise-floor near-miss: brokkr was one step from reporting a 13-point T6 regression sitting inside twice his instrument's own variance. → `persistent-memory.d/2026-08-25-erp-tune-run2-complete.md`
- `[2026-08-25]` **8.6% MFU was an accounting artifact — real utilisation 17-20%, and the cost was attention on AMPERE kernels.** Two independent methods agreed to 2.6 points. Fixed by bucketing (padding 29.9%→0.0%) plus flex_attention. ⚠ Carries the dynamo recompile-ceiling trap that produced two wrong published conclusions. → `persistent-memory.d/2026-08-25-mfu-root-caused-attention.md`
- `[2026-08-25]` **NVFP4A16 serving pipeline built and validated; MERGED WEIGHTS ARE MANDATORY.** vLLM cannot serve a LoRA on ANY Gemma-4 — `get_expert_mapping` is unimplemented and the check branches on MoE-ness, not quantization. Plus the landmine: a `targets=["Linear"]` recipe misses all 11,520 expert tensors silently. → `persistent-memory.d/2026-08-25-nvfp4-serving-pipeline.md`
- `[2026-08-25]` **Refusal retention measured (base 0/100 → tuned 29/100, 71 still complying) — but on the WRONG AXIS.** `harmful_behaviors` is general harm; the abliteration was run for explicit fiction. The convenient set with a recorded baseline was not the right one. → `persistent-memory.d/2026-08-25-refusal-retention-probe.md`
- `[2026-08-25]` **Worldtree b188 + b189 shipped; bridge extracted to `pfi/wt-matrix-bridge` because `vh` is a USER not an ORG** and no service account can ever publish to a user namespace. Plus the selene catalog entry that lied about what answers, and a #411 diagnosis I got wrong twice before a directory probe settled it. → `persistent-memory.d/2026-08-25-worldtree-b188-b189-and-selene.md`
- `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
- `[2026-08-25]` **The ERP/RP tune LAUNCHED after 12 harness defects and an operator override of the corpus gate.** Four of the twelve would have crashed the run; two were INERT GATES that passed because they could not fail. Run is `/tank/erp-tune/run-01`, harness eitri-smithy `997c4a4`. Full arc — override, defects, sizing, the measured MFU — in the in-flight section and `docs/pfi/gemma4-erp-tune-sizing.md`.
- `[2026-08-24]` **char-rp seat swapped to the Gemma-4 26B-A4B MoE; abliterated trainee base staged and measured.** OOM root-caused to `--gpu-memory-utilization` not covering CUDA context (and to gen's footprint GROWING WITH UPTIME); a benchmark finding retracted because it scored below chance; abliteration isolated at −0.6 core points but it MOVES capability rather than removing it. → `persistent-memory.d/2026-08-24-charrp-gemma4-moe-swap-and-trainee.md`
- `[2026-08-24]` **Serving the tuned ERP model: LoRA-on-NVFP4 PREFERRED, merged weights the expected fallback — and the recorded objection may be STALE.** Operator: "if you CAN load it as a lora, all the better, the issue is that we will want to run nvfp4 weights, which we had some serious trouble with loading loras on top of nvfp4." ⚠ **The archived root-cause says it was NOT NVFP4-specific**: `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application was a silent no-op (#47639, regression from #37912) — adapter loads HTTP 200, zero deltas at inference, proven **quant-agnostic (NVFP4 AND FP8 both inert)** and adapter-format-agnostic by a 3-peer dwarf panel. Fix PR #47640 was OPEN then. **ana-ml2 is FAR past 0.24.0 and the box runs a SPREAD, not one version** (measured 2026-08-24): `gen` on `nightly-311b3513` = **0.27.2rc1.dev150**, `mog-sec` on `nightly-e9d1398d` = 0.26.1rc1.dev1102, the small seats still on 0.24.0, and char-rp/trainee-bench pinned to v0.26.0. ⚠ **`vllm/vllm-openai:v0.27.1` is already ON DISK, unused** — a TAGGED release, which is the right retest target: no nightly variance, no pull, ~4 months past the diagnosis. So: RETEST hot-swap LoRA on **v0.27.1** before designing around merge — it is cheap, and if it works the post-tune gate can be two aliases on one engine. If it still no-ops, merged weights it is, which means the harness must EMIT merged weights and Eitri needs that in the contract while he is early. Tracked at this snapshot commit; settle it in the QLoRA sizing conversation.
- `[2026-08-24]` **Homepage rebuilt on Australis Skyfall; light mode shipped.** Two findings worth more than the theme: **(a)** the Skyfall bundle including its canonical light ramp was sitting in this repo's git history at `45c1995` — check `git show` before concluding a vendored design asset is lost; **(b)** removing `theme:` from `settings.yaml` deterministically breaks the dashboard render (six recreates empty, restoring the key fixed it in 12s), which is the first confirmed cause of the "tab bar goes missing" symptom. Retires the `homepage.log` size lead from earlier the same day — it did nothing on this episode. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **Homepage reorganised on the axis "do I open this?" — UI groups expanded on top, API/agent groups collapsed at the bottom** (operator-delegated: "re-categorize however you want"). Load-bearing constraint: `homepage.group` is read at container CREATION, so the 16 GPU-backed model seats keep their unlovely names rather than eat a recreate — `initiallyCollapsed` + order is free. Second rule discovered here: **group members should all have widgets or none should**, because a stat strip adds ~50px and opens a void beside plain cards. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **Homepage columns unified at 4 for every group; the 2026-08-18 "columns = member count" rule is retired.** It was avoiding dead cells in a short last row and bought a worse defect — card width changing at every group boundary. Also carries two CSS traps: `overflow: hidden` clips at the PADDING box (so a `padding-right` gutter is spill room, not a guard), and a `:root` override of a Homepage theme variable is silently outranked by `.theme-slate` on the same `<html>` element. → `persistent-memory.d/2026-08-24-homepage-uniform-grid.md`
- `[2026-08-24]` **AES-128 adopted on both Anaheim tunnels; the per-flow ceiling root-caused to the UDM's software AES-CBC, exonerating the FortiGate.** Proven by an A/B/A cipher swap at identical CPU — hardware offload is not cipher-cost-sensitive. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
- `[2026-08-24]` **ana-gw's public admin surface closed to zero open ports, ACME listener included.** Two of my diagnoses were wrong first (an "ISP proxy" that was the FortiGate, and an "all-port VIP" alarm that was a parser gap) — both from reading config instead of measuring from outside. → `persistent-memory.d/2026-08-24-ana-gw-admin-closed-acme-disabled.md`
- `[2026-08-24]` **Scriberr deployed on ana-ml2 GPU1, image built from source.** Three upstream bugs: the Blackwell image was never published, it must run as uid 10001, and `UV_LINK_MODE=copy` is required or two backends fail silently. → `persistent-memory.d/2026-08-24-scriberr-ana-ml2.md`
- `[2026-08-24]` **ESH DNS fixed at the IPv6 layer and the naming scheme went live on three hosts.** UniFi's RDNSS cannot be disabled but CAN be redirected — the field is only honoured when an explicit server is given. → `persistent-memory.d/2026-08-24-esh-dns-rdnss-and-scheme-live.md`
- `[2026-08-24]` **`speaches` on irv-ml1 stopped, stack retained** — Eyra was abandoned pre-implementation (Scriberr covers the need), leaving it no consumer. Disposition confirmed to eyra-dev; one command to restart. Tracked at althing thread `01M0RRJX8GPZEBDHF1E3W18RZF`.
- `[2026-08-24]` **esh-vm-db brought onto the fleet infra-ops identity and given its first vaulted credential.** It previously had none: root and infra-ops refused key auth and `lkraven`'s sudo wanted a password nobody held, leaving `qm guest exec` from the hypervisor as the only privileged path. Break-glass root password at `secret get esh-vm-db/root-breakglass-password` (console-only; plaintext never crossed the wire — only its SHA-512 hash did).
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction.** brokkr-smithy-dev pre-approved it for "once the FortiGate work settles" and does not need re-asking; the operator declined it in this session's scope. Tracked at althing thread `01M0R46SFYF83099N16WD67KGD`.
- `[2026-08-23]` **Anaheim's IPsec tunnel ceiling — investigated, then CLOSED 2026-08-24.** The 25%-of-2-Gbps framing was wrong (NH3's uplink is 1 Gbps); AES-GCM proved impossible; AES-128 landed instead. → `persistent-memory.d/2026-08-23-anaheim-ipsec-tunnel-ceiling.md`
- `[2026-08-23]` **selene retired after losing a head-to-head on its own job; `chat-judge` moved to gen, the model name 404s by design.** Also surfaced that **7 aliases share one seat** — cross-checking between them is an echo, which caught a real defect in brokkr's 46k-exposure R47 gate. → `persistent-memory.d/2026-08-23-selene-retired-alias-collision.md`
- `[2026-08-23]` **hrafn adopted; its CI reported green for its whole life while deploying nothing.** A staging dir inside the rsync target destroyed its own source mid-copy; the deeper fault was verify steps that asserted uptime, never content. → `persistent-memory.d/2026-08-23-hrafn-adopted-ci-frozen-source.md`
- `[2026-08-23]` **Worldtree b187 shipped; all three instances de-armed from a 69-day-stale `:latest`; Matrix homeserver re-plumbed to personal.** Includes the `:8009`-is-demo port trap that an IP-only fix would have walked into. → `persistent-memory.d/2026-08-23-worldtree-b187-pins-matrix.md`
- `[2026-08-23]` **Every secret-bearing `.env` on ana-docker tightened to 0600** — eight stacks including vaultwarden and traefik, verified exposed by reading one as `nobody`. → `persistent-memory.d/2026-08-23-ana-docker-env-perms-sweep.md`
- `[2026-08-23]` **`pfi` gitea org created; claude-bot is an Owner and creates repos self-serve.** Closes the repo-creation half of the credential-migration directive — `vh` is a USER namespace so no service account could ever create there. Repo creation needs `write:user` + `write:repository` + `write:organization`; `POST /users/{u}/tokens` is basic-auth only, so minting needs the account password. Default new repos to `pfi/`. (`vh/eitri-smithy` was its first tenant, then moved.)
- `[2026-08-23]` **Booth: kept boards are deletable and link rows are prunable.** `release` on a kept card drops the sentinel so the existing × applies; `booth links` / `booth unlink <id|index>` prune one row. Rows are addressed by **content id, never position** — the board is append-only and multi-writer. **Releasing a board RESETS its TTL clock** (unlink bumps the dir mtime), so unkeep-and-wait is a 24h delay, not a delete. (`4be880f`, `0ad332b`)
- `[2026-08-22]` **DFlash2 spec-decode measured on our own stack; `sec` promoted to it.** +18–21% accepted length and +15–18% throughput over MTP k=3, drafter proved model-agnostic across two finetunes to 0.06%, and the k=7 MTP *control* showed deeper MTP is a throughput trap. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md`
- `[2026-08-22]` **Quant pipeline shipped a crippled tokenizer for months — fixed at source.** `quant_mixed_nvfp4.py` baked its calibration truncation (`max_length 2048`) into every mixed-NVFP4 build; latent on old transformers, fatal on new. Both live quants corrected, pipeline now saves a source-pristine tokenizer and asserts it. Playbook §3.14. (`0755ba7`)
- `[2026-08-22]` **`sec` retuned to util 0.52 / 420K after a runtime OOM at 0.55/480K** — `gpu-memory-utilization` is not a hard reservation; activation grows past the dummy-data profile and six vLLM containers share GPU1. Also measured: the KV pool varies ~6.6% between boots, so max-model-len must be sized against the *lower* observation. (`6e82899`)
- `[2026-08-22]` **Max-Q 1.8× spread does NOT apply to LLM decode — measured, not argued.** ana-ml2 draws 256–266 W of 300 W under sustained 100% decode with `SW Power Cap: Not Active` and clocks pinned. Corrected to brokkr-smithy-dev after I had lent the claim credibility; 122B figure (~90–93 tok/s at 262K) stands as a straight number.
- `[2026-08-21]` **ESH internal IPv6 live on two LANs; the Cityside v4 static is a CARRIER problem, proven.** A full gateway reboot forced a fresh DHCP DISCOVER and returned the identical CGNAT address. YaRN was already configured — "1M needs YaRN, absent" was false. → `persistent-memory.d/2026-08-22-dflash2-spec-decode.md` sibling entry in `ad21302`
- `[2026-08-21]` **speaches ASR live on irv-ml1 for Eyra — and `no_speech_prob` alone is a weak hallucination gate.** Silence and room tone both hallucinated "Thank you." under 0.11; `avg_logprob` separates ~6× better. Consumers should gate on a composite. (`aa5863c`, `c7e2187`)
- `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
- `[2026-08-19]` **waterland studio containerised on irv-ml1 — three landmines, all measured.** cupy needs CUDA *headers* the host had by accident; `uv run` re-syncs and prunes cupy at RUNTIME; the A6000 is container-index 0, not the host's 1. → `persistent-memory.d/2026-08-19-waterland-studio-containerised.md`
- `[2026-08-19]` **Homepage cleaned up, then themed with Australis Skyfall + an Arbo-generated background.** Includes the hour lost to a self-healing tab-bar red herring, and the CSS-iteration loop that prevents it recurring. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md`
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
- `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
- `[2026-08-17]` **irv-ml1 cleared of 782 GB, and Homepage brought under version control.** One dead-looking Gradio app pinned three delete targets at once; `/opt/ComfyUI` is NOT the ComfyUI that serves. → `persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md`
- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
- `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
- `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
- `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`.
- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
- `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
- `[2026-08-15]` **RP-seat direction: KEEP MeroMero on `char-rp`; Artemis-31B rejected; next move is Dark-Scarlett on a Qwen3.8 base when it lands (operator).** Evaluated `TheDrummer/Artemis-31B-v1.1` — mechanically a drop-in (same `google/gemma-4-31B-it` base, identical 1188-tensor/356-vision census, same missing-`preprocessor_config.json` trick), so it's purely a quality call, and our own survey already ranked MeroMero **#1** vs Artemis **#6**; Artemis is also unlicensed and its author deprioritizes correctness + warns of token-banning-for-stability, which fights char-rp's tool-calling requirement. **MTP verified impossible on both** (Gemma-4 has no MTP head at all — base/MeroMero/Artemis are all MTP=0; no finetune can add one). **But speculative decoding IS reachable on a Gemma-4 seat via a DETACHED drafter** — vLLM 0.24 supports `eagle3` + `gemma4_mtp`, and real drafters exist: `google/gemma-4-31B-it-assistant` (0.94 GB, 4-layer, 761K dl), `RedHatAI/gemma-4-31B-it-speculator.eagle3` (4.47 GB), `AEON-7/…eagle3-NVFP4` (3.53 GB). ⚠ all list their verifier as **stock** gemma-4-31B-it, not an RP finetune, so acceptance against MeroMero is unmeasured and likely well below the gen seat's ~48%. UNTESTED — parked, ~45 min to measure, needs GPU0 headroom (card is at 94.4/97.9 GB). **Why the Dark-Scarlett 3.8 plan is the strong one:** DS is Qwen3.6-based today, so a 3.8 respin lands on the *gen seat's* architecture → native MTP returns and the whole mixed NVFP4+FP8 recipe + graft ports directly. Watch two things on arrival: `from_pretrained` **silently drops MTP heads during finetuning** (verify 15 `mtp.*` tensors in the index; graft from stock if absent), and DS v1.0 required the `Qwen3_5ForConditionalGeneration` **wrapper class** to save a config vLLM/SGLang accept. Both in `docs/pfi/model-quantization-playbook.md`.
- `[2026-08-15]` **Quant lessons consolidated into `docs/pfi/model-quantization-playbook.md` — the durable home; read it BEFORE any requant.** Survey found quant knowledge scattered across 18 files in 4 trees, with **three** documents having independently written overlapping "landmines" sections (the loader-class trap alone was rediscovered 3×). Playbook owns the **transferable** lessons (scheme choice, landmines, acceptance gate + its 3 measurement traps, hardware/co-residency); per-model artifacts are demoted to worked examples that link up. Carries a **superseded-claims table** — which immediately earned itself: the heretic2 runbook's "use modelopt, compressed-tensors can't load the BF16 MTP" is **false** (the cause was the missing `re:^mtp.*` ignore, not the format) and would have sent the next session down the modelopt dependency-hell path; that runbook now carries a stale-warning header. Maintenance rule in `CLAUDE.md`: model-agnostic → playbook, model-specific → stays put, wrong claim → dated superseded row, never a silent edit. Motivated by Qwen3.8 having just released — the next model swap needs a requant. Commit `a91cc3f`.
- `[2026-08-15]` **Operator ruling: the gen seat's +1.7% perplexity is an acceptable price for the speed — SETTLED, don't re-litigate.** Precise attribution for future reasoning: it is the **activation-quantization** cost (W4A4 MLPs + FP8 attention vs BF16 activations), not an MTP cost — PPL was measured with speculative decoding **off** on both builds, so MTP was not in the loop. Turning MTP off would not recover it; only reverting the quant would (rollback = one `.env` line, old build intact at `…/qwen38-27b-uncensored-nvfp4`).
- `[2026-08-15]` **gen seat requanted to mixed NVFP4+FP8 (+18% decode) + char-rp Gemma-4 tool-calling fixed.** The queued "W4A8" (NVFP4 weights + FP8 activations) is **not servable** — vLLM 0.24 allows NVFP4 weights with only A16 or A4; FP8 activations ValueError at load, and `CompressedTensorsW4A8Fp8` is INT4-weights + sm90-exact (closed on Blackwell twice). FP8 must enter **per-layer-group**. Also: the handoff's "~68 tok/s" baseline didn't reproduce — cache-busted, the incumbent already did **80.12** (≈ the stated W4A8 target), so the premise needed re-measuring before any work. Shortcut: `unsloth/Qwen3.8-27B-NVFP4` was already on-box → served as a probe, measured **+19.1% at identical acceptance**, which both proved the gain was real and handed over the reference recipe. Replicated it on the abliterated weights → **80.12→94.53 tok/s, acceptance unchanged, +1.7% PPL, abliteration 4/4, weights −19%**; surface 6/6 live, 7 aliases routing. char-rp had **no** tool parser at all (every tools request 400'd) → `gemma4` tool + reasoning parser + a **mandatory** `enable_thinking:false` (the parser defaults it True → null `content` for all RP prose; proven byte-identical prompt before deploying). Commits `b8f0f4c`, `74f596b`. Foot-guns banked (llm-compressor prunes unmatched `ignore` entries → the 0%-MTP bug, **fired on this run**; prompt_logprobs uniform under spec-decode; 0600 `.env` silently no-ops compose; GPU0 is zero-sum). → `persistent-memory.d/2026-08-15-gen-seat-mixed-requant.md`
- `[2026-08-15]` **Uncensored gen seat: JonathanColetti/Qwen3.8-27B-Uncensored deployed as `gen-seat`/`vllm-gen` (NVFP4 W4A16 + grafted MTP, 262K); 7 aliases repointed; the definitive `re:^mtp.*`-ignore fix.** 0%-MTP-on-quant (twice) was NOT the abliteration/scheme — the grafted bf16 MTP was missing from `quantization_config.ignore` (vLLM loaded it as quantized → uninitialized). Full arc, the working pipeline, VRAM budget, unsloth speed decomposition, modelopt dead-end. → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-08-12]` **eRP dual-seat overhaul: MeroMero-v2 (`char-rp`) + Dark-Scarlett (`char-rp-reasoning`), both NVFP4A16 @ 256K on ana-ml2; granite retired.** Replaced the GGUF/heretic2 RP seats with two home-quantized vLLM seats. The DS blocker (an `AutoModelForCausalLM` save wrote a flat `Qwen3_5TextConfig` that **both vLLM AND SGLang reject**) was fixed by re-quanting via the `Qwen3_5ForConditionalGeneration` **wrapper class**; ModelOpt was a version deadlock, SGLang lacked the impl (but revealed the fix). MeroMero vision reconstructed by extracting `preprocessor_config.json` from `processor_config.json`. Both models KV-efficient (Gemma-4 sliding-window / Qwen3.6 hybrid linear-attn) → full 256K; GPU-swapped for headroom; compose-ified + committed `f08b6cb`. granite downed + LiteLLM `summarizer`/`classifier`→gen. Full arc, lessons, dead-ends → `persistent-memory.d/2026-08-12-erp-dual-seat-overhaul.md`
- `[2026-08-12]` **infra-ops now holds an all-zones Cloudflare DNS-edit token (vaulted) + wgtunnel Phase-0 DNS landed.** Operator handed over a `Zone·DNS·Edit` (all zones) CF token → `secret put nh3-dev/.config/cloudflare/infra-ops-dns-token` (round-trip verified; /tmp drop shredded). Fleet DNS is now self-serve for infra-ops (⚠ HIGH blast radius — all zones). First use: created `boring.phasefinal.com` CNAME → `ana-srv1.phasefinal.com`, **DNS-only** (proxied:false), verified resolving to 38.120.12.44 on both authoritative NS (louis/wren) + 1.1.1.1 — NOT Cloudflare-proxied. Unblocks wgtunnel's wstunnel ACME cert. phasefinal.com zone id `f812ba74ed9a75cf21bbe7ce9188db50`. auto-memory `reference_infra_ops_cloudflare_dns_token`. (Earlier gap: the only prior vaulted CF token, jackdaw's, had `zone:read`+`worker:edit` but no `dns_records:edit`.)
- `[2026-08-12]` **wgtunnel stood up as its own repo (`vh/wgtunnel`, private) after a live endpoint-verification pass.** Operator directed own-repo (mirrors stonehenge-park/tts-stack). Verified off the fleet before seeding: `ana-wg` WG server = **UDP/31337** (not 51820), subnet 10.30.10.0/24, MTU 1420, active roaming peer proves the public UDP DNAT works; traefik on ana-docker **terminates TLS :443** (ACME `anaprod` http-challenge, docker+file providers, CrowdSec bouncer) → confirms the clean design (wstunnel container on `traefik-net`, Host-routed, WS→UDP to `ana-wg:31337`); edge `38.120.12.44` direct-A, `tunnel.phasefinal.com` free (⚠ must be **direct**, NOT Cloudflare-proxied like vaultwarden). Repo pre-seeded (README/CLAUDE/persistent-memory/ROADMAP + `docs/verified-infrastructure.md` = ground truth) + pushed; commit `9584d38`, Vuong-attributed. vh gitea token pulled from the vault (`secret get`), not persisted to `.git/config`. **NEXT = `/vor-plan` or `/vor` (operator's call, interactive).** Deps to line up in the plan: DNS A-record, FortiGate :443 host-routing, a new ana-wg peer for the laptop, client tooling.
- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
- `[2026-08-11]` **stonehenge-park: new fleet `/park` service repo stood up + designed (`/vor-plan` + `/vor-ui`).** Self-contained SQLite+FastAPI idea-parking service that actively resurfaces (statusline + althing) so nothing dies in a cold repo; `vh/stonehenge-park` pushed + pre-seeded for a fresh agent; build starts at the U1 tracer contract. → `persistent-memory.d/2026-08-11-stonehenge-park.md`
- `[2026-08-12]` **Global `~/.claude/CLAUDE.md`: `secret`/vault tool entry + "store in AND pull from the vault" standing directive** (dotfiles `9db703b`, pushed); statusline reset-countdowns + a latent tab-collapse parse-bug fix, now tracked in the dotfiles stow tree. Dogfooded the directive: created `vh/stonehenge-park` pulling the gitea token via `secret get`. (dotfiles + global config, not eshpfi.)
- `[2026-08-11]` **TTS stack extracted to its own repo (`tts-stack`) + eshpfi stood down on TTS dev.** Operator: hand all TTS tuning/dev to a separate agent with a self-contained repo (knowledge + infra access + a live knowledge list), and move the voice corpus in. New repo `~/development/tts-stack` (commit `9ee3288`) carries: dots-tts stack (canonical intent), `voices/` corpus (MOVED out of eshpfi), `KNOWLEDGE.md` (engine landscape + prosody findings + foot-guns), `docs/infrastructure.md` (irv-ml1 access + gated deploy runbook + rollback), CLAUDE/persistent-memory/ROADMAP, `tools/` (pause-probe + Booth render). Followed the **chatterbox-fast precedent**: eshpfi `stacks/dots-tts/` reduced to a POINTER README; the ~15 experimental TTS compose wrappers stay here as reference (catalogued in tts-stack KNOWLEDGE). Blast-radius check: no eshpfi playbook/script reads the canonical corpus (other `voices/` refs = unrelated host paths). **Reverses** the earlier "Corpus home = eshpfi `voices/` (keep-here)" call. ⚠ tts-stack is LOCAL-ONLY until pushed — needs a gitea remote (`vh/tts-stack`) + push before the separate agent can clone (operator's call — outward-facing + repo-create creds).
- `[2026-08-10]` **dots-tts v3 — clause-break → period pause mapping.** Operator: v2 "sounds good" but donut won't pause at semicolons/dashes. ROOT CAUSE (measured via a pause-probe A/B — synth duration over N runs, non-determinism averaged out): dots' prosody honors a real pause **only for ellipsis (~+0.43s) and period (~+0.3s, capitalization-independent)**; comma/semicolon/colon/dash all run **flat (~+0.03s vs no-punct)**. Two distinct sub-causes: **dashes regressed in v2** (the `—`→`-` fold made em-dashes read as word-joiners), while **semicolons were NEVER a v2 change** — dots ignores them natively, only newly noticeable because v2 made everything else clean. Operator call: ellipsis "too much" → **map `;`, clause `:`, and em-dash `—` → period** in `_sanitize` (believable ~0.3s clause break). GUARDS (pinned by 11 unit tests, `stacks/dots-tts/test_sanitize.py`): digit-guarded colon `(?<!\d)\s*:\s*(?!\d)` so times `3:45` / ratios `2:1` survive; en-dash `–`→hyphen KEPT (numeric-range `10–20` safety — em-dash breaks, en-dash ranges, different jobs); genuine ellipsis left at full strength (author meant a long pause). Gated deploy (redeploy2 pattern → v3): build → throwaway :8199 test container + **pause-gate** (semicolon sentence must run ≥0.12s longer than baseline; measured **+0.427s**) → only then cut live over. LIVE + healthy `local/dots-tts:v3` on :8198. **rollback = `sed -i 's/^DOTS_TAG=.*/DOTS_TAG=v2/' .env + docker compose up -d dots-tts`** (v2 image retained). Booth `dots-pauses` (A=old-flat / C=ellipsis-too-much / D=live-v3). [[reference_chatterbox_fast_repo]]
- `[2026-08-10]` **dots-tts v2 — contraction fix (curly-sanitize) + sentence-chunking + dependency-pin recovery.** Operator: donut read contractions wrong ("you're"→"you ree", "donut's"→"donut ess"). ROOT CAUSE (isolated via A/B booth): **curly/typographic apostrophes** (`’` U+2019 from ratatoskr's LLM) — dots' tokenizer mispronounces them; STRAIGHT apostrophes read clean under `normalize_text=True`. FIX (`app.py`): fold curly→ASCII (`str.maketrans`) before synth, **KEEP `normalize_text=True`** (operator call — retains number/date expansion). Also added **server-side sentence-chunking** (pack ≤280 chars): dots caps one `generate()` at ~500 patches/~40s, so long RP turns (the Zev monologue = 160s audio) truncated; chunking stitches them (verified full 160.3s, not 40s-cut). **⚠ BUILD FOOT-GUNS (both bit this redeploy):** (1) upstream dots.tts `constraints/recommended.txt` now pins **`gradio==6.17.0` — phantom, not on PyPI** → fresh `pip install dots.tts` unsatisfiable; FIX = pin `dots.tts==0.2.1` + **DROP** the `-c recommended.txt` constraints (0.2.1 pulls working gradio 6.17.3). (2) pinning only `torch==2.8.0` let **torchaudio float to 2.11.0 → dots.tts refuses to load** (minor-version match check); FIX = pin `torchaudio==2.8.0`. **⚠ DEPLOY LESSON:** `docker compose up -d` to a new tag swaps the LIVE container BEFORE any health check — a broken image crash-loops production (**ratatoskr TTS down ~1-2min this session**). NEW PATTERN = build → test in a THROWAWAY container on an alt port (:8199) → health+verify → only THEN cut live over (redeploy2.sh). v2 LIVE + healthy on irv-ml1:8198, **CONSUMER-CONFIRMED clean** (ratatoskr verified end-to-end on their :8765 — apostrophe string reads clean, /api/tts 200 @ 48kHz, no client change; the ~1-2min blip didn't hit them, their concurrent auto-audio issue was client-side localStorage). **rollback = `sed DOTS_TAG=v1 + docker compose up -d dots-tts`** (v1 image retained). Also: deployed container GPU crept ~6→13.9GB over 8h serving (cache accumulation; a redeploy resets it — watch item). [[reference_chatterbox_fast_repo]]
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
- `[2026-08-08]` **worldtree-dev #400 CLOSED → fiction-decomp snapshot cleared from nh3-dev.** worldtree-dev signaled #400 done (shipped v1.0.0b185; exact-lexical efficacy 79%→12% on ratatoskr's gate, brokkr no-harm bracket green both ends; the snapshot served 4 probe rounds — rank decomposition, promoted-vs-gold annotation, tie-set falsification, A0/A1/A2 mechanism probe). Cleared `~/snapshots/worldtree-400-fiction-decomp` (208M: chroma + manifest/provenance/stamp) — a read-only rsync copy of PERSONAL Worldtree's Chroma (source on corviduo-dev, so safe to remove). **LEFT INTACT:** `rex393-fiction-index`/`rex393-fiction-snapshot` (separate operator KEEP word, unchanged) + `r42-gate-*`. No config deltas rode this train. Only remaining non-blocking await = ratatoskr-dev's chatterbox-fast knob revert. Replied confirming (`01KZJ9GMCC…`).
_Older entries archived to archival-memory.md._
- `[2026-08-07]` **chatterbox-fast "broken audio" root-caused (T3 AR tail over-run) + FIXED (max_chunk_chars=250 cap, :v2 deployed).** Long saga, operator-driven clean diagnosis. **Symptom:** ratatoskr's migrated RP-surface TTS "swaps to German" / "dead air" / "garbage" on long turns. **NOT** German-leak (Turbo `generate()` has NO language param — plain AutoTokenizer, no `language_id`; the multilingual `language_id="en"` lever lives only in the separate `ChatterboxMultilingualTTS`), **NOT** OOM alone. **Real cause:** the Chatterbox **Turbo T3 model OVER-RUNS its generation tail** — a long single `generate()` degrades into garble/dead-air in its final ~2-3s (lib filters OOV tokens `<6561` + pads silence = messy AR tail). The scheduler's buffer-ratchet builds 300-600 char mega-chunks that land in that zone; streaming concatenates each bad tail (worst case). **ratatoskr's anti-"German" knobs (top_k=80/temp=0.5) made it WORSE** — tight sampling pulls the degradation onset SHORTER (~200 chars vs ~300 at default knobs). **Diagnosis method** (deterministic, no ears-only): single-shot length sweep + **amplitude-gated voiced-ZCR** (garble spikes ZCR; must gate on |x|>500 else trailing silence confounds it) — degraded voiced-tail = 1.58× mid, clean = ~0.64-1.1×. **FIX:** server-side `max_chunk_chars=250` cap on the scheduler (`:v2` image, `CBF_MAX_CHUNK_CHARS=250` env) — bounds each generation to just under the ~300-char onset → clean **3-4 sentence** chunks (max prosodic arc while clean). Operator ear-confirmed clean audio + clean joins; **chatterbox's low emotiveness keeps chunk joins smooth** (the harsh joins that got Zonos rejected are absent — operator's key call). **ratatoskr TODO (relayed msg `01KZER9X7S`):** revert knobs to default (top_k→1000, temp→0.8), send full text (server chunks internally), keep the 503-on-empty guard. **Cap value tunable** per-request (`max_chunk_chars`) + env. **Deeper prosody** (if ever wanted) = scheduler Phase-2 context-priming at joins (feed prior sentence as discarded-audio context; +latency). **⚠ FOOT-GUNS:** (1) acoustic tail-trim is UNRELIABLE — sibilants ('s'/'sh'/'f') spike ZCR like garble, can't cleanly detect the speech→garble boundary. (2) **build-context vs image drift** — the `:v2` image was built from cap source, but after a `:v1` rollback the build context held `:v1` source → a `docker compose build` would've silently produced a cap-less `:v2`; re-synced the flat cap source to `/opt/docker/compose/chatterbox-fast/` (rebuild-verified). **⚠ DIVERGENCE (follow-up):** deployed build context is FLAT (`app.py`/`scheduler.py`, `from scheduler import`, thin-overlay `FROM local/chatterbox:v1`, cap-only) vs the `vh/chatterbox-fast` REPO which is PACKAGE-layout (`chatterbox_fast/`, `from chatterbox_fast.scheduler`, self-contained Dockerfile) + has `norm_loudness` (repo commit `6bc7bf0` = cap; deployed omits norm_loudness deliberately to keep the ear-test unconfounded). Reconcile the two layouts so a repo-based rebuild matches deploy. Rollback: `.bak-cap-20260807-104850` backups on irv-ml1 + `:v1` image both retained. [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]]
_248 older entries archived to archival-memory.md._
- `[2026-08-07]` **Zonos2 TAKEN DOWN on the 3090 (irv-ml1) — operator-directed "for memory", TEMPORARY.** Freed ~17.4 GB (3090: 728 MiB → 18.2 GB free) so chatterbox-fast (co-resident, was OOMing on long generations) has headroom. **⚠ Restore is manual — Zonos2 :1920 was a DETACHED native process (NOT systemd/docker), reparented to init.** GPU memory was held by the `--multiprocessing-fork` CHILDREN (1966165=16.4G, 1966166=1G), which ORPHAN to init when you kill the parent — had to SIGTERM the children explicitly (killing the parent 1965942 + uv-run 1965935 alone left the 16.4G held). **RESTORE CMD** (from irv-ml1, user lkraven): `cd /home/lkraven/tts-audition/models/zonos2 && nohup uv run python -m zonos2 --model-path Zyphra/ZONOS2 --host 0.0.0.0 --port 1920 --tts-default-voices-dir ./default_voices/ --cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3 > /tmp/zonos2.log 2>&1 &` then `docker start zonos-gateway`. **Consumers that lost Zonos:** asset-engine + gateway-chat (via LiteLLM `ext-tts` alias → zonos-gateway :8890, now stopped); ratatoskr already migrated OFF to chatterbox-fast (unaffected). Also unblocks proper drift/cap testing (OOM was blocking it). [[reference_zonos_tts_stack]]
- `[2026-08-07]` **chatterbox-fast: donut voice added + full contract delivered to ratatoskr-dev (their TTS migration off Zonos).** Operator-directed. Copied `zonos-gateway/voices/Donut.wav` → chatterbox `/refs` (`/worktank/chatterbox/reference_audio/donut.wav` — the reference_audio SUBDIR is lkraven-owned so no sudo despite `/worktank` root; container globs `/refs` live → **NO restart**), exposed as `voice:"donut"` (lowercase); verified clean 7.5s synth (24kHz, RTF ~0.31). A/B booth (chatterbox vs zonos donut, same line) at `http://10.100.10.50:8090/b/donut-chatterbox/`. Answered ratatoskr's 8-question contract ask from the live gateway (`local/chatterbox-fast:v1`) + source: **NOT OpenAI-shaped** (`POST /tts`; body `text`/`voice`/`format`/`stream`, not `input`/`model`/`response_format`); **NO affect dials** (Turbo ignores cfg_weight/min_p/exaggeration — the architecture-changing answer they flagged; **Zonos stays the only fleet TTS with real emotion steering**); streaming WAV placeholder-header shape IDENTICAL to Zonos (their per-chunk Web Audio path survives); SR 24000 (Zonos 44100); server chunks arbitrary-length text internally (no client-side chunking, unlike Zonos's 71.2s cap); English-only, no language pin. **FYI-worthy (operator):** ratatoskr is moving its RP-surface TTS OFF Zonos back to chatterbox-fast → loses the live-PAD affect coupling (heavy Zonos emotion investment) — their call, trade-off flagged to them. auto-memory `reference_chatterbox_fast_repo` enriched w/ the live contract. [[reference_zonos_tts_stack]]
- `[2026-08-07]` **Fleet reranker cut over: Qwen3-Reranker-0.6B → BAAI/bge-reranker-v2-m3 (Brokkr R43).** The incumbent was measured HARMING 80/90 fleet queries (no-reranker beat it 89/90 vs 56/90). R43 bake-off: the A2 control (same Qwen weights, seq-cls head) scored identical to the incumbent → proved the fault is a training-prior not the serving head → cancelled the expensive Qwen3-4B arm; A3 (bge-v2-m3) won on multilingual safety + bare-name recovery. LiteLLM `reranker` repointed incumbent→A3 :8013 (boundary 2026-08-06T17:37:48Z, config-edit + ~52s gateway restart); **R42 v13 gate PASSED first-ever** (56/90→90/90). Incumbent kept warm :8002 (rollback via `qwen3-reranker` alias), A4 fallback :8014. Full arc + rollback runbook `docs/pfi/reranker-selection-ledger.md`; commits ad2df89/2c11748/377f8a4 (unpushed). auto-memories: the earlier reranker-serving notes.
- `[2026-08-05]` **Fleet CI resilience flip (`DEFAULT_ACTIONS_URL=self`) — attempted end-to-end, PARKED on a runner action-fetch auth blocker; infra-ops to research it (operator-directed, deferred, NOT now).** 7 gitea action mirrors staged public+populated (orgs `actions`+`astral-sh`); the flip resolves `uses:` correctly but act_runner v0.6.0 can't authenticate its fetch to gitea 1.26 ("Invalid username or token. Password authentication is not supported"). Reverted (CI back on github default); `REQUIRE_SIGNIN_VIEW=false` KEPT as a standing change (operator, internal WG net). Full endeavor, the reliable nh3-dev-egress + git-SSH mirror method, exact config state, smoke method, and next step → `persistent-memory.d/2026-08-05-ci-flip-parked.md`
- `[2026-08-05]` **worldtree herald re-nudge bug root-caused → forseti shipped althing-core v2.1.2 (`d5d33df`, deployed on nh3-dev).** `herald.py:363` rendered the wake command from the empty *fresh* mail set on the re-nudge path (should be `deliver_msgs`) → `messages[0]` IndexError → un-suppressed outer catch-all → 7s crash-loop for 9 days on worldtree-codex's pane route (mimir-dev surfaced it; I traced it from the editable source). Fix + `render_command` empty-guard + outer log-suppress + 3 tests + contract amendment, all forseti's. **nh3-extdev herald 2.1.2 upgrade DEFERRED** (operator, not-now): extdev is a WHEEL install (not editable), unexposed (no pane routes); the verified 2.1.2 wheel is staged on nh3-dev `/tmp` (sha256 `003508…cef27`) — `uv tool install --force` + restart both heralds when un-parked. extdev herald-unit provenance resolved (operator-authorized 2026-07-25 via forseti relay; recorded in this file's 07-25 herald-install entry). auto-memory `reference_nh3_dev_althing_herald`.
- `[2026-07-31]` **muninn-gate (#377 ingestion front door) BUILT + DEPLOYED + healthy on corviduo-dev:8090.** First-boot acceptance passed (watcher:running:true proves ingestion_root byte-identity); submit path deferred to the mimir-inbox era. Full wiring (uid-1000, state-volume mount, staging path-agreement, BuildKit-secret build, deferred repoint + operational guards) → `persistent-memory.d/2026-07-31-muninn-gate-deploy.md`
_209 older entries archived to archival-memory.md._
## Tried and abandoned
- `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
- `[2026-09-04]` **Forcing 10G on the ESH-Media DAC — it linked, then degraded over hours, and I reported a plateau at two minutes.** ⚠ A clean zero-error link at 1G does NOT rule out a marginal cable; autoneg's fallback was protecting something real. → `persistent-memory.d/2026-09-04-dac-forced-10g-failed.md`
- `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
- `[2026-08-25]` **`sample_packing` is NOT strictly better than bucketing on this model, and I told the operator it was before brokkr corrected me.** Packing needs FA2 varlen or a block-diagonal mask; FA2 is unavailable here (head_dim 512 > 256 cap), so packing means an explicit 4D mask on EVERY batch. Bucketing produces **78.3% exactly-zero-pad micro-batches** which recover the `is_causal` fast path on the 5 global layers — measured at 9.4% of step time. Packing forfeits that. ⚠ **The conclusion flips under `flex_attention`**, where a block-diagonal mask is just another BlockMask: do not carry "packing is bad" past the backend decision.
- `[2026-08-25]` **Merging a tune back toward STOCK to fix overfitting would UNDO the abliteration.** brokkr recommended a 50/50 merge-back, then retracted it himself: the published recipes merge into `google/gemma-4-*-it`, and following that literally re-installs exactly the refusal directions the abliteration removed — silently, because the merged model looks *healthier* on general benchmarks. Any merge-back must target the SAME abliterated base. Wider lesson: **recipe cards are per-checkpoint artifacts, not per-family** — the advice came from a card for a DENSE STOCK 31B applied to a MoE ABLITERATED 26B-A4B, three axes apart on a shared name.
- `[2026-08-24]` **AES-GCM on the Anaheim tunnels — impossible, not merely hard.** UniFi's manual site-to-site IPsec implements no AEAD cipher at all: eight GCM spellings rejected `api.err.InvalidPayload` against a passing `aes256` control. Blocks both tunnels since both far ends are UDMs. Accepted enum is `aes128/aes192/aes256/3des` — and 3DES is *slower* (no ARM instructions, 64-bit blocks), so AES-128 is the floor.
- `[2026-08-24]` **Pointing the UDM's `wan_dns1` at AdGuard — silently ignored.** It persists and reads back correctly but the LAN-facing forwarder never uses it; proven with fresh uncached ad domains (AdGuard answers `0.0.0.0`, the UDM returned real IPs). Reverted rather than left in place.
- `[2026-08-24]` **A multi-DUID DHCPv6 VM to claim NH3's seven unclaimed /64s — declined by the operator.** The BGW has no IP-passthrough (confirmed, we hold admin), so the only route needs re-cabling, split-stack routing and **rebuilding the entire v6 firewall policy off the UDM**. The prefixes are easy; the firewall rebuild is why nobody wants them. Do not re-raise on "there are seven free prefixes".
- `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
- `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
- `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
_143 older entries archived to archival-memory.md._
_144 older entries archived to archival-memory.md._
playbooks/ana-ml2-training-window-close.yaml
+175
View File
@@ -0,0 +1,175 @@
# ana-ml2 — CLOSE the ERP/RP tune window: put the fleet back the way it was.
#
# gen GPU1 -> GPU0 -> start mog-sec back onto GPU1
#
# The exact inverse of playbooks/ana-ml2-training-window-open.yaml.
#
# ⚠⚠ ORDER IS LOAD-BEARING, AND IT IS THE MIRROR OF THE OPEN ORDER.
# `gen` must vacate GPU1 BEFORE mog-sec is started. mog-sec runs at
# --gpu-memory-utilization 0.52 = 50,901 MiB that must be free at startup. With
# gen still resident on GPU1 only ~30,000 MiB is free, so mog-sec would fail to
# boot. gen moves back to the (empty) GPU0 first; step 4 waits for GPU1 to
# actually release before mog-sec is started at all.
#
# ⚠ FIRST STEP IS A GATE, NOT A COURTESY. If a training process is still
# resident on GPU0 this playbook REFUSES to run — moving gen back would either
# OOM the run or OOM gen. Override only when you have confirmed the run is
# finished or deliberately abandoned:
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml \
# --var allow_busy_gpu0=true
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-close.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
mog_port: "8019"
# mog-sec's startup requirement: 0.52 x 97,887 MiB, rounded up.
mog_required_free_mib: "50950"
# Set to "true" to close the window even with a process still on GPU0.
allow_busy_gpu0: "false"
steps:
- name: "GATE — GPU0 is idle (refuses to evict a training run mid-flight)"
sudo: true
shell: |
set -e
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $n"
if [ "$n" -eq 0 ]; then exit 0; fi
if [ "{{ allow_busy_gpu0 }}" = "true" ]; then
echo "GPU0 still busy but allow_busy_gpu0=true — proceeding under override"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 0
fi
echo "REFUSING: a process is still resident on GPU0. Confirm the tune has"
echo "finished, then rerun with --var allow_busy_gpu0=true"
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv | grep "$gpu0_uuid" || true
exit 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Point gen back at GPU0 in its .env (GEN_GPU_ID 1 -> 0)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression — a step's `sudo: true` does NOT
# cover its guards, and the root-only .env makes an unsudo'd grep exit 2,
# which silently skips the step. See the open playbook for the incident.
shell: sed -i 's/^GEN_GPU_ID=1$/GEN_GPU_ID=0/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=1' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# ⚠ Parse the JSON, do not regex the YAML — compose emits `- "0"` with
# DOUBLE quotes. See the open playbook for the incident.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["0"]'
changed_when: "false"
- name: "Recreate gen onto GPU0"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health on GPU0"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
- name: "Wait for GPU1 to release gen's memory before mog-sec is started"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ mog_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ mog_required_free_mib }}"; exit 1
changed_when: "false"
- name: "Start mog-sec back up on GPU1 (`sec` / `sec-reasoning`)"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} start vllm-mog-sec
when: "! docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for mog-sec to serve /health"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ mog_port }}/health; then
echo "mog-sec healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: mog-sec did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU0 again"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid` — vLLM V1's EngineCore is a CHILD
# of the container's pid 1, and it is the child nvidia-smi reports.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu0_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU0: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "sec answers a real completion"
sudo: true
shell: |
. {{ mog_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ mog_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${MOG_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "Both cards are back to their normal tenancy"
sudo: true
shell: |
nvidia-smi --query-gpu=index,memory.used,memory.free --format=csv
nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory,process_name --format=csv
changed_when: "false"
playbooks/ana-ml2-training-window-open.yaml
+182
View File
@@ -0,0 +1,182 @@
# ana-ml2 — OPEN the ERP/RP tune window: clear GPU0 completely.
#
# stop mog-sec (GPU1) -> move gen GPU0 -> GPU1 -> GPU0 empty for training
#
# Operator call 2026-08-24: rather than train beside `gen`, move `gen` off GPU0
# entirely and stand `sec` down for the night. Training then gets a whole card
# (95.60 GiB) instead of a shared one, and the fleet's general seat never goes
# dark beyond its own restart.
#
# ⚠⚠ ORDER IS LOAD-BEARING — DO NOT REORDER THE STEPS.
# `gen` runs at --gpu-memory-utilization 0.43, which vLLM reads as a fraction of
# TOTAL card memory: 0.43 x 97,887 MiB = 42,091 MiB that must be FREE at startup
# or the engine refuses to boot. GPU1 has only 19,446 MiB free while mog-sec is
# up. Recreating `gen` onto GPU1 first would take the fleet's main seat down and
# leave it down. mog-sec stops FIRST, and step 3 hard-gates on the freed memory
# before `gen` is touched at all.
#
# ⚠ `stop`, never `down`. `down` removes the container; `stop` leaves it in
# place so the close playbook can `start` it. Both seats are `restart:
# unless-stopped`, which does NOT resurrect a deliberately-stopped container.
#
# ⚠ device_ids vs nvidia-smi ordering was VERIFIED on this host, not assumed:
# gen (GEN_GPU_ID=0) reports under the GPU nvidia-smi indexes 0, mog-sec
# (MOG_GPU_ID=1) under index 1. They agree here. (They do NOT on irv-ml1 —
# never carry that assumption between boxes.)
#
# Restore with: playbooks/ana-ml2-training-window-close.yaml
#
# scripts/elway ana-ml2 --playbook playbooks/ana-ml2-training-window-open.yaml
vars:
gen_dir: /opt/docker/compose/gen-seat
mog_dir: /opt/docker/compose/mog-sec
gen_port: "8015"
# gen's startup requirement: 0.43 x 97,887 MiB, rounded up. If GPU1 has less
# than this free, gen will not boot and the window must not proceed.
gen_required_free_mib: "42100"
steps:
- name: "PREFLIGHT — GPU0 holds vllm-gen and nothing else unexpected"
sudo: true
shell: |
set -e
procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | wc -l)
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
gpu0_procs=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
echo "GPU0 compute procs: $gpu0_procs (total on box: $procs)"
test "$gpu0_procs" -le 1
changed_when: "false"
- name: "PREFLIGHT — record gen's current container id (proves the recreate later)"
sudo: true
shell: docker inspect vllm-gen --format '{{.Id}}' | tee /tmp/gen-container-id-before.txt
changed_when: "false"
- name: "Stop mog-sec (the `sec` / `sec-reasoning` seat) — frees ~55.3 GiB on GPU1"
sudo: true
shell: docker compose --project-directory {{ mog_dir }} stop vllm-mog-sec
# Skip if already stopped, so the playbook is rerunnable.
when: "docker inspect -f '{{.State.Running}}' vllm-mog-sec 2>/dev/null | grep -q true"
- name: "Wait for GPU1 memory to actually release (teardown is not instant)"
sudo: true
shell: |
for i in $(seq 1 60); do
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
if [ "$free" -ge {{ gen_required_free_mib }} ]; then
echo "GPU1 free: ${free} MiB"; exit 0
fi
sleep 2
done
echo "TIMEOUT: GPU1 free is ${free} MiB, need >= {{ gen_required_free_mib }}"; exit 1
changed_when: "false"
- name: "HARD GATE — GPU1 has room for gen's 0.43 budget before we touch gen"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free ${free} MiB vs required {{ gen_required_free_mib }} MiB"
test "$free" -ge {{ gen_required_free_mib }}
changed_when: "false"
- name: "Point gen at GPU1 in its .env (GEN_GPU_ID 0 -> 1)"
sudo: true
# ⚠ `sudo` INSIDE the when: expression. A step's `sudo: true` covers the
# shell, NOT its when/creates/changed_when guards — those run as the login
# user. The .env is root-only 0600, so an unsudo'd grep exits 2
# (permission denied), which is not 0, so the step SILENTLY SKIPS and the
# flip never happens. Caught 2026-08-24 by the effective-value assert below.
shell: sed -i 's/^GEN_GPU_ID=0$/GEN_GPU_ID=1/' {{ gen_dir }}/.env
when: "sudo grep -qx 'GEN_GPU_ID=0' {{ gen_dir }}/.env"
- name: "Assert the EFFECTIVE device id, not the .env line"
sudo: true
# grep on the .env proves a substring is present; only `compose config`
# proves what the container will actually be created with.
# ⚠ Parse the JSON, do not regex the YAML. The first version of this grepped
# for -\s*'?1'? and failed against compose's DOUBLE-quoted `- "1"` — an
# assert that fails for the wrong reason is worse than no assert.
shell: |
docker compose --project-directory {{ gen_dir }} config --format json \
| jq -e '.services["vllm-gen"].deploy.resources.reservations.devices[0].device_ids == ["1"]'
changed_when: "false"
- name: "Recreate gen onto GPU1 (a device change needs up -d, not restart)"
sudo: true
shell: docker compose --project-directory {{ gen_dir }} up -d vllm-gen
- name: "Wait for gen to serve /health (cold start: weights + CUDA graphs + MTP)"
sudo: true
shell: |
for i in $(seq 1 180); do
if curl -sf -o /dev/null http://127.0.0.1:{{ gen_port }}/health; then
echo "gen healthy after $((i*5))s"; exit 0
fi
sleep 5
done
echo "TIMEOUT: gen did not become healthy in 900s"; exit 1
changed_when: "false"
verify:
- name: "gen was genuinely RECREATED (container id changed)"
sudo: true
shell: |
before=$(cat /tmp/gen-container-id-before.txt)
after=$(docker inspect vllm-gen --format '{{.Id}}')
echo "before=${before:0:12} after=${after:0:12}"
test "$before" != "$after"
changed_when: "false"
- name: "gen's process is resident on GPU1"
sudo: true
# ⚠ Match by CGROUP, not by `.State.Pid`. vLLM V1 runs EngineCore as a CHILD
# of the container's pid 1, and it is the child that holds the GPU memory —
# nvidia-smi never reports `.State.Pid`, so comparing against it always fails.
shell: |
cid=$(docker inspect vllm-gen --format '{{.Id}}')
gpu1_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 1)
found=0
for p in $(nvidia-smi --query-compute-apps=gpu_uuid,pid,used_memory --format=csv,noheader \
| grep "$gpu1_uuid" | cut -d, -f2 | tr -d ' '); do
if grep -q "$cid" /proc/$p/cgroup 2>/dev/null; then
echo "gen pid $p resident on GPU1: $(nvidia-smi --query-compute-apps=pid,used_memory --format=csv,noheader | grep "^$p,")"
found=1
fi
done
test "$found" -eq 1
changed_when: "false"
- name: "gen answers a real completion, not just /health"
sudo: true
shell: |
. {{ gen_dir }}/.env
curl -sf -m 120 http://127.0.0.1:{{ gen_port }}/v1/chat/completions \
-H "Authorization: Bearer ${API_KEY}" -H 'Content-Type: application/json' \
-d '{"model":"'"${GEN_SERVED_NAME}"'","messages":[{"role":"user","content":"reply with the single word: ok"}],"max_tokens":16}' \
| grep -q '"content"'
changed_when: "false"
- name: "GPU0 IS EMPTY — zero compute processes"
sudo: true
shell: |
gpu0_uuid=$(nvidia-smi --query-gpu=uuid --format=csv,noheader -i 0)
n=$(nvidia-smi --query-compute-apps=gpu_uuid,pid --format=csv,noheader | grep -c "$gpu0_uuid" || true)
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 0)
echo "GPU0 compute procs=${n} free=${free} MiB"
test "$n" -eq 0 && test "$free" -ge 95000
changed_when: "false"
- name: "mog-sec is stopped (not removed — close depends on `start` working)"
sudo: true
shell: |
docker inspect -f '{{.State.Status}}' vllm-mog-sec | tee /dev/stderr | grep -qx exited
changed_when: "false"
- name: "GPU1 still has headroom for Scriberr's on-demand load"
sudo: true
shell: |
free=$(nvidia-smi --query-gpu=memory.free --format=csv,noheader,nounits -i 1)
echo "GPU1 free after gen landed: ${free} MiB"
test "$free" -ge 12000
changed_when: "false"
playbooks/bootstrap-infra-ops-user.yaml
+4 -1
View File
@@ -20,7 +20,10 @@
# SCOPE — PFI-owned Linux boxes ONLY. NEVER run on:
# - SureFire sf-* / sfsrv-ana (tenant property — coordinate, don't own)
# - corviduo-dev (Worldtree partner app-layer)
# - esh-* (ESH home lab — non-PFI)
# - esh-* app VMs (ESH home lab — non-PFI). EXCEPTION, operator 2026-09-06:
# ALL FOUR PVE HYPERVISORS (pfi-pve, nh3-pve, esh-pve,
# esh-pve-nas) DO get infra-ops — done that day.
# PVE ships without sudo: `apt-get install sudo` first.
# - nh3-nas (Synology DSM) (no standard useradd / sudoers.d)
#
# Idempotent: re-running reconciles the key + sudoers without error.
playbooks/gx10-headless.yaml
+145
View File
@@ -0,0 +1,145 @@
# pfi-gx10 — convert an ASUS Ascent GX10 (NVIDIA GB10, aarch64) from its
# out-of-box desktop configuration to headless server operation.
#
# scripts/elway infra-ops@10.100.10.226 --playbook playbooks/gx10-headless.yaml
#
# CONNECT AS infra-ops. `--sudo` only applies to ad-hoc --shell/--upload, so
# playbook steps run as the connecting user and carry their own sudo. lkraven
# needs a password for sudo on this box; infra-ops is NOPASSWD.
#
# The box ships booting to graphical.target with GDM and GNOME Remote Desktop
# running. That is correct for a desk appliance and wrong for a rack node: a
# display manager holds a GPU context, and a laptop-derived power profile will
# suspend a machine that is meant to answer at 3am.
#
# OUT OF SCOPE — DELIBERATELY:
# Networking. As of 2026-09-01 this box sits on a desk on Wi-Fi with a
# temporary DHCP lease (10.100.10.226) and NO ethernet carrier (enP7s7
# carrier=0). Static addressing, VLAN placement and DNS all belong to the
# rack-install change, not to this one. Configuring an address that is about
# to be wrong is worse than leaving it on DHCP.
#
# REVERSIBLE. Every step here undoes with:
# systemctl set-default graphical.target
# systemctl unmask sleep.target suspend.target hibernate.target
# systemctl enable --now gdm
#
# Idempotent: a second run should report ok/skipped throughout.
vars:
fqdn_hostname: pfi-gx10
# Stop the display manager even if someone is logged in at the console.
# scripts/elway infra-ops@<host> -p ... --var force_dm_stop=true
force_dm_stop: "false"
steps:
# ---------------------------------------------------------------------------
# Identity
# ---------------------------------------------------------------------------
- name: Set the static hostname to match the transient one
# Ships with static=gx10-a745 but transient=pfi-gx10, so `hostname` and
# `hostnamectl` disagree. Anything keying on one of them picks the wrong
# name half the time. pfi-gx10 is the fleet-convention name.
shell: sudo hostnamectl set-hostname {{ fqdn_hostname }}
when: "test \"$(hostnamectl --static)\" != '{{ fqdn_hostname }}'"
# ---------------------------------------------------------------------------
# Stop booting into a desktop
# ---------------------------------------------------------------------------
- name: Default boot target -> multi-user
shell: sudo systemctl set-default multi-user.target
when: "test \"$(systemctl get-default)\" != 'multi-user.target'"
- name: Stop GDM
# ⚠ GUARD ON is-active, NOT is-enabled. Ubuntu ships gdm as a STATIC unit —
# it is pulled in by display-manager.service and is never "enabled". A
# `is-enabled | grep enabled` guard therefore always skips, and a verify
# written the same way passes while gdm is still running. Cost one green
# run that had not done the thing. (2026-09-01)
#
# GNOME stays INSTALLED, only stopped: removing it frees little on a 916G
# disk and makes this hard to walk back from a serial console.
#
# Will NOT run while someone holds a seat session, unless force_dm_stop=true.
# Yanking the display out from under a person at the machine is not a thing
# automation should decide to do.
shell: sudo systemctl stop gdm
when: "systemctl is-active --quiet gdm && { [ '{{ force_dm_stop }}' = 'true' ] || ! loginctl list-sessions --no-legend 2>/dev/null | grep -qE ' seat[0-9]+ '; }"
- name: Stop and disable GNOME Remote Desktop
# Needs a graphical session to be useful; with GDM gone it is a listener
# with nothing behind it.
shell: sudo systemctl disable --now gnome-remote-desktop
when: "systemctl is-enabled gnome-remote-desktop 2>/dev/null | grep -q enabled"
# ---------------------------------------------------------------------------
# A server must never sleep
# ---------------------------------------------------------------------------
- name: Mask sleep / suspend / hibernate / hybrid-sleep
# Masking, not disabling. These are `static` units pulled in by logind and
# by desktop power policy; disable does not stick, mask does.
shell: sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target
when: "! systemctl is-enabled sleep.target 2>/dev/null | grep -q masked"
- name: logind must ignore lid and idle
shell: |
sudo install -d /etc/systemd/logind.conf.d
sudo tee /etc/systemd/logind.conf.d/10-headless.conf >/dev/null <<'EOF'
[Login]
HandleLidSwitch=ignore
HandleLidSwitchExternalPower=ignore
HandleLidSwitchDocked=ignore
IdleAction=ignore
EOF
creates: /etc/systemd/logind.conf.d/10-headless.conf
# ---------------------------------------------------------------------------
# Keep long jobs alive across a dropped SSH session
# ---------------------------------------------------------------------------
- name: sshd keepalives so a NAT/Wi-Fi stall does not kill a running job
shell: |
sudo install -d /etc/ssh/sshd_config.d
sudo tee /etc/ssh/sshd_config.d/10-fleet.conf >/dev/null <<'EOF'
ClientAliveInterval 60
ClientAliveCountMax 10
TCPKeepAlive yes
EOF
sudo sshd -t
sudo systemctl reload ssh 2>/dev/null || sudo systemctl reload sshd
creates: /etc/ssh/sshd_config.d/10-fleet.conf
verify:
- name: Boots to multi-user, not graphical
shell: test "$(systemctl get-default)" = "multi-user.target"
changed_when: "false"
- name: Will boot headless — default target is multi-user
# The DURABLE requirement, true across reboots regardless of what is
# running right now.
shell: test "$(systemctl get-default)" = "multi-user.target"
changed_when: "false"
- name: GDM is not running
# is-ACTIVE. Expected to FAIL while someone holds a console session and
# force_dm_stop is false — that is an honest red, not a broken playbook.
# A reboot (or the rack install) resolves it.
shell: "! systemctl is-active --quiet gdm"
changed_when: "false"
- name: Sleep targets are masked
shell: systemctl is-enabled sleep.target 2>/dev/null | grep -q masked
changed_when: "false"
- name: Static hostname is correct
shell: test "$(hostnamectl --static)" = "{{ fqdn_hostname }}"
changed_when: "false"
- name: GPU still enumerates after the desktop stack stopped
# The point of the exercise. If disabling GDM cost us the GPU, this catches
# it here rather than the next time someone tries to load a model.
shell: nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader | grep -q GB10
changed_when: "false"
- name: sshd config still parses
shell: sudo sshd -t
changed_when: "false"
playbooks/gx10-outfit-privileged.yaml
+131
View File
@@ -0,0 +1,131 @@
# pfi-gx10 — the privileged half of outfitting the box as an inference + training
# machine. The userspace half (uv, ~/ml/.venv, torch 2.14.0+cu130, the whole
# transformers/peft/trl stack) is ALREADY DONE and needed no root; this playbook
# is only the part that does.
#
# scripts/elway infra-ops@10.100.10.226 --playbook playbooks/gx10-outfit-privileged.yaml
#
# ─────────────────────────────────────────────────────────────────────────────
# ⚠ PREREQUISITE — infra-ops has NO NOPASSWD SUDO ON THIS BOX
#
# pfi-gx10 is the fleet exception: infra-ops is in the `sudo` group with
# `(ALL : ALL) ALL`, but every invocation wants a password, and no gx10
# credential is in the vault. elway prompts once via getpass, so an interactive
# operator can just run this. A non-interactive agent cannot.
#
# Step 1 below closes that permanently and makes gx10 consistent with the rest
# of the fleet. Run this playbook once interactively and the exception is gone.
# ─────────────────────────────────────────────────────────────────────────────
#
# WHY BARE METAL AND NOT A VM (settled 2026-09-01, do not re-litigate):
# Proxmox VE has no aarch64 build, and the GB10's GPU sits on an on-package
# NVIDIA root complex cache-coherent with the CPU over NVLink-C2C sharing one
# pool of LPDDR5X. Passing it to a guest partitions the unified memory that is
# the entire reason for the box. See persistent-memory.d/2026-09-01-pfi-gx10-*.
vars:
user: infra-ops
sudoers_file: /etc/sudoers.d/infra-ops-nopasswd
cuda_keyring_url: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/arm64/cuda-keyring_1.1-1_all.deb
steps:
- name: Grant infra-ops NOPASSWD sudo (ends the gx10 fleet exception)
# visudo -cf validates BEFORE the file is put in place. A malformed
# sudoers drop-in locks every sudo user out of the box, and this one is
# on a desk with no iDRAC.
sudo: true
shell: |
printf '%s ALL=(ALL) NOPASSWD:ALL\n' {{ user }} > /tmp/.sudoers-candidate
visudo -cf /tmp/.sudoers-candidate
install -m 0440 -o root -g root /tmp/.sudoers-candidate {{ sudoers_file }}
rm -f /tmp/.sudoers-candidate
when: "! sudo -n true 2>/dev/null"
- name: Verify NOPASSWD actually took before relying on it
# Asserting the effective behaviour, not the presence of a file.
shell: sudo -n true
changed_when: "false"
- name: Add infra-ops to the docker group
# The daemon already runs; infra-ops just gets EACCES on the socket.
sudo: true
shell: usermod -aG docker {{ user }}
when: "! id -nG {{ user }} | tr ' ' '\\n' | grep -qx docker"
- name: Install the NVIDIA CUDA apt repository keyring
sudo: true
shell: |
cd /tmp
curl -fsSLO {{ cuda_keyring_url }}
dpkg -i cuda-keyring_1.1-1_all.deb
rm -f cuda-keyring_1.1-1_all.deb
when: "! test -f /etc/apt/sources.list.d/cuda-ubuntu2404-arm64.list"
- name: apt update
sudo: true
shell: apt-get update -qq
- name: Install the CUDA toolkit (nvcc and friends)
# NOT `cuda` or `cuda-drivers` -- the driver is already installed and
# working (580.173.02) and pulling the metapackage risks replacing it.
# cuda-toolkit is the compiler + libraries only.
sudo: true
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq cuda-toolkit
when: "! test -x /usr/local/cuda/bin/nvcc"
- name: Install python3-dev + build-essential (Triton JITs C at RUNTIME)
# Triton compiles its CUDA-utils shim with gcc on FIRST USE and needs
# Python.h to do it. Without python3-dev, torch imports fine, the model
# loads fine, LoRA attaches fine -- and then the first training step dies
# with a CalledProcessError from gcc whose real message ("fatal error:
# Python.h: No such file or directory") is swallowed, because Triton
# sends the compiler's stdout to DEVNULL. Cost a probe run to find.
# Same shape as the dots-tts container needing a C compiler at runtime.
sudo: true
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3-dev build-essential
when: "! test -f /usr/include/python3.12/Python.h"
- name: Install the NVIDIA Container Toolkit
sudo: true
shell: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq nvidia-container-toolkit
when: "! command -v nvidia-ctk >/dev/null 2>&1"
- name: Wire the container runtime into dockerd
sudo: true
shell: nvidia-ctk runtime configure --runtime=docker && systemctl restart docker
when: "! grep -q nvidia /etc/docker/daemon.json 2>/dev/null"
verify:
- name: NOPASSWD sudo works
shell: sudo -n true
changed_when: "false"
- name: nvcc is on PATH and reports a version
shell: /usr/local/cuda/bin/nvcc --version | grep -q "release"
changed_when: "false"
- name: infra-ops can reach the docker socket
# `sg docker` picks up the new group without needing a fresh login.
shell: sg docker -c 'docker info >/dev/null'
changed_when: "false"
- name: A CUDA container can see the GB10
shell: sg docker -c 'docker run --rm --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi -L' | grep -q GB10
changed_when: "false"
- name: Triton can actually compile its runtime shim
# Asserting the effective behaviour. `dpkg -l python3-dev` would pass while
# the compile still failed for a missing header or library path.
shell: |
V=/home/infra-ops/ml/.venv/lib/python3.12/site-packages/triton/backends/nvidia
test -d "$V" || exit 0
gcc "$V/driver.c" -O3 -shared -fPIC -Wno-psabi -o /tmp/.tritoncheck.so \
-l:libcuda.so.1 -L"$V/lib" -L/lib/aarch64-linux-gnu \
-I"$V/include" -I/usr/include/python3.12
rm -f /tmp/.tritoncheck.so
changed_when: "false"
- name: The userspace torch stack still sees the GPU
# Guards against a toolkit install disturbing the working driver.
shell: /home/infra-ops/ml/.venv/bin/python -c 'import torch;assert torch.cuda.is_available()'
changed_when: "false"
playbooks/gx10-rack-network.yaml
+138
View File
@@ -0,0 +1,138 @@
# pfi-gx10 — RETIRED 2026-09-03. Kept for its reasoning, not for running.
#
# ⚠ THE STATIC-IP APPROACH IN THIS FILE WAS NOT USED. Operator ruling on the
# day of the rack move: put the address on the SWITCH/FIREWALL side as a DHCP
# RESERVATION and leave the host on DHCP, so moving the box later does not mean
# unpicking a stale netplan static on a machine you may not be able to reach.
# The host-static this playbook writes is exactly the "interesting adventure"
# that ruling avoids.
#
# What was actually done instead (see docs/runbooks/gx10-rack-network.md):
# 1. UniFi switch port 22 native network -> nh3-servers (VLAN 50)
# 2. UniFi client reservation: 30:c5:99:3d:a7:45 -> 10.100.50.60
# 3. host DHCP renew; Wi-Fi left UP until the wired path was proven
# 4. dns/internal.yaml + dns-sync
# 5. nmcli radio wifi off, last, as its own step
#
# The safety ORDER below is still right and was followed: verify carrier,
# verify MAC not interface name, keep Wi-Fi up as the escape hatch, prove the
# new address from OUTSIDE, tear the hatch down separately and last.
#
# pfi-gx10 — move from desk (Wi-Fi + DHCP, VLAN 10) to rack (wired static,
# VLAN 50 nh3-servers). Run AFTER the box is racked and the switch port has
# been placed on nh3-servers.
#
# scripts/elway infra-ops@<current-ip> --playbook playbooks/gx10-rack-network.yaml
#
# Find <current-ip> without asking anyone — the wired NIC has its own MAC:
# curl -sk "https://10.100.0.1/proxy/network/api/s/default/stat/sta" \
# -H "X-API-KEY: $(secret get unifi/pfi-udmse-api-key)" \
# | python3 -c "import json,sys;[print(c['ip'],c.get('sw_port')) for c in json.load(sys.stdin)['data'] if c['mac']=='30:c5:99:3d:a7:45']"
#
# ─────────────────────────────────────────────────────────────────────────────
# WHY THIS IS SAFE TO RUN AGAINST A BOX IN A RACK
#
# The lockout risk in remote network reconfiguration is applying a static
# address that does not work and losing the only path back. This playbook never
# has one path: Wi-Fi (wlP9s9) STAYS UP THROUGHOUT. The wired interface is
# configured alongside it, verified from outside on the new address, and only
# then is Wi-Fi torn down — by a SEPARATE playbook run, after a human has seen
# the new address answer.
#
# So a botched netplan costs a retry over Wi-Fi, not a trip to the rack.
# ─────────────────────────────────────────────────────────────────────────────
vars:
wired_if: enP7s7
wired_mac: "30:c5:99:3d:a7:45"
static_ip: 10.100.50.60
prefix: "24"
gateway: 10.100.50.1
dns_servers: "10.100.50.40" # nh3-docker / AdGuard, per dns/internal.yaml
search_domain: nh3.internal
steps:
- name: Refuse to proceed if the cable is not actually in
# carrier=0 means no link. Writing a static config for a dead interface and
# calling it done is the kind of green that costs a rack visit to discover.
shell: test "$(cat /sys/class/net/{{ wired_if }}/carrier 2>/dev/null)" = "1"
- name: Confirm the interface is the NIC we think it is
# Interface names can renumber across kernels. The MAC cannot.
shell: test "$(cat /sys/class/net/{{ wired_if }}/address)" = "{{ wired_mac }}"
- name: Back up the existing netplan
shell: sudo cp -an /etc/netplan /etc/netplan.bak-preRack 2>/dev/null || true
creates: /etc/netplan.bak-preRack
- name: Write the wired static config
# A SEPARATE file from whatever manages Wi-Fi. Wi-Fi keeps working while
# this lands; that is the escape hatch and it is deliberate.
shell: |
sudo tee /etc/netplan/60-wired-static.yaml >/dev/null <<'EOF'
network:
version: 2
ethernets:
{{ wired_if }}:
dhcp4: false
dhcp6: false
addresses: [{{ static_ip }}/{{ prefix }}]
routes:
- to: default
via: {{ gateway }}
metric: 100
nameservers:
addresses: [{{ dns_servers }}]
search: [{{ search_domain }}]
EOF
sudo chmod 600 /etc/netplan/60-wired-static.yaml
creates: /etc/netplan/60-wired-static.yaml
- name: Apply
# `netplan try` would be better but needs a TTY to accept or roll back;
# under elway there is nobody to press a key. Wi-Fi staying up is what
# substitutes for the auto-rollback.
shell: sudo netplan apply
- name: Wait for the address to come up locally
shell: |
for i in $(seq 1 15); do
ip -4 addr show {{ wired_if }} | grep -q "{{ static_ip }}/{{ prefix }}" && exit 0
sleep 2
done
exit 1
verify:
- name: Wired interface holds the static address
shell: ip -4 addr show {{ wired_if }} | grep -q "{{ static_ip }}/{{ prefix }}"
changed_when: "false"
- name: Default route is via the servers-VLAN gateway
shell: ip route get 1.1.1.1 | grep -q "via {{ gateway }}"
changed_when: "false"
- name: Gateway answers
shell: ping -c 2 -W 2 {{ gateway }} >/dev/null
changed_when: "false"
- name: DNS resolves through the fleet resolver
shell: getent hosts nh3-docker.nh3.internal >/dev/null
changed_when: "false"
- name: Wi-Fi is STILL UP — the escape hatch must survive this run
# If this fails, the box may be reachable only on an address that was just
# proven or just broken. Loudly not optional.
shell: ip link show wlP9s9 | grep -q "state UP"
changed_when: "false"
# ─────────────────────────────────────────────────────────────────────────────
# AFTERWARDS, and only once the new address has been confirmed from OUTSIDE:
#
# 1. ssh infra-ops@10.100.50.60 # prove it from another host first
# 2. update servers/pfi-gx10/ssh-target
# 3. add to dns/internal.yaml, then scripts/dns-sync.py --dry-run && ...sync
# 4. tear down Wi-Fi — a server with two default routes picks one at random
# and you find out later, in a way that looks like packet loss:
# sudo nmcli radio wifi off # or delete the wlP9s9 netplan stanza
# Do that as its own change, not bundled in here.
# ─────────────────────────────────────────────────────────────────────────────
playbooks/homepage-regroup-ana-docker.yaml
+226
View File
@@ -0,0 +1,226 @@
# Homepage recategorisation — ana-docker (10.250.50.70), 13 containers.
#
# Splits the dashboard on ONE axis: do you open this thing, or is it an
# endpoint you only want to know is alive? See the layout: block in
# stacks/homepage/conf/settings.yaml for the target shape.
#
# memos, miniflux, nevermore, searxng -> Daily (was Notes / News / Apps)
# zed-fim-proxy -> AI - Inference (no UI; href is /ping)
# adguardhome -> DNS & Filtering
# traefik -> Reverse Proxies
# dockge -> Compose Consoles
# crowdsec, mailrise, rest-server,
# gitea-runner, hbbr (RustDesk relay) -> Agents (no UI)
#
# `homepage.group` is read at container CREATION, so each edit is followed by
# `compose up -d <service>` — a restart would leave the old label in place.
# Both halves are idempotent: the sed is gated on the old value still being
# present, and `up -d` is a no-op when the container already matches its spec.
#
# Run: scripts/elway infra-ops@10.250.50.70 --playbook playbooks/homepage-regroup-ana-docker.yaml
steps:
# ---- label edits -------------------------------------------------------
- name: memos -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=Notes$|homepage.group=Daily|'
/opt/docker/compose/memos/compose.yaml
when: grep -q 'homepage.group=Notes$' /opt/docker/compose/memos/compose.yaml
- name: miniflux -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=News$|homepage.group=Daily|'
/opt/docker/compose/miniflux/compose.yaml
when: grep -q 'homepage.group=News$' /opt/docker/compose/miniflux/compose.yaml
- name: nevermore -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=News$|homepage.group=Daily|'
/opt/docker/compose/nevermore/compose.yaml
when: grep -q 'homepage.group=News$' /opt/docker/compose/nevermore/compose.yaml
- name: searxng -> Daily
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Daily|'
/opt/docker/compose/searxng/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/searxng/compose.yaml
- name: zed-fim-proxy -> AI - Inference
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Gateways . Chat$|homepage.group=AI - Inference|'
/opt/docker/compose/zed-fim-proxy/compose.yaml
when: grep -q 'homepage.group=AI - Gateways . Chat$' /opt/docker/compose/zed-fim-proxy/compose.yaml
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard-ana/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard-ana/compose.yaml
- name: traefik -> Reverse Proxies
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|'
/opt/docker/compose/traefik/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/compose.yaml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: crowdsec -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/crowdsec/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/crowdsec/compose.yaml
- name: mailrise -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/mailrise/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/mailrise/compose.yaml
- name: rest-server -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Agents (no UI)|'
/opt/docker/compose/rest-server-ana/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/rest-server-ana/compose.yaml
- name: gitea-runner -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Toolchain$|homepage.group=Agents (no UI)|'
/opt/docker/compose/gitea-runner/compose.yaml
when: grep -q 'homepage.group=Toolchain$' /opt/docker/compose/gitea-runner/compose.yaml
- name: rustdesk (hbbr) -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|'
/opt/docker/compose/rustdesk/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/rustdesk/compose.yaml
# Two containers were both named plain "Open WebUI" and, once the ESH one
# joined this group, they landed side by side — same name, same icon family,
# only the description telling them apart. Site suffix, like Traefik/Dockge/
# AdGuard already carry.
- name: openwebui (ana) -> "Open WebUI (ana)"
sudo: true
shell: >-
sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (ana)|'
/opt/docker/compose/openwebui/compose.yaml
when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/openwebui/compose.yaml
# ---- recreates ---------------------------------------------------------
# traefik goes LAST: crowdsec is its bouncer, so bounce the bouncer first
# and let traefik come up against a settled agent.
- name: recreate memos
sudo: true
shell: cd /opt/docker/compose/memos && docker compose up -d memos
- name: recreate miniflux
sudo: true
shell: cd /opt/docker/compose/miniflux && docker compose up -d miniflux
- name: recreate nevermore-web
sudo: true
shell: cd /opt/docker/compose/nevermore && docker compose up -d nevermore-web
- name: recreate searxng
sudo: true
shell: cd /opt/docker/compose/searxng && docker compose up -d searxng
- name: recreate zed-fim-proxy
sudo: true
shell: cd /opt/docker/compose/zed-fim-proxy && docker compose up -d zed-fim-proxy
- name: recreate mailrise
sudo: true
shell: cd /opt/docker/compose/mailrise && docker compose up -d mailrise
- name: recreate rest-server
sudo: true
shell: cd /opt/docker/compose/rest-server-ana && docker compose up -d rest-server
- name: recreate gitea-runner
sudo: true
shell: cd /opt/docker/compose/gitea-runner && docker compose up -d runner
- name: recreate rustdesk relay
sudo: true
shell: cd /opt/docker/compose/rustdesk && docker compose up -d hbbr
- name: recreate openwebui (ana)
sudo: true
shell: cd /opt/docker/compose/openwebui && docker compose up -d open-webui
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard-ana && docker compose up -d adguardhome
- name: recreate crowdsec
sudo: true
shell: cd /opt/docker/compose/crowdsec && docker compose up -d crowdsec
- name: recreate traefik
sudo: true
shell: cd /opt/docker/compose/traefik && docker compose up -d traefik
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
memos miniflux nevermore-web searxng zed-fim-proxy adguardhome traefik
dockge crowdsec mailrise rest-server gitea-runner hbbr
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# dig is not installed everywhere in the fleet, so fall back to the AdGuard
# UI — a resolver that serves its own dashboard on :8053 has come back up.
- name: adguard is back (DNS answer, or its UI if dig is absent)
changed_when: "false"
shell: >-
if command -v dig >/dev/null 2>&1;
then dig +short +time=3 +tries=2 @10.250.50.70 gitea.phasefinal.com | grep -q .;
else curl -sf -o /dev/null -m 8 http://10.250.50.70:8053/; fi
# Retried, not one-shot: the first run of this playbook checked 0.12s after
# `Started` and got rc=7 while traefik was still binding. The container was
# fine — `:8380/` 301s to /dashboard/ and both public hostnames answered 200
# seconds later. A recreate needs a moment; assert the settled state.
- name: traefik still routes
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0;
sleep 3; done; exit 1
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' memos miniflux nevermore-web
searxng zed-fim-proxy adguardhome traefik dockge crowdsec mailrise
rest-server gitea-runner hbbr | sort -u)" = "true"
playbooks/homepage-regroup-ana-ml2.yaml
+74
View File
@@ -0,0 +1,74 @@
# Homepage recategorisation — ana-ml2 (10.250.50.54), 2 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# scriberr -> AI - Studios (a transcription UI you open, not an API seat)
# dockge -> Compose Consoles
#
# ⚠ THE vLLM SEATS ON THIS HOST ARE DELIBERATELY NOT TOUCHED. Every one of them
# would need a recreate to change its `homepage.group`, and a recreate means a
# multi-minute model reload on a seat that peers reach through the gateway. The
# separation the operator asked for — UI up top, API endpoints out of the way —
# is achieved for those groups by ORDER and `initiallyCollapsed` in
# stacks/homepage/conf/settings.yaml, which costs nothing. Keep it that way: if
# a future pass wants to rename `AI - Inference`, weigh it against bouncing six
# model seats.
#
# Run: scripts/elway infra-ops@10.250.50.54 --playbook playbooks/homepage-regroup-ana-ml2.yaml
steps:
- name: scriberr -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|'
/opt/docker/compose/scriberr/compose.yaml
when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/scriberr/compose.yaml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate scriberr
sudo: true
shell: cd /opt/docker/compose/scriberr && docker compose up -d scriberr
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
scriberr dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# Container age, not liveness — a seat may be legitimately stopped, so
# "is it running" cannot answer "did I bounce it". See the same step in
# playbooks/homepage-regroup-irv-ml1.yaml for how that distinction was found.
- name: the vLLM seats were NOT recreated by this run
sudo: true
changed_when: "false"
shell: >-
for c in $(docker ps -a --filter 'name=vllm-' --filter 'name=llama-'
--format '{{.Names}}'); do
created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue;
age=$(( $(date +%s) - $(date -d "$created" +%s) ));
if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi;
done
- name: scriberr answers
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 5; done; exit 1
playbooks/homepage-regroup-esh-docker-vm.yaml
+135
View File
@@ -0,0 +1,135 @@
# Homepage recategorisation — esh-docker-vm (10.0.50.45), 6 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; the rationale, the
# label-at-creation constraint and the idempotency scheme are documented there.
#
# lobe-chat, open-webui -> AI - Gateways & Chat (chat frontends belong
# with the other chat frontends, not in Apps)
# adguardhome -> DNS & Filtering
# traefik -> Reverse Proxies
# dockge -> Compose Consoles
# mosquitto -> Agents (no UI) (an MQTT broker has no page)
#
# ⚠ adguard and traefik here use `docker-compose.yml`, not `compose.yaml`.
#
# Run: scripts/elway infra-ops@10.0.50.45 --playbook playbooks/homepage-regroup-esh-docker-vm.yaml
steps:
- name: lobe-chat -> AI - Gateways & Chat
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|'
/opt/docker/compose/lobe-chat/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/lobe-chat/compose.yaml
- name: open-webui -> AI - Gateways & Chat
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=AI - Gateways \& Chat|'
/opt/docker/compose/open-webui/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/open-webui/compose.yaml
# Site suffix — the ana instance is also called "Open WebUI" and the two now
# sit side by side in the same group. See the sibling step in
# playbooks/homepage-regroup-ana-docker.yaml.
- name: open-webui -> "Open WebUI (esh)"
sudo: true
shell: >-
sed -i 's|homepage.name=Open WebUI$|homepage.name=Open WebUI (esh)|'
/opt/docker/compose/open-webui/compose.yaml
when: grep -q 'homepage.name=Open WebUI$' /opt/docker/compose/open-webui/compose.yaml
- name: mosquitto -> Agents (no UI)
sudo: true
shell: >-
sed -i 's|homepage.group=Apps$|homepage.group=Agents (no UI)|'
/opt/docker/compose/mosquitto/compose.yaml
when: grep -q 'homepage.group=Apps$' /opt/docker/compose/mosquitto/compose.yaml
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml
- name: traefik -> Reverse Proxies
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Reverse Proxies|'
/opt/docker/compose/traefik/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/traefik/docker-compose.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
# ---- recreates ---------------------------------------------------------
- name: recreate lobe-chat
sudo: true
shell: cd /opt/docker/compose/lobe-chat && docker compose up -d lobe-chat
- name: recreate open-webui
sudo: true
shell: cd /opt/docker/compose/open-webui && docker compose up -d open-webui
- name: recreate mosquitto
sudo: true
shell: cd /opt/docker/compose/mosquitto && docker compose up -d mosquitto
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome
- name: recreate traefik
sudo: true
shell: cd /opt/docker/compose/traefik && docker compose up -d traefik
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
lobe-chat open-webui mosquitto adguardhome traefik dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
- name: adguard is back
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 3; done; exit 1
- name: traefik still routes
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8380/dashboard/ && exit 0;
sleep 3; done; exit 1
# ⚠ Must be 10.0.50.45, NOT 127.0.0.1. `HOMEPAGE_ALLOWED_HOSTS` matches
# host AND port, and `127.0.0.1:5100` is not in the list — it answers 400
# while the dashboard is perfectly healthy. The first run of this playbook
# failed here on rc=22 for exactly that reason.
- name: the dashboard itself is still served
changed_when: "false"
shell: curl -sf -o /dev/null -m 10 http://10.0.50.45:5100/api/services
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' lobe-chat open-webui
mosquitto adguardhome traefik dockge | sort -u)" = "true"
playbooks/homepage-regroup-irv-ml1.yaml
+115
View File
@@ -0,0 +1,115 @@
# Homepage recategorisation — irv-ml1 (10.100.79.3), 5 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# arbo, comfyui, waterland-studio -> AI - Studios (was AI - Image & Media)
# yt-voice-clipper -> AI - Studios (was AI - Audio Tools)
# dockge -> Compose Consoles
#
# `AI - Studios` is the "you open this and do work in it" group; the ASR and
# TTS API seats stay where they are and get collapsed by settings.yaml instead,
# which is what keeps the GPU seats out of this playbook entirely.
#
# ⚠ yt-voice-clipper carries its homepage labels in `docker-compose.override.yml`,
# not in `docker-compose.yml`, and its compose dir is a git checkout of the
# project — the override is the deploy-local layer, which is the right place
# for it.
#
# ⚠ This host is reached over the WireGuard tunnel. If the run cannot connect,
# check the tunnel before assuming the host is down.
#
# Run: scripts/elway infra-ops@10.100.79.3 --playbook playbooks/homepage-regroup-irv-ml1.yaml
steps:
- name: arbo -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/arbo/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/arbo/compose.yaml
- name: comfyui -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/comfyui/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/comfyui/compose.yaml
- name: waterland-studio -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Image . Media$|homepage.group=AI - Studios|'
/opt/docker/compose/waterland-studio/compose.yaml
when: grep -q 'homepage.group=AI - Image . Media$' /opt/docker/compose/waterland-studio/compose.yaml
- name: yt-voice-clipper -> AI - Studios
sudo: true
shell: >-
sed -i 's|homepage.group=AI - Audio Tools$|homepage.group=AI - Studios|'
/opt/docker/compose/yt-voice-clipper/docker-compose.override.yml
when: grep -q 'homepage.group=AI - Audio Tools$' /opt/docker/compose/yt-voice-clipper/docker-compose.override.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
# ---- recreates ---------------------------------------------------------
- name: recreate arbo
sudo: true
shell: cd /opt/docker/compose/arbo && docker compose up -d engine
- name: recreate comfyui
sudo: true
shell: cd /opt/docker/compose/comfyui && docker compose up -d comfyui
- name: recreate waterland-studio
sudo: true
shell: cd /opt/docker/compose/waterland-studio && docker compose up -d waterland-studio
- name: recreate yt-voice-clipper
sudo: true
shell: cd /opt/docker/compose/yt-voice-clipper && docker compose up -d api
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
arbo comfyui waterland-studio yt-voice-clipper-api-1 dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
# Assert NOT-RECREATED, not RUNNING. The first version of this checked that
# all five seats were up and failed — because chatterbox-fast has been down
# since 2026-08-10 and speaches since earlier the same morning, both long
# before this playbook existed. "Is it running" is the wrong question: a seat
# can be legitimately stopped. The question this step is actually asking is
# "did I bounce a model seat to relabel a dashboard", and container age
# answers it directly.
- name: the TTS and ASR seats were NOT recreated by this run
sudo: true
changed_when: "false"
shell: >-
for c in kokoro dots-tts chatterbox-fast tts-gateway parakeet speaches; do
created=$(docker inspect -f '{{.Created}}' "$c" 2>/dev/null) || continue;
age=$(( $(date +%s) - $(date -d "$created" +%s) ));
if [ "$age" -lt 600 ]; then echo "$c was recreated ${age}s ago"; exit 1; fi;
done
- name: everything relabelled is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' arbo comfyui
waterland-studio yt-voice-clipper-api-1 dockge | sort -u)" = "true"
playbooks/homepage-regroup-nh3-docker.yaml
+58
View File
@@ -0,0 +1,58 @@
# Homepage recategorisation — nh3-docker (10.100.50.40), 2 containers.
# Sibling of playbooks/homepage-regroup-ana-docker.yaml; rationale lives there.
#
# adguardhome -> DNS & Filtering (⚠ uses docker-compose.yml)
# dockge -> Compose Consoles
#
# Run: scripts/elway infra-ops@10.100.50.40 --playbook playbooks/homepage-regroup-nh3-docker.yaml
steps:
- name: adguardhome -> DNS & Filtering
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=DNS \& Filtering|'
/opt/docker/compose/adguard/docker-compose.yml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/adguard/docker-compose.yml
- name: dockge -> Compose Consoles
sudo: true
shell: >-
sed -i 's|homepage.group=Service Networking$|homepage.group=Compose Consoles|'
/opt/docker/compose/dockge/compose.yaml
when: grep -q 'homepage.group=Service Networking$' /opt/docker/compose/dockge/compose.yaml
- name: recreate dockge
sudo: true
shell: cd /opt/docker/compose/dockge && docker compose up -d dockge
- name: recreate adguardhome
sudo: true
shell: cd /opt/docker/compose/adguard && docker compose up -d adguardhome
verify:
- name: every relabelled container now carries its new group
sudo: true
changed_when: "false"
shell: >-
docker inspect -f '{{.Name}} {{index .Config.Labels "homepage.group"}}'
adguardhome dockge
- name: no container is left in the retired Service Networking group
sudo: true
changed_when: "false"
shell: >-
test -z "$(docker ps -q --filter 'label=homepage.group=Service Networking')"
- name: adguard is back
changed_when: "false"
shell: >-
for i in 1 2 3 4 5 6 7 8 9 10; do
curl -sfL -o /dev/null -m 5 http://127.0.0.1:8080/ && exit 0;
sleep 3; done; exit 1
- name: everything is running
sudo: true
changed_when: "false"
shell: >-
test "$(docker inspect -f '{{.State.Running}}' adguardhome dockge
| sort -u)" = "true"
playbooks/install-potrace-headers.yaml
+100
View File
@@ -0,0 +1,100 @@
# Install the potrace + agg development headers so `pypotrace` can build from source.
#
# Why a playbook and not a one-liner: `pypotrace` is an **sdist that compiles at install
# time**, so every machine and every CI runner that resolves it needs these headers present
# FIRST. That makes this a recurring per-box action, not a one-off. Requested by forseti
# (pewpewstudio) for `core.image_pipeline`, which vectorises raster art into laser-ready
# contours; the operator chose pypotrace over shelling out to the potrace binary (2026-08-28).
#
# Run: scripts/elway infra-ops@<host> --playbook playbooks/install-potrace-headers.yaml
# Rerunnable: a second run shows the install `skipped`.
#
# ⚠ TWO THINGS THAT WILL SEND YOU DOWN THE WRONG PATH ON A BOX WHERE THIS FAILS
#
# 1. **Only libagg is a pkg-config consumer. potrace is not.**
#
# libagg /usr/lib/x86_64-linux-gnu/pkgconfig/libagg.pc present
# potrace NO .pc file — found via /usr/include/potracelib.h and the library
#
# So `pkg-config --exists potrace` returns FALSE on a correctly configured box. It looks
# exactly like the cause and never is. The real build error names libagg and only libagg:
#
# Package libagg was not found in the pkg-config search path.
# Package 'libagg', required by 'virtual:world', not found
#
# A wrong model that produces a plausible-looking diagnostic costs more than no model.
#
# 2. **libagg's pkg-config modversion disagrees with its Debian package version.**
#
# pkg-config --modversion libagg -> 2.7.0
# dpkg version -> 1:2.6.1-r134+dfsg1-2+b1
#
# Comparing those two numbers convinces you the wrong package is installed. It is not a
# problem; it is upstream's version vs Debian's packaging of it.
#
# Both of these were learned on the nh3-dev install and are recorded here rather than in an
# althing thread, at forseti's suggestion, because a thread is not where the next person looks.
vars:
probe_venv: /tmp/pypotrace-probe
# The user whose toolchain builds the probe. The headers are installed system-wide
# as root; the build check is a developer action and runs as this user.
dev_user: lkraven
steps:
- name: Install the potrace and agg development headers
shell: sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libpotrace-dev libagg-dev
when: "! dpkg -s libpotrace-dev >/dev/null 2>&1 || ! dpkg -s libagg-dev >/dev/null 2>&1"
verify:
- name: libagg's pkg-config file is discoverable (this is the one that actually gates the build)
shell: pkg-config --exists libagg
changed_when: "false"
- name: potrace's header is present (NOT via pkg-config — it ships no .pc)
shell: test -f /usr/include/potracelib.h
changed_when: "false"
- name: pypotrace COMPILES against them
# ⚠ `uv` is NOT on a non-interactive ssh PATH — infra-ops gets
# /usr/local/bin:/usr/bin:/bin:/usr/games and nothing else. It also lives in a
# different place on every box: /home/lkraven/bin/uv on nh3-dev,
# /home/infra-ops/.local/bin/uv on nh3-extdev. Search rather than assume, and say
# so loudly if it is genuinely absent — a build probe that silently does not run
# is the failure this whole playbook exists to prevent.
shell: |
# ...and on nh3-dev it is inside a 0700 home, so `test -x` from infra-ops fails
# even with the right absolute path — the directory cannot be traversed. The
# headers are system-wide (root's business); the build probe is a DEVELOPER
# action and has to run as the user who owns the toolchain.
RUNAS={{ dev_user }}
UV=""
for c in /home/{{ dev_user }}/bin/uv /home/{{ dev_user }}/.local/bin/uv /usr/local/bin/uv; do
sudo -u "$RUNAS" test -x "$c" && { UV="$c"; break; }
done
[ -n "$UV" ] || { echo "no uv reachable as $RUNAS; cannot run the build probe"; exit 1; }
echo " using uv at $UV (as $RUNAS)"
sudo -u "$RUNAS" rm -rf {{ probe_venv }}
sudo -u "$RUNAS" "$UV" venv {{ probe_venv }} >/dev/null 2>&1
sudo -u "$RUNAS" "$UV" pip install --python {{ probe_venv }}/bin/python pypotrace 2>&1 | tail -2
changed_when: "false"
- name: and the built extension actually TRACES, which a successful build does not prove
# A square must come back as one curve of four CornerSegments. If the extension linked
# against something wrong it can still import and return nonsense; the geometry is the
# assertion, not the import.
shell: |
sudo -u {{ dev_user }} {{ probe_venv }}/bin/python -c "
import numpy as np, potrace
a = np.zeros((40,40), np.uint32); a[10:30,10:30] = 1
curves = list(potrace.Bitmap(a).trace())
segs = [s for c in curves for s in c]
assert len(curves) == 1, curves
assert len(segs) == 4, segs
assert {type(s).__name__ for s in segs} == {'CornerSegment'}, segs
"
changed_when: "false"
- name: Remove the probe venv
shell: sudo -u {{ dev_user }} rm -rf {{ probe_venv }}
changed_when: "false"
playbooks/mog-sec-move-to-gpu0.yaml
+80
View File
@@ -0,0 +1,80 @@
# Move the `sec` pen-test seat (M.O.G.-SEC-27B) from ana-ml2 GPU 1 to GPU 0 and bring it up.
#
# Why: GPU 1 carries the five resident fleet seats (gen 46 GB + embed 9.8 + coder 8.4 +
# rerank 3.5 + reward 2.1 = ~69.9 GB of 97.9), leaving ~28 GB. This seat reserves
# MOG_GPU_MEM_UTIL=0.52 -> ~51 GB, so it could not start on GPU 1 at all. GPU 0 has been
# idle since run 3c was stopped on 2026-08-26. Operator-directed 2026-08-28.
#
# ⚠ POWER. This re-arms the two-GPU load condition that tripped the Anaheim rack breaker
# on 2026-08-26. One circuit feeds the whole rack including ana-gw and ana-wg, so a trip
# costs the site AND the way back in. Idle draw is negligible; the risk materialises when
# sec and gen are under concurrent load. Operator accepted this with the constraint stated.
#
# Labels only apply at container CREATION, so this uses `up -d`, never `restart` --
# the homepage description carries the GPU number and would otherwise stay stale.
#
# Run: scripts/elway infra-ops@ana-ml2 --playbook playbooks/mog-sec-move-to-gpu0.yaml
# Model load is slow (22 GB + 262K ctx + MTP graft); the verify phase polls rather than
# assuming readiness, and the compose healthcheck allows a 900s start_period.
vars:
stack_dir: /opt/docker/compose/mog-sec
container: vllm-mog-sec
service: vllm-mog-sec
gpu_id: "0"
port: "8019"
staging: /tmp/mog-sec-compose.yaml
steps:
- name: Stage the updated compose (GPU pin default + label now say GPU 0)
upload:
src: stacks/mog-sec/compose.yaml
dest: "{{ staging }}"
mode: "0644"
- name: Install it over the deployed copy
# /opt/docker/compose is root-owned, so the scp above lands in /tmp and this
# promotes it. Verified byte-identical against the deployed file beforehand:
# the only diff was these edits, so nothing on the host is being clobbered.
shell: sudo install -o root -g root -m 0644 {{ staging }} {{ stack_dir }}/compose.yaml
changed_when: "! sudo cmp -s {{ staging }} {{ stack_dir }}/compose.yaml"
- name: Pin the seat to GPU {{ gpu_id }} in the host .env
# The .env is the tunable surface and is NOT in git (secrets//tunables are
# excluded both directions). The compose default now matches, but the .env
# is what actually decides, so set it explicitly rather than relying on the
# default resolving.
shell: sudo sed -i 's/^MOG_GPU_ID=.*/MOG_GPU_ID={{ gpu_id }}/' {{ stack_dir }}/.env
when: "! sudo grep -qxF 'MOG_GPU_ID={{ gpu_id }}' {{ stack_dir }}/.env"
- name: Bring the seat up (up -d, not restart — labels apply at creation)
shell: cd {{ stack_dir }} && sudo docker compose up -d {{ service }}
verify:
- name: Container exists and is running
# ⚠ No `docker inspect -f` here. Go templates use {{ }} and so does elway's own
# variable substitution, so an inspect format string gets eaten before it reaches
# the host -- these two checks reported FAILED on a deploy that had in fact
# succeeded. Filter-and-grep has no such collision.
shell: sudo docker ps --filter name={{ container }} --filter status=running --quiet | grep -q .
changed_when: "false"
- name: The container is actually pinned to GPU {{ gpu_id }}
# Assert the EFFECTIVE device reservation on the running container, not the
# .env string we wrote -- the .env is an input, this is the outcome.
shell: sudo docker inspect {{ container }} | tr -d ' \n' | grep -q '"DeviceIDs":\["{{ gpu_id }}"\]'
changed_when: "false"
- name: GPU 0 now holds a vLLM process (the seat really loaded onto that card)
shell: nvidia-smi --id={{ gpu_id }} --query-compute-apps=pid,used_memory --format=csv,noheader | grep -qE '[0-9]'
changed_when: "false"
- name: Health endpoint answers
shell: curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/health >/dev/null
changed_when: "false"
- name: Both served names are advertised (base + thinking)
shell: |
MODELS=$(curl -fsS --max-time 10 http://127.0.0.1:{{ port }}/v1/models)
echo "$MODELS" | grep -q 'mog-sec-27b' && echo "$MODELS" | grep -q 'mog-sec-27b-thinking'
changed_when: "false"
playbooks/nh3-dev-memory-forensics.yaml
+116
View File
@@ -0,0 +1,116 @@
# nh3-dev: make memory exhaustion diagnosable after the fact.
#
# Context: three memory-exhaustion events in 14 days (2026-08-14, 2026-08-26,
# 2026-08-28) with the interval halving. forseti reported that none could be
# attributed because "kernel messages are not being persisted to journald".
#
# ⚠ THAT PREMISE WAS WRONG, and the way it was wrong matters more than the fix.
# journald on this box IS persistent and HAS every OOM report: 15,068 kernel
# entries in the 82-day previous boot, 351 OOM records overall, full task tables
# with per-process RSS. `journalctl -b -1 -k` returned "one entry" because it was
# run by a user in neither `adm` nor `systemd-journal` — journalctl silently shows
# you only your OWN messages and prints the reason as a hint. The same artifact
# also produced the "journal stops mid-line at 05:36:08 with no shutdown
# sequence" claim: the true boot -1 boundary is 05:47:04, and OOM kills are
# recorded at 05:38, 05:40 and 05:42.
#
# So step 1 is a GROUP MEMBERSHIP fix, not a logging fix. The evidence was
# always there and unreadable.
#
# What the evidence says, now that it can be read: the hog is Claude Code.
# `/home/lkraven/.local/share/claude/versions/2.1.220` is the versioned CC binary,
# so OOM victims named `2.1.220` / `2.1.177` / `2.1.168` are CC sessions, as are
# the ones named `claude`. Largest single anon-rss recorded: 18.4 GB (2.1.177,
# Aug 14), with 15.8 GB seen twice. Everything else killed — althing-forseti,
# caddy, ttyd, zellij, the althing daemons — is 30-55 MB collateral.
#
# steps 2-4 add the timeseries that the journal cannot give: the journal records
# the moment of the kill, not the ramp toward it, and the Aug 28 event was a hard
# lockup where the box never got far enough to log a coherent sweep.
# - sysstat -> system-wide memory/CPU timeseries (the ramp)
# - atop -> PER-PROCESS timeseries (which session, and how fast)
# atop is the one that answers "which of the dozen sessions", which sar cannot.
#
# Run: scripts/elway infra-ops@nh3-dev --playbook playbooks/nh3-dev-memory-forensics.yaml
# Rerunnable: a second run shows every step `skipped` or `ok`.
vars:
# The interactive/agent user whose sessions read the journal.
journal_user: lkraven
# Debian's journald ACL grants read to `adm` explicitly (getfacl shows
# group:adm:r-x); `systemd-journal` owns the files. `adm` is the documented
# Debian path and the one the ACL names, so use it.
journal_group: adm
# 5 min, not Debian's default 10 — a CC session can add several GB inside one
# 10-minute bucket, which is exactly the resolution the ramp needs.
sar_interval: "*:00/05"
# 60s per-process sample. ~7 generations keeps this under ~1 GB against 80 GB free.
atop_interval: "60"
atop_generations: "7"
steps:
- name: Grant the agent user journal read access (THE actual fix for "no evidence")
shell: sudo usermod -aG {{ journal_group }} {{ journal_user }}
when: "! id -nG {{ journal_user }} | grep -qw {{ journal_group }}"
- name: Install sysstat and atop
shell: sudo DEBIAN_FRONTEND=noninteractive apt-get install -y sysstat atop
when: "! dpkg -s sysstat >/dev/null 2>&1 || ! dpkg -s atop >/dev/null 2>&1"
- name: Enable sysstat collection in /etc/default/sysstat
# The package ships ENABLED="false" and the timer is a no-op until this flips.
shell: sudo sed -i 's/^ENABLED=.*/ENABLED="true"/' /etc/default/sysstat
when: "! grep -qxF 'ENABLED=\"true\"' /etc/default/sysstat 2>/dev/null"
- name: Tighten the sysstat collection interval to 5 minutes
shell: |
sudo mkdir -p /etc/systemd/system/sysstat-collect.timer.d
printf '[Timer]\n# Default is */10. A CC session can add several GB inside one 10-minute\n# bucket; 5 min is the resolution the memory ramp actually needs.\nOnCalendar=\nOnCalendar=%s\n' '{{ sar_interval }}' | sudo tee /etc/systemd/system/sysstat-collect.timer.d/override.conf >/dev/null
when: "! grep -qxF 'OnCalendar={{ sar_interval }}' /etc/systemd/system/sysstat-collect.timer.d/override.conf 2>/dev/null"
- name: Configure atop for 60s per-process sampling with 7-day retention
shell: |
sudo sed -i 's/^LOGINTERVAL=.*/LOGINTERVAL={{ atop_interval }}/' /etc/default/atop
sudo sed -i 's/^LOGGENERATIONS=.*/LOGGENERATIONS={{ atop_generations }}/' /etc/default/atop
when: "! grep -qxF 'LOGINTERVAL={{ atop_interval }}' /etc/default/atop 2>/dev/null || ! grep -qxF 'LOGGENERATIONS={{ atop_generations }}' /etc/default/atop 2>/dev/null"
- name: Reload systemd and enable the collectors
shell: |
sudo systemctl daemon-reload
sudo systemctl enable --now sysstat.service sysstat-collect.timer sysstat-summary.timer
sudo systemctl enable --now atopacct.service atop.service atop-rotate.timer
sudo systemctl restart atop.service
- name: Seed one sysstat sample so sar has data immediately
shell: sudo /usr/lib/sysstat/sa1 1 1
verify:
- name: Agent user is now in the journal-reading group
# `sg` evaluates the membership WITHOUT waiting for a re-login, so this
# asserts the effective grant rather than the /etc/group substring.
shell: sudo -u {{ journal_user }} sg {{ journal_group }} -c 'journalctl -b -1 -k --no-pager 2>/dev/null | wc -l' | awk '{ if ($1 > 100) exit 0; else exit 1 }'
changed_when: "false"
- name: sysstat collection timer is active
shell: systemctl is-active --quiet sysstat-collect.timer
changed_when: "false"
- name: sysstat is collecting at the 5-minute cadence
# ⚠ Assert the EFFECTIVE value, not the string we wrote. systemd normalises
# `*:00/05` to `*-*-* *:00/5:00`, so grepping for our own input fails while
# the setting is live — which is exactly how this verify failed on the first
# run and briefly looked like the override had not applied.
shell: systemctl show sysstat-collect.timer -p TimersCalendar | grep -qF '*:00/5:00'
changed_when: "false"
- name: sar can actually read a memory timeseries (not just that the timer exists)
shell: sar -r 2>/dev/null | tail -2 | grep -qE '[0-9]'
changed_when: "false"
- name: atop daemon is running
shell: systemctl is-active --quiet atop.service
changed_when: "false"
- name: atop is writing a readable per-process log
shell: sudo test -s /var/log/atop/atop_$(date +%Y%m%d)
changed_when: "false"
playbooks/nh3-extdev-althing-v3.yaml
+138
View File
@@ -0,0 +1,138 @@
# nh3-extdev: cut the system-wide althing install over from v2.1.0 to v3.0.x (U9b flag day; re-run for each release).
#
# nh3-extdev is the one box a `git pull` cannot move: althing lives there as a system WHEEL
# under /opt/uv-tools/althing-core with entry points in /usr/local/bin, installed from a wheel
# that was copied to /tmp -- not from a checkout. So it needs its own install or it goes dark
# at the cutover.
#
# ⚠ Two things about this box that differ from nh3-dev:
# - the v2 daemons are SYSTEM units here (althing-herald, althing-receiver), not user units.
# - `uv` is not on lkraven's PATH; it lives at /home/infra-ops/.local/bin/uv. The original
# install used it under sudo with UV_TOOL_DIR=/opt/uv-tools, per the uv-receipt.toml.
#
# ⚠ There is a live agent session here (ldp-dev) holding a v2 light-monitor. Retiring the v2
# herald does not kill it, but it will never fire again -- that session has to re-arm on
# althing-listen after this. Its handle survives: bare `ldp-dev` is in the authoritative 73;
# only the machine-qualified `ldp-dev@nh3-extdev` was on the legacy exclusion list.
#
# Run: scripts/elway lkraven@10.100.50.42 --playbook playbooks/nh3-extdev-althing-v3.yaml
# Rerunnable: a second run shows the install and unit steps skipped.
vars:
wheel_src: /home/lkraven/development/althing/dist/althing_core-3.1.1-py3-none-any.whl
wheel_dest: /tmp/althing_core-3.1.1-py3-none-any.whl
uv: /home/infra-ops/.local/bin/uv
tool_dir: /opt/uv-tools
bin_dir: /usr/local/bin
# ⚠ Moved off nh3-dev 2026-08-28. A stale value here does not fail loudly at
# install time — it fails in the VERIFY, which then reads as a broken deploy.
post_office: http://10.100.50.40:8390
# ⚠ A release can change more than one file — 3.1.0 changed two. One pair is not
# enough, and a check that asserts only half a release is a check that half-passes
# silently. Space-separated `file:marker` pairs; bump BOTH per release.
# 3.0.1 zellij.py:_PANE_ID session_source.py:_live_pid (pane routes)
# 3.0.3 post_office_herald.py:PANE_SETTLE_S (write/submit race)
# 3.1.0 post_office_herald.py:POST_OFFICE_HINT dev_launch.py:resolve_post_office
# 3.1.1 postbox.py:warn_if_handle_looks_wrong dev_launch.py:handles_launched_at
#
# ⚠ Match on PRESENCE (grep -q), never on a count. 3.1.1 shipped with a stated
# expectation of "grep -c handles_launched_at dev_launch.py # 2+"; the real count
# there is 1 (the definition) with the other two occurrences in postbox.py. A count
# assertion would have reported FAILED on a byte-perfect install.
markers: "postbox.py:warn_if_handle_looks_wrong dev_launch.py:handles_launched_at"
steps:
- name: Stage the v3.0.0 wheel
upload:
src: /home/lkraven/development/althing/dist/althing_core-3.1.1-py3-none-any.whl
dest: "{{ wheel_dest }}"
mode: "0644"
- name: Retire the v2 system daemons BEFORE swapping the package
# Order matters: these run out of /opt/uv-tools/althing-core/bin/python, which the
# install is about to replace. Stopping first means they never see a half-swapped tree.
# v3 has no counterpart to either -- the post office replaced the herald and deleted the
# reason for the receiver, since there is no longer a mailbox per machine to deliver between.
shell: sudo systemctl disable --now althing-herald.service althing-receiver.service
when: "systemctl is-active --quiet althing-herald.service || systemctl is-active --quiet althing-receiver.service"
- name: Install the staged althing-core wheel over the system wheel install
# NOT gated on `postbox` existing — that guard was right for the cutover and
# wrong for every release after it: postbox exists now, so a version bump would
# silently skip. `--force` makes the reinstall idempotent on its own.
shell: sudo env UV_TOOL_DIR={{ tool_dir }} UV_TOOL_BIN_DIR={{ bin_dir }} {{ uv }} tool install --force {{ wheel_dest }}
- name: Install the post-office herald as a system unit
# A system unit rather than a user unit because that is how v2 was supervised here and
# because this box has no lingering user session to hang a --user unit from.
shell: |
printf '%s\n' \
'[Unit]' \
'Description=Althing post-office herald — per-machine relay (v3)' \
'Documentation=https://gitea.phasefinal.com/vh/althing' \
'After=network-online.target' \
'Wants=network-online.target' \
'' \
'[Service]' \
'Type=simple' \
'User=lkraven' \
'Environment=ALTHING_POST_OFFICE={{ post_office }}' \
'ExecStart={{ bin_dir }}/althing-po-herald' \
'Restart=always' \
'RestartSec=5' \
'' \
'# Dials out, opens no port, holds no state. Refuses to start if another herald' \
'# already holds this node — two would double every poke and both write liveness.' \
'# Replaces althing-herald.service + althing-receiver.service, retired 2026-08-28.' \
'' \
'[Install]' \
'WantedBy=multi-user.target' \
| sudo tee /etc/systemd/system/althing-po-herald.service >/dev/null
sudo systemctl daemon-reload
when: "! test -f /etc/systemd/system/althing-po-herald.service"
- name: Enable and (re)start the herald so it picks up the new code
shell: sudo systemctl enable --now althing-po-herald.service && sudo systemctl restart althing-po-herald.service
verify:
- name: postbox is installed and is v3
shell: "{{ bin_dir }}/postbox --help | grep -q 'send,reply,read,peek,thread,search,status,handles,register,sign-off'"
changed_when: "false"
- name: the v2 entry points are GONE, not merely shadowed
# Assert absence of the binaries themselves. A `which` that still resolves would mean the
# old wheel's entry points survived the --force and agents could keep calling a dead CLI.
shell: "! test -e {{ bin_dir }}/althing-cli && ! test -e {{ bin_dir }}/althing-receiver && ! test -e {{ bin_dir }}/althing-herald"
changed_when: "false"
- name: v2 daemons are stopped and disabled
shell: "! systemctl is-active --quiet althing-herald.service && ! systemctl is-active --quiet althing-receiver.service"
changed_when: "false"
- name: the po-herald is running
shell: systemctl is-active --quiet althing-po-herald.service
changed_when: "false"
- name: this box can reach the post office and the roster is populated
# ⚠ --handle is required. postbox resolves its identity from ALTHING_HANDLE, which
# dev-launch sets per pane and which a playbook shell does not have -- without it this
# check fails on identity, not on reachability, and reads as a deployment fault.
shell: ALTHING_POST_OFFICE={{ post_office }} {{ bin_dir }}/postbox --handle operator handles | wc -l | awk '{ if ($1 >= 70) exit 0; else exit 1 }'
changed_when: "false"
- name: This release's markers are ALL present BY CONTENT, not by version string
# forseti's own checks. A dist-info directory records what was INSTALLED, not
# what the files CONTAIN — verify the code, not the label. Bump `marker` and
# `marker_file` with each release rather than trusting the version bumped.
# 3.0.1 _PANE_ID in zellij.py (pane routes)
# 3.0.3 PANE_SETTLE_S in post_office_herald.py (the write/submit race)
shell: |
SP={{ tool_dir }}/althing-core/lib/python3.13/site-packages/althing
rc=0
for pair in {{ markers }}; do
f="${pair%%:*}"; m="${pair##*:}"
if grep -q "$m" "$SP/$f"; then echo " ok $f : $m"
else echo " MISS $f : $m"; rc=1; fi
done
exit $rc
changed_when: "false"
scripts/claude-statusline-command.sh
Executable
+145
View File
@@ -0,0 +1,145 @@
#!/usr/bin/env bash
# Claude Code statusline. Layout:
# [📬N] [🔔/🔕/📵] | <proj> ⎇<branch> *<dirty> ↑<unpushed> | <model> | ctx:<pct> <toks> | $<session-cost> | 5h:% 7d:%
#
# 🔔 reachable — the post office will push to this session
# 🔕 pull-only — nothing will poke it; mail waits until it looks
# 📵 the post office could not be asked — an OUTAGE, not an empty inbox
# ctx% and rate-limit %s are threshold-colored: green <60, yellow 60-90, red >90.
# All segments degrade gracefully (missing tool / non-git dir / no handle => segment omitted).
input=$(cat)
# --- threshold color: $1=numeric pct, $2=display text -> colored text ---
color_pct() {
local p="$1" txt="$2" c
if awk "BEGIN{exit !($p < 60)}"; then c=$'\033[32m' # green <60
elif awk "BEGIN{exit !($p > 90)}"; then c=$'\033[31m' # red >90
else c=$'\033[33m' # yellow 60-90
fi
printf '%s%s\033[0m' "$c" "$txt"
}
# --- reset countdown: $1=unix ts -> "1d3h"/"3h20m"/"45m" (2-unit; "now"/empty edge) ---
reset_in() {
local ts="$1" now delta d h m
[ -z "$ts" ] && return
now=$(date +%s)
delta=$(( ts - now ))
[ "$delta" -le 0 ] && { printf 'now'; return; }
if [ "$delta" -ge 86400 ]; then
d=$(( delta / 86400 )); h=$(( (delta % 86400) / 3600 ))
if [ "$h" -gt 0 ]; then printf '%dd%dh' "$d" "$h"; else printf '%dd' "$d"; fi
elif [ "$delta" -ge 3600 ]; then
h=$(( delta / 3600 )); m=$(( (delta % 3600) / 60 ))
if [ "$m" -gt 0 ]; then printf '%dh%dm' "$h" "$m"; else printf '%dh' "$h"; fi
else
printf '%dm' $(( delta / 60 ))
fi
}
# --- one jq pass for every payload field ---
# \x1f (unit separator) delimiter, NOT tab: tab is IFS-whitespace so `read` would
# collapse consecutive tabs and shift every field after an empty one (e.g. a
# session with no rate_limits). \x1f is non-whitespace -> empty fields preserved.
IFS=$'\x1f' read -r model used_pct input_tok five_pct week_pct cwd fast cost_usd model_id five_reset week_reset < <(
printf '%s' "$input" | jq -r '[
(.model.display_name // "unknown"),
(.context_window.used_percentage // ""),
(.context_window.total_input_tokens // 0),
(.rate_limits.five_hour.used_percentage // ""),
(.rate_limits.seven_day.used_percentage // ""),
(.cwd // .workspace.current_dir // ""),
(.fast_mode // false),
(.cost.total_cost_usd // 0),
(.model.id // ""),
(.rate_limits.five_hour.resets_at // ""),
(.rate_limits.seven_day.resets_at // "")
] | map(tostring) | join("")'
)
[ -z "$model" ] && model="unknown"
# --- model (compact) + fast-mode flag ---
model="${model%% (*}" # "Opus 4.8 (1M context)" -> "Opus 4.8"
[ "$fast" = "true" ] && model="⚡$model"
# --- context % (colored) + absolute input tokens ---
if [ -n "$used_pct" ]; then
ctx_seg=$(color_pct "$used_pct" "ctx:$(printf '%.0f%%' "$used_pct")")
else
ctx_seg="ctx:--"
fi
if [ "${input_tok:-0}" -ge 1000 ] 2>/dev/null; then
toks=$(awk "BEGIN{printf \"%.0fk\", ${input_tok}/1000}")
else
toks="${input_tok:-0}"
fi
# --- per-session cost (Claude Code's own cache/model-aware accounting) ---
# adaptive precision: whole dollars once it's real money, cents when small.
cost=$(awk "BEGIN{c=${cost_usd:-0}; if(c>=100) printf \"%.0f\",c; else if(c>=10) printf \"%.1f\",c; else printf \"%.2f\",c}")
# --- rate limits (each % colored on the same thresholds, + reset countdown) ---
_rl() { # $1=pct $2=label $3=reset_ts -> "<label>:NN%·<reset>"
local seg r; seg=$(color_pct "$1" "$2:$(printf '%.0f' "$1")%")
r=$(reset_in "$3"); [ -n "$r" ] && seg="$seg·$r"
printf '%s' "$seg"
}
rate=""
[ -n "$five_pct" ] && rate=$(_rl "$five_pct" "5h" "$five_reset")
if [ -n "$week_pct" ]; then
wk=$(_rl "$week_pct" "7d" "$week_reset")
[ -n "$rate" ] && rate="$rate "
rate="${rate}${wk}"
fi
# --- project tag + git state (branch, dirty, unpushed) ---
proj=""; gitseg=""
if [ -n "$cwd" ]; then
proj=$(basename "$cwd")
if git -C "$cwd" rev-parse --git-dir >/dev/null 2>&1; then
br=$(git -C "$cwd" branch --show-current 2>/dev/null)
[ -z "$br" ] && br=$(git -C "$cwd" rev-parse --short HEAD 2>/dev/null)
dirty=$(git -C "$cwd" status --porcelain 2>/dev/null | grep -c .)
ahead=$(git -C "$cwd" rev-list --count '@{upstream}..HEAD' 2>/dev/null)
gitseg="⎇${br:-?}"
[ "${dirty:-0}" -gt 0 ] 2>/dev/null && gitseg="$gitseg *$dirty"
[ -n "$ahead" ] && [ "$ahead" -gt 0 ] 2>/dev/null && gitseg="$gitseg ↑$ahead"
fi
fi
# --- althing: unread count (📬 N) + reachability (🔔 push / 🔕 pull / 📵 outage) ---
# NOT implemented here any more. `althing-statusline` is the canonical segment,
# shipped as a console script by the althing package (3.4.0+).
#
# This block used to be one of THREE hand-maintained copies of the same twelve
# lines -- this file, althing's plugin/scripts/statusline.sh, and infra-ops'
# copy in eshpfi-management. An audit on 2026-09-02 found DIFFERENT defects in
# each and all three were fixed separately, by hand, on the same day. That is a
# drift surface with a countdown on it, and it is the same failure that left
# this very block dead for a month after the v3 cutover deleted the binary it
# gated on. A script, unlike a document, has somewhere to be installed: PATH.
#
# The payload goes in on STDIN -- the segment resolves the handle from the cwd
# it carries, and falls back to nothing rather than guessing.
#
# The outer `timeout` is deliberately LOOSER than the program's own 2s budget.
# If the outer one fired first we would get an empty segment, which reads as
# "not an althing directory" -- the outage conflation, reintroduced by the
# guard meant to prevent a hang.
althing=""; mon=""
if command -v althing-statusline >/dev/null 2>&1; then
althing=$(printf '%s' "$input" | timeout 5 althing-statusline 2>/dev/null)
fi
# --- assemble ---
lead="$althing"
[ -n "$mon" ] && lead="${lead:+$lead }$mon"
pg="$proj"
[ -n "$gitseg" ] && pg="${pg:+$pg }$gitseg"
parts="$lead"
[ -n "$pg" ] && parts="${parts:+$parts | }$pg"
parts="${parts:+$parts | }$model | $ctx_seg $toks | \$$cost"
[ -n "$rate" ] && parts="$parts | $rate"
printf '%s' "$parts"

Some files were not shown because too many files have changed in this diff Show More