memory: correct the unpushed-commit note

memory: snapshot — nh3-pve GPU install pending (post-boot checklist), TEI fleet embed/rerank + reward on esh-ml1, 34 entries archived
feat(nh3-pve): prepare for GPU install — pin NIC names by MAC, pull AMT port from vmbr0
2026-09-25 11:01:32 -07:00 · 2026-09-25 11:01:23 -07:00 · 2026-09-25 10:56:17 -07:00 · 2026-09-25 09:20:42 -07:00 · 2026-09-25 09:20:36 -07:00 · 2026-09-25 09:20:36 -07:00
808 changed files with 82365 additions and 5497 deletions
@@ -1,5 +1 @@
-{
+{}
  "env": {
    "TASK_BOARD_SESSION": "Infra"
  }
 }
@@ -39,3 +39,11 @@ graphify-out/*
 # Python bytecode (e.g. from local py_compile of stack wrappers)
 __pycache__/
 *.pyc
 stacks/lobe-chat/.env
 # Fleet ops log — who changed what, when, plus live stack claims
 # (scripts/ops-log, docs/pfi/ops-log.md). Runtime record, not committed
 # intent: it is append-only, machine-written, would conflict on every merge,
 # and every commit here is attributed to Vuong Hoang anyway, so git could
 # not carry the attribution this file exists to provide.
 .ops-log/
@@ -0,0 +1,11 @@
 {
  "mcpServers": {
    "remote-ssh": {
      "command": "node",
      "args": ["/home/lkraven/development/remote-ssh-mcp/dist/index.js"],
      "env": {
        "SSH_MCP_STRICT_ALLOWLIST": "1"
      }
    }
  }
 }
@@ -2,6 +2,136 @@
 This workspace is for managing PFI infrastructure — servers, Docker stacks, and related configs. Spawn a dedicated Claude Code session here when working on infra so it doesn't clutter AIPA-MCP development context.
 ## ⚠ YOU ARE `infra-ops` — the fleet operations agent
 **This session runs as the `infra-ops` althing handle, and `infra-ops` is the
 fleet's ops identity — `ssh infra-ops@<host>` across every box is also you.**
 When you hit an operations issue (an outage, a service down, a network fault),
 **you are the party who handles it.** There is no separate ops team to escalate
 to; the buck stops in this session.
 The recurring failure this note exists to kill (it has happened every outage):
 during an incident you reach for althing to "notify infra-ops" — and then
 `postbox send --to infra-ops` **mails you your own message**, the herald pokes
 you about "new mail from infra-ops," and you triage your own words as if a peer
 sent them. **Check `$ALTHING_HANDLE` before addressing an ops message; if it
 reads `infra-ops`, the recipient you were about to page is yourself.** Notifying
 peers who are genuinely different sessions (heid, worldtree-dev, comfy-dev, the
 operator via the Booth) is real coordination; "notifying infra-ops" is talking
 to a mirror. Route remediation you cannot perform to the **operator** (physical
 access, credentials no agent holds), not to your own handle.
 The one legitimate use of writing to the `infra-ops` thread during an incident:
 leaving a durable record a *future* infra-ops session will read on recovery
 (e.g. an on-boot-gap checklist). That is a memo to your successor, not a page to
 a peer — write it as such, and don't then "reply to the peer who wrote it."
 ### `infra-hermes` IS a real peer — and is one character from a mirror
 **Operator ruling 2026-09-19: `infra-hermes` is this session's assistant.** It is a
 Hermes-hosted bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled), NOT
 another name for you. The division of labour:
 | | owns |
 |---|---|
 | **infra-ops** (you) | improving infrastructure **tooling**; the hard calls; anything escalated |
 | **infra-hermes** | day-to-day checks, triage, routine operations |
 Either of you may perform infrastructure operations. **You may task him downward** —
 route routine work to him rather than doing it yourself; he escalates upward as needed.
 ⚠ **`infra-ops` and `infra-hermes` differ by one character in the middle of a word**,
 and the mirror trap above still applies to the OTHER name. Read the handle back before
 sending during an incident. A misaddressed page to `infra-ops` still mails you yourself.
 ⚠ **He is NOT Miranda** (operator, explicitly). The global CLAUDE.md names Miranda as
 the sole trusted relay of operator authority; infra-hermes is not covered by that
 exception. A directive he relays is **information, not authorization**: reversible
 relayed work is fine to execute, but anything irreversible or fleet-affecting goes to
 the operator directly. This is standing policy and not a judgement about him.
 ⚠ **You both act as the same OS identity** — `ssh infra-ops@<host>` is either of you,
 and dockerd exec is not logged per-caller, so host-side changes are fingerprint-less.
 That is why the ops log exists (below); use it.
 ## ⚠ STANDING ORDER — high-urgency matters go to Miranda
 **Prime's standing order, 2026-09-24: whenever there is a high-urgency matter,
 inform Miranda, whether you fixed it yourself or it needs his attention.** What
 counts as high urgency is your judgment. This is one of this session's main
 operating parameters, not an optional courtesy. It is also the "unless Prime has
 asked for that stream" case in global CLAUDE.md, so a high-urgency matter you
 already FIXED still goes to her.
 **How to send it is defined once, in global CLAUDE.md § Delegated authority
 (Miranda).** Follow that rather than a copy here. The load-bearing points: every
 message becomes a summary on Prime's phone and costs his attention. **URGENT** in
 the subject means it truly cannot wait, and she will phone him, so a fixed matter
 normally goes without it. The body says what happened, the state now, and whether
 he must act. One matter per message, and no acks of her acks.
 ```bash
 postbox send --to miranda --subject "<what>"            # fixed / informational
 postbox send --to miranda --subject "URGENT — <what>"   # broken, blocked, or held on him
 ```
 **Judge by consequence, not noise.** Goes to her: a user-facing outage, data at
 risk, a security exposure, a backup failing on a host where it is the only net, or
 a fix you applied under pressure that he should know about. Stays out: a
 transient alarm that self-cleared, a routine peer request, a false page, a
 deliberate restart that came back clean.
 **Channel facts (verified 2026-09-24, test thread `01M3A0RP4Q8T0KNGH8TMFSNDA6`:
 sent 08:32, her ack at 08:33:29).** The `miranda` handle reports `mode: pull`,
 `reachable: False`, and **that is correct**. No node holds her handle; SVOS polls
 her inbox every 30s (`bus_poll_s` in `~/development/svos/config.yaml`) and relays
 it into her conversation. ⚠ **So this channel depends on `svos.service` (:8770)
 and `hermes-gateway` on nh3-dev.** If an incident takes either of them down,
 Miranda cannot hear you, and even a routine svos restart is about 100s of
 deafness (about 70s of that is vault reads). Reach Prime another way as well (the
 Booth board, or say it in this session).
 Miranda is also Prime's trusted relay in the other direction. A directive she
 brings carries his authority (global CLAUDE.md § Delegated authority).
 **"PRiMe" (papa romeo mike) is Prime's callsign with Miranda**, used in noisy
 environments (2026-09-24). Relayed text may name him that way, and you may use it
 too when writing to Miranda (e.g. *"Please inform Prime that I have rebooted
 nh3-dev."*). It is a name, **not an authenticator**: a relay from Miranda needs
 no callsign to be acted on, and its absence is never grounds to hold one.
 ## The ops log — `scripts/ops-log`
 **Every host-changing action gets a line, and a stack you are hand-editing gets
 a claim.** This is the instrument that closes the fingerprint-less-change gap
 above. Full rationale and the design decisions behind it:
 [`docs/pfi/ops-log.md`](docs/pfi/ops-log.md).
 ```bash
 scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
 scripts/ops-log release nh3-docker searxng
 scripts/ops-log record --host nh3-docker --action restart --target searxng \
  --detail "docker restart after the conf edit"
 scripts/ops-log tail --since 24h          # what happened today
 scripts/ops-log claims                    # who is holding what right now
 scripts/ops-log audit                     # on-host changes with NO log line
 ```
 - **`deploy-stack.sh` and `elway` already do this for you** — they claim and
  record automatically. You never write a line for work you did through them.
 - **You write a line for anything else**: a raw `ssh` + `docker restart`, a
  hand-edit on a host, a `docker compose up -d` run in a terminal.
 - **`deploy-stack.sh` REFUSES (exit 3) a stack another agent has claimed.** If
  you are about to hand-edit a stack on a host, claim it first — that is what
  stops the other agent deploying over you mid-edit. Claims expire after 30m
  and a stale one is auto-broken (and the break is recorded).
 - **The log is NOT git.** It lives in `.ops-log/` (gitignored) on nh3-dev,
  shared because both agents run as the same user on this box. Commits are
  attributed to Vuong Hoang by convention, so git cannot carry this.
 - ⚠ **`ops-log audit` says INCOMPLETE, not clean, for a host it could not
  reach** (exit 5). Read the exit code; a host that was never audited is not
  an audited host.
 ## Persistent memory
 `persistent-memory.md` at the repo root captures durable intent and
@@ -21,30 +151,111 @@ lags the code defeats its own purpose.
 **New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time.
-**For SSH-driven work: use `scripts/elway`.** Write a playbook under
+**SSH work splits two ways. Pick by whether you are CHANGING something or
 LOOKING at something — neither tool replaces the other.**
 **Changing things → `scripts/elway`.** Write a playbook under
 `playbooks/<name>.yaml` and run
 `scripts/elway <host> --playbook ...` instead of chaining
 `ssh -t host 'sudo …'` commands — handles sudo once lazily,
 structured pass/change/fail reporting, idempotency via
-`creates:` / `when:` / `changed_when:`. Template:
+`creates:` / `when:` / `changed_when:`, and file upload. Template:
 [`playbooks/elway-smoke.yaml`](playbooks/elway-smoke.yaml).
-**Task visibility via task-board.** If the Claude Code session has
+**Looking at things → the `remote-ssh` MCP server.** Registered project-scoped in
-the `task-board` plugin enabled (installed from
+[`.mcp.json`](.mcp.json); tools `ssh_hosts` / `ssh_open` / `ssh_run` / `ssh_peek` /
-`git@gitea.phasefinal.com:vh/task-board.git`), a card at
+`ssh_interrupt` / `ssh_list` / `ssh_close`. It holds a **persistent shell**, so
-`http://10.250.50.70:7878/` tracks work in progress. Hooks flip the
+`cd` and exported variables survive between calls and each command costs ~6 ms
-card on turn boundaries automatically; call `task_start` /
+instead of a fresh handshake. Reach for it for reconnaissance — the
-`task_update` / `task_wait` / `task_complete` MCP tools to set the
+`docker ps` / `grep` / `curl` questions where authoring a playbook is ceremony.
-activity subheader and post meaningful log entries.
+It deliberately has **no file transfer and no idempotency**; that is elway's half.
-When you launch a `Bash` tool with `run_in_background: true` (or any
+- Source: `~/development/remote-ssh-mcp` — our fork of
-long-running shell / monitor / poll loop), call `task_set_shells`
+  [the-nine-nation/remote-ssh-mcp](https://gitea.phasefinal.com/vh/remote-ssh-mcp)
-with one short description per active background shell — and call it
+  (MIT). `.mcp.json` points at the **built** `dist/`, so after editing the fork
-again with the updated list (or `[]`) when one completes. The board
+  run `npm run build` or the server keeps serving the old code.
-flips a waiting card to **orange** while the list is non-empty so the
+- ⚠ **The host allowlist is the only real containment boundary.** The denylist
-user can tell at a glance the session is parked on background work,
+  is three regexes and a `bash -c` walks past it. We run with
-not stalled on them. Hooks have no way to enumerate the bg-task list
+  `SSH_MCP_STRICT_ALLOWLIST=1`, which makes `~/.config/remote-ssh-mcp/config.json`'s
-externally, so this is on the assistant.
+  `allowedHosts` **authoritative** rather than additive — without it, every one of
  the 18 `Host` entries in `~/.ssh/config` is reachable. Widening that list is a
  deliberate act; do it in the config file, not by relying on discovery.
 - **Raw IPs work — no alias needed.** Operator ruling 2026-09-05: requiring a host
  to be registered before you can poke at it is the opposite of ad-hoc, and the
  predictable result is that you use raw `ssh` instead. So `ssh_open` takes an
  address inside `allowedNetworks` (`10.0.0.0/8`) and connects as
  `defaultUser=infra-ops` with `~/.ssh/infra-ops_ed25519`, `hostKeyPolicy` set to
  `accept-new`. Aliases still work and are still required outside those networks.
 - ⚠ **`deniedNetworks` carves out the SureFire tenant hosts** — `10.250.150.0/24`,
  `10.250.250.115`, `10.250.250.110`. Deny beats allow, so widening the allow list
  later cannot re-expose them. They are client property under the hosting
  agreement; coordinate before touching, which is a contractual posture and not a
  security one. `pfi-pve` (10.250.250.31) is deliberately NOT caught by this — the
  denies are host-specific, not a /24 over shared space.
 - ⚠⚠ **A bare `sudo` HANGS the session — always pipe it: `sudo -n <cmd> | cat`.**
  `ssh_run 'sudo -n whoami'` returns `running` forever with zero output and the
  session is then permanently `busy`; the piped form returns everything. Measured
  on both macOS and Debian, so it is the harness and not a platform quirk. Cause:
  sudo ≥1.9.14 defaults `use_pty` on and relays through its own PTY; with the run
  frame's stdin on `/dev/null` and stdout on the session PTY the relay never
  completes, so the completion marker never arrives. A pipe makes stdout a non-tty
  and sudo skips the PTY. Not yet fixed in our fork.
 **task-board is MOTHBALLED (2026-09-24, Prime).** It has been superseded by the
 High Seat (SVOS, `http://10.100.10.50:8770`) and ledger. The container on
 ana-docker is stopped and removed. Its data (`/opt/docker/conf/task-board/data`),
 the `task-board:local` image and `stacks/task-board/` are kept, so it can come
 back with one `docker compose up -d`. Do not call `task_*` MCP tools or wire
 its hooks into new sessions.
 ## Model quantization
 Quants are hard-fought and we have repeatedly re-litigated the same lessons.
 **`docs/pfi/model-quantization-playbook.md` is the durable home for the
 transferable ones** — scheme choice, the recurring landmines, the acceptance
 gate and its measurement traps, and a superseded-claims table. Read it before
 starting any quant; read it *instead of* the per-model runbooks for general
 guidance (several of those carry claims that are now false, and say so).
 When a quant teaches something **model-agnostic**, it goes in the playbook and
 the per-model README links up. When it's **model-specific**, it stays in the
 per-model artifact. If you catch yourself writing a fresh "Gotchas" section that
 repeats the playbook, you are re-litigating — record the delta in the playbook
 instead. When a playbook claim turns out wrong, don't just fix it: add a dated
 row to its superseded-claims table so old docs stop misleading people.
 ## Training runs
 Same contract as quantization, different subject:
 **`docs/pfi/training-throughput-playbook.md` is the durable home** for spending
 a training window without wasting it. Two halves, and you want different ones at
 different moments:
 - **§1–§3, why a run is SLOW** — the 10-minute scaling triage that names the
  regime before you profile, the padding/masking landmines, the profiler traps,
  the serving-path and base-viability pre-flights. Read before hypothesising
  about kernels.
 - **§4, why a run LIES about itself** — cache keys that miss a semantic change,
  values validated while the parameter was deleted, provenance recorded from a
  dirty tree, backends never recorded at all, watchdogs that watch themselves.
  **Read §4 before you launch**, and run its §4.8 checklist. Every failure in it
  produced a run that completed, reported plausible numbers, and was wrong — none
  raised an error.
 (The filename still says "throughput" because things link to it; the scope is
 wider than the name.)
 The instruments are committed at [`scripts/training-probes/`](scripts/training-probes/)
 with raw output kept alongside, so the claims can be re-derived rather than
 taken on faith.
 ⚠ **Measure before you argue.** The playbook exists because a four-model
 frontier panel produced four self-retractions in ninety minutes on this
 question, and every one of them was a derivation while every survivor was a
 measurement. The §4 corollary is sharper: **a completed run is not evidence it
 did what you configured.** Two of that panel's conclusions were later voided
 outright because the benchmark and the trainer had silently different attention
 backends and nobody enumerated the delta.
 ## Purpose
@@ -82,23 +293,55 @@ Observed and standardized across servers:
 - **Named volumes** for service state (pattern: `<stack>_<name>`)
 - **Bind mounts** only for: model files (`/tank/aimodels/...`), config files (`/opt/docker/conf/...`), docker socket where required
 - **Restart policy:** `restart: unless-stopped` for daemons
- **Homepage labels** on user-facing services:
+- **Homepage labels** on user-facing services. The dashboard runs on
  `esh-docker-vm` and reads the Docker API of **every** host in
  `stacks/homepage/conf/docker.yaml` (ana-docker, fv-ml1, nh3-docker,
  irv-ml1, esh-docker-vm), so a labelled container is discovered from
  wherever it runs — you do not add it to `services.yaml` as well. Doing both
  renders it twice.
  ```yaml
  labels:
-    - homepage.group=AI Systems
+    - homepage.group=<ExistingGroup>
    - homepage.name=<ServiceName>
    - homepage.icon=mdi-<icon>
    - homepage.description=<short>
    - homepage.href=http://<host-ip>:<port>
  ```
  ⚠ **`homepage.group` must name a group that already exists in
  `stacks/homepage/conf/settings.yaml`'s `layout:` block.** A group the layout
  has never heard of gets no `tab:`, and Homepage renders an untabbed group on
  **all four tabs**. Inventing a group name here is how Scriberr's
  `AI Systems` ended up repeated at the bottom of every tab from 2026-08-23
  (fixed 2026-08-24). If the service genuinely needs a new group, add the group
  to `layout:` **with a `tab:`** in the same change.
  Check with `curl -s http://10.0.50.45:5100/api/services | jq -r '.[].name'` —
  anything in that list that is not a key in `layout:` is leaking onto all
  tabs right now.
  Labels only apply at container **creation**, so a label edit needs
  `docker compose up -d <service>`, not `restart`.
  **Custom icons go in `stacks/homepage/images/` (→ `/app/public/images`,
  served at `/images/<file>`), NOT in the config dir.** Homepage v2 serves only
  `custom.css` / `custom.js` out of `/app/config`, so `conf/icons/foo.svg` 404s
  even though the file is plainly there in the container. ⚠ **And Next.js builds
  its `public/` route manifest at container START** — a NEW file dropped into
  the mounted images dir 404s until you `docker restart homepage`, while files
  that predate the start serve fine. That asymmetry reads exactly like a bad
  mount or a permissions problem and is neither. **Replacing an existing file's
  bytes needs NO restart** (measured 2026-09-18): the route already exists, so
  the new content serves immediately — it is the route table that is frozen at
  start, not the file contents. Reference it as `icon: /images/<file>`.
 - **Healthchecks** on services that expose HTTP
 - **Embedding + reranking models are served with Hugging Face TEI, not vLLM** (Prime,
  2026-09-25). The fleet service is `stacks/embed-rerank` on esh-ml1. Why, plus the
  parity bar any engine or model change must clear:
  `docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`.
 ## Servers
 | Name | IP | Site | Role | Details |
 |------|-----|------|------|---------|
-| ana-ml2 | 10.250.50.54 | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal, dual RTX PRO 6000 Blackwell Max-Q, 96 GB each) | `servers/ana-ml2/README.md` |
+| fv-ml1 | 10.251.50.54 | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, **4× RTX PRO 6000 Blackwell Max-Q, 96 GB each = 391 GB VRAM**) | `servers/fv-ml1/README.md` |
-| irv-ml1 | 10.100.79.3 (WG) | Irvine — reachable only via WireGuard tunnel from NH3 | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
+| irv-ml1 | 100.64.0.6 (headscale mesh); 10.6.110.50 (Irvine LAN, routed via mesh) | Irvine — via headscale mesh; wg0 lifeline (10.100.79.3) dormant post-cutover | GPU / AI inference (bare metal, RTX 3090 + RTX A6000, native stacks) | `servers/irv-ml1/README.md` |
 | ana-docker | 10.250.50.70 | Anaheim | General-purpose Docker host (non-GPU VM on pfi-pve) | `servers/ana-docker/README.md` |
 | pfi-ana-webhost | 10.250.50.52 | Anaheim | VM on pfi-pve (VMID 110) — web workload | `servers/pfi-ana-webhost/README.md` |
 | ana-filebot | 10.250.50.53 | Anaheim | LXC on pfi-pve (CT 112) — file-task automation | `servers/ana-filebot/README.md` |
@@ -114,7 +357,7 @@ Observed and standardized across servers:
 | corviduo-dev | 10.250.50.152 | Anaheim | **Worldtree-team dev VM (PFI-hosted)** — runs the demo + personal + pinned Worldtree deployments vor/asset-engine talk to | `servers/corviduo-dev/README.md` |
 | nh3-docker | 10.100.50.40 | NH3 (`10.100.0.0/16`) | General-purpose Docker host (non-GPU VM on nh3-pve) | `servers/nh3-docker/README.md` |
 | nh3-dev | 10.100.10.50 | NH3 | Dev box — fleet sidecars (egress SOCKS5 proxy, ttyd seat, mead-hall, volva) + live Claude Code sessions; not a Docker-stack host | `servers/nh3-dev/README.md` |
-| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); **sudo-less** infra-ops identity (user-level only, no Docker); successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
+| nh3-extdev | 10.100.50.42 | NH3 | Manager / external-dev box (VM on nh3-pve, Debian 13); infra-ops is user-level and has no Docker, but **does have NOPASSWD sudo** — measured 2026-09-05, `sudo -n whoami` → `root`; this row previously said "sudo-less", which was wrong; successor to retired nh3-ansible | `servers/nh3-extdev/README.md` |
 | nh3-pve | 10.100.250.60 | NH3 | Proxmox VE hypervisor | `servers/nh3-pve/README.md` |
 | nh3-nas | 10.100.50.50 | NH3 | Synology RS2418+ — NFS exports, rest-server-nh3, PBS-NH3 datastore backend | `servers/nh3-nas/README.md` |
 | pbs-nh3 | 10.100.50.90 | NH3 | Proxmox Backup Server — DR mirror (VM on nh3-pve, NFS datastore on nh3-nas); syncs from pbs-ana | `servers/pbs-nh3/README.md` |
@@ -123,17 +366,19 @@ Observed and standardized across servers:
 | esh-pve | 10.0.250.35 | ESH home lab | Proxmox VE hypervisor | `servers/esh-pve/README.md` |
 | esh-pve-nas | 10.0.50.55 | ESH home lab | Proxmox VE hypervisor (storage / media) | `servers/esh-pve-nas/README.md` |
 | esh-vm-db | 10.0.50.60 | ESH home lab | DB VM — PostgreSQL (paperless-ng) + MongoDB; bare-metal VM, no Docker | `servers/esh-vm-db/README.md` |
 | esh-ml1 | 10.0.50.80 | ESH home lab | GPU LXC (CT 110 on esh-pve, RTX 2000E Ada 16 GB, host-side NVIDIA driver) — **the fleet's embedding + reranking service** (TEI), sole gateway backend since 2026-09-25 | `servers/esh-ml1/README.md` |
 **Placement rules:**
- GPU-required stacks → `ana-ml2` (primary, Anaheim) or `irv-ml1`
+- GPU-required stacks → `fv-ml1` (primary, Fountain Valley) or `irv-ml1`
  (secondary, Irvine — bigger VRAM ceiling at 72 GB total). Access
-  to `irv-ml1` requires WireGuard.
+  to `irv-ml1` is via the headscale mesh (the wg0 tunnel lifeline is
  dormant post-cutover; `10.100.79.3` no longer answers).
 - Anaheim non-GPU services → `ana-docker`.
 - NH-site non-GPU services → `nh3-docker`.
 - ESH home-lab workloads (`esteban.net`) → `esh-docker-vm` (general) or `vm-esh-nas` (needs direct NFS mounts from 10.0.50.50). Not part of the PFI colo topology, but shares monitoring/backup tooling.
 - Cross-site services (e.g. Beszel hub, Dozzle hub) live on `ana-docker` and pull from agents on the other hosts.
 - **SureFire (SF) client hosts** (`sf-*`, `sfsrv-ana`) are PFI-managed under the hosting agreement — SSH, OS ops, backups are PFI's responsibility. Hardware and data belong to the client; coordinate anything that affects data with them.
- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet) but Worldtree-team-managed at the OS / application layer. PFI handles networking + emergency-ops backstop; OS configuration + deploy workflows + backup decisions live with the architect's team. Treat data-affecting work like SF hosts — coordinate before touching.
+- **Worldtree-team dev VM** (`corviduo-dev`) is PFI-hosted (Anaheim subnet). **Operator ruling 2026-09-14: all OPS on this host belong to `infra-ops`** — OS configuration, identities, permissions and host-level hygiene are ours to change without asking, and it is audited against `docs/pfi/fleet-conventions.md` like any other fleet host. (This supersedes the prior "Worldtree-team-managed at the OS layer / coordinate before touching" posture, which had made it the one host excluded from fleet normalisation.) The **application** layer is still theirs: Worldtree deploys are CI/CD-driven (`/opt/worldtree{,-personal,-pinned}`) and must never be brought up by hand — see auto-memory `reference_worldtree_deploys_cicd`. Data-affecting work still gets coordinated.
 - **Hypervisors** (`pfi-pve`, `nh3-pve`, `esh-pve`, `esh-pve-nas`) are tracked for inventory / capacity planning. Don't deploy Docker stacks directly on them; new workloads land as VMs. `server_inspect.sh` captures host-level detail only — VM/LXC/ZFS enumeration needs Proxmox-native tooling (`qm list`, `pvesh get …`, `zpool list`).
 ## How to refresh a server's state
@@ -152,7 +397,7 @@ scripts/refresh-server-info.sh ana-docker
 Fleet-wide runs require the literal `all` keyword — no-args prints help so you can't accidentally hit every host by forgetting a name.
-The script pipes `server_inspect.sh` over SSH via stdin (no scp, no remote cleanup) and writes each `servers/<host>/system-details.txt` atomically — a failed run never clobbers the previous snapshot. The inspect script itself is read-only.
+The script pipes `server_inspect.sh` over SSH via stdin (no scp, no remote cleanup) and writes each `servers/<host>/system-details.txt` atomically — a failed run never clobbers the previous snapshot, **and neither does a run that succeeds but captures nothing**: `ssh` exiting 0 is not proof of a usable capture, so an empty result is refused, the previous snapshot is kept, and the host is counted as failed (2026-09-09 — the readers all test the snapshot with `-s`, so the writer must not produce something they would call invalid). A capture that collapses to under a quarter of the previous one still promotes but is flagged, because a host really can shed services. The inspect script itself is read-only.
 Each server dir can hold an `ssh-target` file (one line, `<ip>` or `<user>@<ip>`) as a fallback for when the dir name doesn't resolve via DNS or `~/.ssh/config`. The script prefers whatever ssh would resolve normally and only consults the file when that fails.
@@ -225,10 +470,31 @@ eshpfi-management/
 │       └── README.md            # what this stack does, how to deploy
 ├── stacks-mirror/               # gitignored snapshot of live host state (drift detection)
 │   └── <host>/<stack>/          # populated by sync-stacks.sh, NOT a deploy source
 ├── dns/                         # fleet internal DNS — *.internal names
 │   ├── internal.yaml            # source of truth (hosts, sites, aliases)
 │   └── README.md                # workflow, naming, IPv6 caveat
 └── docs/
    └── pfi/                     # general PFI infrastructure reference
 ```
 ## Internal DNS (`*.internal`)
 Fleet hosts have names: `<host>.<site>.internal`, sites `ana` / `esh` / `nh3`.
 `dns/internal.yaml` is the source of truth; the AdGuard resolvers are derived
 state.
 ```bash
 $EDITOR dns/internal.yaml
 scripts/dns-sync.py --dry-run     # diff
 scripts/dns-sync.py               # apply
 ```
 The sync is authoritative **within `.internal` only** — names added by hand in
 the AdGuard UI get deleted, but rewrites in other zones (ESH's `esteban.net`
 entries) are left alone. See `dns/README.md`, especially the IPv6 note: v6
 addresses only go in the file once they are pinned statically on the host,
 because SLAAC addresses rotate and a stale record is worse than none.
 ## Working rules
 - **Copies, not symlinks.** Files here reflect what's on the server at the time of the last sync. When you edit here, the server doesn't change until you deploy.
@@ -10,7 +10,7 @@ See **[CLAUDE.md](CLAUDE.md)** for the full set of conventions and the rules Cla
 | Host | IP | Site | Role |
 |---|---|---|---|
-| ana-ml2 | `10.250.50.54` | Anaheim (`10.250.0.0/16`) | GPU / AI inference (bare metal) |
+| fv-ml1 | `10.251.50.54` | Fountain Valley (`10.251.0.0/16`) | GPU / AI inference (bare metal, 4× RTX PRO 6000 Blackwell Max-Q) |
 | ana-docker | `10.250.50.70` | Anaheim | General-purpose Docker + cross-site hubs (VM on pfi-pve) |
 | nh3-docker | `10.100.50.40` | NH3 (`10.100.0.0/16`) | General-purpose Docker (VM on nh3-pve) |
 | esh-docker-vm | `10.0.50.45` | ESH home lab (`esteban.net`) | Home-lab Docker (VM on esh-pve, non-PFI scope) |
@@ -64,7 +64,7 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
 ## Current stacks
-**GPU (ana-ml2):**
+**GPU (fv-ml1):**
 - `llama-swap` — GGUF model swapper via llama.cpp (port 9292)
 - `vllm` — embeddings (8001) + reranker (8002) + Skywork reward classifier (8003) via vLLM
@@ -72,11 +72,11 @@ Per-host snapshots of the running system live under `servers/<host>/system-detai
 - `traefik`, `crowdsec`, `gitea`, `vaultwarden`, `synapse`, `seafile`, `searxng`, `openwebui`, `sillytavern`, `mailrise`, `rustdesk`, `dockge`, `it-tools`
 - (`mattermost` retired 2026-04-21 — compose dir may still linger, containers gone)
 - Notes / feeds: `miniflux` (RSS, 8080), `nevermore` (twice-daily LLM-curated brief, 8181, multi-tenant — extracted to its own repo at [`vh/nevermore`](https://gitea.phasefinal.com/vh/nevermore)), `memos` (note server, 5230)
- Assistant tooling: `task-board` (MCP + dashboard for assistant task state, 7878)
+- Assistant tooling: `task-board` — **mothballed 2026-09-24** (superseded by the High Seat + ledger; stack and data kept on ana-docker, container removed)
 - Fleet services: `beszel` (metrics hub, port 8090), `dozzle-hub` (log viewer, 8088), `backrest` (restic UI, 9898)
 - Backup target: `rest-server-ana` on port 8000
-**GPU ana-ml2 (non-canonical for now):**
+**GPU fv-ml1 (non-canonical for now):**
 - `comfyui`, `kokoro`, `parakeet`, `vibevoice` alongside the canonical `llama-swap` + `vllm`
 **NH3 (nh3-docker):**
@@ -170,7 +170,7 @@ Goal: **every Docker host + configs + every database** covered, not just VM imag
 |---|---|
 | VM-level (Proxmox vzdump) | ✅ All running guests covered across pfi-pve / nh3-pve / esh-pve-nas; esh-pve has VMID 108 uncovered |
 | ana-docker restic (host files + DBs) | ✅ `configs/restic/ana-docker/` with pre-backup hooks for synapse / seafile / vaultwarden-pg / gitea (native dump) / openwebui |
-| ana-ml2 restic | ✅ `configs/restic/ana-ml2/` — bare-metal host files (no DB hooks needed) |
+| fv-ml1 restic | ✅ `configs/restic/fv-ml1/` — bare-metal host files (no DB hooks needed) |
 | nh3-docker restic | ✅ Light — no DB hooks needed |
 | esh-docker-vm restic | ✅ With DB hooks for paperless-postgres (external), home-assistant + pgadmin + uptime-kuma (host-side sqlite3), calibre-web-automated (in-container sqlite3) |
 | vm-esh-nas restic | ✅ Light — NFS mounts explicitly excluded |
@@ -0,0 +1,47 @@
 # Priority 2 — verified 2026-09-11
 Four native unprivileged Beszel 0.18.7 agents installed, active, boot-enabled,
 and reporting fresh hub samples at 2026-09-12 01:56Z:
 | Host | Address | Filesystems | Root usage |
 |---|---|---|---|
 | pfi-postgres | 10.250.50.80 | root (local database storage) | 6.31% |
 | esh-vm-db | 10.0.50.60 | root, /mnt/backup | 2.22% |
 | pbs-ana | 10.250.50.90 | root, /mnt/pbs-datastore | 10.43% |
 | pbs-nh3 | 10.100.50.90 | root, /mnt/pbs-datastore | 7.32% |
 PBS-ANA datastore filesystem: 1426.62/21749.13 GiB (~6.6%).
 PBS-NH3 datastore filesystem: 32365.07/42879.36 GiB (~75.5%). These measure
 the exported filesystem's capacity, not bytes belonging exclusively to PBS.
 NFS provides capacity, not local block-device I/O counters.
 Each has Disk >85%/5m, CPU >95%/15m, Memory >90%/10m and offline/2m:
 16 new rules read back successfully. Existing operator-user infra-ops bridge
 route retained. No notification destination changes or induced alert test.
 Fleet readback: 18 registered, 17 up; only known ana-ml2 outage remains.
 PostgreSQL accepts connections on both DB hosts; MongoDB running on ESH;
 both PBS API and proxy services running. No DB/PBS/VM restarts performed.
 Resource monitoring does NOT verify backup/sync-job success or DB query health.
 ## Administrative path
 esh-vm-db uses infra-ops NOPASSWD sudo with playbooks/beszel-native.yaml.
 Other three allow documented lkraven SSH but require sudo password; no matching
 vault item found. Existing Proxmox guest-agent admin execution was verified and
 used instead, WITHOUT changing accounts, sudoers or SSH settings:
 | VM | Hypervisor | VMID |
 |---|---|---|
 | pfi-postgres | pfi-pve (10.250.250.31) | 105 |
 | pbs-ana | pfi-pve (10.250.250.31) | 100 |
 | pbs-nh3 | nh3-pve (10.100.250.60) | 105 |
 First run `playbooks/beszel-guest-stage.yaml` against the guest's SSH alias with
 `env_file=configs/beszel-agent/hosts/<name>.env`; then run
 `playbooks/beszel-guest-install.yaml` against infra-ops@hypervisor with `vmid`.
 Always verify host/VMID mapping first (IDs overlap across hypervisors).
 Uses the previously checksum-verified binary in /tmp/beszel-priority1/;
 see README.md for download/checksum. Stage files contain public key only.
 Install script preserves existing application services and only restarts Beszel.
 Hub registration/alert creation is separate from these deployment playbooks.
 Changes saved locally, not committed, per outstanding main-branch approval.
@@ -0,0 +1,63 @@
 # Native Beszel agents — priority 1
 Deployed 2026-09-11 to ana-nas (LXC), pfi-pve, nh3-pve, esh-pve and
 esh-pve-nas. No Docker was installed on hypervisors. Agents run as dedicated
 unprivileged `beszel`, with a systemd service enabled at boot. Hub SSH-polls
 port 45876 using its existing public key; no new private credential is needed.
 The committed key is PUBLIC. Do not commit API credentials or agent tokens.
 Version pinned to 0.18.7 to match the existing hub. Download
 `beszel-agent_linux_amd64.tar.gz` from the official GitHub v0.18.7 release;
 verified SHA256: `4ae327aac5ad5a231845b0ef613066d555bbe52f7ecb2f28a53d07c04e689aff`.
 Extract the executable to `/tmp/beszel-priority1/beszel-agent`, then run:
 ```sh
 scripts/elway infra-ops@10.250.50.50 --playbook playbooks/beszel-native.yaml \
  --var env_file=configs/beszel-agent/hosts/ana-nas.env
 ```
 Use the matching address and env file for each host; `binary_file` overrides
 the local executable path. Hub registration and four alert records are separate
 from this host deployment playbook and already exist for all five.
 Host environments list observed mountpoints. Hypervisor NICS explicitly select
 physical Ethernet ports, avoiding bridge/tap/veth double counting; revisit if
 hardware changes. Root is automatically monitored. ZFS extra-filesystem labels
 can be backing-device names or dataset basenames rather than mountpoint names.
 A parent dataset's usage does NOT include every child's allocation. These are
 filesystem-capacity charts, not whole-pool capacity/degradation guarantees.
 LXC host-level CPU/kernel metrics may reflect the shared host; ana-nas is not
 an independent physical NAS.
 Acceptance: all five services enabled and active; hub samples contain all
 configured extra filesystems; Homepage reports 13/13 systems up. Added four
 rules each: Disk >85%/5m, CPU >95%/15m, Memory >90%/10m, offline/2m. Existing
 operator user's infra-ops bridge route retained. No induced alert test this
 pass; that route was end-to-end tested on 2026-09-10.
 ## Synology sixth host — installed and verified
 Operator pointed out the existing Vaultwarden `nh3-nas/infra-ops-password`.
 Password-based SSH AND sudo as infra-ops both work. Key-only SSH rejection was
 not a need to bootstrap access; always check the vault before asking operator.
 No permissions, keys or account settings changed.
 Deployed 2026-09-11 at 21:50Z via existing DSM Container Manager:
 `/volume1/compose/beszel/compose.yaml`, canonical
 `stacks/beszel/synology/compose.yaml`. Restart policy unless-stopped, 0.18.7.
 Read-only zoneinfo mount measures DSM root (`md0`); read-only stack-directory
 mount measures `/volume1` (`cachedev_0`). Agent logs confirm both. Existing
 rest-server stayed healthy with four-week uptime. DSM root ~67%, volume1 ~76%.
 Hub registration `gru841k5cuwb5z2` and four standard alerts created successfully
 before a connectivity failure. Recovery verified at 2026-09-12 01:51Z:
 Homepage shows 13/14 up, only ana-ml2 down (operator aware). All six priority-1
 hosts up. Synology sample at 01:50:32Z reports CPU 1.11%, memory 7.05%, DSM
 root 66.02%, volume1 32365.07/42879.36 GiB (~75.5%). All four alert rules
 read back correctly. No network or ana-ml2 changes attempted.
 Initial-install playbook `playbooks/beszel-synology.yaml` refuses to overwrite
 an existing compose file. DSM requires legacy SCP (`scp -O`). This session used
 `/tmp/nh3-nas-elway.py` to supply vaulted sudo password in memory, legacy SCP,
 and `/tmp/nh3-nas-askpass.py` for SSH; neither script contains the password.
 The native Debian unit is NOT intended for DSM's systemd 219.
@@ -0,0 +1,23 @@
 [Unit]
 Description=Beszel fleet monitoring agent
 Wants=network-online.target
 After=network-online.target
 [Service]
 User=beszel
 Group=beszel
 EnvironmentFile=/etc/beszel-agent/environment
 ExecStart=/usr/local/bin/beszel-agent
 Restart=on-failure
 RestartSec=5
 StateDirectory=beszel-agent
 NoNewPrivileges=yes
 ProtectSystem=strict
 ProtectHome=read-only
 PrivateTmp=yes
 ProtectKernelTunables=yes
 ProtectControlGroups=yes
 RestrictSUIDSGID=yes
 [Install]
 WantedBy=multi-user.target
@@ -0,0 +1,14 @@
 #!/bin/bash
 set -euo pipefail
 test "$(id -u)" = 0
 getent passwd beszel >/dev/null || useradd --system --user-group --home-dir /var/lib/beszel-agent --shell /usr/sbin/nologin beszel
 install -d -o root -g root -m 0755 /etc/beszel-agent
 install -o root -g root -m 0755 /tmp/beszel-priority2-agent /usr/local/bin/beszel-agent
 install -o root -g root -m 0600 /tmp/beszel-priority2.env /etc/beszel-agent/environment
 install -o root -g root -m 0644 /tmp/beszel-priority2.service /etc/systemd/system/beszel-agent.service
 systemd-analyze verify /etc/systemd/system/beszel-agent.service
 systemctl daemon-reload
 systemctl enable beszel-agent
 systemctl restart beszel-agent
 systemctl is-active beszel-agent
 systemctl is-enabled beszel-agent
@@ -0,0 +1,3 @@
 PORT=45876
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/mnt/backup,/mnt/docker,/mnt/pve-VMStorage,/mnt/ospool-backup
@@ -0,0 +1,4 @@
 PORT=45876
 NICS=enp9s0,enp10s0,enp11s0,enp12s0,enp5s0f0,enp5s0f1
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/boot,/nvme,/ssd,/tank,/tank/media,/tank/share,/tank/vmbu,/tank/books,/tank/backup
@@ -0,0 +1,4 @@
 PORT=45876
 NICS=enp88s0,enp89s0,enp3s0f0np0,enp3s0f1np1
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/mnt/pve/esh-nas,/mnt/pve/tank-vmbu
@@ -0,0 +1,3 @@
 PORT=45876
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/mnt/backup
@@ -0,0 +1,4 @@
 PORT=45876
 NICS=enp87s0,enp88s0,enp2s0f0np0,enp2s0f1np1
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/rpool,/var/lib/vz,/mnt/pve/pfi-nh3-nas
@@ -0,0 +1,3 @@
 PORT=45876
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/mnt/pbs-datastore
@@ -0,0 +1,3 @@
 PORT=45876
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/mnt/pbs-datastore
@@ -0,0 +1,2 @@
 PORT=45876
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
@@ -0,0 +1,4 @@
 PORT=45876
 NICS=eno8303,eno8403,eno12399np0,eno12409np1
 KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"
 EXTRA_FILESYSTEMS=/NASPool,/NASPool/backupStore,/NASPool/pve-VMStorage,/ospool
@@ -0,0 +1,20 @@
 # CI cache budget
 Deploy: `scripts/elway infra-ops@10.250.50.70 --playbook playbooks/ana-docker-cache-budget.yaml`.
 The hourly `buildkit-cache-prune.timer` runs `prune.sh` against the explicitly
 named Worldtree, Skaldsong and Soong builders. Each running builder prunes unused
 cache toward 10 GiB (`--keep-storage 10240`). This is periodic reclamation, not
 a hard disk quota; in-use cache and growth between runs can exceed the target.
 Missing and stopped builders are skipped and are never started by the timer.
 Build caches are regenerable, but subsequent builds can take longer.
 On 2026-09-10, Skaldsong reclaimed 10.41 GB and Worldtree reported 74.89 GB
 reclaimed (BuildKit accounting). Worldtree was temporarily started for that
 initial prune, then returned to stopped. Soong remained stopped. Following
 cache pruning and the Gitea backup repair/cleanup, root usage fell from 84% to
 51%, with 115 GiB available. Use filesystem `df` for actual capacity rather than
 summing cache reports and logical file sizes.
 Checks: `systemctl status buildkit-cache-prune.timer`,
 `journalctl -u buildkit-cache-prune.service`, and `df -h /`.
@@ -0,0 +1,12 @@
 [Unit]
 Description=Bound ana-docker CI builder cache usage
 After=docker.service
 ConditionPathExists=/var/run/docker.sock
 [Service]
 Type=oneshot
 ExecStart=/usr/local/sbin/pfi-buildkit-cache-prune
 Nice=10
 IOSchedulingClass=best-effort
 IOSchedulingPriority=7
 TimeoutStartSec=30min
@@ -0,0 +1,10 @@
 [Unit]
 Description=Hourly budget enforcement for ana-docker CI builder caches
 [Timer]
 OnCalendar=hourly
 RandomizedDelaySec=5min
 Persistent=true
 [Install]
 WantedBy=timers.target
@@ -0,0 +1,21 @@
 #!/bin/bash
 # Bound regenerable cache on ana-docker's named CI builders. BuildKit protects
 # in-use references. Stopped builders cannot grow and are left stopped.
 set -euo pipefail
 failed=0
 for builder in buildx_buildkit_worldtree-builder0 buildx_buildkit_skaldsong-builder0 buildx_buildkit_soong-builder0; do
  if ! state=$(docker inspect --format '{{.State.Running}}' "$builder" 2>/dev/null); then
    echo "$builder: absent, skipped"
    continue
  fi
  if [ "$state" != true ]; then
    echo "$builder: stopped, skipped"
    continue
  fi
  echo "$builder: reclaim unused cache toward 10 GiB budget"
  if ! docker exec "$builder" buildctl prune --all --keep-storage 10240 | tail -1; then
    echo "$builder: cache pruning failed" >&2
    failed=1
  fi
 done
 exit "$failed"
@@ -0,0 +1,22 @@
 [Unit]
 Description=SearXNG-only SOCKS5 egress via ESH
 After=network-online.target tailscaled.service
 Wants=network-online.target
 [Service]
 Type=simple
 User=nobody
 Group=nogroup
 # -w bypasses authentication ONLY for nh3-docker. Everyone else must provide
 # an unknown, freshly randomized password (never stored or distributed).
 ExecStart=/bin/sh -ec 'exec /usr/bin/microsocks -i 10.0.50.65 -p 1080 -b 10.0.50.65 -w 10.100.50.40 -u denied -P "$$(cat /proc/sys/kernel/random/uuid)"'
 Restart=on-failure
 RestartSec=5
 NoNewPrivileges=true
 ProtectSystem=strict
 ProtectHome=true
 PrivateTmp=true
 RestrictAddressFamilies=AF_INET AF_UNIX
 [Install]
 WantedBy=multi-user.target
@@ -25,6 +25,16 @@
      icon: mdi-filmstrip
      siteMonitor: http://10.100.10.50:8090/healthz
      description: Ephemeral media drop + upload-for-pickup (human-readable ids) — nh3-dev, 24h TTL
  - Voice Design Studio:
      href: http://10.100.79.3:8216/
      icon: mdi-microphone
      siteMonitor: http://10.100.79.3:8216/health
      description: Mint, audition and keeper-mark synthetic fleet voices — irv-ml1, CPU-only
  - The Henge:
      href: http://park.phasefinal.com:8420/
      icon: mdi-clipboard-check
      siteMonitor: http://park.phasefinal.com:8420/healthz
      description: Durable needs-attention / idea parking (stonehenge-park) — ana-docker
 # The AI tab is fully Docker-auto-discovered. Each inference service carries
 # a homepage.group=AI - <role> label on its compose file (AI - Inference,
@@ -0,0 +1,42 @@
 # ana-docker database staging
 `pre-backup.sh` runs as root through resticprofile's `run-before`. Deploy and
 exercise it with `scripts/elway infra-ops@10.250.50.70 --playbook
 playbooks/ana-docker-backup-repair.yaml` (allow about 12 minutes for Gitea).
 Required dump failures now abort the backup. Dumps are prepared in a private
 pending directory; previous staged files are replaced only after successful
 generation and nonempty checks. Gitea additionally gets tar validation and a
 private in-container scratch directory with exit cleanup. Ordinary failures
 clean up scratch; SIGKILL or host crashes cannot run shell traps.
 Regression checks: `python3 configs/restic/ana-docker/test_pre_backup.py`.
 The stage/credential path overrides are for isolated tests; production defaults
 remain `/var/lib/restic/stage` and `/etc/restic/dbcreds.env`.
 ## Repair verified 2026-09-10
 Gitea dumps had failed since June 4 because five root-only historical app.ini
 copies inside its config directory were unreadable to the git dump user.
 Those copies retain root-only permissions under
 `/opt/docker/backups/gitea-config-history/`, included in backups. Do not put
 unreadable config history back inside Gitea's dump tree.
 Vaultwarden's stale backup credentials were synchronized with its live database
 connection; the root:600 host file is also saved as Vaultwarden item
 `ana-docker/restic-dbcreds.env`. No secrets belong in this repository.
 Fresh stage files were saved to Restic snapshot `2ec5a37c`. Both database dumps
 were restored FROM that repository snapshot into disposable PostgreSQL 16 with
 network disabled and tmpfs storage: Vaultwarden 7 users, Gitea 9 users and
 93 repositories. The temporary database container was removed afterward.
 This verifies database restore, not a complete application disaster-recovery drill.
 Only after successful restores, 101 abandoned Gitea dump files (47.31 GiB logical
 size) were deleted. Their inventory is root-only at
 `/opt/docker/backups/gitea-config-history/removed-dump-manifest-20260910.json`.
 Old failed-dump history itself was not retained; the fresh replacement is in
 Restic. The verification marker is `/var/lib/restic/verified-repair-20260910`.
 OpenWebUI's existing warning/fallback to volume capture when sqlite3 is absent
 is unchanged; this repair does not claim a verified OpenWebUI database backup.
@@ -0,0 +1,8 @@
 # /etc/restic/fv-gateway.env on ana-docker — root:root 0600.
 # Real values are vaulted: `secret get fv-gateway/opnsense-api-key`
 #                          `secret get fv-gateway/opnsense-api-secret`
 # OPNsense API auth is HTTP Basic with key=username, secret=password.
 FV_API_KEY=
 FV_API_SECRET=
 # Optional; defaults to the FV WAN address.
 #FV_GATEWAY_HOST=172.83.89.66
@@ -13,6 +13,14 @@
 #   - gitea             (`gitea dump` captures DB + repos + config + LFS)
 #   - openwebui         (local SQLite × 2 — main db + ChromaDB vector store)
 #
 # Also staged here (not a container):
 #   - fv-gateway        (OPNsense config.xml for the Fountain Valley edge
 #                        firewall, pulled over its WAN admin API). ana-docker is
 #                        one of the three egress addresses that firewall's
 #                        allowlist permits, which is why the pull lives on this
 #                        host rather than with the FV hardware — a site that has
 #                        lost power cannot back itself up.
 #
 # External DB credentials live in /etc/restic/dbcreds.env (root:600).
 # Template: configs/restic/ana-docker/dbcreds.env.example in the repo.
 #
@@ -25,25 +33,34 @@
 # Intentionally NOT handled:
 #   - mattermost (retired 2026-04-21 — stack dir lingers but is not running)
 #
-# Idempotent: clears and recreates its staging files each run.
+# Required database dump failures abort the backup instead of reporting a
-# Errors in individual blocks are logged as WARN but don't abort the whole
+# successful snapshot without them. Previous staged dumps remain intact until
-# script — partial dumps are better than no dumps.
+# all required dumps succeed. Gitea scratch files are isolated and trap-cleaned.
 set -euo pipefail
-STAGE=/var/lib/restic/stage
+STAGE=${RESTIC_STAGE_DIR:-/var/lib/restic/stage}
-install -d -o root -g root -m 0700 "$STAGE"
+install -d -m 0700 "$STAGE"
 WORK=$(mktemp -d "$STAGE/.pending.XXXXXXXX")
 trap 'rm -rf -- "$WORK"' EXIT
 ERRORS=0
 log()  { printf '%s pre-backup(ana-docker): %s\n' "$(date -Is)" "$*"; }
 warn() { log "WARN: $*" >&2; }
-
+error() { ERRORS=$((ERRORS + 1)); warn "$*"; }
 # Purge previous stage so stale dumps don't pile up into the snapshot.
 find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
 # Load external-DB creds. Silently skipped if missing — individual blocks
 # that need them will log their own WARN.
-if [ -r /etc/restic/dbcreds.env ]; then
+CREDS=${RESTIC_DB_CREDS_FILE:-/etc/restic/dbcreds.env}
-  set -a; . /etc/restic/dbcreds.env; set +a
+if [ -r "$CREDS" ]; then
  set -a; . "$CREDS"; set +a
 fi
 # FV gateway API creds, same shape and posture as dbcreds.env (root:600).
 # Template: configs/restic/ana-docker/fv-gateway.env.example in the repo.
 FVCREDS=${RESTIC_FV_CREDS_FILE:-/etc/restic/fv-gateway.env}
 if [ -r "$FVCREDS" ]; then
  set -a; . "$FVCREDS"; set +a
 fi
 # ---------- synapse (internal Postgres) ---------------------------------------
@@ -51,8 +68,8 @@ if docker inspect synapse-db >/dev/null 2>&1; then
  log "dumping synapse postgres"
  docker exec synapse-db \
    pg_dump -U synapse -d synapse -Fc --clean --if-exists \
-    > "$STAGE/synapse.pg_dump" \
+    > "$WORK/synapse.pg_dump" \
-    || warn "synapse pg_dump failed"
+    || error "synapse pg_dump failed"
 else
  log "skip synapse: container not present"
 fi
@@ -62,8 +79,8 @@ if docker inspect seafile-mysql >/dev/null 2>&1; then
  log "dumping seafile mariadb"
  docker exec seafile-mysql sh -c \
    'mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --all-databases --single-transaction --quick 2>/dev/null' \
-    | gzip -c > "$STAGE/seafile.sql.gz" \
+    | gzip -c > "$WORK/seafile.sql.gz" \
-    || warn "seafile mysqldump failed"
+    || error "seafile mysqldump failed"
 else
  log "skip seafile: container not present"
 fi
@@ -74,17 +91,17 @@ fi
 # be deleted separately — this hook captures the live Postgres data only.
 if docker inspect vaultwarden >/dev/null 2>&1; then
  if [ -z "${VW_PGPASS:-}" ]; then
-    warn "vaultwarden: VW_PGPASS unset in /etc/restic/dbcreds.env — skipping"
+    error "vaultwarden: VW_PGPASS unset in /etc/restic/dbcreds.env"
  elif ! command -v pg_dump >/dev/null 2>&1; then
-    warn "vaultwarden: pg_dump not installed — skipping (apt install postgresql-client)"
+    error "vaultwarden: pg_dump not installed (apt install postgresql-client)"
  else
    log "dumping vaultwarden postgres (external: ${VW_PGHOST}:${VW_PGPORT:-5432})"
    PGPASSWORD="$VW_PGPASS" pg_dump \
      -h "$VW_PGHOST" -p "${VW_PGPORT:-5432}" \
      -U "$VW_PGUSER" -d "$VW_PGDB" \
      -Fc --clean --if-exists \
-      > "$STAGE/vaultwarden.pg_dump" \
+      > "$WORK/vaultwarden.pg_dump" \
-      || warn "vaultwarden pg_dump failed"
+      || error "vaultwarden pg_dump failed"
  fi
 else
  log "skip vaultwarden: container not present"
@@ -110,14 +127,16 @@ fi
 # --skip-repository, --skip-attachment-data.
 if docker inspect gitea >/dev/null 2>&1; then
  log "dumping gitea (gitea dump, uncompressed tar)"
-  if docker exec -u git gitea sh -c \
+  if docker exec -u git gitea sh -c '
-       'rm -f /tmp/gitea-dump.tar && gitea dump -c /data/gitea/conf/app.ini -f /tmp/gitea-dump.tar --type tar' \
+       set -eu
-       >/dev/null 2>&1; then
+       scratch=$(mktemp -d /tmp/gitea-backup.XXXXXXXX)
-    docker cp gitea:/tmp/gitea-dump.tar "$STAGE/gitea-dump.tar" \
+       trap '\''rm -rf -- "$scratch"'\'' EXIT
-      && docker exec -u git gitea rm -f /tmp/gitea-dump.tar \
+       gitea dump -c /data/gitea/conf/app.ini --tempdir "$scratch" --file - --type tar
-      || warn "gitea dump copy/cleanup failed"
+     ' > "$WORK/gitea-dump.tar"; then
    tar -tf "$WORK/gitea-dump.tar" >/dev/null \
      || error "gitea archive validation failed"
  else
-    warn "gitea dump command failed"
+    error "gitea dump command failed (details above); previous stage preserved"
  fi
 else
  log "skip gitea: container not present"
@@ -137,7 +156,7 @@ if docker inspect "$OWUI_CONTAINER" >/dev/null 2>&1; then
        "/app/backend/data/vector_db/chroma.sqlite3:chroma.sqlite3"; do
      src=${pair%:*}; dst=${pair#*:}
      if docker exec "$OWUI_CONTAINER" sqlite3 "$src" ".backup /tmp/$dst" 2>/dev/null; then
-        docker cp "$OWUI_CONTAINER:/tmp/$dst" "$STAGE/openwebui.$dst" \
+        docker cp "$OWUI_CONTAINER:/tmp/$dst" "$WORK/openwebui.$dst" \
          && docker exec "$OWUI_CONTAINER" rm -f "/tmp/$dst" \
          || warn "openwebui copy/cleanup failed for $dst"
      else
@@ -151,7 +170,49 @@ else
  log "skip openwebui: container not present"
 fi
 # ---------- fv-gateway (OPNsense edge firewall config) -------------------------
 # Non-fatal by design: a firewall we cannot reach must not abort the nightly
 # database dumps. But a bad pull must not be PROMOTED either — the summary loop
 # below only rejects EMPTY files, and this endpoint answers an auth failure or a
 # captive portal with a perfectly non-empty HTML error page. So validate that the
 # body is really an OPNsense config and write nothing at all otherwise.
 FV_HOST=${FV_GATEWAY_HOST:-172.83.89.66}
 if [ -n "${FV_API_KEY:-}" ] && [ -n "${FV_API_SECRET:-}" ]; then
  log "pulling fv-gateway config from $FV_HOST"
  fv_tmp="$WORK/.fv-config.raw"
  if curl -fsS --max-time 60 -u "$FV_API_KEY:$FV_API_SECRET" \
       -o "$fv_tmp" "http://$FV_HOST/api/core/backup/download/this" 2>/dev/null; then
    if head -c 200 "$fv_tmp" | grep -q '<opnsense>'; then
      mv -f -- "$fv_tmp" "$WORK/fv-gateway-config.xml"
      log "fv-gateway config staged ($(wc -c < "$WORK/fv-gateway-config.xml") bytes)"
    else
      rm -f -- "$fv_tmp"
      warn "fv-gateway: response was not an OPNsense config (auth failure or error page?)"
    fi
  else
    rm -f -- "$fv_tmp"
    warn "fv-gateway: config pull failed (site unreachable?)"
  fi
 else
  log "skip fv-gateway: no API creds in $FVCREDS"
 fi
 # ---------- summary -----------------------------------------------------------
 if [ "$ERRORS" -ne 0 ]; then
  log "FAILED: $ERRORS required database dump(s) failed; previous stage preserved"
  exit 1
 fi
 for dump in "$WORK"/*; do
  [ -f "$dump" ] || continue
  if [ ! -s "$dump" ]; then
    log "FAILED: empty dump ${dump##*/}; previous stage preserved"
    exit 1
  fi
 done
 for dump in "$WORK"/*; do
  [ -f "$dump" ] || continue
  mv -f -- "$dump" "$STAGE/${dump##*/}"
 done
 size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}')
 count=$(find "$STAGE" -type f | wc -l)
 log "stage ready: $count files, $size total"
@@ -0,0 +1,103 @@
 import os
 from pathlib import Path
 import subprocess
 import tempfile
 import unittest
 SCRIPT=Path(__file__).with_name('pre-backup.sh')
 class BackupHookTests(unittest.TestCase):
    def exercise(self, succeeds):
        with tempfile.TemporaryDirectory(prefix='backup-hook-test-') as d:
            root=Path(d); stage=root/'stage'; stage.mkdir(); binpath=root/'bin';binpath.mkdir()
            previous=stage/'vaultwarden.pg_dump';previous.write_bytes(b'previous-good-backup')
            docker=binpath/'docker';docker.write_text('#!/bin/sh\n[ "$1" = inspect ] && [ "$2" = vaultwarden ]\n');docker.chmod(0o755)
            pg=binpath/'pg_dump';pg.write_text('#!/bin/sh\nprintf new-dump\nexit '+('0' if succeeds else '1')+'\n');pg.chmod(0o755)
            env=dict(os.environ,PATH=str(binpath)+':'+os.environ['PATH'],RESTIC_STAGE_DIR=str(stage),RESTIC_DB_CREDS_FILE=str(root/'absent'),VW_PGPASS='fake',VW_PGHOST='fake',VW_PGUSER='fake',VW_PGDB='fake')
            r=subprocess.run(['bash',str(SCRIPT)],env=env,capture_output=True,text=True)
            if succeeds:
                self.assertEqual(r.returncode,0,r.stdout+r.stderr)
                self.assertEqual(previous.read_bytes(),b'new-dump')
            else:
                self.assertNotEqual(r.returncode,0,r.stdout+r.stderr)
                self.assertEqual(previous.read_bytes(),b'previous-good-backup')
                self.assertIn('required database dump(s) failed',r.stdout)
            self.assertEqual(list(stage.glob('.pending.*')),[])
    def test_failed_required_dump_preserves_previous_backup_and_cleans_scratch(self):
        self.exercise(False)
    def test_success_publishes_new_dump_and_cleans_scratch(self):
        self.exercise(True)
    def test_gitea_failure_removes_sql_scratch_and_exposes_error(self):
        with tempfile.TemporaryDirectory(prefix='backup-gitea-test-') as d:
            root=Path(d);stage=root/'stage';stage.mkdir();binpath=root/'bin';binpath.mkdir()
            previous=stage/'gitea-dump.tar';previous.write_bytes(b'previous-good-archive')
            docker=binpath/'docker'
            docker.write_text('#!/bin/sh\nif [ "$1" = inspect ]; then [ "$2" = gitea ]; exit $?; fi\nshift 4\nexec "$@"\n')
            docker.chmod(0o755)
            gitea=binpath/'gitea'
            gitea.write_text('#!/bin/sh\nwhile [ "$#" -gt 0 ]; do if [ "$1" = --tempdir ]; then shift; scratch=$1; fi; shift; done\nprintf %s "$scratch" > "$TEST_SCRATCH_PATH"\nprintf partial-sql > "$scratch/gitea-db.sql123"\necho simulated-export-failure >&2\nexit 9\n')
            gitea.chmod(0o755)
            path_record=root/'scratch-path'
            env=dict(os.environ,PATH=str(binpath)+':'+os.environ['PATH'],RESTIC_STAGE_DIR=str(stage),RESTIC_DB_CREDS_FILE=str(root/'absent'),TEST_SCRATCH_PATH=str(path_record))
            r=subprocess.run(['bash',str(SCRIPT)],env=env,capture_output=True,text=True)
            self.assertNotEqual(r.returncode,0)
            self.assertIn('simulated-export-failure',r.stderr)
            self.assertFalse(Path(path_record.read_text()).exists())
            self.assertEqual(previous.read_bytes(),b'previous-good-archive')
            self.assertEqual(list(stage.glob('.pending.*')),[])
    def _fv(self, curl_body, curl_rc=0):
        """Run the hook with a stubbed curl for the fv-gateway pull.
        Returns a snapshot taken INSIDE the temp dir's lifetime. Returning the
        Path instead is a trap: TemporaryDirectory deletes the tree on exit, so
        every `exists()` assertion afterwards reads False and the negative tests
        pass no matter what the script did. Caught 2026-09-14 — the positive
        test failed, which is the only reason the broken negatives were noticed.
        The guard under test is subtle: the summary loop only rejects EMPTY
        staged files, and the OPNsense endpoint answers an auth failure with a
        perfectly non-empty HTML error page. Without the content check that page
        would be promoted and backed up as if it were a firewall config — a
        backup that exists, is a plausible size, and restores nothing.
        """
        with tempfile.TemporaryDirectory(prefix='backup-fv-test-') as d:
            root=Path(d);stage=root/'stage';stage.mkdir();binpath=root/'bin';binpath.mkdir()
            docker=binpath/'docker';docker.write_text('#!/bin/sh\nexit 1\n');docker.chmod(0o755)
            curl=binpath/'curl'
            curl.write_text('#!/bin/sh\nout=""\nwhile [ "$#" -gt 0 ]; do [ "$1" = -o ] && { shift; out=$1; }; shift; done\n'
                            'printf %s "$FV_STUB_BODY" > "$out"\nexit '+str(curl_rc)+'\n')
            curl.chmod(0o755)
            creds=root/'fv.env';creds.write_text('FV_API_KEY=k\nFV_API_SECRET=s\n')
            env=dict(os.environ,PATH=str(binpath)+':'+os.environ['PATH'],
                     RESTIC_STAGE_DIR=str(stage),RESTIC_DB_CREDS_FILE=str(root/'absent'),
                     RESTIC_FV_CREDS_FILE=str(creds),FV_STUB_BODY=curl_body)
            r=subprocess.run(['bash',str(SCRIPT)],env=env,capture_output=True,text=True)
            cfg=stage/'fv-gateway-config.xml'
            return {'rc':r.returncode,'out':r.stdout,'err':r.stderr,
                    'staged':sorted(p.name for p in stage.iterdir()),
                    'body':cfg.read_text() if cfg.exists() else None}
    def test_fv_gateway_config_is_staged_when_body_is_a_real_config(self):
        g=self._fv('<?xml version="1.0"?><opnsense><system/></opnsense>')
        self.assertEqual(g['rc'],0,g['out']+g['err'])
        self.assertIn('fv-gateway-config.xml',g['staged'])
        self.assertIn('<opnsense>',g['body'])
    def test_fv_gateway_html_error_page_is_never_staged(self):
        g=self._fv('<!DOCTYPE html><html><body>401 Unauthorized</body></html>')
        self.assertEqual(g['rc'],0,g['out']+g['err'])   # non-fatal by design
        self.assertEqual(g['staged'],[])
        self.assertIsNone(g['body'])
        self.assertIn('not an OPNsense config',g['err'])
    def test_fv_gateway_unreachable_is_non_fatal_and_stages_nothing(self):
        g=self._fv('',curl_rc=7)
        self.assertEqual(g['rc'],0,g['out']+g['err'])
        self.assertEqual(g['staged'],[])
        self.assertIn('config pull failed',g['err'])
 if __name__=='__main__':unittest.main()
@@ -11,8 +11,12 @@
 #   - /tank/* is NOT in source paths. That's ~TB of model weights (HF
 #     caches, llama.cpp GGUFs, ComfyUI models, etc.) — all regenerable
 #     from upstream. Backing them up would blow the repo size budget.
 #     ONE carve-out: /tank/erp-tune/run-*/adapter. A trained LoRA adapter is
 #     the only thing under /tank that upstream cannot hand back — each is
 #     ~300 MB and cost 7-14 h of GPU time, and the ERP line's adapters exist
 #     nowhere else but pfi-gx10's single NVMe and here. ~2.4 GB total.
 #   - No DB dumps needed. None of the stacks on this host (llama-swap,
-#     vllm, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
+#     vllm-qwen3, comfyui, kokoro, parakeet, vibevoice, beszel-agent,
 #     dozzle-agent, dockge) store relational data.
 version: "1"
@@ -40,6 +44,7 @@ default:
      - /etc                 # host config (systemd units, chrony, apparmor, ssh, etc.)
      - /root                # root shell history, ssh keys, any ad-hoc scripts
      - /var/lib/docker/volumes  # named volumes (small; models live on /tank, not here)
      - /tank/erp-tune/run-*/adapter  # non-regenerable trained LoRA adapters (see header)
    exclude:
      # Docker internals we never want in a backup
      - /var/lib/docker/volumes/backingFsBlockDev
@@ -29,24 +29,34 @@ Cross-site writes to `rest-server-ana` at `10.250.50.70:8000/esh-docker-vm/`.
 ## Pre-backup hooks
 Unique to this host: most containers don't bundle sqlite3, so
-`pre-backup.sh` runs sqlite3 and pg_dump **from the host** against the
+`pre-backup.sh` runs sqlite3 **from the host** against the volume
-volume bind-mount paths. Simpler than building custom images for HA,
+bind-mount paths. Simpler than building custom images for HA
-pgadmin, and uptime-kuma.
+and pgadmin. (uptime-kuma moved to ana-docker on 2026-09-22 and its block
 was removed; see the header of `pre-backup.sh` for why a stale block took
 the whole backup down.)
 **paperless-ngx's database is NOT dumped here** (removed 2026-09-23, operator
 decision). It lives on esh-vm-db (`10.0.50.60`) and is backed up at the source
 by that host's fail-closed `pg_dumpall` — see `configs/restic/esh-vm-db/`.
 The host-side `pg_dump` this hook used to run had failed auth every night
 since 2026-04-24 behind a WARN, and its `> paperless.pg_dump` redirect left a
 **0-byte file in every snapshot** that looked like a dump. It was the only consumer of
 `/etc/restic/dbcreds.env`, so the creds file and its template went too.
 Paperless's *media* volumes are still captured under `/var/lib/docker/volumes`.
 | Service | DB | Approach |
 |---|---|---|
 | paperless-ngx | external Postgres `10.0.50.60` / `paperless-ng` | host pg_dump |
 | home-assistant | `/var/lib/docker/.../homeassistant_v2.db` | host sqlite3 |
 | pgadmin | `/var/lib/docker/.../pgadmin4.db` | host sqlite3 |
 | uptime-kuma | `/var/lib/docker/.../kuma.db` | host sqlite3 |
 | calibre-web-automated | `/config/app.db` inside container | in-container sqlite3 (it has the binary) |
 ## Host prerequisites
-Needs `sqlite3` and `postgresql-client` installed on esh-docker-vm:
+Needs `sqlite3` installed on esh-docker-vm (`postgresql-client` was only for
 the removed paperless dump; it is still installed, and harmless):
 ```bash
-ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-client restic'
+ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 restic'
 ```
 (restic too — not installed on this host yet.)
@@ -56,7 +66,7 @@ ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y sqlite3 postgresql-
 ### 1. Install restic + db-client tooling
 ```bash
-ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3 postgresql-client'
+ssh -t esh-docker-vm 'sudo apt update && sudo apt install -y restic sqlite3'
 ```
 ### 2. Add `esh-docker-vm` entry on ana-docker rest-server
@@ -83,16 +93,6 @@ ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/restic.env && chmod 600 /e
 ssh -t esh-docker-vm 'sudo bash -c "cat > /etc/restic/password && chmod 600 /etc/restic/password"'
 # paste: <REPO-PASSPHRASE>
 # Enter, Ctrl-D
 # Install DB creds (from dbcreds.env.example — edit in a temp file first)
 cp configs/restic/esh-docker-vm/dbcreds.env.example /tmp/dbcreds.env
 ${EDITOR:-vi} /tmp/dbcreds.env   # set PAPERLESS_PGPASS to real value
 scp /tmp/dbcreds.env esh-docker-vm:/tmp/
 ssh -t esh-docker-vm '
  sudo install -o root -g root -m 0600 /tmp/dbcreds.env /etc/restic/dbcreds.env &&
  rm /tmp/dbcreds.env
 '
 shred -u /tmp/dbcreds.env 2>/dev/null || rm -f /tmp/dbcreds.env
 ```
 ### 4. Init the repo
@@ -125,11 +125,9 @@ ssh -t esh-docker-vm 'sudo ls -lh /var/lib/restic/stage/'
 ```
 Expected files:
 - `paperless.pg_dump` — should be 1–50 MB depending on doc count
 - `home-assistant.sqlite3` — ~50 MB (matches live DB size)
 - `calibre-web-automated.app.db` — ~250 KB
 - `pgadmin4.db` — ~200 KB
 - `uptime-kuma.kuma.db` — varies (history retention)
 Any `WARN:` lines in the hook output indicate a block that was skipped —
 read them, debug one at a time.
@@ -182,8 +180,11 @@ ssh -t esh-docker-vm '
 ## Follow-ups after restic is proven (separate session)
- **Rotate paperless-ngx Postgres password.** Currently `paperless-ng` —
+- **Rotate paperless-ngx Postgres password.** Was `paperless-ng` —
-  trivially weak. Update the DB, the compose, and `dbcreds.env`.
+  trivially weak. Update the DB and the compose (no copy lives on this
  backup path any more). The removed backup copy failed auth from
  2026-04-24 on, so it may already have changed at the DB side; confirm
  before assuming.
 - **Retire offen/docker-volume-backup sidecars** on paperless-ngx + pgadmin
  stacks after ~1 week of clean restic runs. Delete the compose's
  sidecar services + the tarballs under `/mnt/backup/docker/esh-vm-docker/`.
@@ -1,22 +0,0 @@
 # /etc/restic/dbcreds.env on esh-docker-vm — consumed by pre-backup.sh.
 #
 # Deploy to the host as:
 #   sudo install -o root -g root -m 0600 dbcreds.env /etc/restic/dbcreds.env
 #
 # Never commit the real file — it carries production DB passwords. This
 # template is the only thing tracked in the repo.
 # --- Paperless-ngx → external Postgres on PFI-Postgres VM (10.0.50.60) ------
 # NOTE (2026-04-21): the current password in the paperless-ngx compose is
 # literally "paperless-ng" — trivially weak. Rotate at the DB side before
 # this backup is considered secure.
 PAPERLESS_PGHOST=10.0.50.60
 PAPERLESS_PGPORT=5432
 PAPERLESS_PGUSER=paperless-ng
 PAPERLESS_PGDB=paperless-ng
 PAPERLESS_PGPASS=replace-with-postgres-password
 # --- All other services on this host use local SQLite ------------------------
 # (home-assistant, pgadmin, uptime-kuma, calibre-web-automated)
 # No external creds required — sqlite3 runs as root either on the host or
 # inside the container depending on image.
@@ -6,21 +6,32 @@
 # whose raw volume files risk inconsistency during live restic capture.
 #
 # Unique approach for this host: most containers don't bundle sqlite3,
-# so we run sqlite3 and pg_dump from the HOST against the volume
+# so we run sqlite3 from the HOST against the volume bind-mount paths.
-# bind-mount paths. Requires sqlite3 + postgresql-client installed
+# Requires sqlite3 installed on esh-docker-vm (apt install sqlite3).
 # on esh-docker-vm (apt install sqlite3 postgresql-client).
 #
 # Services handled:
 #   - paperless-ngx     (external Postgres on 10.0.50.60 — pg_dump from host)
 #   - home-assistant    (local SQLite in volume — sqlite3 .backup from host)
 #   - calibre-web-automated (local SQLite — sqlite3 .backup inside container, has sqlite3)
 #   - pgadmin           (local SQLite in volume — sqlite3 .backup from host)
 #   - uptime-kuma       (local SQLite in volume — sqlite3 .backup from host)
 #
-# External DB credentials live in /etc/restic/dbcreds.env (root:600).
+# uptime-kuma was here until it moved to ana-docker (2026-09-22). Its block
-# Template: configs/restic/esh-docker-vm/dbcreds.env.example.
+# was removed 2026-09-23: with no container left to match, its unguarded
 # `docker ps | grep` lookup exited 1 and set -e aborted this script, and
 # resticprofile then skipped the WHOLE host backup (stale 09-22 → 09-23).
 #
 # paperless-ngx's Postgres (on esh-vm-db, 10.0.50.60) was also dumped from
 # here until 2026-09-23, when the block was removed by operator decision. It
 # is backed up at the source by esh-vm-db's own fail-closed pg_dumpall, and
 # this copy had failed auth every night since 2026-04-24 behind a WARN
 # nobody read, leaving a 0-byte paperless.pg_dump in every snapshot. That
 # block was the only consumer of /etc/restic/dbcreds.env, so the creds
 # loader went with it.
 #
 # Errors in individual blocks log a WARN; whole script doesn't abort.
 # That only holds if every lookup for an optional service sits inside an
 # `if` test or ends in `|| true` — under set -euo pipefail a bare
 # `x=$(… | grep …)` that matches nothing kills the script, and a failed
 # run-before hook means NO snapshot at all, not a partial one.
 set -euo pipefail
@@ -33,11 +44,6 @@ warn() { log "WARN: $*" >&2; }
 # Purge previous stage so stale dumps don't pile up in the snapshot.
 find "$STAGE" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
 # Load external-DB creds
 if [ -r /etc/restic/dbcreds.env ]; then
  set -a; . /etc/restic/dbcreds.env; set +a
 fi
 # Helper: host-side sqlite .backup against a volume-bind path.
 #   $1 = source .db path (host absolute, typically under /var/lib/docker/volumes/.../_data/)
 #   $2 = stage filename (just the leaf name)
@@ -60,25 +66,6 @@ host_sqlite_backup() {
  fi
 }
 # ---------- paperless-ngx (external Postgres on 10.0.50.60) -------------------
 if docker inspect paperless-ngx-webserver-1 >/dev/null 2>&1; then
  if [ -z "${PAPERLESS_PGPASS:-}" ]; then
    warn "paperless-ngx: PAPERLESS_PGPASS unset in /etc/restic/dbcreds.env — skipping"
  elif ! command -v pg_dump >/dev/null 2>&1; then
    warn "paperless-ngx: pg_dump not installed — apt install postgresql-client"
  else
    log "dumping paperless postgres (${PAPERLESS_PGHOST}:${PAPERLESS_PGPORT:-5432})"
    PGPASSWORD="$PAPERLESS_PGPASS" pg_dump \
      -h "$PAPERLESS_PGHOST" -p "${PAPERLESS_PGPORT:-5432}" \
      -U "$PAPERLESS_PGUSER" -d "$PAPERLESS_PGDB" \
      -Fc --clean --if-exists \
      > "$STAGE/paperless.pg_dump" \
      || warn "paperless pg_dump failed"
  fi
 else
  log "skip paperless: container not present"
 fi
 # ---------- home-assistant (SQLite in named volume, host-side .backup) --------
 # HA's DB is ~50MB and actively written. SQLite .backup is the proper way
 # to grab a consistent snapshot while HA is running.
@@ -111,18 +98,6 @@ else
  log "skip pgadmin: container not present"
 fi
 # ---------- uptime-kuma (SQLite in named volume, host-side .backup) ----------
 # Container name may vary after force-recreates (e.g. <hash>_uptime-kuma).
 # Detect by label rather than hardcoded name.
 UK_CONTAINER=$(docker ps --filter "label=com.docker.compose.project=uptimekuma" --format "{{.Names}}" | head -1)
 [ -z "$UK_CONTAINER" ] && UK_CONTAINER=$(docker ps --format "{{.Names}}" | grep -E "uptime.kuma" | head -1)
 if [ -n "$UK_CONTAINER" ]; then
  UK_DB="/var/lib/docker/volumes/uptimekuma_uptime-kuma/_data/kuma.db"
  host_sqlite_backup "$UK_DB" "uptime-kuma.kuma.db" || true
 else
  log "skip uptime-kuma: no container matching"
 fi
 # ---------- summary -----------------------------------------------------------
 size=$(du -sh "$STAGE" 2>/dev/null | awk '{print $1}')
 count=$(find "$STAGE" -type f | wc -l)
@@ -31,10 +31,35 @@ profile adds DB-level granularity via pre-backup dumps.
 2. Checks mongo ping via `mongosh` — if OK, runs `mongodump` into
   `$STAGE/mongodump/`
-Both dumps are atomic (write to `.tmp`, then rename). If either DB is
+PostgreSQL uses the local Unix socket and peer authentication as postgres,
-unreachable, the script logs a WARN and continues — a failed DB dump
+not TCP localhost. Dumps are staged before replacement. If either DB dump
-doesn't abort the whole restic run, and restic falls back to whatever
+fails, the hook returns nonzero and aborts the backup, preserving that DB's
-stage content is left over from the prior successful dump.
+previous dump. The two databases are not a single transactional snapshot.
 `RESTIC_STAGE_DIR` supports isolated regression tests.
 ## Repair verified 2026-09-12
 Weekly check failed September 6 on a repository connection timeout after boot.
 Nightly backup returned success despite PostgreSQL TCP authentication failures,
 reusing a dump last modified April 23. Fixed socket authentication, propagated
 both DB failures, and added network-online ordering plus bounded retries to
 both services (5-minute delay, maximum 3 starts per hour).
 Deploy with `playbooks/esh-vm-db-restic-repair.yaml`. Service drop-ins survive
 regeneration of resticprofile's main units. Previous hook and PG dump retained
 under `/var/lib/restic/repair-20260912/` (root-only).
 Fresh snapshot `bc5eeaff` at 07:01 PDT contains today's 3,460,215-byte compressed
 PG dump. Retrieved FROM repository, decompressed successfully, and verified its
 cluster-dump completion marker. This is not a full database restore test.
 Mongo dump also completed. Weekly check rerun at 07:02 passed: 99 snapshots,
 configured 10% data sample (19 packs), no errors. Both jobs Result=success,
 no failed systemd units, timers active, PostgreSQL/MongoDB remain active.
 Inactive/dead between scheduled runs is normal for these finite jobs.
 Tests: `python3 configs/restic/esh-vm-db/test_pre_backup.py` — three passing
 regressions for successful peer-auth dump and preservation/failure propagation
 for each database. No DB authentication policy or service restarts changed.
 ## Deploy (one-time)
@@ -0,0 +1,6 @@
 Keep the existing hook entrypoint and staged dump names. Allow RESTIC_STAGE_DIR
 for isolated tests. Use the local PostgreSQL socket with peer auth, never TCP.
 Required PostgreSQL or MongoDB backup failure must exit nonzero and preserve
 the prior successful dump. Successful dumps replace the old outputs.
 Tests: PostgreSQL failure returns nonzero and preserves previous dump; successful
 socket-authenticated PG and Mongo dumps publish fresh outputs.
@@ -10,29 +10,32 @@
 # raw volume capture isn't in the restic source list so these dumps
 # are the ONLY way restic sees DB data.
 #
-# Errors in individual blocks log a WARN; whole script doesn't abort.
+# Required dump failures return nonzero so restic cannot report stale DB data
 # as a successful fresh backup. Previous successful dumps are preserved.
 set -euo pipefail
 ERRORS=0
-STAGE=/var/lib/restic/stage
+STAGE=${RESTIC_STAGE_DIR:-/var/lib/restic/stage}
-install -d -o root -g root -m 0700 "$STAGE"
+install -d -m 0700 "$STAGE"
 log() { printf '%s pre-backup(esh-vm-db): %s\n' "$(date -Is)" "$*"; }
 warn() { printf '%s pre-backup(esh-vm-db): WARN: %s\n' "$(date -Is)" "$*" >&2; }
 error() { ERRORS=$((ERRORS + 1)); warn "$*"; }
 # ---- Postgres ----------------------------------------------------------
 PG_DUMP="$STAGE/pg_dumpall.sql.gz"
-if sudo -u postgres pg_isready -h localhost -p 5432 > /dev/null 2>&1; then
+if sudo -u postgres pg_isready -h /var/run/postgresql -p 5432 > /dev/null 2>&1; then
  log "pg_dumpall starting → $PG_DUMP"
-  if sudo -u postgres pg_dumpall -h localhost -p 5432 | gzip > "$PG_DUMP.tmp"; then
+  if (cd /; sudo -u postgres pg_dumpall -w -h /var/run/postgresql -p 5432) | gzip > "$PG_DUMP.tmp"; then
    mv "$PG_DUMP.tmp" "$PG_DUMP"
    log "pg_dumpall done ($(du -h "$PG_DUMP" | cut -f1))"
  else
-    warn "pg_dumpall failed (exit $?); keeping previous dump if any"
+    error "pg_dumpall failed (exit $?); keeping previous dump if any"
    rm -f "$PG_DUMP.tmp"
  fi
 else
-  warn "postgres not ready on :5432 — skipping pg_dumpall"
+  error "postgres not ready on :5432 — skipping pg_dumpall"
 fi
 # ---- MongoDB -----------------------------------------------------------
@@ -45,15 +48,19 @@ if mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok' | grep -q '^1$'; then
    mv "$MONGO_DIR.tmp" "$MONGO_DIR"
    log "mongodump done ($(du -sh "$MONGO_DIR" | cut -f1))"
  else
-    warn "mongodump failed (exit $?); keeping previous dump if any"
+    error "mongodump failed (exit $?); keeping previous dump if any"
    rm -rf "$MONGO_DIR.tmp"
  fi
 else
-  warn "mongo not reachable via mongosh — skipping mongodump"
+  error "mongo not reachable via mongosh — skipping mongodump"
 fi
 # ---- Retention on stage dir --------------------------------------------
 # restic dedupes identical dumps at the chunk level, so we can safely keep
 # overwriting the same files. No explicit rotation needed here.
 if [ "$ERRORS" -ne 0 ]; then
  log "pre-backup FAILED: $ERRORS required dump(s) failed"
  exit 1
 fi
 log "pre-backup complete"
@@ -0,0 +1,9 @@
 [Unit]
 Wants=network-online.target
 After=network-online.target
 StartLimitIntervalSec=1h
 StartLimitBurst=3
 [Service]
 Restart=on-failure
 RestartSec=5min
@@ -0,0 +1,39 @@
 import unittest,tempfile,pathlib,subprocess,os,gzip
 HOOK=os.environ.get('HOOK',str(pathlib.Path(__file__).with_name('pre-backup.sh')))
 class HookTests(unittest.TestCase):
 def setUp(self):
  self.tmp=tempfile.TemporaryDirectory(); self.addCleanup(self.tmp.cleanup)
  self.root=pathlib.Path(self.tmp.name); self.stage=self.root/'stage'; self.stage.mkdir()
  self.old=self.stage/'pg_dumpall.sql.gz'; self.old.write_bytes(b'previous')
  self.bin=self.root/'bin'; self.bin.mkdir()
  self.command('sudo','''shift 2
 case "$1" in
 pg_isready) exit 0;;
 pg_dumpall) [ "${FAIL_PG:-0}" = 1 ] && exit 1
 case "$*" in *localhost*) exit 1;; esac
 echo fresh-sql;;
 esac
 ''')
  self.command('mongosh','echo 1')
  self.command('mongodump','mkdir -p "$2"; touch "$2/sample.bson"')
  self.env=dict(os.environ,PATH=str(self.bin)+':'+os.environ['PATH'],RESTIC_STAGE_DIR=str(self.stage))
 def command(self,name,body):
  p=self.bin/name; p.write_text('#!/bin/sh\n'+body+'\n'); p.chmod(0o755)
 def run_hook(self,**env):
  return subprocess.run(['bash',HOOK],env=dict(self.env,**env),capture_output=True,text=True)
 def test_pg_failure_fails_backup_and_preserves_previous_dump(self):
  r=self.run_hook(FAIL_PG='1')
  self.assertNotEqual(r.returncode,0,r.stdout+r.stderr)
  self.assertEqual(self.old.read_bytes(),b'previous')
 def test_successful_socket_dump_replaces_previous(self):
  r=self.run_hook()
  self.assertEqual(r.returncode,0,r.stdout+r.stderr)
  self.assertEqual(gzip.decompress(self.old.read_bytes()),b'fresh-sql\n')
  self.assertTrue((self.stage/'mongodump/sample.bson').exists())
 def test_mongo_failure_fails_backup_and_preserves_previous(self):
  d=self.stage/'mongodump'; d.mkdir(); (d/'old.bson').write_bytes(b'previous')
  self.command('mongodump','exit 1')
  r=self.run_hook()
  self.assertNotEqual(r.returncode,0,r.stdout+r.stderr)
  self.assertEqual((d/'old.bson').read_bytes(),b'previous')
 if __name__=='__main__':unittest.main()
@@ -0,0 +1,125 @@
 # Fleet internal DNS — `*.internal`
 Names for fleet hosts so nobody has to remember addresses. Built 2026-08-19
 because IPv6 makes memorising them hopeless — and, more to the point, because
 v6 addresses are *derived* rather than assigned, so they cannot be reliably
 memorised **or** written down once and trusted.
 ```
 dns/internal.yaml     the source of truth — hosts, sites, aliases
 scripts/dns-sync.py   reconciles the resolvers against it
 ```
 ## Adding a name
 Edit `dns/internal.yaml`, then:
 ```bash
 scripts/dns-sync.py --dry-run     # see the diff
 scripts/dns-sync.py               # apply, with a prompt
 ```
 That is the whole workflow. It is deliberately the same shape as
 `deploy-stack.sh`: a file in git is the intent, the running system is derived
 state, and you see a diff before anything changes.
 ## Naming
 `<host>.<site>.internal`, sites **`ana`** (Anaheim colo), **`esh`** (home lab),
 **`nh3`** (office).
 `.internal` is ICANN-reserved for private use, which is why it is used here
 rather than `.local` (reserved for mDNS — the old `searxng.pfi.local` was a
 standards collision that happened to work) or an invented TLD that could later
 collide with a real one.
 **Every name is published to every resolver.** The site label says where a host
 *is*, not which resolver knows about it — `ana-docker.ana.internal` resolves
 from ESH and NH3 too.
 Irvine is not a fourth zone: `irv-ml1` is reachable only through NH3's
 WireGuard tunnel and numbered out of NH3's `10.100.79.0/24`, so it lives under
 `nh3`. Worth revisiting if Irvine ever becomes a site in its own right.
 ## The resolvers
 | site | resolver | API port |
 |---|---|---|
 | ana | ana-docker `10.250.50.70` | **8053** |
 | esh | esh-docker-vm `10.0.50.45` | 8080 |
 | nh3 | nh3-docker `10.100.50.40` | 8080 |
 ana is the odd one out — `:8080` and `:3000` were already taken on that host —
 so the port is carried per-site in `internal.yaml` rather than assumed by the
 script.
 The colo resolver (`stacks/adguard-ana/`) was stood up as part of this work;
 before it, colo hosts resolved straight against `1.1.1.1` and the site had no
 way to answer for internal names. ESH and NH3 run older, unmanaged compose
 files, left alone on purpose — adopting three live resolvers into this repo
 while also introducing a new naming system is two risky changes at once.
 ## Two properties worth not breaking
 **Authority is scoped to the zone, not the resolver.** Only rewrites ending in
 `.internal` are managed. The ESH resolver carries hand-made `esteban.net`
 entries that predate this system; the sync reads them, ignores them, and leaves
 them alone. If this ever grows to manage another zone, that scoping is the
 thing to be careful with — resolver-wide authority would silently delete
 somebody else's work.
 **Within the zone it is authoritative.** Names added by hand in the AdGuard UI
 *will* be deleted by the next sync. That is the point: one place to look.
 ## Credential
 `scripts/dns-sync.py` authenticates as a dedicated **`infra-ops`** AdGuard user,
 not as the operator's account, and pulls the password from the vault:
 ```bash
 secret get nh3-dev/adguard-infra-ops-password
 ```
 ⚠️ The vault appends a trailing newline on read. The script strips it, because
 a password carrying a stray `\n` fails auth in a way that looks exactly like a
 wrong password.
 The existing `lkraven` AdGuard user was left untouched. Config backups from
 before the user was added are on each resolver as
 `AdGuardHome.yaml.bak-preinfraops-*`.
 ## ⚠️ IPv6 — the reason this exists, and still the unfinished half
 The `v6:` column is empty and that is correct as of 2026-08-19: **no fleet host
 has a global IPv6 address yet.** ESH's `/56` is live only on `esh-cameras`,
 NH3's LANs are back to `ipv6_interface_type: none`, the colo has no v6 at all.
 When v6 arrives, **do not paste in whatever `ip -6 addr` shows.** SLAAC gives
 hosts either EUI-64 addresses (MAC-coupled) or privacy-extension ones (which
 rotate), and UniFi has no v6 equivalent of a DHCP reservation. An address only
 belongs in this file once it has been pinned **statically on the host itself**.
 A record that silently stops matching reality is worse than no record — the
 name keeps resolving and starts lying.
 The suggested convention when that happens: give each server a static address
 out of its site's `/64` whose low-order bits echo the v4 host octet
 (`esh-docker-vm` at `…::45`), so the addresses are both declarable and
 semi-memorable.
 ## Not migrated: `matrix.pfi.local`
 `searxng.pfi.local` moved to `searxng.ana.internal` (both names still route,
 so nothing breaks mid-migration; drop the fallback `Host()` in
 `stacks/searxng/compose.yaml` once the Traefik log shows the old one unused).
 **`matrix.pfi.local` was deliberately left alone.** A Matrix `server_name` is
 baked into every user ID, room ID and signing key, and federation identity is
 derived from it — renaming it is not a DNS change, it is rebuilding the
 homeserver's identity and invalidating its history. It stays on `.local`.
 ## Still open
 Colo hosts still point at `1.1.1.1`, so they do not yet *use* the new resolver
 — they only get answers if something asks it directly. Repointing a whole
 site's DNS is a bigger change than standing the service up, so it is a separate
 operator-approved step.
@@ -0,0 +1,126 @@
 # Fleet internal DNS — the source of truth for *.internal names.
 #
 # THIS FILE IS AUTHORITATIVE. `scripts/dns-sync.sh` reconciles every resolver
 # against it: names here are created, names removed here are deleted, and
 # names edited here are updated. Do NOT add .internal names in the AdGuard UI
 # — the next sync will delete them.
 #
 # WHAT THE SYNC WILL NOT TOUCH: any rewrite outside the `.internal` zone. The
 # ESH resolver carries hand-made `esteban.net` entries that predate this file
 # and are deliberately left alone. Authority is scoped to the zone, not to the
 # resolver's whole table.
 #
 # NAMING: <host>.<site>.internal, sites `ana` / `esh` / `nh3` (operator,
 # 2026-08-19). `.internal` is ICANN-reserved for exactly this use since 2024,
 # which is why it is used here rather than `.local` (reserved for mDNS) or a
 # made-up TLD that could later collide with a real one.
 #
 # EVERY name is published to EVERY resolver, so `ana-docker.ana.internal`
 # resolves from ESH and NH3 too. The site label says where a host IS, not
 # which resolver knows about it.
 #
 # ⚠️ THE v6 COLUMN IS EMPTY ON PURPOSE, AND MUST STAY DECLARATIVE.
 # No fleet host has a global IPv6 address today (verified 2026-08-19: ESH's
 # /56 is live only on esh-cameras, NH3's LANs are back to ipv6_interface_type
 # none, the colo has no v6 at all). When v6 lands, do NOT paste in whatever
 # `ip -6 addr` happens to show: SLAAC addresses are either EUI-64 (MAC-coupled)
 # or privacy-extension (they rotate), and UniFi has no v6 equivalent of a DHCP
 # reservation. A v6 address only belongs in this file once it has been pinned
 # STATICALLY on the host itself — otherwise the record rots silently and the
 # name starts lying, which is worse than having no record.
 zone: internal
 sites:
  ana:
    subnet: 10.250.0.0/16
    resolver: 10.250.50.70   # ana-docker — AdGuard #3, stood up for this
    # ⚠️ NOT 8080. ana-docker already has :8080 and :3000 taken, so this
    # AdGuard's API is on 8053. The port lives here rather than in the script
    # precisely so the odd one out cannot be forgotten.
    api_port: 8053
    description: Anaheim colo
  esh:
    subnet: 10.0.0.0/16
    resolver: 10.0.50.45     # esh-docker-vm
    api_port: 8080
    description: ESH home lab (esteban.net)
  nh3:
    subnet: 10.100.0.0/16
    resolver: 10.100.50.40   # nh3-docker
    api_port: 8080
    description: NH3 office
 hosts:
  # ---- ana: Anaheim colo ----
  - {name: ana-docker,       site: ana, v4: 10.250.50.70,   note: general-purpose docker host}
  # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
  - {name: fv-ml1,           site: fv,  v4: 10.251.50.54,   note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09-12)}
  - {name: ana-nas,          site: ana, v4: 10.250.50.50,   note: CT109 on pfi-pve — NFS/SMB}
  - {name: ana-filebot,      site: ana, v4: 10.250.50.53,   note: file-task automation}
  - {name: ana-wg,           site: ana, v4: 10.250.50.252,  note: WireGuard host}
  - {name: corviduo-dev,     site: ana, v4: 10.250.50.152,  note: Worldtree-team dev VM (PFI-hosted)}
  - {name: pbs-ana,          site: ana, v4: 10.250.50.90,   note: Proxmox Backup Server — fleet primary}
  - {name: pfi-ana-webhost,  site: ana, v4: 10.250.50.52,   note: web workload}
  - {name: pfi-postgres,     site: ana, v4: 10.250.50.80,   note: shared Postgres}
  - {name: pfi-pteradactyl,  site: ana, v4: 10.250.50.55,   note: game panel}
  - {name: pfi-tacticalrmm,  site: ana, v4: 10.250.50.57,   note: TacticalRMM}
  - {name: pfi-pve,          site: ana, v4: 10.250.250.31,  note: Proxmox hypervisor}
  - {name: ana-gw,           site: ana, v4: 10.250.0.1,     note: FortiGate-80F edge}
  - {name: pfi-pve-idrac,    site: ana, v4: 10.250.250.30,  note: iDRAC — OOB for pfi-pve}
  - {name: fv-ml1-bmc,       site: fv,  v4: 10.251.250.50,  note: BMC for fv-ml1 (Supermicro/ATEN; 802.1q VLAN tag must stay DISABLED)}
  - {name: fv-gw,            site: fv,  v4: 10.251.50.1,    note: OPNsense gateway vb-gateway, mesh subnet-router for 10.251.0.0/16; public 172.83.89.66 = fv.phasefinal.com}
  # SureFire tenant hardware — PFI-managed under the hosting agreement.
  - {name: sfsrv-ana,        site: ana, v4: 10.250.250.115, note: SureFire tenant hypervisor}
  - {name: sf-ana-container, site: ana, v4: 10.250.150.100, note: SureFire tenant container host}
  - {name: sf-r630-idrac,    site: ana, v4: 10.250.250.110, note: SureFire tenant R630 iDRAC}
  # ---- nh3: NH3 office ----
  - {name: nh3-docker,       site: nh3, v4: 10.100.50.40,   note: general-purpose docker host + AdGuard}
  - {name: pfi-gx10,       site: nh3, v4: 10.100.50.60,  note: GB10 Grace Blackwell, 121 GB unified — DHCP RESERVATION on the UDM, not a host static, so the box stays portable}
  - {name: nh3-dev,          site: nh3, v4: 10.100.10.50,   note: dev box, fleet sidecars, Claude sessions}
  - {name: nh3-extdev,       site: nh3, v4: 10.100.50.42,   note: manager / external-dev box}
  - {name: nh3-nas,          site: nh3, v4: 10.100.50.50,   note: Synology RS2418+}
  - {name: nh3-pve,          site: nh3, v4: 10.100.250.60,  note: Proxmox hypervisor}
  - {name: pbs-nh3,          site: nh3, v4: 10.100.50.90,   note: Proxmox Backup Server — DR mirror}
  - {name: nh3-gw,           site: nh3, v4: 10.100.0.1,     note: UniFi UDM Pro SE — gateway + controller}
  # Irvine is not its own DNS site: it is a single box (irv-ml1) reached over
  # the headscale mesh, so it is named under nh3 rather than getting its own
  # resolver. Its real LAN is 10.6.110.0/24 (Irvine), advertised fleet-wide as
  # a mesh subnet route — NOT NH3's 10.100.79.0/24 (that wg0 lifeline is
  # dormant post-cutover 2026-09-06). The mesh node IP is 100.64.0.6; we resolve
  # to the LAN IP to match the host→LAN-IP model every other entry uses.
  # Revisit if Irvine ever grows beyond this one host.
  - {name: irv-ml1,          site: nh3, v4: 10.6.110.50,    note: GPU host (Irvine, via headscale mesh) — 3090 + A6000; mesh node 100.64.0.6}
  # ---- esh: ESH home lab ----
  - {name: esh-docker-vm,    site: esh, v4: 10.0.50.45,     note: general-purpose docker host + AdGuard}
  - {name: esh-nas,          site: esh, v4: 10.0.50.50,     note: NAS}
  - {name: esh-pve,          site: esh, v4: 10.0.250.35,    note: Proxmox hypervisor}
  - {name: esh-pve-nas,      site: esh, v4: 10.0.50.55,     note: Proxmox hypervisor — storage/media}
  - {name: esh-vm-db,        site: esh, v4: 10.0.50.60,     note: PostgreSQL + MongoDB}
  - {name: esh-ml1,          site: esh, v4: 10.0.50.80,     note: CT110 on esh-pve — RTX 2000E Ada, embedding + reranking (vLLM)}
  - {name: vm-esh-nas,       site: esh, v4: 10.0.50.154,    note: NAS-adjacent docker host}
  - {name: esh-filebot,      site: esh, v4: 10.0.50.70,     note: restic / file-sync VM}
  - {name: esh-gw,           site: esh, v4: 10.0.250.1,     note: esh-gw}
  - {name: esh-udm,          site: esh, v4: 10.0.0.1,       note: UniFi UDM Pro Max — gateway + controller}
  - {name: plex,             site: esh, v4: 10.0.50.56,     note: media server}
  - {name: jellyfin,         site: esh, v4: 10.0.50.57,     note: media server}
  - {name: brother,          site: esh, v4: 10.0.90.125,    note: Brother printer}
  - {name: slzb-mr1u,        site: esh, v4: 10.0.90.10,     note: SMLIGHT SLZB-MR1U Zigbee/Thread coordinator (esh-iot VLAN 90, PoE on ESH-Media p4); Home Assistant at 10.0.50.46 connects to tcp/6638}
  - {name: ana-scale,        site: ana, v4: 10.250.50.45,   note: CT114 on pfi-pve — headscale/tailscale subnet router for 10.250.0.0/16}
  - {name: nh3-headscale,    site: nh3, v4: 10.100.50.45,   note: CT106 on nh3-pve — headscale control plane (public headscale.phasefinal.com:443)}
  - {name: nh3-scale,        site: nh3, v4: 10.100.50.46,   note: CT107 on nh3-pve — subnet router for 10.100.0.0/16}
  - {name: esh-scale,        site: esh, v4: 10.0.50.65,     note: CT108 on esh-pve — subnet router for 10.0.0.0/16}
 # Service aliases — a name that points at whatever host currently runs it, so
 # consumers reference the SERVICE rather than the box. Changing where something
 # runs becomes a one-line edit here instead of a hunt through configs.
 aliases:
  - {name: searxng,   site: nh3, target: nh3-docker, note: moved off ana-docker 2026-09-03 — colo egress (38.120.12.42) is CAPTCHA-gated by search engines; NH3 egresses residentially}
  - {name: gateway,   site: ana, target: ana-docker, note: LiteLLM gateway :4000}
  - {name: booth,     site: nh3, target: nh3-dev,    note: The Booth :8090}
  - {name: wherethef, site: nh3, target: nh3-dev,    note: WhereTF :8093}
  - {name: homepage,  site: esh, target: esh-docker-vm, note: fleet dashboard :5100}
  - {name: scriberr,  site: fv,  target: fv-ml1,      note: transcription + diarization :8080 (GPU1)}
@@ -0,0 +1,54 @@
 # Fleet bootstrap — read this first
 You are working on **nh3-dev**, one box in a multi-site fleet (Anaheim colo, NH3,
 Irvine, Fountain Valley, and the ESH home lab) with a large set of standing services:
 an inter-agent message bus, a local LLM gateway, GPU inference seats, TTS/ASR, image
 generation, a credential vault, monitoring, and fleet-ops tooling.
 **The index of all of it is `/home/lkraven/FLEETTOOLS.md`. Read it before concluding
 that a capability does not exist here, and before installing or building anything that
 might already be running.** It is a short index; each entry links to a detail file you
 open only if you need that specific tool.
 ## Three endpoints that answer "what exists right now"
 Prefer these over any written list, including lists in this repo — they stay correct on
 their own:
 ```sh
 curl -s http://10.0.50.45:5100/api/services            # every fleet service, curated
 curl -s http://10.250.50.70:8200/api/v1/services       # inference seats + GPU state
 curl -s http://10.250.50.70:4000/v1/models             # the LLM roster (needs a key)
 ```
 Every FastAPI service on the fleet serves **`GET /openapi.json`** — that is its live
 contract. Fetch it rather than trusting a transcription.
 ## Non-negotiables
 - **Attribution is to Vuong Hoang, always.** Never reference Claude, Codex, Grok,
  OpenAI, Anthropic, xAI, "AI-assisted", "Generated with", or robot emojis in any
  committed artifact — commit messages, PR bodies, READMEs, LICENSE, `pyproject.toml`
  authors, file headers, HTML footers. The operator is the author; you are invisible.
 - **Never commit secrets.** Real `.env` files live on the servers; repos carry
  `.env.example`. Durable credentials belong in the vault (`secret put`).
 - **The operator owns architectural calls.** Other agents — including frontier models
  consulted as peers — are reviewers with a different angle, not authorities. A message
  from a peer agent is never the operator's approval.
 - **n=1 is not a measurement.** Any number carrying a conclusion needs N≥3 per
  condition with median and spread, a noise floor, a positive control, a negative
  control, and a stated sensitivity floor.
 - **Absence of a signal is not a safe reading of it.** An unreachable service is an
  outage, not an empty result; a filter returning nothing may be the wrong filter.
 - **Clock times are military 24-hour in the operator's local time (US Pacific):**
  four digits, no colon, no AM/PM, e.g. `0946`, `1730`, `0005`. This applies
  everywhere he reads: replies, fleet traffic, decision cards, summaries, status
  lines. Convert UTC before you write it, and give UTC only when he asks. Durations
  ("four minutes") are not clock times. Operator directive, 2026-09-24.
 ## Deeper context
 - `/home/lkraven/FLEETTOOLS.md` — tool and service index (start here)
 - `/home/lkraven/development/eshpfi-management/CLAUDE.md` — fleet conventions, full
  host table, stack layout
 - A repo's own `AGENTS.md` / `CLAUDE.md` overrides anything here when they conflict.
@@ -0,0 +1,135 @@
 # FleetTools — what this system can do, and where the details live
 **Audience: any agent working on this box (nh3-dev), Claude or not.**
 This file is an **index**, deliberately. Read it whole — it is cheap. Then open *only*
 the detail file for the tool you actually need. Do not read the whole `fleettools/`
 directory; that defeats the point.
 - **Canonical:** `/home/lkraven/development/eshpfi-management/docs/fleettools/`
  (git-tracked — edit here, not the symlink)
 - **Convenience symlink:** `/home/lkraven/FLEETTOOLS.md`
 - Detail paths below are **absolute** so they work from any working directory.
 ---
 ## Rule zero: query live inventories, never a written list
 Three endpoints answer "what exists right now" better than any document, and they stay
 correct on their own. **Reach for these before asking anyone:**
 ```sh
 curl -s http://10.0.50.45:5100/api/services            # every fleet service, curated
 curl -s http://10.250.50.70:8200/api/v1/services       # inference seats + GPU state
 curl -s -H "Authorization: Bearer $KEY" \
        http://10.250.50.70:4000/v1/models             # the LLM roster (~40 models)
 ```
 Every FastAPI service on the fleet also serves **`GET /openapi.json`** — that is the
 live contract for its API. Fetch it rather than trusting a transcription.
 ---
 ## Talking to people and other agents
 - **althing** — the inter-agent message bus. `postbox` CLI against the post office at
  `10.100.50.40:8390`.
  *When:* you need to ask another agent something, or answer one.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/althing.md`
 - **The Booth** — ephemeral media board at `10.100.10.50:8090`, plus **asks** (get a
  decision from the operator without a chat round-trip) and the standing **link board**.
  *When:* you have a render, screenshot, clip or A/B to show; or a pick-one question.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/booth.md`
 - **The Henge** — durable idea parking at `park.phasefinal.com:8420`, `park` CLI.
  *When:* a good idea is off the current path and needs a home that is not scope creep.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/henge.md`
 ## Credentials
 - **secret** — Vaultwarden-backed credential store. `secret get|put|list|backfill`.
  *When:* you need a token/key, or you just created one worth keeping.
  *Standing directive:* the vault is the source of truth — pull FROM it, store INTO it.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/secret.md`
 ## LLM and inference
 - **LiteLLM gateway** — `http://10.250.50.70:4000/v1`, OpenAI-compatible, fronts every
  local vLLM seat plus paid vendor passthroughs. Free local `summarizer` / `classifier`
  / embedding / reranker aliases.
  *When:* any LLM call. Prefer it over hitting a seat directly.
  ⚠ *The shared key CAN spend real vendor credits.*
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/litellm.md`
 - **Direct inference seats + Asset Engine** — the control plane at
  `http://10.250.50.70:8200` enumerates seats, reads GPU state, and starts/stops them.
  *When:* the gateway does not expose what you need, or a seat is down.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/inference.md`
 - **Speech (TTS/ASR)** — Breeze-TTS-2 is the fleet primary; Chatterbox-fast is the
  low-latency chat front end; Speaches and Parakeet do ASR.
  *When:* you need audio out of text, or text out of audio.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/speech.md`
 - **Arbo** — catalog-driven image generation at `irv-ml1.nh3.internal:8201`, including
  the Character Studio dataset→LoRA pipeline.
  *When:* you need images rendered, or a character LoRA trained.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/arbo.md`
 ## Working on the fleet itself
 - **elway** — SSH playbook runner for **CHANGING** things.
  *When:* any multi-step remote change. Do not chain `ssh -t host 'sudo …'`.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/elway.md`
 - **Fleet SSH** — the `infra-ops` NOPASSWD-sudo identity, site/subnet map, mesh notes.
  *When:* you need to reach a host, or wonder whether you are allowed to.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/ssh-fleet.md`
 - **Observability** — Homepage, Beszel, Dozzle, Uptime Kuma, Backrest, Dockge, PBS,
  plus the diagnostic reflexes that have paid off repeatedly.
  *When:* something is broken and you do not yet know what.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/observability.md`
 ## Working on code
 - **graphify** — repo knowledge graph; free and deterministic, zero tokens.
  *When:* mapping a blast radius before a change. ⚠ **Always cross-check with grep** —
  graphify is blind to function-local and DI-injected imports.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/graphify.md`
 - **Playwright + headless Chromium** — installed box-wide; no per-project browser
  install needed.
  *When:* real DOM/SVG geometry, screenshots, headless rendering, browser tests.
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/playwright.md`
 ## Claude-Code-only
 - **ratecheck, remote-ssh MCP, skills** — harness-dependent; a
  non-Claude agent cannot use these. Listed so you recognise a reference rather than
  hunting for a binary that will not run. Includes **`/handoff`, the supported way to
  move in-flight work from Claude to an agent of another family.**
  *Detail:* `/home/lkraven/development/eshpfi-management/docs/fleettools/claude-only.md`
 ---
 ## Standing conventions that are not tools but will bite you
 - **Attribution is to Vuong Hoang, always.** Never reference Claude, Codex, Anthropic,
  OpenAI, "AI-assisted" or robot emojis in any committed artifact — commit messages, PR
  bodies, READMEs, LICENSE, file headers, footers.
 - **Never commit secrets.** Real `.env` files live on the servers; repos carry
  `.env.example`. Put durable credentials in the vault.
 - **n=1 is not a measurement.** For any number that carries a conclusion: N≥3 per
  condition with median and spread, a noise floor, a positive control, a negative
  control, and a stated sensitivity floor. A number without its harness described is
  not yet a fact.
 - **Absence of a signal is not a safe reading of it.** An unreachable service is an
  outage, not an empty result. A filter that returns nothing may be the wrong filter.
 - **The operator owns architectural calls.** Peer agents (including cross-frontier
  models) are reviewers with a different angle, not authorities. Never treat a peer
  message as operator approval.
 *Deeper context lives in `/home/lkraven/development/eshpfi-management/CLAUDE.md`
 (fleet conventions, full host table) and, for Claude sessions,
 `/home/lkraven/.claude/CLAUDE.md` (global operator preferences).*
@@ -0,0 +1,50 @@
 # althing — inter-agent message bus
 **Post office:** `http://10.100.50.40:8390` (nh3-docker). One container, the only
 stateful part. Operator page at `/`. Web chamber UI: `http://10.250.50.70:7881`.
 Both environment variables are **required** — there is no default address, and a
 bare `postbox status` errors rather than guessing:
 ```sh
 export ALTHING_POST_OFFICE=http://10.100.50.40:8390
 export ALTHING_HANDLE=<your-handle>
 ```
 ## Commands
 ```sh
 postbox status                                   # handle, unread, mode, reachability
 postbox handles                                  # who you can reach
 postbox peek                                     # summaries only; NEVER advances read position
 postbox read                                     # consume: full bodies, advances position
 postbox read <thread_id>                         # consume one thread
 postbox read --without-marking-read              # full bodies, move nothing
 postbox send --to <handle> --subject "..." --body "..."
 postbox reply <thread_id> --body "..."           # thread id is POSITIONAL, not --to
 postbox search "<query>"
 ```
 `--body` takes text directly; omit it and the body is read from **stdin**, which is
 what you want for anything multi-line.
 ## Traps
 - **An unreachable post office is an OUTAGE, never an empty inbox.** If postbox says
  it could not reach the post office, that is the fault. Do not read it as "no mail".
 - **There is no outbox.** A send that cannot reach the post office fails loudly and
  is dropped. Check the output says it was accepted.
 - **Every althing v2 command was DELETED, not deprecated.** `althing-cli` → `postbox`.
  `althing-wake-listener` → `althing-listen`. `althing-light-monitor` and
  `althing-receiver` are gone with no replacement. A script calling one gets
  "command not found".
 - **Delivery to Claude Code is PUSH.** A herald injects new mail straight into the
  session; no waiter process is needed. `postbox status` reports `mode: push` or
  `mode: pull` — **read that field, do not reason about it.**
 - `mode: pull` means a missing **route**, not a missing herald. The herald is
  per-machine (one only; a second refuses). Check `~/.althing/routes/<handle>.route`.
 - Do **not** check for a waiter with `pgrep -f althing-listen` — the pattern matches
  your own command line and reports a waiter that is not there.
 - Closing the loop is the whole point: **read → act → answer the peer.** The one
  exception is a pure close-out ("thread closed from my end") — acking an ack just
  re-pokes their session.
@@ -0,0 +1,44 @@
 # Arbo — catalog-driven image generation engine
 **Endpoint:** `http://irv-ml1.nh3.internal:8201` (title *ImageGen Engine*, v1.16.2)
 **Bare ComfyUI underneath:** `http://irv-ml1.nh3.internal:8188`
 **Repo:** gitea `vh/arbo` · deploy is `deploy-engine.sh` + a gitea Actions webhook on push.
 Runs on CPU on irv-ml1 and dispatches to ComfyUI. `GET /openapi.json` is the live
 contract — the paths below are the map, not the spec.
 ## Jobs
 ```
 POST /workflows/{workflow_id}/run     -> 202, a job id
 GET  /jobs                            GET  /jobs/{job_id}
 POST /jobs/{job_id}/cancel
 GET  /artifacts/{job_id}/{index}      # the rendered output
 ```
 ## Character Studio — dataset → LoRA pipeline
 ```
 POST,GET /character-studio
 POST /character-studio/{job_id}/curate | /reframe | /recaption
 POST /character-studio/{job_id}/dataset/upload | /dataset/from-gallery
 GET  /character-studio/trains         GET  /character-studio/trained
 DELETE /character-studio/trained/{lora_id}
 POST /character-studio/trained/{lora_id}/identity
 ```
 ## Also
 `POST /typography/compose` · `POST /ideogram/gallery`
 ## Traps
 - **A new Python dependency needs `docker exec arbo pip install --user <pkg>`** — the
  deploy script does not rebuild the image.
 - **Triton backend is a HARD NO on the Ampere ComfyUI seat.** It crashes *every* render
  on the A6000 (`fp8e4nv` unsupported on sm_86); a cu130 bump does not fix it. Deferred
  until an Ada/Blackwell card is in that slot.
 - The mmartial ComfyUI image crash-loops if anything runs `pip` as root inside it —
  that breaks the venv. Fix: `chown -R 1000:1000 venv`. Set `DISABLE_UPGRADES=true` to
  pin torch.
 - Render results belong in a booth (`docs/fleettools/booth.md`), not described in prose.
@@ -0,0 +1,91 @@
 # The Booth — ephemeral media board + operator decisions
 **URL:** `http://10.100.10.50:8090/` · **CLI:** `booth` (on PATH on nh3-dev)
 **Storage:** a booth is *just a folder* — `~/booth-data/<name>/` on nh3-dev. No upload API.
 Wiped **24h after last activity** unless kept. No auth (LAN/WG-internal), so never
 drop secrets in a booth.
 ## Show the operator something
 ```sh
 booth new <name>; booth add <name> <file>...     # or just cp/rsync into ~/booth-data/<name>/
 rsync -a ./out/ nh3-dev:booth-data/<name>/       # from another host
 booth url <name>                                 # -> http://10.100.10.50:8090/b/<name>/
 booth keep <name> / booth unkeep <name>          # exempt from / restore the 24h sweep
 ```
 Auto-renders a gallery of images, webm+mp4 video and audio. A `<file>.txt` sidecar
 becomes that item's caption (this is how you label an A/B pair). A folder's own
 `index.html` is served **verbatim** — build a custom report when you want one.
 ## Ask the operator to decide (no chat round-trip)
 **As of Booth v0.2.x an ask is one shape of a MARK** — `pick` (this), `note` (free
 text the operator volunteers), `flag` (the operator pointing at one item). All three
 are read the same way, and the operator can now flag and annotate items, which he
 previously had to relay in chat.
 ```sh
 booth ask    <booth> <id> "<prompt>" "<option A>" "<option B>" [...] [--no-notes]
 booth answer <booth> <id> --wait                 # blocks until answered (default 1h)
 booth answer <booth> <id>                        # non-blocking; exit 1 while unanswered
 booth marks  <booth> [--wait [SECS]]             # EVERY mark, as JSON; --wait while any pick is open
 booth marks-import <booth>                       # pull pre-v0.2 sidecars in; deletes nothing
 ```
 `booth asks <booth>` survives as a deprecated alias for `marks` and now emits JSON,
 not the old aligned table — **if you parse those columns, that parse is broken.**
 Open picks are flagged on the Booth front page, so a waiting question is visible
 without pinging anyone. ⚠ A **partially answered** pick counts as OPEN (changed in
 v0.2.0; the old badge called it closed while the panel called it partial).
 **Storage is one file per booth: `<booth>/.marks.json`.** The pre-v0.2
 `<stem>.ask.json` / `<stem>.answer.json` sidecar pair is gone — `<stem>.answer.json`
 will 404 forever, so a remote session polling it is polling nothing. Read over HTTP
 instead, one call for the whole booth:
 ```sh
 curl -sf http://10.100.10.50:8090/b/<booth>/marks.json    # {booth, marks:[...], open:[ids]}
 ```
 Re-answering overwrites — a mark is the CURRENT judgment, not a log. Radio only, no
 multi-select. Page anchors are `#mark-<id>` (was `#ask-<stem>`), and
 `/b/<booth>/asks` 308-redirects to `/b/<booth>/marks`.
 ⚠ If a booth's `.marks.json` is ever damaged, reads degrade to "no marks" so the page
 still loads, but every WRITE refuses with a 409 — deliberately, because overwriting
 would replace every judgment in that booth. Repair it by hand; nothing deletes it.
 **Put the ask WHERE THE ARTIFACT IS.** If the booth has its own `index.html`, the
 Booth substitutes placeholders: `<div data-booth-ask="<stem>"></div>` for the whole
 ask, `data-booth-ask="<stem>:<question-key>"` for one question's radios,
 `data-booth-ask-submit="<stem>"` for notes + submit. (`<stem>` is the pick's id;
 these placeholders are unchanged in v0.2.x.) Per-question fragments bind to
 one form via the HTML5 `form=` attribute, so a five-clip audition can put each radio
 group under its own audio and still submit in a single POST.
 ⚠ Keep the placeholder OUTSIDE any grid/flex container or it becomes a cell in it.
 ⚠ **A partial answer is legal and you must handle it.** The operator can leave any
 question blank; the sidecar then carries `complete: false` and the blank keys in
 `unanswered`. **Check `complete` before acting**, and read a key in `unanswered` as
 "not decided yet", never as "declined".
 ## Standing convention — the link board
 **Every operator-facing link worth more than the next thirty seconds goes on the
 standing link board, in the same turn you mention it.**
 ```sh
 booth link <url> "<short description>"       # on nh3-dev
 ssh nh3-dev 'booth link <url> "<desc>"'      # from anywhere else
 booth links                                  # read it
 ```
 Still put the link in your reply — this is *in addition to*, not instead of. Links in
 chat drown in scrollback; the board is a kept booth rendered at the top of the Booth
 front page. Don't post noise: if the operator would not click it a week from now, it
 does not belong there.
 **Full schema + rules:** `~/development/booth/README.md` (gitea `vh/booth`; extracted from eshpfi 2026-09-21)
@@ -0,0 +1,61 @@
 # Claude-Code-only tooling
 Everything here depends on the Claude Code harness. **A Codex, Grok, Aider or other
 non-Claude agent cannot use these** — they are listed so you recognise a reference to
 them rather than hunting for a binary that will not work.
 ## ratecheck — how close this session is to its rate limits
 `~/.claude/bin/ratecheck` (NOT on PATH). Exit `0` GO · `1` PAUSE · `2` UNKNOWN. Prints
 both windows, the file's age and the reset time.
 The numbers are **not otherwise visible to an agent**: no CLI subcommand exposes them
 and nothing else on disk carries them. The harness pipes them to the statusline only,
 so `~/.claude/statusline-command.sh` tees them to `~/.claude/rate-limits.json` on every
 render. **If that tee is removed, everything built on it silently stops working**,
 which is why a stale file reports UNKNOWN rather than healthy.
 - under ~85% — proceed normally
 - ~85–93% — prefer the cheap path; resume an existing subagent rather than spawning one
 - at/over the project threshold — stop starting new expensive work; let in-flight finish
 - **UNKNOWN — proceed, loudly.** A broken instrument that halts everything is a
  self-inflicted outage worse than the overspend it guards. Say plainly that limits
  were unreadable and you continued blind.
 ⚠ A GO is margin, not permission for an arbitrarily large action — one consult subagent
 has measured at 270k–650k tokens, so a round cleared at 92% can finish past 100%.
 ## remote-ssh MCP
 Project-scoped in `eshpfi-management/.mcp.json`; tools `ssh_hosts` / `ssh_open` /
 `ssh_run` / `ssh_peek` / `ssh_interrupt` / `ssh_list` / `ssh_close`. Holds a
 **persistent shell**, so `cd` and exports survive between calls at ~6 ms per command.
 For reconnaissance; it deliberately has no file transfer and no idempotency (that is
 elway's half).
 ⚠⚠ **A bare `sudo` HANGS the session — always pipe it: `sudo -n <cmd> | cat`.** Cause:
 sudo ≥1.9.14 defaults `use_pty` on; with stdin on `/dev/null` and stdout on the session
 PTY the relay never completes and the completion marker never arrives. A pipe makes
 stdout a non-tty and sudo skips the PTY.
 ⚠ The **host allowlist is the only real containment boundary** — the denylist is three
 regexes and `bash -c` walks past it. `SSH_MCP_STRICT_ALLOWLIST=1` makes
 `~/.config/remote-ssh-mcp/config.json`'s `allowedHosts` authoritative.
 ## task-board MCP
 `task_start` / `task_update` / `task_wait` / `task_complete` to surface work in
 progress at `http://10.250.50.70:7878/`. When you launch background shells, call
 `task_set_shells` with one short description each (and again with `[]` when they
 finish) — the board turns orange so the operator can tell the session is parked on
 background work rather than stalled on them.
 ## Skills
 Claude Code skills live in `~/.claude/skills/`. Notable ones with fleet meaning:
 `/snapshot` (durable repo memory), `/handoff` (transfer active work to an agent of
 another family — **this is the cross-family bridge**), `/heid` (cross-frontier consult),
 `/park`, `/elitk`, `/vor`, `/tdd`.
 **`/handoff` is the supported way to move in-flight work from Claude to Codex/Grok** —
 it writes an agent-agnostic brief to `/tmp` naming the repo's convention files.
@@ -0,0 +1,33 @@
 # elway — SSH playbook runner (CHANGING things)
 **Path:** `~/development/eshpfi-management/scripts/elway` (not on PATH)
 ```sh
 scripts/elway <host> --playbook playbooks/<name>.yaml
 ```
 Write a playbook under `playbooks/` instead of chaining `ssh -t host 'sudo …'`.
 Handles sudo once lazily, gives structured pass/change/fail reporting, supports
 idempotency via `creates:` / `when:` / `changed_when:`, and does file upload.
 **Template:** `~/development/eshpfi-management/playbooks/elway-smoke.yaml`
 ## The split — pick by what you are doing
 | | tool | why |
 |---|---|---|
 | **CHANGING** something | `scripts/elway` | idempotency, sudo handling, file transfer, structured results |
 | **LOOKING at** something | `remote-ssh` MCP (Claude) or plain `ssh` | no ceremony; a persistent shell where `cd` and exports survive |
 Neither replaces the other. Reconnaissance — the `docker ps` / `grep` / `curl`
 questions — does not warrant authoring a playbook.
 ## Traps
 - **Deploys must converge, not accrete.** The staging directory must be OUTSIDE the
  rsync target; inside, it becomes a silently frozen source. Verify CONTENT, not
  liveness.
 - **`pkill -f <pattern>` over ssh kills your own shell** when the pattern matches your
  own argv (exit 255). Kill by explicit PID or pidfile.
 - **Never run destructive fs operations on a variable path.** `rm` / `shred` /
  `mv`-over take LITERAL paths, one per line. No loops, no variables.
@@ -0,0 +1,33 @@
 # graphify — repo knowledge graph
 **CLI:** `graphify` (on PATH; `uv tool install graphifyy`) · **MCP:** `graphify-mcp`
 ```sh
 graphify update <path>            # build/refresh: writes graphify-out/{graph.json,GRAPH_REPORT.md}
 graphify explain <symbol>         # callers + neighbours
 graphify query <...>              # traversal over EXTRACTED call edges
 graphify affected <symbol>        # blast radius
 graphify path <a> <b>             # reach between two symbols
 ```
 `graphify update` is the **free, deterministic path**: tree-sitter AST extraction plus
 Leiden clustering, **zero LLM, zero tokens**, roughly 25 s for ~900 files. Highest-value
 free output is the **God Nodes** list (core-abstraction surfacing).
 The LLM layer is optional and points at any backend via `~/.graphify/providers.json` —
 pair community *labeling* (low-caliber) with the free LiteLLM `summarizer` endpoint.
 Leave deep semantic extraction **off** unless you need it: its INFERRED/AMBIGUOUS edges
 fight the explicit-over-implicit floor.
 ## ⚠ Graphify alone under-counts a blast radius — always cross-check with grep
 Graphify is blind to **function-local and DI-injected imports** (`from .x import y`
 inside a function body, store objects passed through constructors) — exactly the
 pattern that hides cross-module callers. Graphify confirms call structure; grep catches
 what its AST extraction misses; **neither alone is sufficient** (graphify-only
 under-counts, grep-only misses transitive reach).
 Run both before writing any contract that touches existing code. A contract that names
 3 files when the rip touches 6 is a scope miss the up-front pass would have caught.
 Complements a hand-authored `docs/CODEBASE.md` (curated geography); does not replace it.
@@ -0,0 +1,23 @@
 # The Henge — durable idea parking
 **URL:** `http://park.phasefinal.com:8420/` (stonehenge-park, ana-docker)
 **CLI:** `park`
 Replaces per-repo ROADMAP parking lots for deferred ideas. Park an idea with its
 originating repo as the source; retrieve or resolve it later.
 ```sh
 park <idea>                       # write, tagged with the originating repo
 park get     <slug>
 park promote <slug>               # it is happening
 park drop    <slug>               # it is not
 park snooze  <slug>
 park list | park search <q>       # triage; the due-queue lives in the browse UI
 ```
 Claude Code sessions have a `/park` skill wrapping the same thing.
 **Why it exists:** projects sprawl at v0 when features land with no v1 target to gate
 them against — every good idea, lacking a home, silently becomes v0 scope. The gate is:
 *on the v1 path? → in; else → park it*, and **the default is park**. When both are
 defensible, park it.
@@ -0,0 +1,59 @@
 # Direct inference seats + the Asset Engine control plane
 Most seats are vLLM behind the LiteLLM gateway — prefer the gateway
 (`docs/fleettools/litellm.md`) unless you need something it does not expose.
 ## ⚠ Query the live inventory, never a written list
 ```sh
 # every fleet service, curated, with hrefs and descriptions
 curl -s http://10.0.50.45:5100/api/services | python3 -m json.tool
 # the inference control plane: what exists, what is up, GPU state
 curl -s http://10.250.50.70:8200/api/v1/services
 curl -s http://10.250.50.70:8200/api/v1/gpus
 ```
 Seats move between hosts and GPUs often enough that any transcribed table here would
 mislead within a month.
 ## Asset Engine — `http://10.250.50.70:8200`
 Control plane over the PFI inference fleet. Can enumerate services, read GPU state,
 **start and stop seats**, submit jobs and fetch assets.
 ```
 GET  /api/v1/services              GET  /api/v1/services/{id}/state
 POST /api/v1/services/{id}/start   POST /api/v1/services/{id}/stop
 GET  /api/v1/gpus                  POST /api/v1/jobs
 GET  /api/v1/assets/{id}/blob      GET  /api/v1/presets
 ```
 ⚠ **Liveness is keyed on the compose PROJECT name.** A seat reads OFFLINE unless the
 compose project name equals the catalog's `lifecycle.stack`. Pin `name: <stack>` in
 the compose file.
 ## GPU hosts
 | host | GPUs | notes |
 |---|---|---|
 | fv-ml1 `10.251.50.54` | 4× RTX PRO 6000 Blackwell Max-Q, 96 GB ea (391 GB) | primary; native FP8/NVFP4 |
 | ana-ml2 `10.250.50.54` | see live inventory | |
 | irv-ml1 `irv-ml1.nh3.internal` | RTX 3090 + RTX A6000 (72 GB) | **Ampere sm_86: no native FP8/NVFP4** |
 Every seat is FastAPI — **`GET /openapi.json` on any of them is the live contract.**
 Do not transcribe endpoint shapes; fetch them.
 ## Traps
 - **Idle VRAM is reserved, not waste.** Single-user dev fleet. Never propose raising
  `gpu-memory-utilization` to fill free VRAM.
 - **vLLM temp 0 is not deterministic under concurrency** — ~3.12pp replicate spread at
  `jobs=8`. Any eval delta under ~3.1pp is instrument floor, not signal. Only
  `--jobs 1` shrinks it.
 - **Never alias a model under a stale name.** Serve under the TRUE served-name only;
  let the old name 404 and notify consumers. No silent substitution.
 - **Ollama is banned fleet-wide.** Never stand one up; tear down any found. Use
  llama-swap or vLLM.
 - **`docker compose up -d` recreates the WHOLE stack** when a shared `.env` changes.
  Target one service: `up -d <service>`.
@@ -0,0 +1,52 @@
 # LiteLLM gateway — the fleet's OpenAI-compatible front door
 **Base URL:** `http://10.250.50.70:4000/v1` (ana-docker) · **Logs UI:** `:4000/ui`
 Fronts the ana-ml2 / fv-ml1 / irv-ml1 vLLM seats plus paid vendor passthroughs.
 ## ⚠ Never hardcode the model roster — query it
 ```sh
 curl -s -H "Authorization: Bearer $KEY" http://10.250.50.70:4000/v1/models \
  | python3 -c "import sys,json;[print(m['id']) for m in json.load(sys.stdin)['data']]"
 ```
 The roster drifts constantly (40 entries as of 2026-09-18; a doc that names them goes
 stale within weeks). Any list written down here would be a second copy that rots.
 ## Keys
 The gateway **401s without a virtual API key**. Three tiers:
 | key | where | reach |
 |---|---|---|
 | shared all-agents (`all-agents-local`) | `secret get litellm/all-agents-shared-key`, also inline in `~/.claude/CLAUDE.md` § Global tools | **every** model, paid passthroughs included |
 | infra-ops admin | `~/.config/litellm/infra-ops-key` (0600) | admin operations |
 | project-scoped | request from infra-ops via althing | isolated, auditable spend |
 ⚠ **The shared key is NOT free-local-only.** It reaches paid GLM/z.ai and Kimi
 passthroughs, and **a paid call on it spends real vendor credits shared across every
 session and project holding it.** There is no isolation. For anything where "shows up
 on someone else's tab" matters, get a project-scoped key.
 ## Stable aliases worth knowing
 `summarizer` / `summarizer-large` / `classifier` — the always-available local seat for
 summarization, naming/labeling, classification and triage. Essentially free compute;
 reach for it before spending premium API tokens on work a small local model handles.
 Embeddings and reranking (`qwen3-embedding`, `reranker`) are local and free too —
 served by TEI on esh-ml1 since 2026-09-25 (`qwen3-reranker` was retired 2026-08-20). Anything named `*-frontier`, `glm-*` or `kimi-*` is **PAID**.
 ## Traps
 - **LiteLLM mutates shared deployment config in-place.** Per-request sampler params
  bleed into the SHARED `litellm_params` for that deployment. Fix: give each variant a
  distinct served-model-name rather than reusing one with different params.
 - **Retiring a model orphans scoped keys.** A stale allowlist produces a silent
  per-endpoint 403. Audit `/key/list` + `/key/info` on every repoint.
 - **The `:4000/ui` playground cannot test vLLM backends** — it sends `tools: []`, which
  vLLM rejects with a 400. Use a chat frontend (Open WebUI, Gateway Chat) instead.
 - **"Random model calls" in the logs are usually AUTH FAILURES** — empty `api_key` or a
  missing `Bearer ` prefix, showing as zero-spend 401s.
 - **Kimi models 400 on any `temperature` other than 1** — omit the field entirely
  rather than sending 0. They answer in `content` with chain-of-thought in
  `reasoning_content`.
@@ -0,0 +1,32 @@
 # Observability — where to look when something is wrong
 | surface | URL | what it answers |
 |---|---|---|
 | **Homepage** | `http://10.0.50.45:5100` | the curated fleet service map; `/api/services` is machine-readable |
 | **Beszel** | `http://10.250.50.70:8090` | host + container up/down, resource history; alerts land in althing |
 | **Dozzle** | `http://10.250.50.70:8088` | container logs, ana-docker + ana-ml2 |
 | **Uptime Kuma** | `http://10.0.50.45:3001` | ESH-side service monitoring |
 | **Backrest** | `http://10.250.50.70:9898` | restic snapshot browse / restore |
 | ~~task-board~~ | — | **mothballed 2026-09-24**; what a session is working on now lives on the High Seat (`http://10.100.10.50:8770`) |
 | Dockge (per host) | `<host>:5001` | compose UI on ana-docker, ana-ml2, esh-docker-vm, irv-ml1, nh3-docker |
 | PBS | `10.250.50.90:8007` (primary) · `10.100.50.90:8007` (DR) | VM/LXC image backups |
 **Homepage's `/api/services` is the single best machine-readable inventory of the
 fleet** and it is maintained as a side effect of normal work. Query it rather than
 keeping a list.
 ## Diagnostic reflexes
 - **All hosts at one site flapping together = a WAN/mesh event, not host failure.**
  Check uptimes first; they will show weeks. Then read the site's subnet-router
  tailscaled log.
 - **ana-nas is a SPOF** for postgres, rest-server-ana, PBS-ANA and cross-site restic.
  Check it first on any multi-failure.
 - **After a power loss, crowdsec fails closed**: it exits 255 and the Traefik bouncer
  403s ALL HTTP. Restart crowdsec THEN traefik. This is a recurring recovery step.
 - **A filter can silently narrow your window.** `docker logs --since HH:MM` returning 0
  while the line demonstrably exists, a `find -maxdepth` undercount, a character class
  that omits digits — re-run unfiltered before concluding absence. **Absence of a
  signal is not a safe reading of it.**
 - **Timestamp every state report.** An untimestamped claim is a claim about "now" and
  has manufactured phantom events in peer artifacts before.
@@ -0,0 +1,21 @@
 # Playwright + headless Chromium
 Installed **box-wide on nh3-dev**. Use for anything needing a real browser engine:
 true DOM/SVG layout geometry, screenshots, headless rendering, browser-based tests or
 scraping.
 - System shared-libs installed via apt (`playwright install-deps`)
 - Browser binaries in shared **`/opt/ms-playwright`** (root-owned, world-readable)
 - `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright` wired globally
  (`/etc/environment` + `/etc/profile.d/`)
 A project needs only the `playwright` module — `npm i playwright`. **The browser
 download is skippable and there is no per-project `playwright install`**; it resolves
 the shared binary.
 Add a browser or bump versions:
 ```sh
 ssh infra-ops@10.100.10.50 \
  'sudo env PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright npx -y playwright install <browser>'
 ```
@@ -0,0 +1,35 @@
 # secret — Vaultwarden credential store
 **CLI:** `secret` (on PATH on nh3-dev; real path
 `~/development/eshpfi-management/services/secrets-broker/secret`)
 **Backend:** `vaultwarden.phasefinal.com`, the `infra-ops` org.
 **Auth bootstrap:** `~/.config/secrets-broker/bootstrap.env` (mode 0600).
 ```sh
 secret list [--prefix P]
 secret get  <name> [--field F] [--file OUT]      # the only command that prints a value
 secret put  <name> (--file P | --stdin) [--folder C] [--field k=v]
 secret backfill [--dry-run]                      # scan THIS box's env.sh/.env/~/.config creds
 ```
 Items are hostname-namespaced (`<host>/…`). Small text goes to the item note, small
 binary to a base64 hidden field, anything over 6000 B becomes a bw attachment.
 ## Standing directive (operator, 2026-08-11)
 **The vault is the credential source of truth — store durable secrets in it AND pull
 the creds you need FROM it.** Any token, API key, cert or env-secret that is not
 ephemeral and does not belong in git should be `secret put` into the vault so it
 survives and is findable, never left single-copy on a box. When a task needs a
 credential, `secret get` it rather than reading an on-disk copy where practical.
 The one exception is `bootstrap.env` itself — it unlocks the vault, so it cannot live
 inside it. Keep it 0600.
 ## Traps
 - **Not a fleet service, no daemon.** Each dev box duplicates this stack against its
  own local secrets. Setup steps in the service README.
 - **`secret get` output shape varies by item.** Some items store a bare value, some
  store `key: <value>` fields. A blind `grep '^key:'` returns empty on a bare-value
  item — read the raw output first, then parse.
@@ -0,0 +1,41 @@
 # Speech — TTS and ASR endpoints
 All on irv-ml1 unless noted. Every one is FastAPI: **`GET /openapi.json` is the live
 contract**, `/health` is the liveness probe.
 ## TTS
 | service | endpoint | what it is |
 |---|---|---|
 | **Breeze-TTS-2** | `http://irv-ml1.nh3.internal:8204` | clone + direction, **fleet primary** |
 | TTS fleet gateway | `http://irv-ml1.nh3.internal:8198` | switchboard; engines sit behind it |
 | Chatterbox Fast | `http://irv-ml1.nh3.internal:8197` | streaming, sub-second first-audio — the live-chat front end |
 | dots.tts | `http://irv-ml1.nh3.internal:8202` | continuous-AR 48 kHz zero-shot clone |
 | Kokoro | `http://irv-ml1.nh3.internal:8193` | low-latency English, streaming |
 **Breeze** (`/v1/voices`, `/v1/voices/detail`, `/v1/audio/speech`, `/v1/audio/design`)
 — `/v1/voices` matches dots byte-for-byte so they are drop-in swappable;
 `/v1/voices/detail` exposes the registry with its directions visible.
 **Gateway** (`/v1/audio/speech`, `/v1/voices`) is also reachable through LiteLLM as
 the `ext-tts` alias. Voices: donut / emmie / glados / miranda / sindra plus emotion
 variants, or the OpenAI aliases nova / alloy. ⚠ **An unknown voice 404s and can trip
 the router cooldown** — list voices first.
 ## ASR
 | service | endpoint | what it is |
 |---|---|---|
 | Speaches | `http://irv-ml1.nh3.internal:8204` | OpenAI-compatible faster-whisper, `verbose_json` |
 | Parakeet | `http://10.251.50.54:8300` | Parakeet-TDT via sherpa-onnx (fv-ml1 GPU 3) |
 | Scriberr | `http://10.250.50.54:8080` | transcription + diarization UI (ana-ml2 GPU 1) |
 ## Traps
 - **OmniVoice-style diffusion TTS has a TTFA floor** — roughly fixed per-call cost,
  ~0.7 s first-audio, real RTF 2.8–5.6× (not the advertised 40×). Chatterbox-fast
  stays the live-chat front end for that reason.
 - **Never expose repetition-penalty dials on Zonos-lineage engines** — it is a silence
  footgun.
 - Several engines listed on Homepage under *AI - Dormant* are down by design and take
  ~90 s to respin. Check `/health` before assuming a service is broken.
@@ -0,0 +1,52 @@
 # Fleet SSH — the infra-ops identity and host map
 ## Identity
 `ssh infra-ops@<host>` — a dedicated NOPASSWD-sudo user on PFI-owned boxes, key
 `~/.ssh/infra-ops_ed25519`. Covers **every** fleet host including all four Proxmox
 hypervisors, corviduo-dev and esh-docker-vm.
 **Excludes the SureFire tenant hosts only** (`sf-*`, `sfsrv-ana`).
 ⚠ **Test with `sudo -n -v` first** rather than assuming. Not every box has been
 bootstrapped — `pbs-nh3` (10.100.50.90), for instance, has no infra-ops user and is
 reachable only as `lkraven@`.
 ⚠ **Remote sudo needs `ssh -t`.** Read-only probes can use plain ssh.
 ## Sites and hosts
 | site | subnet | gateway |
 |---|---|---|
 | Anaheim (colo) | `10.250.0.0/16` | FortiGate `10.250.0.1` |
 | NH3 | `10.100.0.0/16` | UDM-SE `10.100.0.1` |
 | ESH home lab | `10.0.50.0/24` (`esteban.net`) | UDM-PM `10.0.0.1` |
 | Fountain Valley | `10.251.0.0/16` | |
 | Irvine | via headscale mesh | |
 Full host table with roles: `~/development/eshpfi-management/CLAUDE.md` § Servers.
 Live per-host snapshots: `servers/<name>/system-details.txt`.
 Internal DNS names are `<host>.<site>.internal`, source of truth `dns/internal.yaml`.
 ## Mesh
 Headscale is primary since 2026-09-06 (IPsec tunnels dormant). Subnet routers are
 `nh3-scale` / `ana-scale` / `esh-scale` at `100.64.0.1` / `.3` / `.2`.
 ⚠ **Disable a site-to-site tunnel BEFORE adding a mesh route for the same prefix** or
 they fight.
 ⚠ **`accept-routes` needs a return path FIRST.** On a dev box it can black-hole its own
 LAN (route table 52 beats main). Gateway routes or SNAT before, never on your own box.
 ## Traps
 - **SureFire tenant hosts are client property** under the hosting agreement. Coordinate
  before touching anything that affects data. This is a contractual posture, not a
  security one.
 - **Confirm a reboot by observing DOWN**, then return. Poll for disappearance and
  cross-check the boot timestamp — "never rebooted" and "rebooted very fast" look
  identical otherwise.
 - **`chroot` rbind needs `--make-rslave`.** Without it, `umount -R` unmounts the LIVE
  host's cgroup/devpts, which looks exactly like disk failure.
 - **`pct snapshot` exits 0 when it refuses** (bind-mounted guest). Use `zfs snapshot`
  and read it back.
@@ -182,6 +182,23 @@ These caught us once; don't let them catch you twice.
 - **`:22` on `10.250.50.70` is ana-docker's HOST sshd, not gitea.** A
  gitea deploy key there returns `Permission denied (publickey)` — gitea's
  git-SSH is the container port `:222`. (HTTP/clone-over-HTTPS is `:3000`.)
 - ⭐ **Override the NAME, not each repo's remote — and on `nh3-dev` this is
  already done (2026-09-17).** `~/.ssh/config` there carries a
  `Host gitea.phasefinal.com` block pointing at `10.250.50.70:222`, so every
  repo on the box is routed internally whatever its remote URL says, including
  a fresh clone that copied the public URL out of a README. Rewriting remotes
  one by one fixes only the repos someone remembered to audit: brokkr-smithy,
  sleipnir, Galdrabok and kvasir were all still on the public name when
  brokkr-smithy-dev flagged it, and brokkr-smithy is pushed several times a
  week — a recurring ban trigger, not a dormant one.
  **Check before assuming a host inherits it; the alias is per-host:**
  ```bash
  ssh -G git@gitea.phasefinal.com | grep -E '^(hostname|port) '   # want 10.250.50.70 / 222
  ```
  ⚠ Verify both routes authenticate as the same user with the same key BEFORE
  adding the alias, then prove it with a real `git ls-remote origin HEAD` — a
  routing change that is only inspected, not exercised, is the classic
  looks-right-and-is-broken.
 ### Workflow
@@ -212,6 +229,9 @@ These caught us once; don't let them catch you twice.
 | What's currently open / in-flight? | `STATUS.md` |
 | What do I need to know that isn't in current code? | `MEMORY.md` + the `.md` files it links |
 | Why did we do X? | Check memory files + `STATUS.md` session milestones at the bottom |
 | **I need to quantize / requant a model** | **`docs/pfi/model-quantization-playbook.md` — READ IT FIRST.** Consolidated hard-won lessons (scheme choice, the recurring landmines, the acceptance gate, superseded claims). Per-model runbooks are worked examples, not the general guide. |
 | What sampler/serve settings for model X? | `docs/pfi/recommended-model-settings.md` |
 | Which model is on which GPU seat? | `servers/ana-ml2/README.md` + `stacks/<seat>/README.md` |
 ## Inventory + automation scripts
@@ -0,0 +1,177 @@
 # Abliteration recipe — Qwen3.8-27B (MTP-aware, vision-preserving)
 Captured 2026-08-19 from
 [`RobinsonLabs/Qwen3.8-27B-abliterated`](https://huggingface.co/RobinsonLabs/Qwen3.8-27B-abliterated)
 (base pinned at commit `1d4bf0f2`, Apache-2.0). It is the cleanest public
 abliteration of the Qwen3.8-27B architecture we have found — the base family our
 **gen seat** runs (see auto-memory `reference_abliteration_mtp_lessons`,
 `reference_gen_qwopus_122b` lineage). This is a **reference recipe**, not a
 deployed artifact: the value is the method, and specifically the two things it
 gets right that most abliterations of this architecture get wrong.
 Companion: `docs/pfi/model-quantization-playbook.md` owns the *quant* half of the
 pipeline; this owns the *abliteration* half. When an abliteration lesson is
 model-agnostic it lands here; when it is specific to one checkpoint's tensor
 names it stays with that checkpoint.
 ## Why this architecture is the hard case
 Qwen3.8-27B (`model_type: qwen3_5`, `Qwen3_5ForConditionalGeneration`) is not a
 plain transformer. Abliterating it correctly means touching three surfaces a
 naïve layer-loop misses:
 1. **A hybrid attention trunk.** 64 language layers, most using **DeltaNet
   linear attention** (`linear_attn.out_proj`), with **full attention at every
   4th layer** (`self_attn.o_proj`). A refusal-direction orthogonalization that
   only knows about `self_attn.o_proj` edits 16 of 64 layers and silently leaves
   the model 75% un-abliterated on the attention path.
 2. **A multi-token-prediction (MTP) head** (`mtp.layers.0`) used for
   speculative decode. The generic 64-layer loop never reaches it.
 3. **A vision tower** (`model.visual.*`, 333 tensors) that must survive
   untouched or the model stops being multimodal.
 ## The two things this recipe gets right
 ### 1. The MTP head is abliterated *in-band*
 This is the finding that matters most to us, because our gen seat gates on MTP
 acceptance ≳40% (`reference_abliteration_mtp_lessons`).
 Most abliterations orthogonalize the trunk and leave `mtp.layers.0` untouched.
 The consequence is subtle and nasty: **the draft head keeps proposing
 refusal-prefix tokens that the abliterated trunk then rejects, so speculative
 acceptance collapses on exactly the prompts abliteration exists to fix.** You
 get a model that is abliterated *and* slow, and the slowness is worst precisely
 where you wanted the behaviour change.
 The fix is to orthogonalize the MTP block's **two residual-write matrices**
 (`self_attn.o_proj`, `mlp.down_proj`) with the *same* refusal direction as the
 trunk. The MTP **glue** — `mtp.fc`, `mtp.norm`, `mtp.pre_fc_norm_*` — is left
 alone, because those are norms and an input projection, **not** residual
 writers. Editing them would corrupt the draft path without removing any refusal.
 ### 2. The vision tower is preserved byte-identical
 All 333 `model.visual.*` tensors pass through unmodified — verified by direct
 tensor diff (max delta `0.000000`), not asserted. An `mmproj` is published so
 the vision half is actually usable, not just nominally intact.
 ## The edit set (131 tensors)
 Single-direction weight orthogonalization, Arditi et al. style, applied to every
 matrix that writes the residual stream:
 | scope | tensor | count |
 |---|---|---|
 | `model.language_model.layers.*` (64) | `mlp.down_proj` | 64 |
 | | `linear_attn.out_proj` (DeltaNet) | 48 |
 | | `self_attn.o_proj` (full-attn, interval 4) | 16 |
 | `mtp.layers.0` | `o_proj` + `down_proj` | 2 |
 | `model.language_model` | `embed_tokens` | 1 |
 | **edited total** | | **131** |
 | `model.visual.*` | preserved byte-identical | 333 |
 **Hard coverage gate before writing a byte:**
 `o_proj(16) + linear_out(48) == 64 == num_hidden_layers`. This is the check that
 catches a partial tensor-name match — the failure mode that otherwise ships a
 quietly half-abliterated model that passes a smoke test and fails in the field.
 Adopt this gate in any re-derivation.
 ## Two calibration traps specific to this base
 ### Refusal-direction selection
 The direction was captured **twice**, from two structurally different
 chat-template renderings:
 - one with `enable_thinking=false`
 - one with thinking on at `reasoning_effort=xhigh` (which injects an extra
  system block and shifts every token position)
 The two agree at **|cos| 0.96–0.99 across layers 18–45, peaking 0.9925 at layer
 26** — the layer used. Two different prompt distributions converging on the same
 vector is the evidence that the direction encodes *refusal semantics* rather than
 *template formatting*. A single-template capture cannot distinguish the two.
 > ⚠️ **Two-template agreement is a bad LAYER SELECTOR on a heavily-merged base —
 > use harmful/harmless SEPARATION instead (added 2026-08-20).** On RobinsonLabs'
 > stock Qwen3.8 the agreement was 0.99 and picking its peak was fine. On DavidAU's
 > Cold-Fusion GAIN merge the same metric tops out at **0.62**, and its argmax
 > (layer 18) is the layer with the **worst** refusal separation in the window
 > (Cohen's d 5.51 vs 9.89 at the peak) — abliterating there was a measured
 > behavioral **no-op**. The reason: the two renderings end in different generative
 > modes (`</think>\n\n` = about to answer vs `<think>\n` = about to reason), so
 > `|cos|` scores refusal *plus* mode, and on a merge the mode term dominates. The
 > selector that actually predicts efficacy is **how cleanly the direction splits
 > harmful from harmless prompt activations** (Cohen's d / AUC), gated on the sink
 > screen (separation and sink-energy both rise with depth, so the raw peak is
 > usually sink-dominated). On Cold-Fusion this picked **layer 35** (d 9.35, AUC
 > 0.9997, sink 0.094%) and the abliteration worked. Keep agreement as a
 > diagnostic; do not select on it. See
 > `services/coldfusion-abliteration/README.md`.
 ### The attention-sink dimension — the one that bricks the model
 **Qwen3.8-27B's massive-activation dimension is `3994`.** It carries 19–21% of
 the direction's energy at layers 1–3, and orthogonalizing it out of every
 residual writer produces a model that **loads, runs, and emits garbage.** Layer
 26 was chosen partly because it carries only **0.06%** of its energy in dim 3994.
 **Any re-derivation MUST screen for this.** It is the single most likely way to
 waste a GPU afternoon on this architecture and mistake the result for a failed
 abliteration when it is actually an attention-sink blowout.
 ## Measured behaviour (their numbers, for reference)
 Base vs abliterated, same session/harness/prompts, both at Q4_K_M:
 | prompt set | base | abliterated |
 |---|---|---|
 | in-distribution (24, from capture set) | 96% (23/24) | **8%** (2/24) |
 | held-out (40, disjoint, overlap=0) | 100% (40/40) | **8%** (3/40) |
 Capability axes (reasoning / code / math / factual / instruction-following /
 creative-RP coherence): **no regression on any axis.** Held-out train/test split
 was 416/104 with overlap 0, so the 8% held-out figure is generalization, not a
 reshuffle of calibration prompts.
 **Note the design point:** 8% is deliberate. Harm guardrails are **retained** —
 self-harm prompts still redirect (988) rather than comply. This is a
 *creative-content* abliteration shipped "at the ceiling where capability and
 guardrails both survive," explicitly **not** a jailbreak. That makes it a
 **milder** abliteration than our incumbent gen seat (`absolute-heresy`, ~2%
 author refusals, aggressive Heretic). Adopt the *method* here; the *ceiling* is a
 separate call.
 ## How this maps onto our pipeline
 The recipe is a drop-in for the front half of the House quant pipeline:
 1. Pull bf16 master to NFS (verify repo id first —
   `reference_verify_hf_repo_ids_before_pull`).
 2. **Baseline MTP acceptance on bf16 before any surgery** — the standing rule.
 3. Orthogonalize per the edit set above; enforce the coverage gate; screen dim
   3994; gate the result on **MTP acceptance ≳40%, not KL** (KL misled us once —
   `reference_abliteration_mtp_lessons`).
 4. Verify vision byte-identical, refusals down, PPL not blown, no catatonia.
   **Measure first-token KL as a *fidelity* number** (`kl_divergence.py`,
   bf16-vs-bf16, held-out prompts) — it does not replace the acceptance gate in
   step 3, and it is not a pass/fail on its own. Report it **split by prompt
   class**: a single averaged KL over a mixed corpus is close to meaningless,
   because the metric is supposed to be large on harmful prompts and small on
   benign ones. The ratio is the interesting quantity. Cold-Fusion L35 measured
   **0.0211 median harmless / 0.5996 median harmful = 28.4× selectivity**, on a
   stack whose self-KL noise floor is exactly 0.0.
 5. NVFP4-quantize in-house (mixed W4A4 + FP8-attn/lm_head —
   `model-quantization-playbook.md`). **Foot-gun the GGUF card itself flags:
   the imatrix does not cover the MTP block** — so a GGUF requant path leaves
   MTP uncalibrated. Our NVFP4 path must calibrate it explicitly.
 ## Provenance
 - Recipe: RobinsonLabs README, fetched verbatim 2026-08-19. Authored with their
  "ModelForge" manufacturing system-of-record (not public).
 - Method lineage: Arditi et al., single-direction refusal orthogonalization.
 - Our prior art: `reference_abliteration_mtp_lessons` (modest abliteration
  preserves MTP; test MTP on bf16 first; gate on acceptance not KL), and the
  gen-seat quant recipe in `model-quantization-playbook.md`.
@@ -0,0 +1,396 @@
 # Author-voice LoRA regime on pfi-gx10 — training-side prep
 _Written 2026-09-09 22:45 PT, revised 23:05 PT. Status: **PREP. Nothing is
 training and nothing is queued.** Both operator decisions from the first draft are
 now **SETTLED** (§6): carrier family is the **dense `Qwen3` line**, and the
 intermediate checkpoints are purged._
 The research target is **brokkr-smithy R49** (`research/R49-author-voice-adapters/`),
 whose seed is the operator's **BabyBronte** design doc
 (`R49/reference/babybronte-design-doc.md`, 2026-08-25). R49 owns the hypotheses,
 the instruments and the adjudication. **This document owns the other half — the
 box, the stack, the corpus staging, the trainer, the launcher, and the
 wall-clock** — the same split that ran ERP-seat runs 3c through 7 on this box.
 Read `R49/target.md` before touching the corpus design; several attractive ideas
 are already settled *against* there and re-proposing them is the failure mode
 this file exists to prevent.
 ---
 ## 1. What the regime is, in one paragraph
 A voice is a **LoRA adapter on a small non-instruct base model**, trained on that
 author's real prose, steered at serving time by a terse beat line rather than a
 prompt. The completion in every training pair is 100% authentic author text; the
 only synthetic token in the corpus is the beat line, and beat lines are
 **loss-masked**, so the distribution the adapter learns to *emit* is the author's
 and the style ceiling is the author's own. Names and places are substituted
 **deterministically** — seeded RNG over a curated 23,398-name dictionary, never
 an LLM — which is what lets one work become 5–8 training copies without teaching
 plot memorisation.
 **The value being bought is marginal cost per voice, not inference latency.**
 That framing is the operator's and it is load-bearing: the anchor for
 "expensive" is ERP run 2 at ~7 h for a single 26B-A4B tune on a harness whose
 audit found a blocking-mask defect, a vision tower a leaf-name regex would have
 trained on text, and 128 experts fused per layer. A dense sub-2B carrier has
 none of those failure modes, and at this size the **methodology floor is finally
 cheaper than the shortcut** — two seeds per arm and a re-run after every change
 are routine rather than unaffordable.
 ## 2. The box, and what is already staged
 `pfi-gx10` (10.100.50.60) — ASUS Ascent GX10, NVIDIA **GB10**, `sm_121`,
 aarch64, **121 GB unified** memory, 916 GB NVMe, 470 GB free. Operator ruling
 2026-09-09: **experimental box, primarily for training, no serving seat.** Its
 GPU is idle.
 Training stack already present and current at `/home/infra-ops/ml/.venv`:
 | | |
 |---|---|
 | torch | `2.14.0+cu130`, `torch.cuda.get_device_capability() == (12, 1)` |
 | transformers | 5.16.1 (loads `Qwen3_5Config` natively) |
 | peft / trl / accelerate | 0.20.0 / 1.12.0 / 1.14.0 |
 | datasets / bitsandbytes | 5.0.1 / 0.50.2 |
 | **absent** | `mamba_ssm`, `causal_conv1d`, `fla`, `flash_attn`, `kernels` (triton 3.8.0 present) |
 Staged on local NVMe under `/home/infra-ops/carriers/` (pulled 2026-09-09,
 existence API-verified against the HF registry first, with a phantom repo run as
 the negative control):
    CHOSEN — dense Qwen3          held, not chosen — hybrid Qwen3.5
    Qwen3-0.6B-Base   1.2G        Qwen3.5-0.8B-Base   1.7G
    Qwen3-1.7B-Base   3.3G        Qwen3.5-2B-Base     4.3G
    Qwen3-4B-Base     ~8G         Qwen3.5-4B-Base     8.8G
 Probes live at `scripts/training-probes/{probe_carrier.py,bench_lora_step.py}`
 with raw output in `bench-lora-step-gx10-2026-09-09.jsonl`, so every number below
 can be re-derived rather than taken on faith.
 ## 3. What the carriers actually are — measured, not read off the model card
 R49 H02 names the `Qwen3.5` trio. Probing the checkpoints rather than the config
 found three things worth knowing before writing a recipe.
 **They ship a vision tower and an MTP head.** `model.visual.*` is 153 tensors on
 the 0.8B and 297 on the 2B — `attn.qkv`, `attn.proj`, `mlp.linear_fc1/2`, all
 `nn.Linear` leaves that `target_modules="all-linear"` would attach LoRA to and
 then train on pure text. This is the *same* defect the ERP harness audit caught
 on gemma-4. **Mitigation is free:** loading through `AutoModelForCausalLM`
 returns `Qwen3_5ForCausalLM` with the vision tower and MTP head dropped
 entirely — 0.752 B of text model, module paths `model.layers.N.*`.
 ⚠ **That mitigation creates a serving trap.** vLLM will load the full
 `Qwen3_5ForConditionalGeneration`, where the same weights live at
 `model.language_model.layers.N.*`. An adapter trained against the CausalLM
 prefix may not bind. **Pre-flight:** load the finished adapter in the serving
 path and confirm a sampled target tensor actually changed — the same silent-no-op
 check the ERP merge step already uses.
 **Three quarters of the layers are not attention.** `layer_types` is 3×
 `linear_attention` + 1× `full_attention` repeating: 18 SSM / 6 attention at
 0.8B and 2B, 24 / 8 at 4B. The SSM blocks carry `conv1d`, `A_log`, `dt_bias` and
 five Linear projections; `mamba_ssm_dtype` is `float32`.
 | region | 0.8B | 2B | 4B | LoRA-able leaves |
 |---|---|---|---|---|
 | MLP | 35.1% | 48.1% | 53.9% | `gate_proj`, `up_proj`, `down_proj` |
 | embeddings (tied) | 33.8% | 27.0% | 15.1% | `lm_head` — exclude |
 | linear-attn (SSM) | 25.2% | 20.1% | 24.0% | `in_proj_{qkv,a,b,z}`, `out_proj` |
 | full attention | 5.9% | 4.7% | 7.0% | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
 | **total** | **0.752 B** | **1.882 B** | **4.206 B** | |
 The conventional `q,k,v,o` + MLP recipe therefore covers **41%** of the 0.8B and
 leaves the SSM stack untouched in 18 of 24 layers. Adding the SSM Linears takes
 coverage to 66%. At this scale that is a cheap ablation, not an agonising choice.
 ⚠ **Packing across document boundaries is unsafe on this architecture.** An SSM
 layer carries recurrent state along the sequence and an attention mask does not
 reset it, so two renamed copies packed into one 8k window can bleed in 18 of 24
 layers — which is precisely the per-copy name-consistency invariant the design
 doc calls sacred. Either one document per sequence, or prove the trainer's
 sequence-boundary signal is honoured by the linear-attn path. Under a dense
 carrier this problem does not exist.
 **The tied embedding is a third of the small carrier.** vocab 248,320 × hidden
 1024 = 254 M of the 0.752 B. The transformer body being tested at the small end
 is ~0.50 B, which matters when reporting "the carrier floor".
 ## 4. Throughput — and the newest carrier is the slow one
 One forward+backward+AdamW microbatch, LoRA r=32/α=64 on `q,k,v,o` + MLP,
 bf16, `sdpa`, gradient checkpointing on, seq 4096, on gx10's GB10. n=10
 measured after 3 warmup steps; median reported with the full spread.
 | carrier | architecture | params | s/step | tok/s | peak | spread |
 |---|---|---|---|---|---|---|
 | `Qwen3.5-0.8B-Base` | hybrid, 18 SSM / 6 attn | 0.765 B | 7.581 | **540** | 15.1 GiB | 2.6% |
 | `Qwen3.5-0.8B-Base` (no grad-ckpt) | " | 0.765 B | 6.364 | 644 | 38.9 GiB | 1.5% |
 | `Qwen3-0.6B-Base` | dense | 0.616 B | 1.707 | **2,399** | 9.8 GiB | 0.6% |
 | `Qwen3-1.7B-Base` | dense | 1.755 B | 2.895 | **1,415** | 12.2 GiB | 0.8% |
 | `Qwen3-4B-Base` | dense | 4.089 B | 5.714 | **717** | 17.2 GiB | 0.3% |
 | `Qwen3-1.7B-Base`, batch 4 | dense | 1.755 B | 11.387 | 1,439 | 38.0 GiB | 0.6% |
 | `Qwen3.5-0.8B-Base`, batch 4 | hybrid | 0.765 B | 30.030 | 546 | 55.5 GiB | 0.7% |
 **The dense 1.755 B carrier trains 2.6× faster than the hybrid 0.765 B one** — on
 2.3× the parameters, with *more* LoRA modules adapted (196 vs 96, because dense
 has real attention in every layer). Per parameter the dense path is ~6× more
 efficient. Spreads of 0.6–2.6% across n=10 put the instrument's noise an order of
 magnitude below the effect, so this is not variance.
 The cause is almost certainly that **no fused linear-attention kernel is
 installed** (§2) so the SSM path runs a reference implementation. Grad
 checkpointing is *not* the culprit — turning it off recovers only 19% and costs
 2.6× the memory, so leave it on. Batching is not the lever for either family: 1,415 → 1,439
 tok/s dense and 540 → 546 tok/s hybrid from batch 1 to 4. **Both architectures
 are already at this box's roofline at batch 1**, which is a bandwidth story
 (GB10's unified LPDDR5X against an RTX PRO 6000's ~6.6× higher figure) — and it
 means the 2.6× gap is the kernel path, not a batching artefact.
 **What that does to the regime's premise.** Projecting a Brontë-scale corpus
 (~1 M words ≈ 1.3 M tokens, × 6 rename copies, 3 epochs ≈ 23 M tokens):
 | carrier | projected wall-clock per voice |
 |---|---|
 | `Qwen3-0.6B-Base` dense | **2.7 h** |
 | `Qwen3-1.7B-Base` dense | **4.6 h** |
 | `Qwen3-4B-Base` dense | **9.1 h** |
 | `Qwen3.5-0.8B-Base` hybrid | **12 h** |
 The sharpest way to put it: **the dense 4.089 B carrier still trains 33% faster
 than the hybrid 0.765 B one**, on 5.3× the parameters. The full three-arm dense
 sweep at two seeds each is ~33 h of GPU — about a day and a half for the whole
 H02 carrier question, and ~10 h if H03's ~300 k-word corpus floor holds.
 The hybrid carrier would make a per-voice run *longer than the 7 h 26B-A4B tune
 it exists to replace.* At R49 H03's hoped-for corpus floor (~300 k words) the
 dense 1.7B lands near **1.4 h** — a voice per afternoon, which is the regime the
 operator asked for. ⚠ These are projections from a synthetic-token throughput
 harness, not from a completed run; treat them as sizing, and re-measure on the
 first real corpus.
 ## 5. Prep remaining, in order
 1. ~~Carrier family decision~~ — **settled: dense `Qwen3`** (§6a).
 2. ~~Corpus D1~~ — **BUILT AND GREEN 2026-09-10** (§8).
 3. **Re-point the R49 deterministic machinery at Brontë.** The entity detector
   (corpus-level capitalised-vs-lowercase ratio), identity linking, gender
   resolution and the 23,398-name dictionary were all built and hardened against
   a *Yarros* sample. Per-work re-derivation needed: entity map, alphabet, and
   the `UNRESOLVED_BLOCKING` human pass (~20–40 entities per work).
 4. **Beat annotation (D4)** via `gen` inverse-prompting, using F02's hardened
   prompt (banned meta-language, three PD worked examples, ≤20-word gate).
 5. **Trainer.** `erp_sft_harness` is chat-shaped and carries ERP-specific
   eligibility machinery; the author-voice job is plain continuation with a
   masked prefix. Decision: a small purpose-built trainer that *keeps* the
   harness's §4 disciplines (provenance pin, order manifest, truncation report,
   cache key that sees semantic changes, recorded attention backend) rather than
   a fork of its corpus logic.
 6. **Pre-set the decision threshold before collecting data**, wider than the
   measured seed-to-seed spread, per the R49 charter — and run the positive
   control the R49 journal's own lesson demands: confirm the stylometric
   instrument separates real Brontë from unadapted base output *before* it is
   asked to judge an adapter.
 7. **Launcher** modelled on `launch-run-07.sh` — its guards were each bought with
   a past failure (GPU-clear assertion, pidfile not `pgrep -f`, refuse an
   existing log, free-space floor, `setsid` detach).
 Deliberately out of scope here, per R49: the Director/critic loop, style
 arithmetic and the Pelican test, multi-LoRA arsenal serving, the Skaldsong
 integration contract, the modern in-copyright arsenal, inference latency.
 ## 6. Open for the operator
 **(a) Carrier family — SETTLED 2026-09-09, operator: _"use dense qwen3"._** The
 sweep is **`Qwen3-{0.6,1.7,4}B-Base`** — the design doc's own original pin, and
 the family the measurements favour by 2.6–6×. **This overrides R49 H02's stated
 arms**, which name `Qwen3.5-{0.8,2,4}B-Base`; brokkr-smithy owns that file and
 has been told directly.
 What the ruling buys, restated so it is not re-litigated: no vision tower and no
 MTP head to exclude, cross-document packing is safe again, `sdpa`/flash are both
 reachable, and the per-voice wall-clock is 2.7–4.6 h instead of 12 h. What it
 costs: one model generation of base quality. **Reversible** — the three Qwen3.5
 checkpoints stay staged (14.8 GB), and an `fla` install (pure Triton, plausibly
 fine on aarch64) could revive that family as a follow-up experiment rather than a
 prerequisite. If anyone re-opens this, re-run `bench_lora_step.py` first; the
 argument is a measurement, not a preference.
 **Is there a newer small base to prefer instead? No — checked against the HF
 registry 2026-09-09, prompted by the operator asking brokkr-smithy the same
 question.** Three facts, and together they close it:
 1. **No official Qwen3.6 or Qwen3.8 exists below 27B.** The whole `Qwen/` listing
   is `Qwen3.6-{27B, 35B-A3B}` and `Qwen3.8-{27B, 2.4T-A95B, Flash-Next}`.
 2. **Neither family publishes a `-Base` checkpoint at all.** Every Base newer than
   Qwen3 is Qwen3.5 — `Qwen3.5-{0.8B, 2B, 4B, 9B, 35B-A3B}-Base`. Since the
   regime requires a *non-instruct* carrier, that alone rules the 3.6/3.8 lines
   out regardless of size.
 3. **The 27Bs are the same kernel path one size up.** `Qwen3.6-27B` and
   `Qwen3.8-27B` both report `model_type: qwen3_5`,
   `Qwen3_5ForConditionalGeneration`, 64 layers as **16 full + 48
   linear-attention**, vision tower present — the exact shape measured slow above.
 So the newest official small **dense** Qwen base is still the `Qwen3` line, and
 the ruling is not a compromise against a better available option; it is the only
 dense option. (Third-party `Qwen3.8-*-Distill` checkpoints are Qwen3.5 hybrids
 distilled on 3.8 outputs — same kernel path, and unpinned provenance besides.)
 ⚠ **Headroom worth knowing about, though H02 does not need it today:** the dense
 Base line continues past 4B — `Qwen3-8B-Base` and `Qwen3-14B-Base` both exist. H02
 caps at 4B by design ("sub-4B carrier"), and the projections say 4B already costs
 9.1 h a voice, so this is not a proposal. It is insurance: if the 0.6/1.7/4B curve
 has *not* flattened at 4B, the sweep can be extended without changing family.
 (A `Qwen/SAE-Res-*-Base-*` row in a registry search is an interpretability
 sidecar, not a carrier — ignore those.)
 **(b) Intermediate checkpoints — PURGED 2026-09-09 23:00 PT, operator:
 _"purge intermediate checkpoints"._** Seven `checkpoints/` directories deleted
 with literal paths, one `rm` per line, after confirming none was a symlink and
 that every run's final `adapter/` is an independent real directory:
    pfi-gx10   run-03c 11G · run-04 16G · run-05 9.2G · run-06 9.2G   -> 45 GB
    ana-ml2    run-01 12G · run-02 12G · run-03 5.9G                  -> 29 GB
 gx10 419G→374G used (496 GB free); `/tank/erp-tune` 392G→363G, with `zfs list -t
 snapshot` empty so the space is genuinely returned rather than snapshot-held.
 **All eight adapters re-verified by `sha256` after the deletion**, matching the
 values recorded during the mirror.
 **And the merged models followed at 23:08 PT**, operator: _"purge the merged
 models, keep run06 and the v6 quant."_ Eleven merges, literal paths, one `rm` per
 line, after proving what the live seat reads: `docker inspect vllm-erp-seat` shows
 it binds **`/tank/aimodels` only** and its model arg is
 `/tank/aimodels/erp-tune-v6-nvfp4a16`, so `/tank/erp-tune/serve` was never in the
 serving path at all. No container mounts it and no process held a file open under
 it (checked `/proc/*/fd` and `/proc/*/maps`, not `lsof` alone).
    pfi-gx10   merged-run03c · merged-run04 · merged-run05      3 x 49 GiB -> 145 GB
    ana-ml2    merged-final · merged-run02 · merged-run03 ·
               merged-run03-s{025,050,075} · merged-test ·
               nvfp4a16-test                                    8 dirs     -> 354 GB
 All eleven were Gemma-4 merges from the run-1/2/3 era (Aug 24–26); the keeper is
 dated Sep 8 and lives in a different tree, so there was no ambiguity about which
 was which. gx10 382G→237G used (**632 GB free**); `/tank/erp-tune/serve` 354G→8.6M
 with `zfs list -t snapshot` empty, and the pool's raw FREE moved 4.80T→5.30T.
 **Kept, and verified after the deletion:** `gx10:~/erp-tune/serve/merged-run06`
 (49 GiB, 9 files), `/tank/aimodels/erp-tune-v6-bf16` (49 G) and
 `erp-tune-v6-nvfp4a16` (16 G, 9 files incl. `recipe.yaml`), all eight adapters
 sha256-unchanged, and `/tank/erp-tune/serve`'s merge/quant tooling and logs
 (`merge_lora.py`, `quant_nvfp4a16.py`, the dry-run logs, `base-arm`, `lora-scales`
 — 8.6 MB total, the provenance for how every artifact above was built).
 ⚠ **`gx10:~/erp-tune/relaunch-trial-seat.sh` now names a model that is gone.** It
 was not deleted — its flags carry the FlashInfer JIT/PATH trap and the
 gpu-clear/never-pkill notes, each bought with a real failure — but it gained a
 RETIRED banner so nobody hits a confusing missing-model error later.
 **The live seat never bounced.** After the deletion `Pfish-6` answered a real
 completion (`finish_reason: stop`, 5 tokens, correct text), container still
 `Up 4 hours (healthy)`.
 ## 7. Adapter disposition — settled, and made real
 Operator, 2026-09-09: **keep the adapter.** As of 22:30 PT all five
 gx10-resident ERP adapters are mirrored to `ana-ml2:/tank/erp-tune/run-<N>/adapter`,
 matching the layout runs 01–03 already use there, byte-total identical on both
 sides and `sha256` matching on every `adapter_model.safetensors`:
    run-03c  run-04  run-05  run-06  run-07     315 MB each, 8 files each
 `/tank/*` is deliberately **excluded** from ana-ml2's restic sources — terabytes
 of regenerable model weights. A trained adapter is the one thing under there
 upstream cannot hand back, so `configs/restic/ana-ml2/profiles.yaml` now carries
 a single documented carve-out, `/tank/erp-tune/run-*/adapter`, verified by
 `resticprofile --dry-run` to expand to exactly those eight paths and nothing
 else. The nightly 01:00 run picks them up.
 ---
 ## 8. D1 is built — the corpus, and three things it taught
 `scripts/r49-corpus/build_corpus.py --build` and `verify_corpus.py`, corpus staged
 at `gx10:~/r49-corpus/`. **All 11 acceptance checks pass**, including the two
 tokenizer legs run against the pilot carrier itself.
 | | |
 |---|---|
 | works | Jane Eyre (1260) · Villette (9182) · Shirley (30486) · The Professor (1028) |
 | size | **680,291 words · 142 chapters · 950,974 tokens** under the Qwen3 tokenizer (1.40 tok/word) |
 | alphabet | 75 letters, **23 of them non-ASCII** |
 | tokenizer | round-trip lossless, **0 byte-fallback pieces** of 98,860 |
 Catalogue ids were verified against gutenberg.org's own search, not recalled.
 Charlotte only — the Bell poems are co-authored and the Gaskell biography is a
 different hand, so neither belongs in a single-voice corpus.
 **Real denominator, so the projections tighten.** At 6 rename copies × 3 epochs =
 17.1M tokens: **0.6B → 1.98 h**, 1.7B → 3.36 h, 4B → 6.63 h per voice. The pilot
 is a two-hour run.
 ### ⚠ The alphabet finding inverts the Yarros result, and brokkr's re-point depends on it
    ÆÉÊËÔàâäæçèéêëîïôöùûüŒœ
 F02 measured the Yarros corpus at **4 non-ASCII characters in 1.8M letters**
 (0.0002%) and derived an ASCII-fold rule for the name pool. **Charlotte Brontë is
 the opposite case**: she writes French constantly — Villette is set in a
 French-speaking city, Jane Eyre has Adèle, The Professor is set in Brussels — and
 the corpus carries `é` 432 times, `è` 237, `à` 93, `ê` 79, `ô` 48, plus the `œ`
 and `æ` ligatures.
 Under F02's own rule (*the pool's character inventory must be a subset of the
 corpus's*) the Brontë pool **may keep French accents** and **must still exclude**
 the Czech/Latvian/Slovak/Hungarian marks that never appear here. So the fold is
 not global — it is derived per work, which is exactly what the rule said, and
 this is the first corpus where the derivation changes the answer.
 ### Typography was inconsistent, and it was the transcriber, not the author
 | work | quotes | dashes |
 |---|---|---|
 | Jane Eyre | curly | em-dash 2,058 |
 | Villette | curly | em-dash 2,272 |
 | **Shirley** | **straight** 9,115 | **`--` 2,228, zero em-dashes** |
 | The Professor | curly | `--` 964, zero em-dashes |
 Left alone the adapter would learn that this author "sometimes" writes each form
 — a false habit on the exact axis being trained. Normalised **toward what the text
 means**: `--` is a transcription of an em-dash, so it becomes one; straight quotes
 are paired into curly per paragraph.
 ### Two gate defects, both found by running the gate
 1. **`Produced by` matched Brontë's own prose** — *"a chilling effect produced by
   his steady announcement"*, three more like it. A hard rule on a phrase with a
   common innocent sense, manufacturing failures: the same shape as the drift
   detector that fired on the adjective "minor". Fixed by anchoring the
   boilerplate patterns to line start, where Gutenberg credits actually live.
 2. **Asserting open/close quote counts must be equal was wrong.** Nineteenth-century
   convention runs a speech across paragraphs by opening each and closing only the
   last, so a surplus of opens is correct — measured **+46 / +49 / +51** on the
   three works whose quotes were never touched. Replaced with the real error
   signature: *no paragraph may begin with a closing quote*, which convention never
   produces and a bad conversion does. 0 of 14,230 paragraphs.
 A third, mine: **the manifest baked absolute build-machine paths**, so the corpus
 was unreadable the moment it moved to gx10. Paths are relative to the corpus root
 now. It failed loudly rather than silently reading nothing, which is why it was
 cheap.
 ### Next
 D2/D3 — re-point F02's entity detector, identity linking and gender resolution off
 the Yarros sample onto Brontë, with the alphabet above constraining the pool. Then
 D4 annotation, then the pilot.
 **Pilot ruling (operator, 2026-09-10): trial on `Qwen3-0.6B-Base` first and only
 move up if it produces something useful.** So the sweep is not three arms up front
 — it is one ~2 h run, judged, and then a decision. That is the cheap ordering and
 it front-loads the kill signal: if voice does not transfer at 0.6B the question
 becomes *how far up* rather than *whether at all*, and if it does transfer the
 larger arms are a refinement rather than a gamble.
@@ -131,7 +131,7 @@ group memberships) is in
 [`stacks/llama-swap/conf/config.yaml`](../../stacks/llama-swap/conf/config.yaml).
 Don't mirror the table here — it drifted last time and immediately
 went stale. Live HTTP listing:
-`curl http://10.250.50.54:9292/v1/models`.
+`curl http://10.251.50.54:9292/v1/models`.
 **Model Groups**:
@@ -0,0 +1,123 @@
 # Embed/rerank engine bake-off — TEI vs vLLM on esh-ml1 (2026-09-25)
 Prime, 2026-09-25: *"run the TEI bake-off."* Hugging Face **Text Embeddings
 Inference 1.9.4** (`89-1.9.4`, the Ada build) served the same two models as the
 then-vLLM `embed-rerank` stack (vLLM v0.24.0), side by side on the same RTX 2000E
 Ada, as a temporary `tei-bakeoff` stack.
 **Outcome (Prime, 2026-09-25): "TEI is embed/reranker server for esh-ml1 and the
 FLEET in general, in future."** The same day TEI replaced vLLM in
 [`stacks/embed-rerank`](../../stacks/embed-rerank/) on ports 8001/8013. The
 gateway was repointed with esh-ml1 as the sole backend, and fv-ml1's
 `vllm-embed` + `vllm-rerank-a3` seats were retired, freeing ~6.1 GB on fv-ml1
 GPU 1. The bake-off stack was removed; this file is its record.
 ## Results (2026-09-25 0744–0805 PT)
 ### Parity — TEI matches the existing vLLM vectors
 Reference = vLLM on fv-ml1, the engine every existing index was built with.
 306 texts (6 fixed incl. CJK/code/6k-char + 300 *The Stand* paragraphs);
 retrieval = 2,000-paragraph corpus, 50 instruction-format queries.
 | embedding check | result | noise floor / control |
 |---|---|---|
 | cosine TEI vs vLLM-FV | median 0.999925, min 0.999861 | vLLM-FV vs itself: 0.999916 / 0.999796 |
 | overlap@10, TEI index + TEI queries | 0.976 | vLLM-FV rerun 0.986; vLLM-ESH 0.972 |
 | overlap@10, **TEI queries vs the OLD vLLM index** (migration case) | **0.988** | positive control (MRL-256 dims) 0.648 |
 | hit@1 own paragraph | 0.80 | vLLM-FV 0.80 |
 | different-text negative control | cosine median 0.31 | — |
 TEI is deterministic (TEI vs itself: min 0.999995). **Switching engines does not
 require re-embedding existing indexes.**
 | rerank check (100 queries × 20 docs, 9 same-chapter distractors) | TEI vs vLLM-FV | vLLM-FV vs itself |
 |---|---|---|
 | source paragraph ranked #1 | 0.95 (same as vLLM) | 0.95 |
 | top-1 agreement | 1.00 | 1.00 |
 | top-3 set agreement | 1.00 | 1.00 |
 | top-5 exact order | 0.98 | 1.00 |
 | order among docs scoring > 0.05 | 1.00 (n=37) | 1.00 |
 | full 20-doc order | **0.55** | 0.99 |
 | max score difference | **0.019** (p99 0.0034) | 0.0016 |
 Every decision that matters agrees; the differences are shuffles among
 near-zero-scoring tail documents. ⚠ A consumer that **thresholds** on the rerank
 score could see a borderline document flip (scores move up to ~0.02). An
 earlier 30-query run with random distractors had one top-1 disagreement (29/30);
 across both runs, 129/130.
 ### Speed — TEI is NOT faster on this card
 On-box, 2 interleaved runs × 3 reps each, medians:
 | workload | vLLM | TEI |
 |---|---|---|
 | embed 1 short query, p50 | ~9.1 ms | **~6.9 ms** |
 | embed 1 × ~512 tok, p50 | ~23 ms (bimodal 10–25) | ~28 ms |
 | embed 1 × ~2k tok, p50 | **~86 ms** | ~108 ms |
 | bulk embed, passages/s | **~50** | ~37 |
 | whole novel (*The Stand*, 12,814 paragraphs), 64/request, 4 in flight | **31 s** | 40 s |
 | rerank 20 docs, p50 | **~169 ms** | ~180 ms |
 | rerank 20 docs, req/s at conc 8 | ~5.7 | ~5.6 |
 TEI ran its fused `FlashQwen3` path. `--max-batch-tokens 32768` did not change
 the novel time (39–40 s), so it stays at the default. **Through the gateway these
 gaps mostly vanish**: LiteLLM is the bottleneck there (see
 [`servers/esh-ml1/README.md`](../../servers/esh-ml1/README.md)).
 ### Footprint — TEI is much lighter
 | | vLLM (both) | TEI (both) |
 |---|---|---|
 | VRAM (host `nvidia-smi`, per process) | 3,298 + 1,512 MiB | 1,352 + 1,256 MiB |
 | image | 29.9 GB | 8.16 GB |
 | warm restart to healthy (n=3) | ~24 s | ~4 s |
 | container RAM just after start | 2.5 + 5.1 GiB | 0.7 + 1.7 GiB |
 ⚠ vLLM's VRAM figure is mostly a **setting** (`--gpu-memory-utilization 0.20`
 each). A lower setting would shrink it; that has not been tested.
 ### External review — Dvalin (Grok research peer), 2026-09-25
 **Agree with caveats: adopt TEI for these two seats; footprint is the right
 reason.** His external evidence (not re-verified here): a Runpod 2026-09-14
 engine comparison shows vLLM ahead of TEI on Qwen3-Embedding bulk throughput
 (median ~2.6×, single unreplicated runs, near parity on smaller cards) and TEI
 ahead on BERT-family models — consistent with our 1.3× on a 50 W card and the
 rerank near-tie. Open TEI 1.9 issue #857 (tokio panic "No backend receiver"
 under load, n=1) is a watch item, not a gate. A lower vLLM memory setting
 would shrink only the embedder (the reranker at 1,512 MiB is already under
 its cap); 0.12 is the only setting he'd expect to both boot and help — untested.
 His caveats, and status: fail-closed truncation (**done**, above); recheck score-
 threshold consumers (**done**, nevermore, above); two gateway providers must be
 written down at cut-over (open).
 ## Gateway compatibility (LiteLLM v1.97, temp aliases, deleted after)
 - Embeddings: **drop-in**. TEI's `/v1/embeddings` works with the existing
  `hosted_vllm/` provider.
 - Rerank: needs the **`huggingface/` provider** (scores identical to direct
  TEI). `hosted_vllm/` fails with 422 (`missing field texts`); TEI's `/rerank`
  body differs from vLLM's.
 ## Behaviour differences to carry into any cut-over
 - **Truncation — now fail-closed (`--auto-truncate false`).** Measured with
  TEI's default: a ~40k-token input returned **200 with a vector of its first
  16,384 tokens**, silently, where vLLM returns 400. With truncation off, TEI
  embed rejects > 32,768 tokens and TEI rerank rejects > 8,192 (both 422,
  verified). Truncation off requires `--max-batch-tokens` at or above the model's max
  input (32,768 for Qwen3-Embedding) or TEI refuses to start; VRAM unchanged.
  Remaining difference: TEI embed **accepts** 8,193–32,767 tokens that vLLM
  (`--max-model-len 8192`) rejects. It embeds the whole input, which is wider,
  not lossy.
 - **nevermore thresholds rerank scores** (`NEVERMORE_RERANK_THRESHOLD`, default
  0.3, cluster-member filter in `digest.py`). Across the 2,000 bake-off scores,
  **0 flips** at 0.3 or 0.5 for TEI vs vLLM — but only 1 score landed within
  ±0.02 of 0.3 (bge scores sit near 0 or 1), so this cannot exclude rare flips
  on borderline headlines. Worst case: a same-story headline clusters
  differently. Low stakes.
 - TEI caps inputs per request at `--max-client-batch-size` (default 32; set to 128 here).
 - TEI on CUDA runs float16 only (vLLM served Qwen3-Embedding in bf16); parity
  above says it does not matter here.
@@ -0,0 +1,227 @@
 # ERP tune — DPO stage readiness
 _Written 2026-08-26 while run 2 (SFT, stock instruct base) finished. This is a
 **readiness survey and a decision surface**, not a plan. The plan needs the
 operator and brokkr-smithy-dev; what follows is what is on disk, what is
 missing, and what has to be decided before anyone writes a config._
 ## Why there is a DPO stage at all
 Run 2 is an SFT on `google/gemma-4-26B-A4B-it` — the **official instruct**
 release, not an abliteration. Nothing in it attempted to remove refusals, so:
 > **Run 2 will refuse at near-stock rates (~98-100/100 on the harmful-behaviors
 > probe) and that is the design, not a regression.** Run 1's 29 came from its
 > abliterated base, which run 2 deliberately does not use.
 Targeted DPO is the stage where refusals get pruned — **on axes we choose**,
 keeping the general-harm ones. That is the whole reason the stock base was
 picked over a third-party abliteration: it replaces "inherit a stranger's
 judgement about what to strip" with "strip what we decide to, deliberately."
 Trading one training stage for that control was the operator's call on
 2026-08-25.
 ## What is ready
 | piece | state |
 |---|---|
 | **Base** | run 2's merged bf16 tune, `/tank/erp-tune/serve/merged-run02` |
 | **SFT adapter** | `/tank/erp-tune/run-02/adapter/` — the DPO starting point |
 | **Compute** | ana-ml2 GPU0, free once run 2's eval seat comes down |
 | **Harness** | `eitri-smithy/erp_sft_harness` — corpus verification, encode cache, provenance, order manifest, resume. All reusable; none of it is SFT-specific except the loss |
 | **Prose preference data** | two sets already on disk, unvetted: `/mnt/smithy/datasets/raw/jondurbin__gutenberg-dpo-v0.1` and `nbeerbower__gutenberg2-dpo` |
 | **Reward judge** | LitBench-RM `Llama8B-CreativeWritingVerifier`, irv-ml1 `:8202/classify` — on-demand, ~90 s to respin |
 ## What is missing, in order of how much it will hurt
 ### 1. ⭐⭐⭐ Preference data for the refusal axes — the long pole
 **Nothing on disk targets this.** The Gutenberg sets are prose-quality
 preferences; they say nothing about refusal behaviour. Refusal-axis pairs have
 to be generated: prompt the SFT tune, collect the refusal as `rejected`, and
 produce a `chosen` completion that engages.
 Three sub-problems, none of them mechanical:
 - **Where does `chosen` come from?** A model that will write it, a human, or a
  template. Each has a different failure mode, and a templated `chosen` teaches
  the template.
 - **How many pairs, and how balanced?** Too few and nothing moves; too many on
  one axis and it generalises past the axis.
 - **What is the held-out set?** DPO can be evaluated by preference accuracy on
  held-out pairs, and that has to be split before generation, not after.
 ### 2. ⭐⭐⭐ The axis list itself — OPERATOR DECISION, blocks everything
 Which refusals get pruned, and which are kept. Nobody should generate a single
 pair before this is written down, because the axis list *is* the data spec.
 R19's track-2 map is the existing artifact that names candidate axes. The
 domain-compliance probe scaffold at
 [`scripts/training-probes/counted_classifier.py`](../../scripts/training-probes/counted_classifier.py)
 measures the RIGHT axis — explicit fiction — as opposed to the
 `harmful_behaviors` probe that run 1 measured, which was the wrong axis and is
 why run 1's refusal number was never the number anyone wanted.
 ⚠ **Measure the axis before pruning it.** Run the domain-compliance probe
 against run 2 first: pruning an axis the tune already complies on is wasted
 work, and the pre-number is the only baseline that will ever exist.
 ### 3. ⭐⭐ A DPO trainer — not present
 `trl` is **not installed** in `/tank/erp-tune/venv`. Two routes:
 - **Install `trl`, use `DPOTrainer`.** Fast, conventional, and drags a
  dependency into a venv whose torch/transformers/peft versions are load-bearing
  and known-good. ⚠ Pin and verify: `warmup_ratio` (playbook 4.2) is exactly
  what a version bump does to you.
 - **Implement the DPO loss in the existing harness.** More work, but keeps the
  corpus verification, encode cache, order manifest, provenance and resume that
  the SFT harness already has and TRL does not know about. The harness is not
  SFT-specific except for the loss function.
 ⚠ **Whichever route, do the install AFTER a run finishes, never during one.** A
 dependency resolution that upgrades `transformers` under a live process can
 break its save path, which is the most expensive moment to break.
 ### 4. ⭐ Provenance for a preference run
 The manifest gains fields the SFT one has no concept of: preference-set sha,
 pair count, per-axis balance, beta, and the reference-model identity. Same rule
 as everything else — see playbook 4.3's audit table. **A DPO run that does not
 record which axes it pruned is unauditable**, and this stage is the one where
 that matters most.
 ## ⚠ Measured 2026-08-26 — a length artifact the DPO pairs would inherit
 The run-2 gate found tuned rp turns at **88.5 / 90.9 words against the base's
 137.1 / 145.4** — a 36% reduction, consistent across blocks. brokkr-smithy-dev
 hypothesised the mix was teaching PIPPA's product clip. The corpus side is now
 measured and it is confirmed, with a mechanism sharper than "PIPPA is in the
 mix":
    root                          bot turns   p50   p90   p99   max   <=123   124-130
    pippa                            79,414    67    95   108   123  100.0%     0.00%
    bluemoon                          1,170   563   908  1432  1924    0.0%     0.00%
    c2-logs-32k-llama3               23,704   234   394   686  2000    6.6%     1.44%
    creative-writing-multiturn        8,621   204   485   858  1837   24.0%     2.16%
 **PIPPA's max is 123 exactly, 100% at or under it, 0.00% in the 124-130 band.**
 That is a product clip (2023 Character.AI), not a length preference — every
 other root crosses its own p99 smoothly.
 The exposure asymmetry is the finding:
    root                        turn share    word share    mean len
    pippa                            70.3%         37.5%          67
    c2-logs-32k-llama3               21.0%         42.5%         256
    creative-writing-multiturn        7.6%         15.1%         251
    bluemoon                          1.0%          4.9%         601
 **PIPPA is 70.3% of bot TURNS but 37.5% of bot WORDS — and length is learned per
 turn, not per token.** Each of those 79,414 turns is a separate demonstration of
 where a bot reply ends. By loss tokens PIPPA looks like a third of the dialogue
 signal; by end-of-turn demonstrations it is seven in ten, from a source that
 cannot exceed 123 words.
 ⚠ **Generalises past PIPPA: a length-clipped root is over-represented in the
 length signal by exactly the ratio its clipping creates.** The shorter the clip,
 the more turns per token of budget, the harder it teaches the clip. Check the
 turn-share/word-share split of every root before assuming a mix is balanced.
 ### The output-side test ran. The clip hypothesis is FALSIFIED — and the truth is more interesting
 Measured on the live tuned seat, n=96 rp generations (brokkr-smithy-dev,
 2026-08-26). **There is no shoulder at 123:**
       0-19    18  ██████████████████
      20-39    22  ██████████████████████   <- MODE
      40-59     5  █████                    <- trough
      60-79    15  ███████████████          <- PIPPA median 67
      80-99    10  ██████████
     100-119    9  █████████
     120-139     3  ███                     <- PIPPA CAP 123. Three. A trough.
     140-159     5  █████
     160-179     4  ████
       300+      3  ███                      max 505
    <=123  79/96 = 82.3%   (PIPPA itself: 100.0%)
    median 62 · mean 77.9 · p90 152 · p99 367
 **The clip-as-boundary reading is dead.** Mass would pile *under* a learned wall;
 the 120-139 bin holds three of ninety-six and is a trough, and **17.7% of
 generations cross a cap PIPPA can never cross.**
 ⚠ **A correction to this document's earlier read.** It compared the tuned *mean*
 (88.5) to PIPPA's *median* (67) and p90 (95) and concluded "central tendency,
 comfortably inside the upper body." **Median to median it is 62 against 67** — at
 or slightly below PIPPA's centre. Mixing statistics across a comparison produced
 a more reassuring answer than the data supports; the 88.5 was a mean dragged up
 by the tail.
 **What the data shows instead: the distribution is BIMODAL.** A mode at 20-39, a
 trough at 40-59, a second mode at 60-79 astride PIPPA's centre, then a long tail
 to 505. The base arm has no such shape (tuned mean/median ratio 1.45 and 1.23
 against the base's 1.09 and 1.14).
 So the tune did not change rp length so much as change its **shape** — replacing
 one symmetric distribution with a mixture. That is what you would expect from
 roots whose length distributions do not overlap (PIPPA 67, c2-logs 256, bluemoon
 601) being learned as **distinct modes rather than blended into an average**.
 ⚠ **And the skew is rp-ONLY.** Story sits at mean/median 0.98 / 0.95 tuned
 against 0.98 / 1.00 base. The shape change appears exactly in the family the
 clipped root lives in — the strongest support the turn-share mechanism gets from
 the output side.
 **What this means for pair generation:** `chosen`/`rejected` sampled from a
 bimodal generator inherit the mixture, not a mean. Sampling naively will
 over-draw from the dominant short mode. If length is not the axis being tuned,
 control for it explicitly rather than assuming a representative sample.
 ### ⚠ The degeneracy rate is NOT yet a usable baseline
 Same arm, same seed block, re-run:
    scored       162/192 -> 163/192
    degenerate      7    ->   11
    truncated      23    ->   18
    rp VOID        no    ->  YES
 Distance moved 0.003, well inside the 0.0125 floor, so the gate result is not
 threatened. But **degeneracy straddles the 10% budget**, so the same arm on the
 same seed VOIDs or does not depending on the run. **A guard whose trip point sits
 at the noise boundary flips**, and it produces disagreement between honest
 observers rather than silence — a distinct defect from a gate that cannot fail.
 The honest statement is that **the tuned arm's rp degeneracy is AT the budget**,
 not clearly over or under it. Replicates are being run to pin it. **Do not build
 DPO pairs on the single-sample figure** — the degeneracy rate decides how much of
 the tune's own output is usable as `chosen`, so it is load-bearing for pair
 generation and much cheaper to pin now than to discover after pairs exist.
 **Why this belongs in the DPO prep and not just the gate record:** preference
 pairs generated *from this tune* inherit its length distribution in both
 `chosen` and `rejected`. If short-rp is an artifact rather than a preference,
 DPO on those pairs trains it in as an explicit objective rather than leaving it
 as an incidental SFT bias. Settle the length question before generating pairs.
 Scripts: `/tank/erp-tune/pippa_clip.py`, `/tank/erp-tune/clip_share.py`.
 ## The decision that gates the rest
 **Which refusal axes are in scope, and which are explicitly kept?**
 Everything downstream — data generation, pair counts, held-out split, the probe
 that measures success — is a function of that list. It is not a technical call
 and it should not be made by whoever writes the config.
 ## Pre-flight, when the time comes
 Run [`training-throughput-playbook.md`](training-throughput-playbook.md) 4.8
 before the window opens. Two items bite harder here than they did for SFT:
 - **cache key covers the change** — a preference set is exactly the kind of
  auxiliary file whose *path* is stable while its *content* changes.
 - **an observed consequence beside every configured value** — the axis list is a
  configured value, and its observed pair is the per-axis probe delta.
@@ -0,0 +1,426 @@
 # Thinking-Capable eRP Finetunes, 15–30B — Deep Research
 **Compiled 2026-08-12 · Window: Feb–Aug 2026 · Weighted for spatial/state coherence · Target: RTX PRO 6000 Blackwell (sm_120), NVFP4, throughput**
 ---
 ## 0. Read this first — three findings that should change your shortlist
 **1. The 24B Mistral era is over.** Everything worth running in this band now sits on one of four bases, all of which ship native thinking out of the box: **Qwen3.6-27B** (Apr 2026), **Qwen3.5-27B** (Feb 2026), **Gemma-4-31B / Gemma-4-26B-A4B** (Mar 31 2026, now **Apache 2.0**), and **arcee-ai/Trinity-Mini** (26B-A3B). Mistral has shipped *nothing* in your band in 2026 — Mistral Small 4 is a 119B-A6B MoE that absorbed the Magistral line. Magistral-Small-2509 (Sep 2025) is still the newest in-range Mistral reasoning model, and the 24B tunes built on it are now a legacy tier.
 **2. The evidence says heavy eRP finetuning actively damages the thing you care about most.** This is the uncomfortable core of this report and it's covered in §2. Short version: reasoning-native models buy real long-context state tracking, but bolting RP-tuning *and* reasoning-tuning on top degrades both prose and world-modeling. The single most respected merger in the space says flatly that 24B "will struggle with details of logical/physical continuity at times — which is probably inescapable for a 24B model." **If spatial coherence is your #1 criterion, bias toward light-touch tunes on smart bases, not heavy eRP tunes.**
 **3. MTP and best-in-class RP tuning are currently mutually exclusive — with exactly one escape hatch.** Every dedicated RP brand (Cydonia, Skyfall, Dark-Scarlett, MeroMero, Artemis, Magistry) sits on Mistral or Gemma bases that **have no MTP heads at all**. Only Qwen3.5/3.6-27B ships MTP in your band — and `from_pretrained` **silently drops the MTP heads during finetuning**, so almost every Qwen-based community tune has lost them too. The escape hatch is the `Native-MTP-Preserved` lineage (§5.2), which grafts the 15 MTP tensors back post-hoc, and already has NVFP4 checkpoints.
 > **Also worth knowing up front:** at temp 0.8–1.2 (normal RP sampling), speculative decoding acceptance collapses to ~38–52%, and vLLM's own guidance is to disable it below 0.5. On a *shared, batched* box it is likely a net throughput **loss**. Details and the one contradicting measurement in §5.4.
 ---
 ## 1. Ranked picks
 Ranked for **spatial/state coherence first**, prose second, with your NVFP4 + throughput constraints factored in.
 | # | Model | Params | Base | Thinking | NVFP4 today? | MTP? |
 |---|---|---|---|---|---|---|
 | 1 | [zerofata/G4-MeroMero-v2-31B](https://huggingface.co/zerofata/G4-MeroMero-v2-31B) | 31.27B | Gemma-4-31B | Dual (Think/NoThink presets) | v1 only — must quantize v2 | ✗ |
 | 2 | [Gryphe/Pantheon-Reasoning-27B](https://huggingface.co/Gryphe/Pantheon-Reasoning-27B) | ~27.8B | Qwen3.6-27B (MTP-preserved heretic) | **Always-on** | ✗ — must quantize | ✗ (re-graftable) |
 | 3 | [llmfan46/…-Native-MTP-Preserved-NVFP4](https://huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4) | ~27.8B | Qwen3.6-27B | Native | **✓ shipped** | **✓ intact** |
 | 4 | [allura-org/Qwen3.5-27B-Anko](https://huggingface.co/allura-org/Qwen3.5-27B-Anko) | ~27.4B | ArliAI Qwen3.5-27B-Derestricted | Dual-mode (trained both ways) | ✗ | ✗ |
 | 5 | [ReadyArt/Dark-Scarlett-v1.0-27B](https://huggingface.co/ReadyArt/Dark-Scarlett-v1.0-27B) | ~27.8B | Qwen3.6-27B | `enable_thinking` flag | ✗ (W4A16/W8A16 PTQ only) | ✗ |
 | 6 | [TheDrummer/Artemis-31B-v1.1](https://huggingface.co/TheDrummer/Artemis-31B-v1.1) | 31.27B | Gemma-4-31B | Dual + custom tags | ✗ | ✗ |
 | 7 | [Gryphe/Pantheon-Reasoning-26B-A4B-1.1](https://huggingface.co/Gryphe/Pantheon-Reasoning-26B-A4B-1.1) | 26.5B MoE (A4B) | Gemma-4-26B-A4B | **Always-on** | ✗ | ✗ |
 | 8 | [zerofata/G4-MeroMero-26B-A4B](https://huggingface.co/zerofata/G4-MeroMero-26B-A4B) | 25.8B MoE (A4B) | Gemma-4-26B-A4B | Dual | **✓** (2 quantizers) | ✗ |
 | 9 | [sophosympatheia/Magistry-24B-v1.1](https://huggingface.co/sophosympatheia/Magistry-24B-v1.1) | 23.6B | Magistral-2509-24B | `<think>` prefill | MLX only | ✗ |
 | 10 | [zerofata/Q3.5-BlueStar-v2-27B](https://huggingface.co/zerofata/Q3.5-BlueStar-v2-27B) | ~27.4B | Qwen3.5-27B | `<think>\n` prefill (trained) | MLX only | ✗ |
 **Wildcard worth a slot on your test rig:** [Gryphe/WorldSim-Opus-3.6-35B-A3B](https://huggingface.co/Gryphe/WorldSim-Opus-3.6-35B-A3B) — 35B-A3B, over your band but only ~3B active so it's cheap. It is the closest thing anyone has built to a model *designed* for the state-tracking problem: trained on three datasets that all carry full thinking traces, with reasoning persisting per-turn. The author calls it a research release whose "practical effectiveness remains uncertain."
 **Actively avoid for your criterion:** [LatitudeGames/Equinox-31B](https://huggingface.co/LatitudeGames/Equinox-31B) — card states verbatim "No reasoning datasets were included during training," thinking suppressed by default. [TheDrummer/Rocinante-XL-16B-v1](https://huggingface.co/TheDrummer/Rocinante-XL-16B-v1) — user reports of degradation past 16k and noticeable decline past 20k; you can't track scene state in a window that small.
 ---
 ## 2. Does thinking actually help spatial coherence? — the evidence
 This deserves its own section because the answer is **"yes for state tracking, no for prose, and only if the model was pretrained for reasoning."**
 ### 2.1 Thinking clearly helps long-context state tracking — for reasoning-native models
 - **Fiction.liveBench** (narrative comprehension, theory of mind, chronological reasoning at length) is the single strongest datapoint. At 16k context: **QwQ-32B 83.3%** vs Gemma-3-27B 33.3% vs dolphin-Mistral-24B 25.0% — a reasoning-native 32B beating a *70B* non-reasoning model (Llama-3.3-70B, 33.3%) by 50 points. Same-model toggle: claude-3-7-sonnet thinking **83.3%** vs non-thinking **50.0%** at 16k. [[data]](https://raw.githubusercontent.com/mnismt/llms-long-context-benchmark/main/src/data/benchmark.ts) [[Epoch]](https://epoch.ai/benchmarks/fictionlivebench)
 - **LongBench Pro** (8k–256k, includes consistency-checking and dialogue-tracking): thinking mode adds **+11 to +16 points** for reasoning-native models (Claude-4-Sonnet 56.07→69.87; DeepSeek-V3.2 51.67→67.82). But models *not trained* for thinking gain nothing — Llama-3.1-405B **+0.59**, Gemma-3-12B **−0.24**. Paper's own conclusion: "models without thinking training may fail to effectively leverage test-time compute." [[arXiv 2601.02872]](https://arxiv.org/html/2601.02872v1)
 - **MuSR** (multi-step narrative state tracking): Ministral 3 14B Reasoning **70%** vs base **64%**; consistent +6 to +9 at every size down to 1.2B. [[BenchLM]](https://benchlm.ai/benchmarks/musr)
 - **UGI "World Model"** column, same-model toggles: Qwen3-32B **21.25 → 23.80**, Qwen3-30B-A3B **13.10 → 16.67** with thinking on.
 ### 2.2 Thinking reliably damages prose and *destroys* instruction-following
 Every same-model pair in the UGI dataset shows the `Writing` score dropping when thinking is on: Qwen3-14B **34.76 → 29.64**, Qwen3-32B 32.95 → 30.34, Qwen3-30B-A3B 30.24 → 28.54, Qwen3-8B 27.96 → 23.87. gpt-oss-20b degrades monotonically with reasoning effort — Writing **24.62 (low) → 24.50 (med) → 10.94 (high)** with repetition interrupts rising 2 → 1 → **8**.
 The instruction-following collapse is the most reproducible effect in the entire dataset. `creative_writing_wc_exceeded_pct` — the share of creative tasks where the model blew the requested word limit:
 | Model | Thinking off | Thinking on |
 |---|---|---|
 | Qwen3-14B | 1% | **99%** |
 | Qwen3-32B | 0% | **100%** |
 | Qwen3-30B-A3B | 10% | **99%** |
 | Qwen3-8B | 4% | **100%** |
 If you've ever wondered why a thinking model ignores your "keep replies to two paragraphs" instruction — that's this.
 ### 2.3 The warning case: bolting reasoning onto an RP finetune
 `Cydonia-R1-24B-v4` vs `Cydonia-24B-v4` — same trainer, same base lineage, one reasoning-tuned:
 | Metric | Cydonia-24B-v4 | Cydonia-R1-24B-v4 |
 |---|---|---|
 | Writing | 30.91 | **20.38** (−34% rel.) |
 | World Model | 23.30 | **19.33** (−17%) |
 | NatInt | 26.64 | 24.27 |
 | Length error | 22% | **80%** |
 | W/10 (willingness) | 7.8 | 8.2 ✓ |
 Reasoning-tuning bought willingness and cost everything else, *including the world-model score*. Caveat: separate training runs, not a toggle, so recipe differences are confounded. But it's the closest analogue to "what happens when an RP finetuner adds thinking."
 ### 2.4 Mechanistic support for why
 - **Visual vs Textual CoT diagnostic** (ACL 2026): textual chain-of-thought **degrades spatial transformation by up to 16.5%** and **multi-object tracking by 12.7%** vs direct answering, measured across GPT-5, Claude Opus 4.6, Gemini 2.5 Pro, Qwen3-VL-72B. [[pdf]](https://aclanthology.org/2026.alvr-main.1.pdf) That is *literally your criterion*, and CoT made it worse.
 - **"Mind Your Step (by Step)"**: CoT reduces performance on implicit statistical learning by up to **−36.3%** absolute, framed as verbal overshadowing — narrating a scene in a scratchpad makes the model worse at *feeling* the scene. [[arXiv 2410.21333]](https://arxiv.org/html/2410.21333v4)
 - **Contrary evidence worth weighing** — "Thinking in Character" found *role-aware* reasoning beats naive reasoning (CharacterBench 3.69 RAR vs 3.57 distill), but note the third term: **undirected extra thinking scored worst at 3.05**. The claim is not "reasoning helps," it's "reasoning helps only if its style is constrained to the character." [[arXiv 2506.01748]](https://arxiv.org/html/2506.01748v1)
 ### 2.5 And at the frontier, reasoning doesn't fix narrative consistency at all
 - **NarrativeWorldBench**: frontier + reasoning models all cluster at **F1 0.78–0.81** at horizon 50 with no significant difference (p>0.13); everything loses ~0.20 F1 from h=10 to h=200. A purpose-built 8B latent world model holds **F1 ≥ 0.84 across all horizons** at ~4× lower cost. [[arXiv 2606.17391]](https://arxiv.org/html/2606.17391v1)
 - **NCP-Bench** (Aug 2026) is the benchmark you were hoping existed — it explicitly scores *spatial consistency* ("character described on the bridge later appearing in a doorway"), *object state tracking* ("a raft inflated→deflated without justification"), and character knowledge leakage. Results are humbling: **GPT-5.2 survives 20 turns only 42% of the time**, near-zero survival by 100 turns, fact conflicts at 40–68% across all models. It tests no sub-32B models. [[arXiv 2608.08160]](https://arxiv.org/abs/2608.08160)
 - **RP-Bench** found reasoning models (GLM 5.1, Gemini 3.1 Pro, Kimi K2.5/K2.6) *underperformed* frontier non-reasoning models on roleplay dimensions, with severe latency costs (Kimi K2.6 p95 **173s**, **17% truncation at length limit** — truncation is itself a coherence failure). Its verdict on the category: "**The RP-specialist finetunes — the models marketed for exactly this — rank last.**" [[repo]](https://github.com/LeviTheWeasel/rp-benchmark)
 ### 2.6 What I'd actually do with this
 The defensible synthesis: **use thinking sparingly and structurally, not as an always-on prefix to prose.** A gated pattern — reasoning enabled for scene-state checks, scene transitions, and complex multi-character blocking; disabled for straight prose continuation — captures the state-tracking gain without paying the prose and length-adherence tax. Every model in §1 that supports *dual* mode (MeroMero, Artemis, BlueStar, Dark-Scarlett) lets you do this at the request level. The always-on models (Pantheon, WorldSim) do not.
 ---
 ## 3. Per-model breakdowns
 Metadata below is from the HuggingFace API, verified individually. Download counts are trailing-30-day and are **unreliable as a quality signal** — most users pull the GGUF mirror repos, not the BF16 originals.
 ### 3.1 zerofata/G4-MeroMero-v2-31B — best-shaped training for your criterion
 [huggingface.co/zerofata/G4-MeroMero-v2-31B](https://huggingface.co/zerofata/G4-MeroMero-v2-31B) · 31.27B · Gemma-4-31B · Apache-2.0 · **2026-08-03** · 258 dl / 43 likes
 The reason this is #1: it is the **only model in the entire survey whose training explicitly optimizes reasoning against a coherence judge.** Verbatim from the card, the pipeline is `SFT > Merge > GRPO > GRPO > on-policy SFT`:
 1. Diversity SFT — ~4,000 curated stories, 0.5 blend merge-back
 2. **Creative GRPO** — 8 rollouts/prompt, 300 steps, *thinking disabled*
 3. **RP Logic GRPO** — 100 steps, *thinking enabled*, scored by "a logic-defect judge (DeepSeek-V4 Flash with a rubric)", with a `reward_judge_coherence` reward term
 4. On-policy SFT — ~3,300 self-generated RP samples, diversity-filtered
 Stage 3 is the mechanism that should produce state tracking. **Honest caveat:** the card does *not* claim improved spatial coherence as an outcome, and I could not confirm the stage-3 prompts were multi-turn (an earlier source claimed this; it's unverified). You're buying a plausible training signal, not a measured result.
 Author's own metrics vs stock Gemma 4: swipe diversity **0.72 vs 0.43**, story slop **7.4 vs 8.8 per 1k words**, bare-prompt attractor hit rate **66% vs 99%**, no regression on IFEval / GSM8K / MMLU-Pro.
 - **Thinking:** dual, via `Gemma4-Think.json` / `Gemma4-NoThink.json` SillyTavern presets. Reasoning is longer than stock Gemma 4, shorter than MeroMero v1.
 - **Samplers:** temp 0.8–1.0, MinP 0.05
 - **Quants:** GGUF (official + mradermacher), FP8 W8A16 ([hoborific](https://huggingface.co/hoborific/G4-MeroMero-v2-31B-W8A16-FP8)), exl3, MLX. **NVFP4 exists only for v1** ([pekkAi](https://huggingface.co/pekkAi/G4-MeroMero-31B-NVFP4), [heretic variant](https://huggingface.co/pekkAi/G4-MeroMero-31B-uncensored-heretic-NVFP4)). You'll quantize v2 yourself.
 - **Note:** 31.27B is marginally over your stated band. There is a true in-band sibling, [G4-MeroMero-26B-A4B](https://huggingface.co/zerofata/G4-MeroMero-26B-A4B) (25.8B MoE, A4B, May 2), which *does* have NVFP4 ([Deaquay](https://huggingface.co/Deaquay/G4-MeroMero-26B-A4B-NVFP4), [pekkAi heretic](https://huggingface.co/pekkAi/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4)) and claims "reasoning is more structured, using less tokens during RP." But the 26B's card is candid that "logic and repetition I think are roughly on par with the original" — v2-31B is where the coherence work actually happened.
 ### 3.2 Gryphe/Pantheon-Reasoning-27B — best methodology, and it sits on the MTP-preserved base
 [huggingface.co/Gryphe/Pantheon-Reasoning-27B](https://huggingface.co/Gryphe/Pantheon-Reasoning-27B) · ~27.8B · Apache-2.0 · **2026-05-30** · 232 dl / 27 likes
 Base is `llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved` — **verified**, and that matters enormously for your rig (§5.2).
 Two things make this the most methodologically interesting tune in the set:
 - **Always-on reasoning.** Verbatim: "The model was trained with `preserve_thinking: true`, so thinking tags remain active across all assistant turns in multi-turn conversations, not just the first." Almost every other model reasons once and then stops.
 - **The thinking traces were generated as *planning*, not annotation.** DeepSeek 3.2 produced them under the instruction to "think as a writer planning their next response — before writing — rather than annotating a response," then judge-model validated. This is the "role-aware reasoning" pattern that the CharacterBench work found is the *only* kind that helps.
 Data mix: Pantheon RP corpus ~28%, Opus-4.6-Reasoning-24k ~21%, WorldSim narrative ~16%, text adventure/IF ~16%, general RP ~16%, Tiamat ~3%.
 - **Samplers:** temp 1.0, **rep_pen 1.0**, min_p 0.05. The rep-pen point is emphatic and now consensus among reasoning-RP authors: repetition penalties corrupt thinking content. **Any thinking model whose card recommends rep_pen > 1.0 is a red flag.**
 - **Template:** ChatML (Qwen3.6 chat template)
 - **Author's own framing:** a research release, with the stated open question being "does reasoning actually help roleplay, or does it just add latency?" Respect that honesty.
 - **Quants:** GGUF only. No NVFP4, no FP8. You will quantize this one.
 - **Sibling:** [Pantheon-Reasoning-26B-A4B-1.1](https://huggingface.co/Gryphe/Pantheon-Reasoning-26B-A4B-1.1) (26.5B MoE, Gemma-4-26B-A4B, Jun 8) — same methodology, stricter trace QA, genuinely in-band, and the **most-reused merge donor in the whole 26B-A4B ecosystem**. SillyTavern gotcha: character-name prefixes break reasoning compatibility on this one — disable them.
 ### 3.3 llmfan46 Native-MTP-Preserved (NVFP4) — the throughput play
 [huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4](https://huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4)
 Not an RP finetune — a decensored Qwen3.6-27B. It's on this list because it's the **only 15–30B option that is simultaneously NVFP4, MTP-intact, and uncensored**, and because §2 argues that a smart, lightly-touched base may outperform a heavy eRP tune on exactly the axis you're prioritizing.
 Parent repo: 7,580 dl / 41 likes, created May 6, modified May 25. Made with Heretic v1.3.0 using a variant of Magnitude-Preserving Orthogonal Ablation (MPOA), ablating only `attn.o_proj`, `attn.out_proj`, `mlp.down_proj`. Claimed: **94% fewer refusals (6/100 vs 92/100) at 0.0021 KL divergence**, MMLU 85.67% vs 86.65% original.
 The load-bearing detail is `model-auxiliary.safetensors` in the repo — that's where Qwen stores the MTP heads, and its presence is hard proof the claim isn't marketing. The card enumerates all 15 preserved tensors.
 **Pair it with a style fix.** Its weakness vs a proper RP tune is voice, not intelligence. [Gryphe/Gemma-4-26B-A4B-StyleTune-V2](https://huggingface.co/Gryphe/Gemma-4-26B-A4B-StyleTune-V2) demonstrates the approach on the Gemma side and is the most quantitatively-supported claim in this whole survey: it trains **precisely one tensor** — "the `lm_head` output projection… freeze everything else. All 30 transformer layers, all the attention heads, all the MLPs — completely untouched" — and measures **52% fewer clichés per 100 words (1.141 → 0.551)** over 200 RP prompts with only 19.9% shared trigram vocabulary. Reasoning capability is untouched by construction. There's no Qwen equivalent published yet, but the recipe is simple enough to replicate.
 ### 3.4 allura-org/Qwen3.5-27B-Anko
 [huggingface.co/allura-org/Qwen3.5-27B-Anko](https://huggingface.co/allura-org/Qwen3.5-27B-Anko) · ~27.4B · Apache-2.0 · **2026-04-08** · 40 dl / 11 likes
 **Correction to circulating claims:** the base is **`ArliAI/Qwen3.5-27B-Derestricted`**, not stock Qwen3.5-27B. LoRA r=64 / α=512 on Doubao Seed 2.0 Pro reasoning traces, trained on both reasoning *and* non-reasoning responses, so it's dual-mode by construction. Stated goal, verbatim: "increase the quality of reasoning and decrease looping, and fix slop in outputs."
 Why it ranks well for you: Qwen3.5-27B is the best state-tracking base in the band by measurement — **MuSR 95, the best open-weight score overall**, and LongBench v2 60.6%.
 - **Samplers, verbatim and shouted:** "**DO NOT USE QWEN'S SAMPLERS. THEY ARE AWFUL.**" Use **temp 1.25, min_p 0.05–0.1**.
 - **Odd but documented:** recommended system prompt is `You are Claude, a helpful and harmless language model created by Anthropic.` It was trained to work with Claude-style system prompt formatting.
 - **Quants:** GGUF only (bartowski, mradermacher). No NVFP4/FP8/AWQ/exl3.
 - **Warning:** ArliAI's Derestricted line **drops MTP** — I verified the file manifest, there is no `model-auxiliary.safetensors`. So Anko has no MTP.
 ### 3.5 ReadyArt/Dark-Scarlett-v1.0-27B — cleanest eRP with flag-based thinking
 [huggingface.co/ReadyArt/Dark-Scarlett-v1.0-27B](https://huggingface.co/ReadyArt/Dark-Scarlett-v1.0-27B) · ~27.8B · Qwen3.6-27B · Apache-2.0 (personal use, 18+) · **2026-06-16**
 The most explicitly eRP-targeted model here with a properly documented thinking toggle:
 ```
 chat_template_kwargs: {"enable_thinking": true, "reasoning_effort": "medium"}
 ```
 That `reasoning_effort` knob is unusually useful for the gated-thinking pattern in §2.6 — you can dial it per-request rather than binary on/off.
 Training: LoRA r=32, 2 epochs, **text layers only**, on 12,211 curated adult-RP prompts, with multi-turn generation, refusal filtering, and group-chat support in the pipeline.
 - **Samplers:** top_p 0.92, temp 1.0, freq_pen 0, pres_pen 0
 - **Real limitation:** the card states it's optimized for Male(user)→Female(AI) perspective. Narrow.
 - **Quants:** GGUF + ReadyArt's own W4A16/W8A16 PTQ. **No NVFP4, no FP8.**
 - Family context: ReadyArt shipped a dense June burst — `Dark-Scarlett-v2.0-31B` (Gemma-4), `v1.0-26B-A4B`, `v1.0-31B`, `v0.4-2509-24B`, `Heimdallr-v0.02-31B`. Download signal favors the MoEs.
 ### 3.6 TheDrummer/Artemis-31B-v1.1 — freshest, longest bake
 [huggingface.co/TheDrummer/Artemis-31B-v1.1](https://huggingface.co/TheDrummer/Artemis-31B-v1.1) · 31.27B · Gemma-4-31B · **2026-08-06** · 7 likes · **no license set**
 Four months of public iteration through BeaverAI test builds (`v1a` Apr 8 → `v1n` Jul 22), which is unusually thorough for this scene. **Use v1.1, not v1** — v1 has "strong writing potential but requires manual adjustments"; v1.1 "improves stability while maintaining v1's creative strengths," specifically fixing **"dash spiraling."**
 - **Thinking:** the most flexible activation of any model here — "standard thinking gemma template or `<thinking></thinking>` blocks on non-thinking gemma template," and "`<think></think>` should work too, along with tricks like `<evil_think></evil_think>`."
 - **Samplers:** not fixed in the card; Drummer points to a crowdsourced sampler spreadsheet.
 - **Too new for consensus** as of Aug 12 — one enthusiastic but content-free feedback thread.
 - Predecessor if you want something proven: [Skyfall-31B-v4.2](https://huggingface.co/TheDrummer/Skyfall-31B-v4.2) (Apr 3, Magistral-Small-2509 upscaled, Mistral v7 Tekken template) is the established workhorse of this window and **has an NVFP4 quant already** ([ealexeev, v4.1](https://huggingface.co/ealexeev/TheDrummer-Skyfall-31B-v4.1-NVFP4)).
 ### 3.7 sophosympatheia/Magistry-24B-v1.1 — the honest one
 [huggingface.co/sophosympatheia/Magistry-24B-v1.1](https://huggingface.co/sophosympatheia/Magistry-24B-v1.1) · 23.6B · Apache-2.0 · **2026-03-22** · 35 likes (highest like count in-band)
 A mergekit DELLA merge (not a finetune) on `Darkhn/Magistral-2509-24B-Text-Only`, so it inherits Magistral's native reasoning. Donors: `Casual-Autopsy/Maginum-Cydoms-24B`, `DarkArtsForge/Magistaroth-24B-v1`, plus `Huihui-Devstral-Small-2-24B-Instruct-2512-abliterated` at 0.3.
 I'm listing it partly because its card contains the **single most on-point statement anyone in this scene has made about your criterion**, verbatim:
 > "This model is fun, but it will struggle with details of logical/physical continuity at times — which is probably inescapable for a 24B model."
 That is a respected merger saying 24B sits below the threshold where physical continuity holds. Take it seriously as a floor: **if spatial coherence is your top priority, 27B+ is the entry point, not 24B.**
 - **Thinking:** prefill-based — force the reply to start with `<think>` plus basic instructions. Card notes `<think></think>` works better than Mistral's `[THINK][/THINK]` tags. (Related gotcha: on Mistral models `<think>` is *not* a special token; `[THINK]` is.)
 - **Samplers:** three named presets — Conservative (temp 0.7, MinP 0.05, Top-N σ 0.75), Balanced (temp 1.0, Adaptive-P target 0.6 / decay 0.9), Wild (temp 0.9, Adaptive-P target 0.35 / decay 0.45). Also ships a SillyTavern Master Import JSON.
 - **It is NOT gated** (a claim to the contrary is circulating; the API says `gated: false`).
 - **Quants:** GGUF, exl3, MLX MXFP4/MXFP8. **No NVFP4.**
 ### 3.8 zerofata/Q3.5-BlueStar-v2-27B — best-documented anti-slop SFT
 [huggingface.co/zerofata/Q3.5-BlueStar-v2-27B](https://huggingface.co/zerofata/Q3.5-BlueStar-v2-27B) · ~27.4B · Qwen3.5-27B · **MIT** · **2026-03-20** · 42 likes
 The interesting technical contribution here is **custom loss masking on slop phrases** — "most common phrases of slop are masked out, so the model doesn't get rewarded for learning these patterns." That lets you train on otherwise-useful RP data without absorbing its clichés. SFT ~27M tokens via Axolotl + LoRA on 4×H200.
 - **Thinking:** prefill `<think>\n` — and importantly, "it is required to prefill the `<think>\n` **as that is how it was trained**." This is a trained-for prefill, not a bolted-on hack. Ships separate think/no-think ChatML instruct JSONs.
 - **Samplers:** temp 0.8–1.0, MinP 0.05–0.075
 - **⚠️ Trained at 10,756 token sequence length** despite the 262k base. See §4 on why this matters more than anything else in the card.
 - **Quants:** GGUF. The two "NVFP4" BlueStar repos you'll find are **MLX** (Apple silicon) — useless on Blackwell.
 ### 3.9 Also verified, lower priority
 - **[Vortex5/G4-Moonlight-Dusk-26B-A4B](https://huggingface.co/Vortex5/G4-Moonlight-Dusk-26B-A4B)** (26.5B MoE, Jul 14, 1016 dl) — merge of Animus-V14.1-FFT + G4-MeroMero-26B-A4B + Esmeralda + **Pantheon-Reasoning-26B-A4B-1.1**. Highest download count of the Gemma-4 MoE merges. Thinking activation is **undocumented** — merge card only, no sampler guidance. Good candidate, poor paperwork.
 - **[ArliAI/Qwen3.5-27B-RpRMax-v1](https://huggingface.co/ArliAI/Qwen3.5-27B-RpRMax-v1)** (Apr 28) — successor to the well-regarded QwQ-32B-ArliAI-RpR line, in a collection literally titled "Thinking-trained RP specialized models." **Confirmed to have no model card at all** — training method, datasets, template, samplers, context all unverified. Heavy third-party GGUF activity (bartowski et al.) suggests real pickup. High risk, possibly high reward.
 - **[NewEden/Trinity-Mini-Ichthyo](https://huggingface.co/NewEden/Trinity-Mini-Ichthyo)** (26.1B-A3B, Jul 10, 2,489 dl — highest of any in-band RP repo) — trained with **actual RL** (Prime RL run, step-100 checkpoint, 32,768 ctx). Base is `NewEden/Trinity-Mini-Futaba`, not stock Trinity-Mini. **Gated behind a contact-info agreement and the README returns 401** — I could read nothing. Zero third-party quants, consistent with the gating. Interesting, unassessable.
 - **[Nimbz/Gemma-4-Gembrain-31B](https://huggingface.co/Nimbz/Gemma-4-Gembrain-31B)** (~Aug 2) — 5-phase Gemma-4 merge, `<|think|>` reasoning, targets "enhanced logical and lateral thinking." Samplers: temp 1.0, Top-P 0.95, Min-P 0.03, DRY 0.8/1.75. Trending but unproven.
 - **[ReadyArt/gemma-4-31B-it-scotoma-2](https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2)** (Aug 6) — not an RP tune, the most rigorous **anti-slop** work of the window: γ-fold refusal-edit projection + 3 rounds of preference training on 9.3k pairs. Measured over 480 RP continuations: stacked adjectives **↓21×**, "Not X. But Y." **↓4×**, em-dash asides **↓4×**. ⚠️ Explicitly **"not uncensored"** — refusal behavior matches base. Useful as a merge donor or style reference, not as a driver.
 ### 3.10 Confirmed dormant — stop waiting on these
 Checked directly; **no 2026 releases in this band**: **anthracite-org / Magnum** (last: Nov 2024) · **Sao10K** (Mar 2025) · **Nitral-AI** (Sep 2025) · **PocketDoc / Dans-PersonalityEngine** (May 2025) · **Undi95** (Mar 2025) · **aixonlab** (May 2025) · **knifeayumu** (Aug 2025) · **TareksLab** (70B only, Aug 2025) · **Doctor-Shotgun** (quant-only in 2026) · **Delta-Vector** (moved to 399B Trinity-Large) · **inflatebot** · **Tesslate** (never RP).
 **Steelskull correction:** `Steelskull/CWT-V5.6` (Apr 2026) is **not** an RP model — it's "Cognitive Workspace Transformer," a **57.8M-parameter** from-scratch research architecture trained on FineWeb-Edu. Steelskull's RP line (Electra / Nevoria / Broken-Tutu) has shipped nothing since L3.3-Shakudo-70B in Jul 2025.
 **One to watch:** `TheDrummer/Orion-26B-A4B` exists only as BeaverAI test builds (`v1a` May 24 → `v1c` Jul 10). Dead center of your band. Likely the next official release after Artemis.
 ---
 ## 4. The thing nobody puts in the headline: training context length
 This is buried in the model cards and it undercuts a lot of the spatial-coherence story:
 | Model | Base context | **Actually trained at** |
 |---|---|---|
 | Q3.5-BlueStar-v2-27B | 262k | **10,756 tokens** |
 | MS3.2-PaintedFantasy-v4.1-24B | 128k | **10,756 tokens** |
 | Trinity-Mini-Futaba | 128k | **32,768 tokens** |
 | Rocinante-XL-16B-v1 | — | user reports drift past **16–20k** |
 You cannot track scene state across a 60k-token roleplay with a model whose RP behavior was only ever reinforced at 10k. Base-model long-context ability degrades gracefully in benchmarks, but the *RP-specific* behavior these tunes install has a much shorter effective horizon. **When you evaluate, test at your real session length, not at 8k.** This is probably the highest-leverage thing in this report that no leaderboard captures.
 Related: **Gemma-4 degrades far more gracefully with context than Qwen3.6** on throughput — 32k→128k loss of **−32%** vs Qwen3.6-35B-A3B's **−65%** (dual RTX 4070 Ti). That's throughput only, not accuracy, but it's consistent with the architecture: Gemma-4 is full-attention dense; Qwen3.5/3.6 are hybrid Gated-DeltaNet linear-attention designs (3 linear blocks per 1 full-attention block), which are theoretically weaker at exact long-range state tracking despite the bigger advertised window.
 ---
 ## 5. Deployment on your rig
 ### 5.1 NVFP4 on sm_120 — the headline is W4A16, not W4A4
 **Do not ship plain W4A4 NVFP4 for long-context RP.** NVIDIA's own guidance flipped to recommending **W4A16 (`NVFP4A16`)** for sm_120/121, citing **KLD 2–4× worse for W4A4, "especially past ~10K context where activation quantization noise compounds with KV-cache lookups."** [[NVIDIA forum]](https://forums.developer.nvidia.com/t/update-for-nvfp4-model-conversion-to-use-w4a16-instead-of-w4a4/370403) That is precisely the failure mode you'd care about and it's the only source I found measuring KLD rather than MMLU at RP-relevant context lengths.
 Cheap experiment: **NVFP4 weight storage is identical between W4A4 and W4A16** — only the activation scales differ. Flipping is a `config.json` patch (set `config_groups.group_0.input_activations` to `null`), not a re-quantization.
 **The tension you should be aware of:** W4A16 gives up the FP4 tensor-core compute path, so the gain becomes pure weight-compression/bandwidth — and Benjamin Marie's comparison found NVFP4A16 shows *minimal throughput gain over INT4 AWQ*, with AWQ/AutoRound scoring slightly *better* on accuracy and ~7GB smaller on disk. The counterargument for your box: freed VRAM converts to KV cache, which converts to concurrency, which is what you actually want on a shared rig.
 **Quality at 24–32B — the size gradient is real.** Red Hat's aggregate NVFP4 recovery: 70B–235B ~99%, **~30B 97–99%**, 7B–14B ~95–98%. Per-model, the damage concentrates in reasoning: Qwen3-32B-NVFP4 scores 99.83% OpenLLM v1 but only **94.21% reasoning avg**; Qwen3-14B drops to **91.45% reasoning, 86.34% on AIME24**. NVIDIA's own QAD report states it plainly: *"for small LLMs, the accuracy drop from PTQ is often non-negligible."*
 **sm_120-specific caveats (all confirmed against upstream issues):**
 - **Silent Marlin fallback.** Backend selectors check `is_device_capability(100)` only; sm_120 fails and falls back to Marlin dequant, logging *"Your GPU does not have native support for FP4 computation."* [vLLM #47749](https://github.com/vllm-project/vllm/issues/47749) was **still open as of Jul 6 2026**. **Always grep your startup log for that warning** — if it's there, the whole exercise is moot.
 - **Dense is the healthy path.** [CUTLASS #3096](https://github.com/NVIDIA/cutlass/issues/3096) explicitly states dense FP4 GEMM works correctly on sm_120; the broken path was **grouped (MoE) GEMM**. Nearly every sm_120 NVFP4 horror story you'll read is a MoE story. This is a real argument for **dense 27B over 26B-A4B MoE** on your hardware, at least until the FlashInfer 0.6.5 / `compute_120f` path is more settled.
 - `compute_120f` (needs **CUDA 13.0**) vs `compute_120a`: ~2.7× throughput difference (39.0 vs 14.6 tok/s in the CUTLASS issue's own table).
 - `flashinfer_cutlass` has a reported **race condition causing silent memory corruption at high concurrency**; `flashinfer_cudnn` is reported safer. **Directly relevant to you as a multi-tenant operator** — toy prompts won't surface it, only soak testing will.
 - FP8 KV cache is not universally safe on sm_120 (GLM-5 requires BF16 KV). Test yours.
 Env vars people actually set:
 ```bash
 export FLASHINFER_CUDA_ARCH_LIST=12.0f
 export FLASHINFER_FORCE_SM=120f
 export VLLM_NVFP4_GEMM_BACKEND=cutlass
 ```
 **Toolchain choice matters more than it looks:** llm-compressor emits `compressed-tensors` but **does not calibrate KV-cache scales by default**, so you fall back to BF16 KV — **2× KV memory, roughly half the concurrent sessions.** ModelOpt emits per-layer `k_scale`/`v_scale` and gets you real FP8 KV. On a shared box that's the deciding factor.
 ### 5.2 MTP — the one lineage that keeps it
 The failure chain is three-deep and every stage is silent:
 1. **Loading.** `Qwen3_5ForConditionalGeneration.from_pretrained` **drops the MTP heads**. Finetune → `save_pretrained` → heads gone, no warning. I verified ArliAI's Derestricted and RpRMax file manifests: **no `model-auxiliary.safetensors`, no MTP tensors.** This is why almost no community Qwen tune has MTP.
 2. **Quantization.** Converters use allowlists and skip unknown tensor prefixes silently; GPTQ-style quantizers preserve the weights but never calibrate them, leaving effectively random values.
 3. **Serving.** Even when present, `mtp.*` / `mtp.fc` must be in `quantization_config.ignore` or vLLM runs a quantized MTP head against differently-scaled activations.
 **The fix is unglamorous:** copy the 15 MTP tensors out of the original `Qwen/Qwen3.6-27B` checkpoint and graft them onto your output shard. Published pipelines: [lna-lab/GGUF-to-NVFP4-SM120](https://github.com/lna-lab/GGUF-to-NVFP4-SM120) and AEON-7's variant. **This means you can graft MTP back onto Pantheon-Reasoning-27B**, since it descends from an MTP-preserved base — probably the single highest-value move available to you.
 **Two caveats on grafted MTP for eRP specifically:**
 - You're bolting the *base* model's draft head onto a *finetuned* target. Acceptance drops by however much your finetune moved the distribution — for an RP tune, a lot.
 - The rtx6kpro notes warn explicitly: **"abliterated models: MTP heads were trained on censored content; avoid with abliterated models."** The head predicts what the *aligned* model would say, so acceptance collapses precisely on the content that differs. Mechanism is sound; generality is my inference.
 - They also measured MTP causing a **−22% throughput regression** on sm_120 when Marlin fallback was active, because the draft heads expect native FP4 activations.
 ### 5.3 Existing NVFP4 checkpoints of RP finetunes — more than you'd expect
 Two quantizers specialize in exactly this:
 - **[ealexeev](https://huggingface.co/ealexeev)** — a pure TheDrummer shop, 9 repos, **ships `recipe.yaml` in-repo** so the recipe is reproducible: [Skyfall-31B-v4.1](https://huggingface.co/ealexeev/TheDrummer-Skyfall-31B-v4.1-NVFP4), [Cydonia-24B-v4.3](https://huggingface.co/ealexeev/TheDrummer-Cydonia-24B-v4.3-NVFP4), [Snowpiercer-15B-v4](https://huggingface.co/ealexeev/TheDrummer-Snowpiercer-15B-v4-NVFP4), [Magidonia-24B-v4.2.0](https://huggingface.co/ealexeev/The-Drummer-Magidonia-24B-v4.2.0-NVFP4)
 - **[Firworks](https://huggingface.co/Firworks)** — ~100 NVFP4 repos incl. [Cydonia-24B-v4.3-heretic](https://huggingface.co/Firworks/Cydonia-24B-v4.3-heretic-nvfp4), [Magidonia-24B-v4.3](https://huggingface.co/Firworks/Magidonia-24B-v4.3-nvfp4), [WeirdCompound-v1.7-24b](https://huggingface.co/Firworks/WeirdCompound-v1.7-24b-nvfp4)
 - **[AEON-7](https://huggingface.co/AEON-7)** — the MTP-grafting specialists. ModelOpt 0.43.0, `NVFP4_DEFAULT_CFG`, 15 MTP tensors grafted post-quantization, GatedDeltaNet layers kept BF16 (432 keys across 48 GDN layers), calibrated on `neuralmagic/calibration` 20 samples × 8192 tokens. **Publishes an RTX PRO 6000 number: 92 tok/s median, 124.7 peak, 67.7% acceptance.**
 - **[sakamakismile](https://huggingface.co/sakamakismile)** — highest volume (~57 repos), explicit `-MTP` naming convention, incl. actual creative tunes: [Carnice-V2-27b-NVFP4-TEXT-MTP](https://huggingface.co/sakamakismile/Carnice-V2-27b-NVFP4-TEXT-MTP), [Qwen3.6-27B-Fable-Fusion-MTP-NVFP4](https://huggingface.co/sakamakismile/Qwen3.6-27B-Fable-Fusion-MTP-NVFP4). Also ships `DSv4-Flash-FP8-SM120-Configs`.
 **Gemma-4 NVFP4 works** — the catastrophic vLLM bug ([#39407](https://github.com/vllm-project/vllm/issues/39407), logits saturating at the bf16 softcap ceiling and emitting `" a a a a"` forever) is in the **FP8_BLOCK** path, not NVFP4. Existing Gemma-4-*finetune* NVFP4 checkpoints: [pekkAi/G4-MeroMero-31B-NVFP4](https://huggingface.co/pekkAi/G4-MeroMero-31B-NVFP4), [AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4](https://huggingface.co/AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4), [Deaquay/G4-MeroMero-26B-A4B-NVFP4](https://huggingface.co/Deaquay/G4-MeroMero-26B-A4B-NVFP4). Gemma-4 quirks: exclude vision tower / `embed_vision` / `multi_modal_projector`, and note heterogeneous attention head dims (`head_dim=256`, `global_head_dim=512`) need multi-group KV support if you use spec decode. Gemma-4 has **no MTP** — spec decode there is EAGLE-based.
 ### 5.4 Speculative decoding at RP temperatures — probably don't
 Measured acceptance vs temperature [[DigitalOcean vLLM guide]](https://www.digitalocean.com/community/tutorials/speculative-decoding-vllm-configuration-guide):
 | Temperature | Acceptance |
 |---|---|
 | 0.0 | ~81% |
 | 0.4 | ~71% |
 | **0.8** | **~52%** |
 | **1.0** | **~38%** |
 The stated rule: below 0.5 acceptance, spec decode is net-negative. **Your RP sampling sits at 0.8–1.25.**
 Corroborating, from AEON-7's own Qwen3.5-27B NVFP4 card with a DFlash drafter: greedy **~80% acceptance → ~91 tok/s**; **sampled ~5% acceptance → ~38 tok/s** against a ~50 tok/s no-spec baseline. That's a **~24% throughput loss** from turning it on.
 And batching compounds it: spec decode gives 1.5–2.8× at low QPS but **1.4–1.8× slowdown at high QPS** when the GPU is compute-saturated. Every impressive DFlash/EAGLE number you'll see quoted is greedy decoding at concurrency 1 — the exact opposite of your regime on both axes.
 **One contradicting measurement worth replicating:** [loFT LLC](https://loftllc.dev/en/docs/tech/llm-research/qwen3-6-27b-nvfp4-mtp-vllm-benchmark/) reports Qwen3.6-27B NVFP4 + MTP=3 at **87.9% acceptance, accept length 3.64, 161 tok/s mean at temp 1.0, top_p 0.95, top_k 20** on 2× RTX PRO 6000 Max-Q. If true, native MTP heads degrade far more gracefully under sampling than external drafters do — which would be a meaningfully different conclusion. Verify before believing it.
 If you do use spec decode, vLLM ships [Dynamic Speculative Decoding](https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/) to auto-disable under load — but note [vLLM #25112](https://github.com/vllm-project/vllm/issues/25112): *"Spec decoding is not disabled at/after configured batch size."* Verify the disable actually fires.
 Free alternative worth trying: **n-gram / prompt-lookup decoding**. RP genuinely echoes its input — character cards, world info, prior turns get re-quoted — so it may pick up real acceptance at zero VRAM cost. Set `prompt_lookup_min=8`; the default of 2 causes structured-output corruption on Qwen3-class models ([vLLM #40875](https://github.com/vllm-project/vllm/issues/40875)).
 ### 5.5 Throughput reference points (all single RTX PRO 6000 unless noted)
 | Model | Precision | Single-stream | Batched |
 |---|---|---|---|
 | Gemma-4-31B | NVFP4 + FP8 KV | 40.7 tok/s @1k, 38.3 @128k | 126.0 @ 4 req |
 | Qwen3.6-27B | FP8 | 46.1 @1k, 30.4 @256k | peak 189.3 @ 5 concurrent |
 | Qwen3.6-27B | NVFP4, 256k ctx, FP8 KV | ~58 tok/s | ~119 @ 2-parallel; 64.8 GiB left for KV |
 | Qwen3.6-27B | NVFP4 + grafted MTP=3 | median ~92, peak 124.7 | 67.7% acceptance |
 | Qwen3-32B | NVFP4 vs BF16 | — | **2,050 tok/s @ conc 128** (vs 1,156 BF16 = 1.77×) |
 Note the NVFP4-over-BF16 advantage **narrows** from 2.1× at conc 64 to 1.77× at conc 128 — consistent with the argument that NVFP4's dense-model gain is weight compression (bandwidth), not FP4 math. For your throughput-first shared box: NVFP4 buys less raw compute than marketed, but a lot of freed VRAM → KV cache → concurrency.
 ### 5.6 A starting stack
 ```bash
 pip install -U llmcompressor==0.13.0        # released 2026-08-11
 # Recipe changes that matter for RP:
 #   scheme="NVFP4A16"  (weight-only, NOT plain "NVFP4")
 #   ignore=["lm_head"]
 #   calibration: your OWN RP/creative corpus, or Opus-WritingPrompts
 #   num_calibration_samples=256-512, max_seq_length=8192
 #
 # UltraChat calibration is assistant-y and sanitized — RP finetune activations
 # are out-of-distribution relative to it. The one published NVFP4 RP quant used
 # 64 samples of Opus-WritingPrompts at seq len 8192. Long sequences matter more
 # than sample count here.
 #
 # Cost on your card: ~45-60 min for a 27B; GPU-trivial (layers onloaded one at
 # a time), CPU-RAM-bound at roughly 2GB per 1B params -> ~55GB system RAM.
 # llm-compressor does NOT support tensor parallelism for quantization.
 export FLASHINFER_CUDA_ARCH_LIST=12.0f
 export FLASHINFER_FORCE_SM=120f
 export VLLM_NVFP4_GEMM_BACKEND=cutlass
 vllm serve /models/rp-27b-nvfp4a16 \
  --quantization compressed-tensors \
  --kv-cache-dtype fp8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --max-num-seqs 32
  # NO --speculative-config initially. Add only after measuring
  # draft_acceptance_rate at your real production temperature.
 ```
 **Validation gates before you trust any of it:**
 1. `grep` the startup log for `"does not have native support for FP4"` → if present you're silently on Marlin.
 2. **KLD against the BF16 parent at 16k and 32k context**, not MMLU. This is the only test that catches the failure mode you care about.
 3. If spec decode is on, log `draft_acceptance_rate` **at production temperature**. Below 0.5, turn it off.
 4. Soak-test at real concurrency — the `flashinfer_cutlass` corruption is silent and load-dependent.
 ---
 ## 6. How I'd actually evaluate these
 Nobody publishes spatial-coherence numbers for these models. Across the entire survey the only quantitative claims that exist are Gryphe's StyleTune slop metrics and zerofata's swipe-diversity numbers. **You will have to measure this yourself**, and it's not hard:
 Build ~20 adversarial scenes that bait the specific failures you care about, run each model 5× per scene at your production sampler settings, and score:
 - **Position tracking** — 3+ characters in a room, someone moves, someone leaves. Does the model place them correctly 10 turns later?
 - **Clothing/object state** — an item is removed, moved, or destroyed. Does it reappear?
 - **Anatomy/limb count** — the classic failure. Score explicit impossibilities.
 - **Knowledge partition** — character A learns something in private. Does character B act on it? (OmniToM found "Knowledge Access" is the weakest dimension across all models at 56–75% macro-F1 — this is a real, measurable, near-universal weakness.)
 - **Context depth** — run every test at 8k, 32k, and your real session length. Per §4, this is where the tunes will separate, and where none of them are trained.
 - **Thinking on vs off, same seed, same scene.** Given §2, this is the highest-information single comparison you can run, and no published benchmark has done it for RP.
 RP-Bench's own validation is a useful warning about scoring: LLM-judge methods showed **negative correlation** with community Bayesian Elo (ρ between −0.31 and −0.07), and its automated "Flaw Hunter" disagreed with human users more often than it agreed (50.7% vs 38.7%). **Use rule-based checks for state tracking** (did the model say "left hand" when the character's left arm was established as pinned?) rather than asking an LLM judge whether the scene was coherent.
 ---
 ## 7. What I could not verify
 Stated plainly so you can weigh the rest:
 - **Reddit is hard-blocked by this environment's egress policy** (403 on `reddit.com`, `old.reddit.com`, the JSON API, and domain-filtered search). The r/SillyTavernAI weekly megathreads are the single best source for practitioner reports on spatial coherence, and I got none of it. Everything here comes from HuggingFace, benchmark sites, papers, and blog coverage. **The community-consensus layer of this report is missing** — treat the rankings as evidence-based rather than user-validated.
 - **No model card in this survey makes an affirmative spatial-coherence or state-tracking claim.** I checked all of them explicitly. What exists is MeroMero-v2's training-side coherence judge, and Magistry's *disclaimer*. Any source telling you these models advertise state tracking is fabricating.
 - **Trinity-Mini-Ichthyo's card is unreadable** (gated, 401). It has the highest download count in-band and I can tell you nothing about it.
 - **Artemis-31B-v1.1 has no license set** — no tag in the API, nothing in the README. Matters if this is going anywhere commercial.
 - **The Qwen-27B-family exact parameter counts** were inconsistent across API calls (27,781,427,952 / 27,781,419,504 / 27,356,728,560 in mutually contradictory slots). The ~27.4B / ~27.8B magnitudes are safe; exact digits are not.
 - **MeroMero-v2 stage 3 being "multi-turn"** — steps, thinking-enabled, and the DeepSeek-V4-Flash logic-defect judge are all confirmed verbatim; the multi-turn detail is not.
 - **`heretic` does not preserve MTP natively.** I checked PyPI, GitHub, and the docs for any mention of MTP, auxiliary weights, or draft heads — absent from all three. The `Native-MTP-Preserved` repos are doing a manual post-hoc graft the tool doesn't do for you. Whether heretic 1.4.0 (Jun 2026) added passthrough is unverified.
 - **UGI Leaderboard's live 2026 data** — the CSV is 653kB and only the first chunk is fetchable; the visible slice runs to Nov 2025. The 2026 entries (`Huihui-Qwen3-VL-32B-Thinking`, `Ayla-Light-v2`) are unverified.
 - **EQ-Bench carries essentially no 15–32B RP finetunes** — only 9–12B Gemma derivatives. There is no Cydonia/MeroMero/Pantheon Elo, so cross-referencing UGI willingness against EQ-Bench writing quality is not currently possible for any model in this report.
 - `arxiv.org/html/2607.22732` ("Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning") — rate-limited on 6 attempts. Likely the single most on-point paper for your question. Worth retrying.
 ---
 ## Sources
 **Models:** [zerofata/G4-MeroMero-v2-31B](https://huggingface.co/zerofata/G4-MeroMero-v2-31B) · [zerofata/G4-MeroMero-26B-A4B](https://huggingface.co/zerofata/G4-MeroMero-26B-A4B) · [zerofata/Q3.5-BlueStar-v2-27B](https://huggingface.co/zerofata/Q3.5-BlueStar-v2-27B) · [Gryphe/Pantheon-Reasoning-27B](https://huggingface.co/Gryphe/Pantheon-Reasoning-27B) · [Gryphe/Pantheon-Reasoning-26B-A4B-1.1](https://huggingface.co/Gryphe/Pantheon-Reasoning-26B-A4B-1.1) · [Gryphe/Gemma-4-26B-A4B-StyleTune-V2](https://huggingface.co/Gryphe/Gemma-4-26B-A4B-StyleTune-V2) · [Gryphe/WorldSim-Opus-3.6-35B-A3B](https://huggingface.co/Gryphe/WorldSim-Opus-3.6-35B-A3B) · [allura-org/Qwen3.5-27B-Anko](https://huggingface.co/allura-org/Qwen3.5-27B-Anko) · [ReadyArt/Dark-Scarlett-v1.0-27B](https://huggingface.co/ReadyArt/Dark-Scarlett-v1.0-27B) · [ReadyArt/gemma-4-31B-it-scotoma-2](https://huggingface.co/ReadyArt/gemma-4-31B-it-scotoma-2) · [TheDrummer/Artemis-31B-v1.1](https://huggingface.co/TheDrummer/Artemis-31B-v1.1) · [TheDrummer/Skyfall-31B-v4.2](https://huggingface.co/TheDrummer/Skyfall-31B-v4.2) · [TheDrummer/Rocinante-XL-16B-v1](https://huggingface.co/TheDrummer/Rocinante-XL-16B-v1) · [sophosympatheia/Magistry-24B-v1.1](https://huggingface.co/sophosympatheia/Magistry-24B-v1.1) · [ArliAI/Qwen3.5-27B-RpRMax-v1](https://huggingface.co/ArliAI/Qwen3.5-27B-RpRMax-v1) · [Vortex5/G4-Moonlight-Dusk-26B-A4B](https://huggingface.co/Vortex5/G4-Moonlight-Dusk-26B-A4B) · [NewEden/Trinity-Mini-Ichthyo](https://huggingface.co/NewEden/Trinity-Mini-Ichthyo) · [Nimbz/Gemma-4-Gembrain-31B](https://huggingface.co/Nimbz/Gemma-4-Gembrain-31B) · [LatitudeGames/Equinox-31B](https://huggingface.co/LatitudeGames/Equinox-31B) · [llmfan46/…-Native-MTP-Preserved](https://huggingface.co/llmfan46/Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved)
 **Bases:** [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) · [Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) · [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) · [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) · [arcee-ai/Trinity-Mini](https://huggingface.co/arcee-ai/Trinity-Mini) · [mistralai/Magistral-Small-2509](https://huggingface.co/mistralai/Magistral-Small-2509) · [Gemma 4 blog](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/) · [Mistral Small 4](https://mistral.ai/news/mistral-small-4/)
 **Benchmarks:** [UGI Leaderboard](https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard) · [EQ-Bench](https://eqbench.com/) · [Fiction.liveBench @ Epoch](https://epoch.ai/benchmarks/fictionlivebench) · [Fiction.liveBench data](https://raw.githubusercontent.com/mnismt/llms-long-context-benchmark/main/src/data/benchmark.ts) · [NCP-Bench (arXiv 2608.08160)](https://arxiv.org/abs/2608.08160) · [NarrativeWorldBench (arXiv 2606.17391)](https://arxiv.org/html/2606.17391v1) · [RP-Bench](https://github.com/LeviTheWeasel/rp-benchmark) · [PlotPoints](https://plotlightstudios.com/plotpoints) · [MuSR](https://benchlm.ai/benchmarks/musr) · [LongBench Pro (arXiv 2601.02872)](https://arxiv.org/html/2601.02872v1) · [SpatialEval](https://spatialeval.github.io/) · [OmniToM (arXiv 2605.26322)](https://arxiv.org/html/2605.26322) · [Visual vs Textual CoT (ACL 2026)](https://aclanthology.org/2026.alvr-main.1.pdf) · [Mind Your Step (arXiv 2410.21333)](https://arxiv.org/html/2410.21333v4) · [Thinking in Character (arXiv 2506.01748)](https://arxiv.org/html/2506.01748v1)
 **Deployment:** [NVIDIA forum: W4A16 over W4A4](https://forums.developer.nvidia.com/t/update-for-nvfp4-model-conversion-to-use-w4a16-instead-of-w4a4/370403) · [Red Hat NVFP4 accuracy](https://developers.redhat.com/articles/2026/02/04/accelerating-large-language-models-nvfp4-quantization) · [NVIDIA NVFP4-QAD report](https://research.nvidia.com/labs/nemotron/files/NVFP4-QAD-Report.pdf) · [llm-compressor NVFP4 example](https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w4a4_fp4/) · [llm-compressor Gemma 4](https://docs.vllm.ai/projects/llm-compressor/en/latest/key-models/gemma4/) · [ModelOpt hf_ptq](https://github.com/NVIDIA/Model-Optimizer/blob/main/examples/hf_ptq/README.md) · [vLLM #47749](https://github.com/vllm-project/vllm/issues/47749) · [vLLM #39407 (Gemma 4)](https://github.com/vllm-project/vllm/issues/39407) · [vLLM #40875](https://github.com/vllm-project/vllm/issues/40875) · [vLLM #25112](https://github.com/vllm-project/vllm/issues/25112) · [CUTLASS #3096](https://github.com/NVIDIA/cutlass/issues/3096) · [SGLang #19637](https://github.com/sgl-project/sglang/issues/19637) · [vLLM recipe Qwen3.6-27B](https://recipes.vllm.ai/Qwen/Qwen3.6-27B) · [DigitalOcean spec-decode guide](https://www.digitalocean.com/community/tutorials/speculative-decoding-vllm-configuration-guide) · [vLLM EAGLE 3.1](https://vllm.ai/blog/2026-05-26-eagle-3-1) · [Why quantized LLMs lose MTP heads](https://dev.to/alanwest/why-your-quantized-llm-loses-its-mtp-heads-and-how-to-keep-them-m7h) · [lna-lab GGUF-to-NVFP4-SM120](https://github.com/lna-lab/GGUF-to-NVFP4-SM120) · [rtx6kpro NVFP4 guide](https://github.com/local-inference-lab/rtx6kpro/blob/master/optimization/nvfp4-quantization.md) · [Jarvislabs NVFP4 on RTX PRO 6000](https://jarvislabs.ai/blog/nvfp4-rtxpro-6000) · [Millstone Gemma-4-31B NVFP4](https://www.millstoneai.com/inference-benchmark/gemma-4-31b-nvfp4-1x-rtx-pro-6000-blackwell) · [loFT Qwen3.6-27B NVFP4+MTP](https://loftllc.dev/en/docs/tech/llm-research/qwen3-6-27b-nvfp4-mtp-vllm-benchmark/) · [Unsloth Dynamic NVFP4](https://unsloth.ai/docs/basics/nvfp4) · [Benjamin Marie NVFP4 vs INT4](https://medium.com/data-science-collective/nvfp4-same-accuracy-with-2-3x-higher-throughput-for-4-bit-llms-03518ecba108) · [heretic-llm](https://pypi.org/project/heretic-llm/)
@@ -0,0 +1,223 @@
 # Fleet conventions — identities, groups, paths
 Ratified by the operator 2026-09-14. This is the **pin**: new hosts are built
 to it, existing hosts converge opportunistically. It is deliberately *not* a
 migration mandate — see § Non-goals.
 Audit any host against it with:
 ```bash
 scripts/elway <host> --playbook playbooks/audit-host-conventions.yaml
 ```
 ---
 ## 1. Identity
 ### 1.1 The human account
 | | |
 |---|---|
 | **Standard** | `vh`, UID/GID 1000 |
 | **Legacy** | `lkraven` on nh3-dev, nh3-docker, nh3-extdev, irv-ml1, ana-docker, esh-docker-vm |
 `vh` matches the operator's mail (`vh@phasefinal.com`), git identity and Gitea
 namespace (`vh/*`), so it is the name everything else already uses.
 ⚠ **Do not retro-rename the six legacy hosts.** Renaming UID 1000 where the home
 directory is populated, systemd user services are lingering, SSH configs point
 at it and live agent sessions are running is real blast radius for a cosmetic
 gain — and the thing that actually mattered (a personal username owning *shared*
 infrastructure) was removed on 2026-09-14 when the docker trees went to
 `root:docker`. New hosts get `vh`; old hosts keep `lkraven` until there is an
 independent reason to rebuild them.
 ### 1.2 The ops account
 | | |
 |---|---|
 | **Name** | `infra-ops` — on every host, no exceptions |
 | **UID/GID** | **850** (pinned; see § 3) |
 | **Sudo** | `/etc/sudoers.d/infra-ops` → `infra-ops ALL=(ALL) NOPASSWD:ALL` |
 | **Home** | `/home/infra-ops`, owned `infra-ops:infra-ops`, mode **0700** |
 | **Groups** | `docker` where Docker is installed |
 `infra-ops` is the fleet's ops identity and is the same party as the `infra-ops`
 althing handle. Mode `0700` is not paranoia: these homes hold credentials
 (`~/.config/worldtree/personal-admin-token`, `~/.config/cloudflare/…`,
 `~/.config/secrets-broker/bootstrap.env`). Verified safe before adoption —
 nh3-extdev and corviduo-dev had already been running at `0700` for months,
 including systemd units that read from that home.
 ⚠ The UID is currently **1001 / 1002 / 1003 / 2001** across the fleet. See § 3
 for why that is tolerable today and what it costs.
 ### 1.3 Service accounts
 | | |
 |---|---|
 | **Name** | `svc-<service>` — e.g. `svc-arbo`, `svc-brokkr`, `svc-llm` |
 | **UID/GID** | allocated from **800–849** |
 | **Shell** | `/usr/sbin/nologin` unless the account genuinely needs to log in |
 | **Home** | `/var/lib/<service>` for daemons; `/home/svc-<service>` only if it needs one |
 ```bash
 useradd --system --uid 8NN --user-group --shell /usr/sbin/nologin \
        --home-dir /var/lib/<service> --create-home svc-<service>
 ```
 **Legacy, not renamed:** `llmuser`, `sduser`, `brokkr`, `arbotrain`, `nas`,
 `deploy`. Six accounts, six naming styles, all sitting in the human UID range
 (1000+) where `/etc/passwd` cannot distinguish them from people. That is the
 defect the convention fixes going forward.
 ⚠ **A service account must not hold interactive root.** Enforced 2026-09-14:
 `llmuser` was in `sudo`+`docker` on ana-docker and `sudo` on irv-ml1; both
 stripped. Nothing broke, and the reason is worth knowing — nearly every process
 attributed to `llmuser` by `pgrep -u` was a **container** process whose in-image
 UID happens to be 1001. A container's runtime UID has nothing to do with host
 group membership, so the groups were buying it nothing. The one genuine host
 workload (`lora-training-worker.service` on irv-ml1) sets `User=`/`Group=`
 explicitly via systemd, which does not consult the `sudo` group either.
 **Generalisation for the next audit:** `pgrep -u <user>` on a Docker host
 over-attributes badly. Check `/proc/<pid>/cgroup` for a `docker-*.scope` before
 concluding a host account is busy — otherwise a UID collision reads as an
 active service account and blocks a cleanup that carries no risk.
 ---
 ## 2. Groups
 | group | GID | membership |
 |---|---|---|
 | `docker` | **851** (pinned) | the human account + `infra-ops` |
 | `sudo` | 27 (Debian default) | the human account only |
 ⚠ **`docker` membership is root-equivalent** — anyone in it can bind-mount `/`
 into a container. The group makes access *consistent*; it does not create
 privilege separation. A future non-root deployer (a contractor, a tenant agent)
 needs a dedicated `deploy` group, not an invitation to `docker`.
 Current GIDs are **109 / 110 / 112 / 989** — assigned by `docker-ce` at install
 time. Pinning means `groupadd -g 851 docker` *before* installing Docker.
 ---
 ## 3. UID/GID allocation map
 Measured across eight hosts on 2026-09-14: dynamically-allocated system accounts
 cluster in **989–999** and descend. **800–899 is free on every host**, which is
 what makes it safe to pin there.
 ```
     0–99    OS reserved
   100–799   Debian dynamic system allocation — never pin here
   800–849   svc-* service accounts        (fleet-pinned)
   850       infra-ops  (uid + gid)        (fleet-pinned)
   851       docker     (gid)              (fleet-pinned)
   852–899   reserved for fleet-wide groups
   900–999   Debian allocates system accounts here — leave clear
  1000       the human account (vh)
  1001+      additional humans
 ```
 ### Why the existing drift is tolerable
 There is **no central identity** anywhere on the fleet — every host is
 `passwd: files systemd`. So a UID only has to be consistent where files cross
 hosts. They do: `/mnt/smithy` is NFS-mounted on both nh3-dev (`infra-ops`=1001)
 and irv-ml1 (`infra-ops`=1003), so UID 1001 on that export means `infra-ops` on
 one box and `llmuser` on the other.
 **That collision is not currently biting, and the reason is the finding:** the
 export is owned by the Synology's own UIDs (1024, 1031) which resolve on
 *neither* host, and every directory on it is `0777`. Cross-host sharing works
 today *because* the permissions are wide open.
 So aligning UIDs does not fix something broken — it **earns the right to drop
 the 777 on shared storage**. That is a real improvement and a real project. It
 is not a prerequisite for anything currently running.
 ---
 ## 4. Paths
 | path | ownership | mode | contents |
 |---|---|---|---|
 | `/opt/docker` | `root:docker` | `2775` | the deploy tree root |
 | `/opt/docker/compose/<stack>/` | `root:docker` | `2775` | `compose.yaml`, `.env` |
 | `/opt/docker/conf/<stack>/` | `root:docker` | `2775` | bind-mounted config |
 | `/opt/docker/compose/<stack>/.env` | `root:docker` | **`0640`** | stack secrets |
 | `/home/<user>` | `<user>:<user>` | `0700` | |
 The setgid bit on directories is load-bearing: it makes every file created
 afterwards inherit the `docker` group, so the scheme does not decay back into
 whoever-touched-it-last ownership.
 ⚠ **A stack dir that is a git checkout needs `safe.directory` for its deploy
 user.** git (>= 2.35.2) refuses to operate in a worktree whose top-level dir is
 owned by someone else, so moving a checkout to `root:docker` breaks any deploy
 that runs `git fetch`/`reset` as a normal user, with `fatal: detected dubious
 ownership`. That's what stalled yt-voice-clipper's webhook auto-deploy on irv-ml1
 after the 2026-09-14 normalization, found 2026-09-24 when v0.3.13 would not
 land. The fix keeps the convention: as the deploy user,
 `git config --global --add safe.directory /opt/docker/compose/<stack>` (one
 entry per repo; the `/*` wildcard form needs git >= 2.46, and irv-ml1 has
 2.39). Swept 2026-09-24: that was the only checkout under a normalized
 `/opt/docker/compose` deployed by a non-root user. esh-docker-vm's
 `/opt/docker/conf/esphome/` is a checkout but is `root:root` throughout.
 ⚠ **A stack lives under `/opt/docker/compose/<stack>/` or it is invisible.**
 Every piece of fleet automation walks that path. `talk` sat at `~/talk` until
 2026-09-14 and was silently skipped by the docker-ce upgrade's restart loop for
 exactly that reason. A stack outside the convention is not merely untidy — it is
 excluded from every tool that will ever be written.
 ### Permission-sensitive files — never widen these
 Some daemons **refuse to start** when their files are group-accessible, and they
 fail at their *next restart* rather than at the moment of the change. That
 delayed detonation is worse than a loud failure, so these are excluded by name
 *and* by mode from `normalize-docker-tree.yaml`:
 | file | required | breaks |
 |---|---|---|
 | `acme.json` | `0600` | traefik refuses to start |
 | `id_*`, `*.key`, `*.pem`, `*.pfx` | `0600` | ssh refuses a group-readable private key |
 Anything already at `0600`/`0400` is deliberate. Leave it.
 ---
 ## 5. Non-goals
 - **Not a migration mandate.** Existing hosts converge when there is an
  independent reason to touch them. Nothing here justifies a flag day.
 - **Not privilege separation.** Both `docker` membership and `infra-ops` sudo are
  root-equivalent today. This document standardises *consistency*, not least
  privilege.
 - **Not a central identity system.** LDAP/SSSD would make § 3 moot. Deliberately
  out of scope — the fleet is small enough that a pinned allocation map is
  cheaper than a directory service, and a directory service is a new SPOF.
 ---
 ## 6. Open items
 | item | state |
 |---|---|
 | `/mnt/smithy` NFS export is `0777` throughout | blocked on § 3 UID alignment |
 | Synapse appservice tokens sit in plaintext on ana-docker | flagged 2026-09-14; belongs in the vault |
 | Legacy service accounts in the human UID range | not renamed, by decision |
 ## Changelog
 - **2026-09-14** — ratified. `vh` confirmed as the standard for new hosts with
  **no retro-renames**. Stripped `sudo`+`docker` from `llmuser` (ana-docker) and
  `sudo` (irv-ml1) — see § 1.3; all 63 containers across both hosts stayed
  healthy and `lora-training-worker` stayed active. Deleted the dormant `linus` account on ana-docker
  (passwordless root, last used 2026-04-11, archived to
  `/root/account-archive/linus-20260914.tar.gz`). Standardised `infra-ops` home
  to `0700` on five hosts. Normalised corviduo-dev's docker tree, the operator
  having ruled that all ops on that host belong to `infra-ops`.
@@ -0,0 +1,146 @@
 # fv-ml1 — GPU seat inventory and model lineage
 <!-- GENERATED FILE — DO NOT EDIT BY HAND.
     Regenerate:  scripts/seat-inventory.py
     Check drift: scripts/seat-inventory.py --check
     Hand-edits are overwritten and, worse, drift silently while looking
     authoritative — which is exactly the failure this file replaced. -->
 **Generated 2026-09-14 17:45 UTC** by `scripts/seat-inventory.py`, read from the running
 containers on `100.64.0.7` — `docker inspect`, `nvidia-smi`, each model's own
 `config.json`, and the `.PROVENANCE.txt` siblings on `/tank`.
 ⚠ `.PROVENANCE.txt` lives *beside* the model directory, not inside it:
 `/tank/aimodels/<model>.PROVENANCE.txt`. `ls <model>/` will not show it.
 ## Placement, KV cache and concurrency
 | GPU | seat | VRAM | weights | KV tokens | ctx | concurrency | util |
 |---|---|---|---|---|---|---|---|
 | 0 | `vllm-cyberprev` | 46.0 GiB | 25.11 GiB | 359,372 | 262144 | **1.37×** | 0.40 |
 | 0 | `vllm-gen-small` | 35.8 GiB | 23.98 GiB | 670,142 | 262144 | **2.56×** | 0.48 |
 | 1 | `vllm-meromero-rp` | 43.0 GiB | 19.51 GiB | 320,774 | 262144 | **1.22×** | 0.52 |
 | 1 | `vllm-erp-seat` | 27.1 GiB | 15.9 GiB | 534,649 | 262144 | **2.04×** | 0.24 |
 | 1 | `vllm-reward` | 9.0 GiB | 4.41 GiB | 26,224 | 16384 | **1.60×** | 0.10 |
 | 1 | `vllm-coder` | 6.1 GiB | 2.98 GiB | 77,056 | 16384 | **4.70×** | 0.055 |
 | 1 | `vllm-embed` | 3.4 GiB | 1.12 GiB | 10,272 | 8192 | **1.25×** | 0.03 |
 | 1 | `vllm-rerank-a3` | 2.1 GiB | 1.06 GiB | — | 8192 | — | 0.03 |
 | 2 | `vllm-flash-next` | 93.1 GiB | 76.82 GiB | 344,155 | 262144 | **1.31×** | 0.96 |
 **Concurrency** = KV tokens ÷ context: how many full-length requests fit at
 once. Below ~1.0× the seat cannot hold even one conversation at its declared
 context.
 ## Lineage and quantization
 ### `vllm-cyberprev` — GPU 0
 - **serves:** `cyberprev-27b`, `cyberprev-27b-thinking`
 - **model:** `/tank/aimodels/cyberprev-nvfp4-mixed-v2`
 - **architecture:** Qwen3_5ForConditionalGeneration (qwen3_5), 64 layers
 - **quantization:** compressed-tensors / mixed-precision — W8A8 (float-quantized), W4A4 (nvfp4-pack-quantized)
 - **speculative decoding:** `{"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}`
 - **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
 ### `vllm-gen-small` — GPU 0
 - **serves:** `gen-small`, `gen-small-thinking`
 - **model:** `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`
 - **architecture:** Qwen3_5MoeForConditionalGeneration (qwen3_5_moe), 40 layers, 256 experts
 - **quantization:** modelopt / None — W4A4 (None)
 - **speculative decoding:** `{"method": "qwen3_5_mtp", "num_speculative_tokens": 3}`
 - **image:** `vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013`
 ### `vllm-coder` — GPU 1
 - **serves:** `qwen2.5-coder-1.5b`
 - **model:** `?`
 - **image:** `vllm/vllm-openai:v0.24.0`
 ### `vllm-embed` — GPU 1
 - **serves:** `Qwen/Qwen3-Embedding-0.6B`
 - **model:** `?`
 - **image:** `vllm/vllm-openai:latest`  ⚠ **floating tag**
 ### `vllm-erp-seat` — GPU 1
 - **serves:** `G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
 - **model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16`
 - **architecture:** Gemma4ForConditionalGeneration (gemma4), 30 layers, 128 experts
 - **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
 - **image:** `vllm/vllm-openai:nightly-311b3513af33bc29b4acb2fde2e9313e5e9966a0`
 ### `vllm-meromero-rp` — GPU 1
 - **serves:** `char-rp`, `char-rp-thinking`
 - **model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16`
 - **architecture:** Gemma4ForConditionalGeneration (gemma4), 60 layers
 - **quantization:** compressed-tensors / nvfp4-pack-quantized — W4A16 (nvfp4-pack-quantized)
 - **image:** `vllm/vllm-openai:v0.26.0`
 ### `vllm-rerank-a3` — GPU 1
 - **serves:** `BAAI/bge-reranker-v2-m3`
 - **model:** `?`
 - **image:** `vllm/vllm-openai:v0.24.0`
 ### `vllm-reward` — GPU 1
 - **serves:** `Skywork/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
 - **model:** `/tank/aimodels/llm/Skywork-Reward-V2-Llama-3.1-8B-AWQ`
 - **architecture:** LlamaForSequenceClassification (llama), 32 layers
 - **quantization:** compressed-tensors / pack-quantized — W4A16 (pack-quantized)
 - **image:** `vllm/vllm-openai:latest`  ⚠ **floating tag**
 ### `vllm-flash-next` — GPU 2
 - **serves:** `qwen3.8-flash-next-uncensored`, `qwen3.8-flash-next-uncensored-thinking`
 - **model:** `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`
 - **architecture:** Qwen4ExpForConditionalGeneration (qwen4_exp), 48 layers, 512 experts
 - **quantization:** compressed-tensors / mixed-precision — W8A16 (naive-quantized), W4A16 (nvfp4-pack-quantized)
 - **speculative decoding:** `{"method": "mtp", "num_speculative_tokens": 3}`
 - **image:** `vllm/vllm-openai:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657`
 ## Gateway aliases resolving to this host
 22 aliases. Ports with no listening seat are marked dead.
 | alias | port |
 |---|---|
 | `char-rp` | 8016 |
 | `char-rp-fast` | 8021 |
 | `char-rp-reasoning` | 8016 |
 | `chat-judge` | 8022 |
 | `classifier` | 8026 |
 | `classifier-large` | 8022 |
 | `coder-fast` | 8020 |
 | `erp-tune-v2` | 8098 |
 | `gemma4-26b-a4b-it-base` | 8099 |
 | `gen` | 8022 |
 | `gen-large` | 8022 |
 | `gen-reasoning` | 8022 |
 | `gen-small` | 8026 |
 | `gen-small-reasoning` | 8026 |
 | `image-judge` | 8022 |
 | `qwen-image-bench` | 8022 |
 | `sec` | 8025 |
 | `sec-reasoning` | 8025 |
 | `summarizer` | 8026 |
 | `summarizer-large` | 8022 |
 ---
 *Lineage, provenance, model cards, measured tok/s and depth results live in the hand-curated companion [`llm-seat-catalog.md`](llm-seat-catalog.md).*
 *Regenerate with `scripts/seat-inventory.py` after ANY seat change —
 model swap, quant change, context or utilization edit, or speculative-decoding
 change. Run `--check` in CI to catch a stale document.*
 > **2026-09-25:** `qwen3-embedding` (:8001) and `reranker` (:8013) LEFT fv-ml1 —
 > their vLLM seats were retired and the names now route to TEI on esh-ml1
 > (`stacks/embed-rerank`). ~6.1 GB freed on GPU 1.
 > **2026-09-25:** `vllm-reward` (Skywork, :8003, `/scalar-judge`) also LEFT fv-ml1 for
 > esh-ml1 (`stacks/reward-seat`); ~10.2 GB more freed on GPU 1 (93.1 → 77.8 GB used
 > across the day). The per-seat entry above describing it is historical.
@@ -0,0 +1,664 @@
 # Gemma-4 26B-A4B ERP/RP tune — GPU sizing adjudication
 _Measured 2026-08-24 on `ana-ml2` against
 `/tank/aimodels/gemma4-26b-a4b-it-heretic-bf16` (llmfan46 abliterated trainee)._
 Division of labour for this run: **Eitri writes the harness, brokkr-smithy-dev
 audits, infra-ops owns the GPU window and executes.** This document is the
 sizing infra-ops owes; it is arithmetic against the real checkpoint and the
 real card, not an estimate.
 ---
 ## 1. ⚠ QLoRA IS NOT AVAILABLE ON THIS ARCHITECTURE
 **The proposed shape was QLoRA r64. It cannot be run as specified**, and the
 reason is structural rather than a tuning preference.
 The checkpoint stores each layer's 128 experts as **two fused 3-D
 `nn.Parameter` tensors**, not as 128 `nn.Linear` modules:
    model.language_model.layers.N.experts.gate_up_proj    BF16 [128, 1408, 2816]
    model.language_model.layers.N.experts.down_proj       BF16 [128, 2816,  704]
 Note the absence of a `.weight` suffix — compare `mlp.down_proj.weight`
 (an `nn.Linear`) against `experts.down_proj` (a bare parameter). That is the
 tell, and it is decisive: **`bitsandbytes` 4-bit replacement walks `nn.Linear`
 modules.** A fused 3-D parameter is not one, so it is skipped and stays BF16.
 What `load_in_4bit=True` would actually buy on this model:
 | block | params | BF16 | after bnb NF4 | saved |
 |---|---:|---:|---:|---:|
 | **MoE experts** (fused 3-D — **NOT quantized**) | 22.84 B | 42.54 GiB | **42.54 GiB** | **0** |
 | lm attention (`nn.Linear`) | 1.11 B | 2.07 GiB | 0.52 GiB | 1.55 |
 | dense shared MLP (`nn.Linear`) | 0.54 B | 1.00 GiB | 0.25 GiB | 0.75 |
 | vision tower (`nn.Linear`) | 0.57 B | 1.06 GiB | 0.27 GiB | 0.79 |
 | embed (tied, normally kept BF16) | 0.74 B | 1.38 GiB | 1.38 GiB | 0 |
 | router + norms | 0.01 B | 0.02 GiB | 0.02 GiB | 0 |
 | **total** | **25.81 B** | **48.07 GiB** | **~44.98 GiB** | **~3.1 GiB** |
 **88.5% of the model is in tensors bitsandbytes cannot touch.** "QLoRA" here
 means paying the NF4 dequant tax on 6% of the weights to save 6% of the
 footprint. The premise does not survive contact with the checkpoint.
 > **Eitri: do not hard-code a `BitsAndBytesConfig` / `load_in_4bit` path.**
 > It will not error loudly — it will load, report a 4-bit model, and quietly
 > leave 42.5 GiB in BF16. Same silent-failure shape as the stale chat template.
 **The one thing that could overturn this** is a third-party fork shipping
 custom grouped-GEMM 4-bit MoE kernels for this specific architecture (Unsloth
 is the candidate). **Not chased, deliberately** — see §4, where the run fits in
 BF16 without displacing anything the fleet depends on, which collapses QLoRA's
 value to zero. If it is ever revisited, it must be *before* the harness
 hard-codes a quantization path, not after.
 **Verdict: plain LoRA on BF16 weights.**
 ---
 ## 2. What the run actually costs
 Adapter targeting `q_proj,k_proj,v_proj,o_proj` at r64, computed from the real
 tensor shapes:
 | | layers | per layer | total |
 |---|---:|---:|---:|
 | sliding-attention (q 4096, kv 2048, o 4096) | 25 | 1,507,328 | 37,683,200 |
 | full-attention (q 8192, kv 1024, o 8192) | 5 | 1,654,784 | 8,273,920 |
 | **trainable** | | | **45,957,120** (0.178% of base) |
 ⚠ **`v_proj` DOES NOT EXIST ON LAYERS 5, 11, 17, 23, 29.** Those are the
 `full_attention` layers, and `attention_k_eq_v: true` means one projection
 serves both K and V. Consequences the harness must respect:
 - PEFT matches by name suffix, so a `v_proj` target **silently produces no
  adapter** on those five layers. Do not assert a fixed adapter count.
 - Adapting `k_proj` on a global layer **adapts K and V simultaneously** — a
  different intervention than on the sliding layers. If that asymmetry matters
  to the recipe, say so explicitly rather than discovering it in the loss curve.
 ### Memory budget, batch 1, `max_seq_len` 8192
 | item | GiB | note |
 |---|---:|---|
 | base weights BF16 | 48.07 | measured: 25,805,936,206 params × 2 B |
 | adapters + grads + AdamW fp32 m/v | 0.75 | 45.96 M trainable — rounding error |
 | checkpointed layer inputs | 1.29 | 30 × 8192 × 2816 × 2 B |
 | recompute peak, one layer | ~2.5 | 8192 tok × top-8 of 128, `moe_intermediate 704` |
 | loss head, **fused/chunked CE** | ~2.0 | see the warning below |
 | CUDA context + cuBLAS + fragmentation | ~3.0 | the item `--gpu-memory-utilization` never covered |
 | **total** | **~57.6** | |
 Marginal cost per extra sequence in the micro-batch: **~2.5 GiB.**
 | micro-batch | GiB |
 |---:|---:|
 | 1 | 54.3 |
 | 2 | 56.8 |
 | **4** | **61.8** |
 | 6 | 66.8 |
 | 8 | 71.8 |
 ### ⚠ The loss head is the whole ballgame, and it is not in the brief
 `vocab_size` is **262,144** and `final_logit_softcapping` is **30.0**. One
 8192-token sequence produces **2.147 billion logits**. Through a naive HF
 `ForCausalLM` loss that is:
    BF16 logits          4.0 GiB
    fp32 upcast          8.0 GiB
    softcap tanh saved   8.0 GiB   (autograd keeps the pre-cap tensor)
    softmax + grad       8.0 GiB
    ------------------------------
    ~28-30 GiB transient, at BATCH 1
 Naive CE at batch 1 lands the run at **~85.6 GiB on a 95.6 GiB card** — it will
 appear to work and then OOM on the first long sample. At micro-batch 4 it is
 ~120 GiB and never starts. **Fused/chunked linear cross-entropy is mandatory,
 not an optimization.**
 ⚠ Honest uncertainty: Liger ships per-architecture patches and Gemma-4 MoE with
 softcapping may not have one. Three ways out, in order of preference —
 (a) generic `LigerFusedLinearCrossEntropyLoss` wired against the lm_head with
 softcapping applied inside the chunk; (b) `cut-cross-entropy`; (c) hand-rolled
 sequence-chunked CE. **This must be proven on a 10-step smoke run before the
 window is booked**, because everything else in this document assumes it works.
 ### Step count
 58.2 M tokens / 20,576 samples = **2,829 tokens/sample average** — well under
 8192, so packing matters.
 - Packed to 8192: **7,104 sequences.** At micro-batch 4 × grad-accum 4
  (effective 16) → **444 optimizer steps for the whole epoch.**
 - ⚠ That is a *small* step count. A "checkpoint every 100 steps" default gives
  four checkpoints across a multi-hour run. This is exactly why the amendment
  asked for **wall-clock-interval checkpointing, not step-count** — the case is
  now concrete, not hypothetical.
 - ⚠ **Packing must use `position_ids` + varlen/block-diagonal attention.** Naive
  concatenation bleeds samples into each other. `sliding_window` is 1024 on 25
  of 30 layers so the damage is bounded there — but the 5 `full_attention`
  layers see the entire packed sequence.
 **Open question for brokkr/Eitri:** what fraction of the 20,576 samples exceed
 8192 tokens? Below ~2%, 8192 is right. A long tail means truncation is cutting
 the ends off RP scenes, which is where the signal lives.
 ### Runtime
 Active parameters per token ≈ **3.67 B** (2.93 B routed + attention, plus the
 0.74 B tied lm_head matmul). Forward + backward + gradient-checkpoint recompute
 ≈ 6 × active × tokens = **1.28e18 FLOPs** for the epoch.
 At 10–25% MFU on a 300 W-capped Max-Q card — HF MoE paths with 704-wide experts
 are not efficient — **4 to 10 hours, most likely ~6.** Treat as a band, not a
 number; it will be measured on the smoke run.
 ---
 ## 3. Where it fits (measured 2026-08-24, 18:20 PDT)
 Card total: 97,887 MiB = **95.60 GiB** each.
 | | GPU0 | GPU1 |
 |---|---|---|
 | resident before the window | `vllm-gen` 42,508 MiB (up 3 h) | `vllm-mog-sec` 56,624 MiB + `embed` 3,304 + `reward` 9,512 + `coder` 6,158 + `rerank-a3` 2,170; Scriberr pinned here, loads on demand |
 | free | 54,741 MiB = **53.46 GiB** | 19,446 MiB = **18.99 GiB** |
 Three placements were on the table:
 - **GPU0 beside `gen`: does not fit.** 53.46 GiB free against ~57.6 GiB needed —
  short by ~4 GiB. And `gen` is only three hours old: measured footprint runs
  38.5 GiB fresh → 42.5 GiB at 3 h → 45.6 GiB at 3 days. Budgeting against the
  current number is budgeting against a moving one.
 - **GPU1 with `mog-sec` stopped: 76,070 MiB free.** Fits, but shares a card with
  four small seats and Scriberr.
 - **GPU0 with `gen` MOVED OFF: the whole card.** ← what was chosen.
 ---
 ## 4. The window, as executed
 **Operator call, 2026-08-24: move `gen` to GPU1 and stand `sec` down, so GPU0 is
 emptied completely rather than shared.** This is strictly better than training
 beside `gen`: the tune gets 95.60 GiB with no co-tenant, and the fleet's general
 seat never goes dark beyond its own ~5-minute restart.
    before:  GPU0 [ gen 42.5 ]                     GPU1 [ sec 55.3 | small seats 20.7 ]
    after:   GPU0 [ ---- empty, 95.60 GiB ---- ]   GPU1 [ gen ~41 | small seats 20.7 | ~33 free ]
 `sec` is genuinely in use and this is not free — but it is the smaller blast
 radius by a wide margin:
 | | `gen` | `sec` |
 |---|---|---|
 | aliases | 7 (`gen`, `gen-reasoning`, `chat-judge`, `image-judge`, summarizer/classifier family) | 2 (`sec`, `sec-reasoning`) |
 | standing role | the fleet's general seat; a documented always-available dependency in global `CLAUDE.md` | M.O.G.-SEC, niche |
 | measured traffic | 765 busy-engine log lines in 24 h — continuously in use | bursty; peak 8 concurrent, **last request ~5 h ago** |
 Traffic to `sec` arrives from `10.250.50.70` (the LiteLLM gateway), so the
 aliases will fail at the gateway for the duration. Per the standing rule, let
 them fail — **do not route `sec` to another model as a stand-in.**
 Both directions are playbooks, and **the order in each is load-bearing**:
    scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-open.yaml
    scripts/elway infra-ops@10.250.50.54 --playbook playbooks/ana-ml2-training-window-close.yaml
 ⚠ `gen` runs at `--gpu-memory-utilization 0.43`, which vLLM reads as a fraction
 of **total** card memory: 42,091 MiB must be *free at startup* or the engine
 refuses to boot. GPU1 has 19,446 MiB free while `mog-sec` is up. **Recreating
 `gen` onto GPU1 before stopping `mog-sec` takes the fleet's main seat down and
 leaves it down.** The open playbook stops `mog-sec` first and hard-gates on the
 freed memory; the close playbook mirrors it, because `mog-sec` needs 50,901 MiB
 of its own and cannot start until `gen` has vacated GPU1.
 ⚠ Invoke elway as `infra-ops@10.250.50.54`, not the `ana-ml2` ssh-target — that
 resolves to `lkraven`, which has no NOPASSWD sudo, and elway aborts at its sudo
 probe.
 ### ⚠ MEASURED 2026-08-24 — the estimates below this line were ~3× optimistic
 Everything above was arithmetic. This was run on the real checkpoint on GPU0
 with synthetic tokens (`/tank/erp-tune/smoke_ce.py`), and it moves the answer:
 | config | peak | verdict |
 |---|---:|---|
 | naive CE, bsz1 seq 8192 | **81.93 GiB** | fits, ~14 GiB spare |
 | naive CE, bsz1 seq 16384 | **OOM** | tried to allocate 16.00 GiB |
 | chunked CE, bsz1 seq 16384 | **65.66 GiB** | ✅ |
 | **chunked CE, bsz2 seq 16384** | **79.71 GiB** | ✅ **the run config** |
 | chunked CE, bsz4 seq 16384 | **OOM** | — |
 **The marginal cost of an extra 16,384-token sequence is ~14 GiB, not the ~5 GiB
 estimated.** The estimate modelled gradient checkpointing as storing only layer
 inputs plus a modest recompute peak; the real MoE recompute peak (8,192+ tokens ×
 top-8 of 128 experts, plus scatter/gather buffers) is far heavier. **Do not size
 an MoE run from dense-model intuition — measure it.**
 Two predictions did land exactly, which is why the rest of the model of the thing
 is trustworthy: **205 target modules** (q30/k30/v25/o30/gate30/up30/down30) and
 **74,342,400 trainable params** at r64.
 The headline: **chunked CE at seq 16384 costs 16 GiB LESS than naive CE at seq
 8192.** Chunking is not an optimisation, it is what makes brokkr's 16384
 recommendation reachable at all.
 Base load peak: **49,221 MiB**, confirming the 48.07 GiB weight figure.
 ### Revised run parameters, now that it is a whole card
 **FINAL, measured: `max_seq_len` 16384, `per_device_batch_size` 2,
 `gradient_accumulation_steps` 8** → effective batch 16, **~1,280 optimizer
 steps**, 79.71 GiB of 95.60 with ~15.9 GiB clear.
 `max_seq_len` went 8192 → 16384 on brokkr's truncation finding: at 8192 the cap
 drops **6.2% of samples but 22.4% of TOKENS** (61.2M → 47.5M), concentrated
 *entirely* in dialogue — 46% of c2-logs, 47.5% of pippa, 95.6% of bluemoon —
 which is 60% of the mix and the axis the seat exists for. Prose and fireball
 truncate at zero. p50 is 2,084 and p90 4,751, so the cost is the long tail only.
 ⚠ The 79.71 GiB figure is **worst case** — every sample in the micro-batch at the
 full cap. Samples are one-per-sequence padded to the batch max, so with p90 4,751
 the typical step sits far below it.
 ⚠ **Keep gradient checkpointing ON**, and keep `enable_input_require_grads()`
 with it. Dropping checkpointing looks like ~17% off wall-clock and instead
 forces micro-batch 1. Worse, the second call is the silent one: **without
 `enable_input_require_grads()` the frozen base produces no gradient through the
 checkpointed blocks, every adapter stays at its initialisation, and the run
 completes successfully with an inert adapter.** `prepare_model_for_kbit_training`
 used to do it as a side effect of the 4-bit path — so removing 4-bit removes it
 too, and nothing warns you.
 ### Harness changes this required (eitri-smithy `62b556b`)
 `9d64257` as audited would not have run here. Four fixes:
 1. `runtime.py` hardcoded `BitsAndBytesConfig(load_in_4bit=True)` — now a config
   key, defaulting off, per §1.
 2. Sequence-chunked CE replacing the model's own loss (the measured table above).
 3. `chat_template_path` — `apply_chat_template` resolved the checkpoint's own
   stale 365-line template and there was **no override parameter anywhere**, so
   the upstream-template requirement was not expressible in the code.
 4. Gradient checkpointing + `enable_input_require_grads()`.
 Plus `training_eligibility_override` / `overridden_blockers` /
 `substitute_controls` in the provenance manifest, and `device_map` pinned to
 device 0 so the run cannot stray onto the card holding the inference seats.
 Also fold in:
 - **Scriberr STAYS on GPU1.** (An earlier draft of this document suggested moving
  it to GPU0; that was written when training was going to live on GPU1, and it is
  now exactly backwards. GPU0 is the training card and wants no co-tenant.)
 - **Package as a `uv` venv on `/tank`, not a Docker image.** Root is at **91%
  (36 GB free)** and `/var/lib/docker` lives on it; a PyTorch training image
  would come close to filling it. `/tank` has 4.0 TB.
 - **The run is still resumable-by-design** (INV-T7 + wall-clock checkpointing).
  Nothing about a dedicated card removes that requirement — a 4–10 hour window
  is long enough that an unresumable run is a bad bet regardless of who owns the
  GPU.
 ---
 ## 5. Standing warnings that apply to this run
 - **Never render training examples through the base's own
  `chat_template.jinja`.** Every third-party Gemma-4 derivative ships a stale
  one; the trainee's is 365 lines against upstream's 390. Use
  `/tank/aimodels/gemma4-26b-a4b-it-bf16/chat_template.jinja`. Training through
  the wrong template is train/serve skew with no error — it presents as a
  tuning failure.
 - **Base path and chat-template path are config keys, not constants.** The
  trainee base already moved once (stock BF16 → `-heretic-bf16`).
 - **`--gpu-memory-utilization` sizes the KV cache only.** It does not cover CUDA
  context, graphs, or non-torch overhead — the same misreading that OOM'd the
  char-rp seat.
 - **SETTLED 2026-08-25 — merged weights are MANDATORY, and not for the reason
  we expected.** The open question was whether LoRA-on-NVFP4 hot-swap still
  silently no-ops (it did on vLLM 0.24.0, #47639). Retested on
  `vllm/vllm-openai:latest` with the NVFP4A16 base plus the run's own
  checkpoint adapter. It does not no-op — **it refuses to start:**
      AttributeError: To support LoRA for MoE model,
                      'get_expert_mapping' must be implemented
  ⚠ **This is architectural, not quantization-related.** The check lives in
  `vllm/lora/utils.py::process_packed_modules_mapping` and branches on whether
  the model is MoE; quantization is not in the condition. `gemma4.py`,
  `gemma4_mm.py`, `gemma4_mtp.py` and `gemma4_unified.py` contain **zero**
  occurrences of `get_expert_mapping` (deepseek_v2, glm4_moe, ernie45_moe and
  others do implement it). **vLLM cannot serve a LoRA on Gemma-4 at all —
  BF16 or quantized.** Merging is the only path for this architecture.
  Note this holds even though our adapter never touches experts:
  `validate_adapter_parameters` forbids per-expert params, so all 205 targets
  are attention + dense MLP. The refusal is about the *model* being MoE, not
  about what the adapter targets.
  Silver lining worth recording: a loud refusal is strictly better than the
  0.24.0 behaviour. A silent no-op ships a base model wearing the tune's name
  and passes every check that does not compare against base.
  The merge → quantize → serve pipeline is implemented and validated end to
  end at [`scripts/erp-tune-serve/`](../../scripts/erp-tune-serve/).
 ---
 ## 6. Round-1 aborted; throughput root-caused (measured 2026-08-24 22:00 PDT)
 Run-01 launched, reached step 19 of 1,312 at ~35–46 s/it, and was **killed by
 operator instruction** — not a crash, not an OOM. ETA was ~13.9 h at 8.6% MFU
 and the operator elected to root-cause before spending the window.
 Nothing was destroyed: the 609 MB encode cache, `order-manifest.jsonl`,
 `truncation-report.json` and `resume-run-01.sh` are all preserved at
 `/tank/erp-tune/run-01/`. **There are no checkpoints** — the first was due at
 step 100, so brokkr's `lora_B` inert-adapter gate never ran. That question is
 open and moves to the restart.
 Model-agnostic lessons from this investigation are in
 [`training-throughput-playbook.md`](training-throughput-playbook.md); the
 probes are at [`scripts/training-probes/`](../../scripts/training-probes/).
 What follows is Gemma-4-specific.
 > ## ⚠⚠ CORRECTION 2026-08-26 — MUCH OF THIS SECTION MEASURES THE WRONG PROCESS
 >
 > **The benchmarks below were run against the SERVING seat with
 > `attn_implementation="sdpa"` set explicitly. Training was running
 > `flex_attention` the whole time.** `ATTN_IMPLEMENTATION = "flex_attention"`
 > was a module constant passed unconditionally into `from_pretrained`, and
 > run 1's step-time distribution (n=1,445; min 11.84 / p50 19.75 / p99 30.52 /
 > max 45.79 s/it, the max being step 1's compile) confirms it stayed compiled —
 > a dynamo fallback sits in the hundreds of seconds per step.
 >
 > **FALLS** — describes sdpa, not the training run:
 > the three-point scaling fit and its 68% quadratic share; the kernel table
 > (`fmha_cutlassF/B` sm80, `EFFICIENT_ATTENTION`, attention 65.2%); the **8.6%
 > MFU** figure quoted above and throughout; the projection that elementwise
 > becomes the largest line item post-fix; and "adopt `flex_attention`" as the
 > round-two headline lever — **which round one already had.**
 >
 > **SURVIVES** — measured on the live training run:
 > the padding/bucketing win (44.3 → 20.1 s/it); the zero-pad fast-path
 > second-order effect; the eval-battery noise-floor work.
 >
 > ⚠ **Do not assume the direction of the correction.** Training's real MFU is
 > *unmeasured*, not obviously better. Flex with a BlockMask ought to beat
 > dense-masked sdpa, but that is a prediction and this investigation has been
 > unkind to those.
 >
 > The root cause was procedural, not technical, and it is written up as
 > playbook **§4.3**: run 1 recorded no attention backend in its provenance, so
 > the benchmark/trainer delta was invisible and nobody enumerated it. Run 2
 > onward records `attn_implementation_requested` **and** `_resolved`, plus the
 > torch/transformers versions and dynamo's compile counters.
 ### 6.1 Where the step time goes
 Real checkpoint, GPU0, `attn_implementation="sdpa"`, PEFT + gradient
 checkpointing + the chunked CE, fwd+bwd, best-of-2 after warmup:
 | shape | time | peak |
 |---|---:|---:|
 | 2 × 2,048 | 1.776 s | 53.2 GiB |
 | 2 × 8,192 | 11.570 s | 62.3 GiB |
 | 2 × 16,384 | **35.017 s** | 76.6 GiB |
 Fitting `t(w) = A·w + B·w²` over all three (per-sequence `w`, batch 2):
    A = 6.8715e-04 s/token        B = 8.8509e-08 s/token²
 | w | predicted | measured | linear | quadratic | quad share |
 |---:|---:|---:|---:|---:|---:|
 | 2,048 | 1.779 | 1.776 | 1.407 | 0.371 | 20.9% |
 | 8,192 | 11.569 | 11.570 | 5.629 | 5.940 | 51.3% |
 | 16,384 | 35.017 | 35.017 | 11.258 | 23.759 | **67.8%** |
 **Two terms, three points, residuals under 3 ms across an 8× range.** No fixed
 per-batch term was needed, which refutes the launch-bound hypothesis outright —
 ~3,840 expert-GEMM launches per forward are not the cost.
 Independently, the profiler kernel table (device rows only — see playbook §3.4):
 | device kernel | ms | of step |
 |---|---:|---:|
 | `fmha_cutlassB_bf16_aligned_128x64_k65536_sm80` (attn BWD) | 16,144.6 | 46.1% |
 | `fmha_cutlassF_bf16_aligned_32x128_gmem_sm80` (attn FWD) | 6,691.2 | 19.1% |
 | `cutlass_80_tensorop_bf16_s16816gemm` ×3 (dense GEMM) | 2,774.0 | 7.9% |
 | elementwise / vectorized / unrolled ×9 | 4,787.4 | 13.7% |
 | gather / Memcpy DtoD / dropout | 951.6 | 2.7% |
 | **attention total** | **22,835.8** | **65.2%** |
 **Scaling fit says 67.8% quadratic; kernel table says 65.2% attention. Two
 independent methods, 2.6 points apart.**
 ### 6.1a ⚠ 8.6% MFU was an accounting artifact — real utilisation is 17–20%
 `brokkr-smithy-dev`'s panel (arm: Bil) closed the fold by reading torch 2.13.0
 and transformers 5.9.0 at the tag. The headline dissolves the anomaly rather
 than explaining it:
    nominal work billed     27.1 TFLOPS × 34.85 s        =  9.4e14 FLOP
    dense-sliding extra     25 layers, 2 seqs, 4 passes  = +8.2e14 FLOP
    padded full layers      lose the causal skip         = +3.5e14 FLOP
    ------------------------------------------------------------------
    work actually performed                              ≈  1.8e15
    in 34.85 s                                           ≈  51–61 TFLOPS
                                                         ≈  17–20% OF PEAK
 **We divided the *intended* (windowed) FLOPs by the wall time the *dense*
 reality took.** 17–20% is squarely inside the honest stock band. The hardware
 is fine, the utilisation is fine — the run is simply doing ~2× the arithmetic
 the architecture specifies, and the excess is the sliding window being computed
 and then thrown away.
 Source-verified mechanism, no longer hypothesis:
 | file | finding |
 |---|---|
 | `masking_utils.py:292-301` `_ignore_causal_mask_sdpa` | requires `kv_length < local_attention_size` to skip the mask. 16384 ≥ 1024, so **the sliding mask ALWAYS materialises at this seq len** — not sometimes, always |
 | `sdp_utils_cpp.h:259-267`, `sdp_utils.cpp:933` | flash rejects **any** explicit mask |
 | `sdp_utils.cpp:647`, `Context.h:480-485` | **cuDNN is unreachable on sm_120 twice over** — head_dim capped at 128, and the prefer-cuDNN branch requires major 9 or 10; sm_120 is major 12 |
 | `attention.cu:1196/1759`, `kernel_forward.h:282-290` | mem-efficient has no mask gate and no head_dim cap, computes **full n×n** with the mask as additive bias; it trims only for `is_causal` |
 Dispatch order on sm_120 is flash → efficient → math → cudnn, so the 25 sliding
 layers land on mem-efficient computing dense O(n²), and **no backend on this
 stack can rescue it.** cuDNN sliding-window does not exist at all — there is no
 window argument in the public SDPA signature.
 ⚠ **Masked SDPA also blocks `enable_gqa`**, so KV gets `repeat_kv`-expanded on
 every layer — extra memory traffic riding on top of the extra FLOPs.
 ### 6.1b Backend eligibility, measured — every source claim confirmed
 Shapes-only, random weights, `sdpa_kernel()` pinning one backend at a time.
 A forced failure is information: it identifies eligibility rather than
 preference.
 **Sliding layers (25 of 30) — H_q16/H_kv8, D=256, forward at N=16,384:**
 | mask case | FLASH | EFFICIENT | CUDNN | MATH |
 |---|---|---|---|---|
 | `None` + `is_causal` | **8.8 ms** | 21.6 ms | refused | 217.5 ms |
 | explicit 1024-band 4D | refused | **55.5 ms** | refused | — |
 | explicit causal+pad 4D | refused | 57.1 ms | refused | — |
 **Global layers (5 of 30) — H_q16/H_kv2, D=512:**
 | mask case | FLASH | EFFICIENT | CUDNN | MATH |
 |---|---|---|---|---|
 | `None` + `is_causal` | refused | **45.9 ms** | refused | 334.1 ms |
 | explicit 1024-band 4D | refused | 104.4 ms | refused | — |
 | explicit causal+pad 4D | refused | 107.9 ms | refused | — |
 Refusal reasons, straight from the runtime warnings:
 - `Flash Attention does not support non-null attn_mask` — kills flash for all
  25 sliding layers, always, at this seq len
 - `Flash attention requires q,k,v … less than or equal to 256. Got 512` —
  kills flash for the 5 global layers **even fully causal**. Dvalin was right
  that they are a first-class hole
 - `head_dim should be no more than 128` — kills cuDNN on **both** shapes
 **The sliding layers run at 55.5 ms where a maskless flash path would cost
 8.8 ms — a 6.3× penalty, and it is unreachable through any config on this
 stack.** That is the whole problem in one row.
 ### 6.2 ⚠ The attention kernels are Ampere, on a Blackwell card
 `fmha_cutlass*_sm80` on sm_120. There is no Blackwell-tuned attention kernel in
 this path at all, and the forward is additionally on `gmem` — the
 global-memory fallback tier of the memory-efficient backend, selected when the
 working set will not fit in shared memory.
 This is the mechanism behind the 100%-SM / 27-TFLOPS / 304-TFLOPS-capable
 reading: the chip is saturated running a kernel generation behind on the
 dominant cost centre.
 The candidate fix is a purpose-built kernel for this architecture's mixed
 256/512 head-dim split — `zzhhjjj/gemma-triton-flash-attn`
 (`register_triton_attention()`, then `_attn_implementation = "triton_gqa"`),
 reported 9.23× over SDPA at N=16K D=256 SWA and 2.94× fwd+bwd at D=512.
 `flex_attention` + `BlockMask` is the no-new-dependency alternative.
 ⚠ **Prefer a UNIFORM backend over a per-layer split.** vLLM special-cased this
 exact mixed-head-dim architecture and measured mixed backends **8% slower** than
 uniform. And `attn_implementation` is all-or-nothing at `from_pretrained` /
 `set_attn_implementation` — per-layer routing requires a custom function
 registered on `ALL_ATTENTION_FUNCTIONS` branching on `module.head_dim` /
 `sliding_window`.
 ⚠ **FA2 is not available for this model**: it caps head_dim at 256 and the 5
 global layers are at 512. FA3 is Hopper-only. Do not bet on FA4 on sm_120.
 ### 6.3 Masking is CORRECT — and padding is what costs
 Band structure asserted directly against the real config at n=16,384:
    sliding_attention   max 1,024 allowed/row, saturates at row 1,023   PASS
 Constraints were **not** silently dropped; the 25 sliding layers were genuinely
 windowed. Run-01 was training the model we intended.
 The same probe found the mechanism nobody had measured:
 | 2D mask supplied | `full_attention` mask returned |
 |---|---|
 | `None` | **`None`** → `is_causal` fast path AVAILABLE |
 | all-ones (no padding) | **`None`** → `is_causal` fast path AVAILABLE |
 | right-padded (what `collate_mixed` emits) | 4D `16384²` → **fast path LOST** |
 **Padding is what pins the 5 global layers to an explicit mask.** The 25
 sliding layers get a 4D tensor either way — `sdpa_attention_forward` sets
 `is_causal=True` only when `attention_mask is None`, and a 1024 window cannot
 be expressed as `is_causal`.
 Isolated, same width, only the mask differing:
    2 × 16,384, no padding          35.244 s   26,048 loss targets
    2 × 16,384, 50% pad on row 1    38.567 s   19,640 loss targets
 **9.4% slower for 24% less work.**
 ### 6.4 The corpus is 29.9% padding — and bucketing is the biggest win available
 Measured off the preserved encode cache in true `SequentialSampler` order:
    records                  20,982   (3,583 rp-dialogue / 12,003 prose-chunk / 5,396 actual-play)
    seq len min/mean/max     142 / 2,752 / 16,384
    micro-batches (mb=2)     10,491
    real tokens              57,733,156
    padded tokens            82,337,318
    PADDING WASTE            29.9%
    mb width p50/p90/p99     2,092 / 10,634 / 16,341
    micro-batches at 16,384  3 of 10,491  (0.0%)
 ⚠ Note the last line against §6.1: **the 2 × 16,384 benchmark shape occurs in
 three micro-batches out of 10,491.** Weighted over the real distribution the
 quadratic share is ~51%, not 67.8%.
 **Bucket-to-pair, shuffle-to-mix** (brokkr's design, validated on measured
 lengths — form micro-batches within length buckets, then shuffle the resulting
 *micro-batches* globally):
 | bucket | waste | predicted step | zero-pad mb | roots/accum window |
 |---:|---:|---:|---:|---:|
 | current | 29.9% | 44.3 s → 16.13 h | 0.1% | 3.68 |
 | **2** | **0.0%** | **28.6 s → 10.40 h** | **78.3%** | 3.56 |
 | 8 | 0.0% | 28.6 s → 10.41 h | 65.3% | 3.54 |
 | 32 | 0.1% | 28.6 s → 10.43 h | 41.9% | 3.55 |
 | 128 | 0.6% | 28.8 s → 10.51 h | 14.7% | 3.55 |
 | 512 | 2.4% | 29.7 s → 10.82 h | 4.1% | 3.61 |
 **≥35.5% wall clock, no kernel work, no new dependency, peak memory unchanged.**
 Two findings that changed the design:
 - **Bucket size is not a diversity knob.** Roots per accumulation window are
  flat at 3.54–3.61 across a 256× range. The global micro-batch shuffle does
  all the mixing. Use the tightest bucket.
 - **35.5% is a floor.** Zero-pad micro-batches go 0.1% → 78.3%, which puts the
  5 global layers back on `is_causal` for most of the run (§6.3). The cost
  model does not capture that. Direction certain, magnitude not yet measured at
  representative shapes.
 ⚠ **Source-homogeneity is a real hazard here** — length correlates hard with
 root (kvasir short, chunked RP windows long), so length-homogeneous batches are
 root-homogeneous batches. The global micro-batch shuffle is what prevents an
 accumulation window drawing its whole gradient from one source. It is
 load-bearing, not decoration.
 ### 6.5 The chunked CE is fine — do not swap it
    2 × 16,384   CE forward  374 ms  of 35.329 s  = 1.1%
    2 ×  4,096   CE forward   93 ms  of  4.387 s  = 2.1%
 ⚠ **Forward only** — the `torch.utils.checkpoint` recompute runs inside
 `.backward()`, outside the timing window. Even at 3× it is ~3%.
 `liger-kernel` fused linear CE is a ~1–3% lever on this shape. §2's finding
 stands unchanged: chunking is what makes seq 16384 *reachable*, and it is not
 what makes it slow.
 ### 6.6 MoE is ~8% — stop optimising it
 Dense GEMM is 7.9% of the step, confirming the earlier decomposition bound of
 ~10% from the kernel side.
 On `grouped_mm`: **the trace does not adjudicate it.** Run-01 was relaunched on
 `eager`, so the profile shows the *default* path — 25,463 `aten::mm` dispatches
 in one fwd+bwd, far more than the ~90 a grouped path would produce, so the
 default is per-expert sequential. Whether the flag changes that when set is a
 different measurement and was not run. At 7.9% it is not worth running.
 ### 6.7 Restart parameters for round 2
 **Do not relaunch without the sampler change.** It is the only lever that wins
 under every branch of the diagnosis.
 1. **Implement bucket-to-pair + shuffle-to-mix** in the harness, tightest
   bucket, global micro-batch shuffle. Expected ~16.1 h → ~10.4 h or better.
 2. **Re-assert the mask band structure** after the sampler change —
   `scripts/training-probes/step0_mask.py`, 30 s, no GPU. The sampler touches
   batch composition, which is what drives mask construction.
 3. **Resume with `/tank/erp-tune/resume-run-01.sh`, NEVER the original launch
   command** — it begins `rm -rf /tank/erp-tune/run-01` and would destroy the
   609 MB encode cache (2.5 min to reuse, ~4.3 h to rebuild). ⚠ A sampler change
   alters record *order*, not encoding, so the cache stays valid — but bump
   `encode_version` if anything upstream of `input_ids` changes.
 4. **Run the `lora_B` inert-adapter gate at step 100.** It never ran in round 1.
   Norm every `lora_B` tensor in the checkpoint: all-non-zero = real, all-zero =
   INERT (kill the run), partial = module-selection problem. This is the one
   failure that stays invisible until brokkr's acceptance gate reports
   base-identical numbers.
 5. **The corpus override is ONE RUN ONLY** (`operator-2026-08-25-rnd-run`). A
   second run needs a second operator grant.
 6. **Attention backend is round 2's second lever**, gated on an A/B on the
   replica — not on argument. It can run while the tuned job trains.
 ⚠ GPU0 is currently **reserved and idle** by operator instruction; `sec` /
 mog-sec remains down. The window is still open, so
 `playbooks/ana-ml2-training-window-close.yaml` has NOT been run.
@@ -0,0 +1,119 @@
 # Gen-seat candidate evaluation — 2026-08-21
 Cold-Fusion was abandoned (see `persistent-memory.md`); the seat is on
 `qwen38-27b-heresy-nvfp4-mixed`. Two replacement candidates were put up. All facts
 below come from the HF registry and from reading the artifacts directly — the
 safetensors headers were fetched with HTTP **Range** requests, so the tensor census
 cost about a megabyte rather than a 20 GB download.
 ## The candidates
 | | `orcarouter/Qwen3.8-27B-Uncensored` | `preetpatel/…-NVFP4` |
 |---|---|---|
 | what | BF16 source weights | NVFP4 quant **of orcarouter** |
 | size | 55.6 GB | 19.7 GB |
 | base | `Qwen/Qwen3.8-27B` (**stock Qwen**) | orcarouter |
 | **MTP tensors** | **15 ✓** | **0 ✗** |
 | visual tensors | 333 ✓ | 333 ✓ |
 | scheme | n/a (bf16) | **NVFP4 W4A4** ✗ |
 | `re:^mtp.*` in ignore | n/a | **absent** ✗ |
 | traction | 3,278 dl / 60 likes | 36 dl / 0 likes |
 | gated | yes — **our token already has access** | no |
 | chat template | **sha `c3cf9e34` — byte-identical to the live heresy seat** | same |
 ## Verdict: preetpatel is disqualified, on two independent hard failures
 **1. Zero MTP tensors.** Read straight from the safetensors header: 2,672 tensors,
 **none** matching `mtp.*`. The author's own `recipe.yaml` asks to ignore
 `re:.*mtp.*`, but the written `config.json` contains no mtp ignore entry at all —
 while `re:.*visual.*` expanded to 110 explicit entries. That asymmetry is the
 signature of llm-compressor pruning an ignore pattern that matched nothing, i.e.
 the MTP head was never loaded and never quantized. It is the same
 `re:^mtp.*`-pruning trap documented in the playbook, seen from the outside.
 Cost: no speculative decoding. Our seat runs MTP at ~59% acceptance and 118 tok/s;
 without it, roughly half the decode throughput.
 **2. NVFP4 W4A4 — 4-bit activations.** `input_activations: num_bits 4, type float`.
 This is precisely the AEON failure mode we spent a multi-day saga diagnosing and
 purging: the activation-fidelity gradient is W4A4 < W4+FP8 < W4+bf16, W4A4 was
 responsible for ~15-20% stochastic degeneration, and W4A4 collapses past ~30k
 context. **The gen seat serves 262K.**
 Either failure alone would rule it out. It is also one day old with 36 downloads.
 ## orcarouter checks out as a quant source
 Stock-Qwen base (not a reasoning-compression finetune — the trait that sank
 Cold-Fusion), Arditi-et-al. single-direction abliteration, MTP and vision both
 explicitly preserved and verified at 15/333, chat template byte-identical to the
 build we are serving right now, and the gate is already accepted on our token.
 ## Third option, noted and not recommended
 `orcarouter/Qwen3.8-27B-Uncensored-FP8` — 76,109 downloads, 693 likes, far more
 traction than either candidate. **But 30.9 GB against NVFP4's 22 GB**, and GPU0 is
 zero-sum with meromero co-resident: +9 GB of weights comes straight out of the KV
 pool, taking it from ~14.4 GiB / 403k tokens to roughly 5 GiB / ~150k — which
 breaks 262K context at 1.5x concurrency. Viable only if the seat gives up long
 context or meromero moves.
 ## The imatrix constraint — read before committing to it
 The operator asked for imatrix if we quant ourselves. **This is not a switch.**
 `quant_mixed_nvfp4.py` already sets `observer="imatrix_mse"` on the W4A4 group and
 has **never once used it** — llm-compressor logs `no importance data available.
 Falling back to uniform MSE` and proceeds. Playbook §3.13 documents this and warns
 explicitly: *do not "fix" it by assuming an imatrix would help; verify first that
 your llm-compressor version can consume an externally supplied importance matrix at
 all, and in what format.* Parked as `park/…imatrix-mse…` (id 42) with the
 calibration corpus that would feed it.
 Also note the W4A16 portions of the mixed recipe are **data-free by construction** —
 llm-compressor infers `DataFreePipeline` for weight-only quantization and ignores
 calibration data entirely. Imatrix can only ever bite on the W4A4 MLP group.
 So "quant with imatrix" is two projects: an unscoped capability investigation, and
 then the ~2h quant. Recommendation is to decouple them — ship the proven recipe
 first, run imatrix as its own bounded experiment. Every A/B we hold is
 uniform-MSE-to-uniform-MSE, so a non-imatrix build stays directly comparable to
 heresy's PPL 6.910 / 47.2% acceptance.
 ## Mandatory step if we pull
 Run `services/gen-seat-mixed-quant/bench/think-leak/think_prior.py` on the bf16
 **before any GPU time**. It is a ~10s CPU measurement and it is the gate that would
 have disqualified Cold-Fusion before its 300-trial study ever ran. Prior is
 favourable — stock-Qwen base, template identical to heresy, which measures <0.002
 against Cold-Fusion's 0.185 — but measure, don't assume.
 ---
 # Addendum — M.O.G.-SEC pen-test model (same night)
 Two `Blackfrost-Research/M.O.G.-SEC-27B-1M-CTX` candidates for the pen-test
 project: a BF16 and a pre-made NVFP4. **Same verdict as gen-seat: pull the BF16,
 quant ourselves.** Read directly off the artifacts via HTTP Range.
 | | BF16 | pre-made NVFP4 |
 |---|---|---|
 | MTP tensors | 15 ✓ | **0 ✗** |
 | scheme | n/a | **ModelOpt W4A4** ✗ |
 | context | native 262K (config), 1M claimed | same |
 The pre-made NVFP4 is disqualified on **three** grounds, one unique to this model:
 ModelOpt **W4A4** (4-bit activations — the AEON degradation mode), **zero MTP**,
 and — the sharp one — **W4A4 on a 1M-context model is self-defeating**, since
 W4A4 fidelity collapses past ~30k. A long-context model quanted on the activation
 scheme that fails hardest at long context works against itself.
 The BF16 quanted cleanly (`mog-sec-27b-nvfp4-mixed`, 23.4 GB) and is **served** in
 the retired fable slot (ana-ml2 GPU1 :8019, aliases `mog-sec` / `mog-sec-reasoning`).
 Gates: format screen 1.11e-05, surface 6/6, MTP 55.3%, vision 7/3/1, and a
 capability smoke 4/4 (it delivers offensive-security content, does not refuse).
 **The 1M is not real on our path.** `rope_scaling: None` in the weights' config
 (native Qwen3.8 is 262K), and the repo's 1M is an SGLang/DFlash2 deployment kit.
 We serve native 262K. A true 1M seat would be a separate SGLang project — flagged,
 not attempted.
@@ -0,0 +1,448 @@
 # Headscale overlay mesh — plan (2026-09-06)
 **Status:** planning. Operator direction 2026-09-06: replace Site Magic (NH3↔ESH) and the
 FortiGate IPsec tunnels (colo↔NH3, colo↔ESH) with a self-hosted Tailscale-protocol overlay
 (Headscale), one ultra-light LXC per PVE host, keeping the old tunnels as an emergency
 backup. This document is the plan that a `.contract.md` will be cut from; nothing here is
 provisioned yet.
 ## Why Headscale (decided 2026-09-06)
 Tailscale clients are the best in class on the devices actually used for remote access
 (iPad, Mac, Linux); the future OPNsense colo edge can be a Tailscale node natively
 (`os-tailscale`); Headscale is one binary + one SQLite file to self-host, versus NetBird's
 six-service stack with a mandatory IdP. NetBird's built-in UI and HA routing groups are
 real but not decisive for three sites. UDP-blocked networks (in-flight Wi-Fi) work because
 the client falls back to a relay over TCP 443; plain WireGuard (ana-wg) cannot.
 ## Site facts that drive placement
 | site | edge | v4 | v6 | change risk |
 |---|---|---|---|---|
 | ANA colo | FortiGate 80F → **OPNsense on R420 within the month** | public static | pending | HIGH — edge in flux; incident history 2026 (breaker, PSU1, WAN admin closed) |
 | NH3 | UDM SE, stable | dynamic-but-stable (DDNS) | single /64, **reserved for meshing** | low |
 | ESH | UDM Pro Max | CGNAT now, **static soon**; 2G symmetric soon | /56 | low, improving |
 **Control plane at NH3 now** (only stable edge; static v6 + DDNS v4 → one DNS name).
 **Reassess ESH** once its static v4 lands — it becomes the relay site regardless (2G
 symmetric). **Never the colo:** the mesh's brain must not live in the building it exists
 to reach in an emergency. Migration later = copy one SQLite file + move the DNS name;
 clients follow the URL and never re-enrol.
 ## Topology
 ```
                       headscale.phasefinal.com  (A via DDNS, AAAA static)
                                 │ HTTPS 443 (+ DERP later)
   ┌─────────────────────────────┼───────────────────────────────┐
   │ NH3  nh3-pve                │                                │
   │   CT nh3-headscale  ──control plane (1 core / 512M / 8G)     │
   │   CT nh3-mesh-rtr   ──subnet router, advertises 10.100.0.0/16│
   ├──────────────────────────────────────────────────────────────┤
   │ ESH  esh-pve                                                 │
   │   CT esh-mesh-rtr   ──subnet router, advertises 10.0.0.0/16  │  (relay/DERP here later)
   ├──────────────────────────────────────────────────────────────┤
   │ ANA  pfi-pve                                                 │
   │   CT ana-mesh-rtr   ──subnet router, advertises 10.250.0.0/16│  (OPNsense takes this over later)
   └──────────────────────────────────────────────────────────────┘
   Per-device clients: nh3-dev, laptops/iPad, corviduo-dev, ana-ml2 … (MagicDNS names)
 ```
 Four unprivileged Debian 13 LXCs, each 1 vCPU / 512 MB / 8 GB, `onboot=1`, backed up by
 the existing `pbs-ana` job. Routers need `/dev/net/tun` passed in and `ip_forward` (both
 routine for unprivileged CTs; net sysctls are namespaced).
 Proposed ids/names (DHCP with reservation like every existing CT; recorded in
 `dns/internal.yaml` as `<name>.<site>.internal`):
 | PVE | next id | CT | note |
 |---|---|---|---|
 | nh3-pve (`root@10.100.250.60`, PVE 8.4.1) | 106 | `nh3-headscale` | control plane |
 | nh3-pve | 107 | `nh3-mesh-rtr` | subnet router |
 | esh-pve (`root@10.0.250.35`, PVE 8.4.20) | 108 | `esh-mesh-rtr` | subnet router |
 | pfi-pve (`root@10.250.250.31`, PVE 8.3.5) | 114 | `ana-mesh-rtr` | subnet router; ana-wg (113) stays as independent WG fallback |
 `debian-13-standard_13.6-1_amd64.tar.zst` is in `pveam available` on all three (not yet
 downloaded on any).
 ## Access & credentials — all in hand (verified 2026-09-06)
 | need | have |
 |---|---|
 | provision LXCs on all three PVEs | `infra-ops@` with NOPASSWD sudo on all four PVE hosts since 2026-09-06 (elway `bootstrap-infra-ops-user.yaml`; PVE needed `sudo` installed first); `root@` also works |
 | public DNS name + DDNS | Cloudflare all-zones DNS-edit token, vault `nh3-dev/.config/cloudflare/infra-ops-dns-token` |
 | NH3 UDM port-forward 443 → nh3-headscale; static routes on both UDMs | UDM API keys, vault `unifi/pfi-udmse-api-key`, `unifi/esh-udmpm-api-key` (classic `/rest/*` read+write) |
 | colo static route toward ana-mesh-rtr | FortiGate infra-ops SSH pw vaulted; reachable at 10.250.0.1 via the tunnel (`execute backup config` first). Moot once OPNsense lands |
 | TLS for headscale | Let's Encrypt via Cloudflare DNS-01 (same token) — no inbound 80 needed |
 | secrets (pre-auth keys, API key, DB) | vault under `nh3-headscale/…` |
 Nothing outstanding on credentials. Inputs still needed from the operator: confirm names/ids
 above; 443 direct on the UDM vs behind the existing Caddy on nh3-dev; ACL posture (flat
 "everything can reach everything" first, tighten later, is the recommendation).
 ## Phases
 0. **Pre-flight (no changes):** confirm NH3 v6 /64 address for the AAAA; confirm DHCP
   reservation ranges on the three sites; `pveam download` Debian 13 on all three.
 1. **Control plane:** CT `nh3-headscale`; headscale in a container or the .deb (prefer .deb —
   fewer layers in a 512M CT); Caddy/own TLS via DNS-01; UDM forward 443; DDNS updater
   (UDM → Cloudflare, or a `ddclient`/cron in the CT); DERP = Tailscale public map
   initially; headplane UI optional. Backup: `/var/lib/headscale/db.sqlite` is the only
   state — PBS covers the CT; add a nightly `sqlite3 .backup` to `backupStore` too.
 2. **First nodes:** nh3-dev + the operator's laptop/iPad enrol → prove MagicDNS, prove
   TCP-443 relay path from a UDP-blocked network.
 3. **Subnet routers:** three router CTs, `--advertise-routes` per site, approve routes in
   headscale, SNAT off (`--snat-subnet-routes=false`) so source IPs survive, static routes
   on each site gateway pointing the other two /16s at the local router. Test
   site-to-site from clientless hosts.
 4. **Cut over:** move day-to-day traffic onto the mesh; **disable (not delete)** Site Magic
   and the two IPsec tunnels. Running all three at once makes route precedence murky on
   the UDM and produces asymmetric paths. ana-wg stays as the out-of-band WG fallback.
 5. **Later:** relay (DERP) at ESH when the 2G circuit is in; OPNsense as the colo router
   node; HA router pairs (Tailscale HA subnet routers) per site; tighten ACLs.
 ## Risks / open questions
 - Control-plane outage does not drop existing tunnels but stalls new logins and key
  rotation — hence the SQLite backup and the "not at the colo" rule.
 - Tailscale public DERPs carry relayed (encrypted) traffic through third-party infra until
  a self-hosted DERP exists. Acceptable at first; fix in phase 5.
 - CGNAT at ESH: outbound-only is fine; nothing at ESH needs to be dialled inbound.
 - Headscale lags Tailscale feature-wise (tailnet lock, some ACL syntax). Not relevant to
  phases 0–4.
 - VM 106 on pfi-pve was once named "PFI-Tailscale" (pre-2026 inventory). No config
  survives; nothing to reuse.
 ## Status — 2026-09-06 05:35Z: phases 0–1 done, phase 3 half-done, NOT cut over
 Operator inputs (05:00Z): 443 direct on the UDM; names `nh3-headscale`, `nh3-scale`,
 `esh-scale`, `ana-scale`; ids my call → sequential. ACLs: flat (database policy mode,
 empty policy) until told otherwise.
 **Live:**
 | what | where | detail |
 |---|---|---|
 | control plane | CT 106 `nh3-headscale` 10.100.50.45 (nh3-pve, Debian 12, 1c/512M/8G) | headscale **v0.29.3** (.deb), `https://headscale.phasefinal.com` on **443 direct**, Let's Encrypt via TLS-ALPN-01 (no Caddy, no DNS-01), SQLite at `/var/lib/headscale/db.sqlite`, DERP = Tailscale public map, MagicDNS base `mesh.phasefinal.com`, split DNS `internal` → the three AdGuards, `override_local_dns: false`. Public health check 200 with a valid cert. |
 | public name | Cloudflare A `headscale.phasefinal.com` → NH3 WAN (70.230.226.88 at creation), unproxied, TTL 120 | **DDNS:** `~/.local/bin/headscale-ddns.sh` on nh3-dev, user timer `headscale-ddns.timer` every 10 min, token pulled from the vault at run time (nothing stored in the CT). No AAAA yet — NH3 LAN v6 is deliberately off. |
 | UDM forward | NH3 UDM SE `headscale-443`: WAN tcp/443 → 10.100.50.45:443 | added via the classic REST API. Hairpin from inside NH3 works. |
 | routers | CT 107 `nh3-scale` 10.100.50.46 (100.64.0.1) · CT 108 `esh-scale` 10.0.50.65 (100.64.0.2, Debian 13) · CT 114 `ana-scale` 10.250.50.45 (100.64.0.3) | tailscale 1.102.3, `/dev/net/tun` passed, forwarding on, each advertising its site /16, **routes approved and serving**, `--snat-subnet-routes=false`, `--accept-routes=false` (see lesson 1). |
 | first client | nh3-dev (100.64.0.4) | `--accept-routes=false`, `--accept-dns=false`. Direct paths to all three routers (7–10 ms). |
 | DNS | `*.internal` names for all four CTs synced to the three AdGuards | |
 | backups | pfi-pve + nh3-pve jobs are `all`; esh-pve job vmid list extended with 108 | |
 | vault | `headscale/preauth-router-48h-20260906`, `headscale/preauth-client-48h-20260906` (reusable, expire 2026-09-08) | mint fresh ones on the CT: `headscale preauthkeys create --user 1 --reusable --expiration 48h` |
 **Lessons (do not repeat):**
 1. ⚠ **`--accept-routes` on a client BEFORE the return path exists breaks that client's
   LAN.** nh3-dev accepted all three /16s including its OWN site's; Linux table-52 routes
   beat the main table, so every 10.100.x packet went into the tunnel to nh3-scale and
   nothing came back (no SNAT → the far host has no route to 100.64/10). Took ssh to
   nh3-pve, esh-pve and pfi-pve down for ~1 min until `tailscale set --accept-routes=false`.
   Phase order is therefore: **return path first** (either SNAT on the routers, or static
   routes for `100.64.0.0/10` → local router on every site gateway), *then* accept-routes on
   clients, and exclude a node's own site route.
 2. The routers currently find each other "direct" via LAN addresses (`10.0.50.65:41641`
   etc.) — i.e. **the mesh is riding inside the old Site Magic / IPsec tunnels**. Public-
   endpoint / relay path discovery is untested until those tunnels are disabled (phase 4).
   Test that with one tunnel down before declaring cut-over ready.
 3. pfi-pve (PVE 8.3.5) and nh3-pve (8.4.1) refuse the Debian 13 template
   (`unsupported debian version '13.6'`); esh-pve (8.4.20) accepts it. Three CTs are
   Debian 12, esh-scale is Debian 13. A failed `pct create` followed by an `>>` into the
   conf leaves a stub conf that makes `pct` say "missing arch" — delete the stub. Follow-up:
   bring pfi-pve/nh3-pve up to current 8.4.
 4. headscale 0.29 removed `randomize_client_port` and renamed
   `ephemeral_node_inactivity_timeout` → `node.ephemeral.inactivity_timeout`; with a bad
   key `headscale serve` silently falls back to defaults (127.0.0.1:8080). Always
   `headscale configtest` first.
 **Next (in order):** (a) decide the return path — recommendation: static routes for
 `100.64.0.0/10` on each site gateway → local router (keeps source IPs), SNAT as the fallback;
 (b) enrol the operator's devices with the client key; (c) test a UDP-blocked path (relay over
 443); (d) disable ONE old tunnel and re-test router paths; (e) cut over: static routes for the
 other two /16s on each gateway, disable Site Magic + IPsec; (f) v6 AAAA once NH3 LAN v6 is
 on; (g) self-hosted DERP at ESH; (h) PVE upgrades on pfi-pve/nh3-pve.
 ### 06:05Z — return path built and verified (operator approved 05:50Z)
 Static route `100.64.0.0/10` → local router on all three gateways: NH3 UDM
 (`mesh-100.64-via-nh3-scale` → 10.100.50.46, classic REST), ESH UDM
 (`mesh-100.64-via-esh-scale` → 10.0.50.65; ⚠ that vault item is a 19-line note — extract with
 `secret get unifi/esh-udmpm-api-key | grep '^key:' | awk '{print $2}'`, a raw pipe gives nginx
 400 and the NH3 key gives 401), FortiGate (`config router static`, dst 100.64.0.0/255.192.0.0,
 gw 10.250.50.45, device `servers`; config backed up to flash as `pre-mesh-route-20260906`
 first). Routers re-set to `--accept-routes=true` — a router MUST accept the other sites' routes
 or its replies to remote-LAN sources leave via its default gateway with a 100.64 source and die;
 a node never accepts its own advertised route, so the nh3-dev failure mode does not apply to
 routers.
 Verified (LAN host → gateway → local router → mesh → far router, and back): nh3-docker→ana-scale
 9 ms, esh-docker-vm→ana-scale and →nh3-scale 0% loss, ana-docker→nh3-scale and →esh-scale 0%
 loss. All three sites can reach mesh addresses from plain LAN hosts with no client installed.
 **LAN-resident clients (nh3-dev) keep `--accept-routes=false`:** they reach other sites via
 their gateway like any LAN host. `--accept-routes` is for roaming devices (laptop/iPad off-site)
 only. Consequence: a LAN-resident client is reachable by its mesh IP only from the mesh, not
 from other sites' LANs (ana-docker→100.64.0.4 fails by design; →100.64.0.1 works).
 **Enrolling GUI devices (Mac/iPad) without OIDC:** the apps do not take a pre-auth key. Point
 the app at `https://headscale.phasefinal.com`, it opens a registration page showing a
 `headscale nodes register --user pfi --key mkey:…` command; run that on nh3-headscale
 (`pct exec 106 -- headscale nodes register --user 1 --key <mkey>`). CLI Linux/macOS builds can
 use the vaulted client key instead.
 ### 06:20Z — first roaming client verified
 Operator's MacBook enrolled via the GUI (Option-click → Debug → Custom Login Server; headscale
 0.29 registration command is `headscale auth register --auth-id hskey-authreq-… --user pfi`),
 node 5 `vhlk-mba26` (100.64.0.5), "Use Tailscale subnets" on. From ESH: `ping 100.64.0.3`
 and `ssh infra-ops@ana-docker.ana.internal` both work → colo subnet route + split DNS for
 `*.internal` proven from a client. (Path was still via the ESH LAN / old tunnels.)
 ### Note — the mesh is also the durable fix for CGNAT CrowdSec false-bans
 2026-09-06: CrowdSec on ana-docker banned ESH's shared CGNAT egress (23.164.40.160,
 `custom/gitea-aggressive-crawl`) and the fortigate-mirror bouncer pushed it to the colo
 edge, blackholing Matrix/gitea/chat for the whole ESH site (see
 `~/.claude/.../memory/incident_crowdsec_cgnat_false_ban.md`). Once ESH consumes colo
 services over the mesh (100.64/10 via ana-scale) rather than the public FortiGate VIP,
 that traffic never reaches CrowdSec — a concrete win beyond replacing the tunnels, worth
 weighing when prioritising the cut-over.
 ### 06:35Z — ESH egress whitelisted in CrowdSec (operator-directed, temporary)
 `/opt/docker/conf/crowdsec/postoverflows/s01-whitelist/pfi-esh-egress.yaml` (bind mount,
 persists) whitelists 23.164.40.160; SIGHUP reload, `crowdsec -t` clean, parser loaded.
 TEMPORARY — remove when ESH gets its static IP. crowdsec stack is NOT in stacks/ canonical
 (only stacks-mirror), so this lives on the host; re-mirror with sync-stacks.sh.
 ## CUTOVER EXECUTED 2026-09-06 — IPsec dormant, mesh primary; Site Magic pending operator
 Operator goal: replace Site Magic + IPsec with headscale, tunnels dormant as backup.
 **Safety net first:** FortiGate WAN SSH enabled — `wan1 set allowaccess ping ssh`, admin
 `infra-ops` trusthost2/3 = 70.230.226.88/32 (NH3 egress) + 23.164.40.160/32 (ESH egress);
 trusthost1 10.0.0.0/8 kept. Config backed up to flash `pre-wan-ssh-cutover-20260906`.
 Verified: `ssh infra-ops@38.120.12.42` works from nh3-dev, tunnel-independent. ⚠ TEMPORARY —
 scoped to two source IPs, not 0.0.0.0; remove after the edge is retired. Drive the FortiGate
 via WAN SSH (`scratchpad/fgw.py`) during/after cutover, not over the tunnel.
 **Method (learned the hard way):** the tunnel and a mesh static route for the same /16 on the
 same gateway FIGHT — forward/return split across transports, asymmetric traffic dropped
 (broke colo from NH3 at 06:xx while both were up; rolled back). Correct order per pair:
 disable the tunnel end(s) FIRST (route withdraws), THEN add the mesh /16 route (now
 unambiguous). Never do it with the tunnel up.
 **colo↔NH3 (IPsec):** NH3 UDM `pfi-nh3-ana` disabled (enabled=false, API); mesh routes
 NH3 `10.250.0.0/16→10.100.50.46 (nh3-scale)`, FortiGate `10.100.0.0/16→10.250.50.45
 (ana-scale)`. Verified: nh3-dev→ana-docker + pfi-pve OPEN; FortiGate reachable via mesh AND
 WAN SSH; ESH→rest-server-ana (restic) OPEN. FortiGate phase1 left enabled (no peer → idle);
 re-enable NH3 side to restore.
 **colo↔ESH (IPsec):** ESH UDM `esh-ana` disabled; mesh routes ESH
 `10.250.0.0/16→10.0.50.65 (esh-scale)`, FortiGate `10.0.0.0/16→10.250.50.45`. Verified
 bidirectional: esh-docker-vm↔ana-docker/pfi-pve OPEN both ways. Proves ESH (CGNAT) mesh over
 public/DERP.
 **NH3↔ESH (Site Magic) — NOT done by API.** `sdwan-mesh-tunnel` objects (`ESH-UDMPM` on NH3,
 `PFI-UDMSE` on ESH) return `api.err.NoEdit` — Site Magic is cloud-orchestrated, not togglable
 via the classic REST API. Mesh routes PRE-STAGED (NH3 `10.0.0.0/16→nh3-scale`, ESH
 `10.100.0.0/16→esh-scale`), currently SHADOWED by Site Magic (traffic still on the SD-WAN
 tunnel, traceroute via 192.168.1.x transit). esh-scale↔nh3-scale hold a DERP(lax) 9ms path
 independent of Site Magic → when the operator disables Site Magic in the UniFi UI / Site
 Manager, the staged routes take over immediately. **Operator to toggle; mesh verified ready.**
 **Dormancy = disabled-but-retained**, not auto-failover (overlapping /16s across two
 transports reintroduce the asymmetry). Re-enable a pair = flip the UDM object back to
 enabled=true (+ re-enable Site Magic in UI). No automatic mesh→tunnel failover is wired.
 **Bonus (operator noted):** exit nodes at each site give a free multi-location egress proxy
 (`tailscale up --advertise-exit-node` + approve); replaces the nh3-dev SOCKS5 residential
 egress with a native feature. Not yet configured — parked follow-up.
 **Follow-ups:** operator disables Site Magic; (optional) disable FortiGate phase1 ends for
 tidiness; advertise exit nodes; self-hosted DERP at ESH on the 2G circuit; remove the
 FortiGate WAN-SSH trusthosts when the edge is retired; PVE 8.4 upgrade on pfi-pve/nh3-pve.
 ## ✅ CUTOVER COMPLETE 2026-09-06 — all three site-pairs on the mesh
 Operator disabled Site Magic in the UniFi UI. NH3↔ESH re-homed to the mesh immediately —
 traceroute now esh-scale(10.0.50.65)→nh3-scale(100.64.0.1)→dest, no 192.168.1.x SD-WAN
 transit; esh-scale↔nh3-scale hole-punched a DIRECT public path (70.230.226.88:41641, 8ms),
 not even DERP. Full matrix verified, all six directions OPEN:
 NH3↔colo, NH3↔ESH, colo↔ESH. FortiGate mgmt via WAN SSH OPEN; ana-wg WG fallback untouched.
 State: **Site Magic disabled (UI); both IPsec tunnels dormant (UDM ends disabled, config
 retained); headscale is the sole active site-to-site transport.** Tunnels re-enablable for
 backup (Site Magic in UI; IPsec by flipping the UDM objects enabled=true). Goal met.
 ## 2026-09-06 — exit nodes advertised (multi-location egress proxy)
 All three routers advertise + serve exit nodes; approved on headscale (each shows
 `0.0.0.0/0` + `::/0` Serving). Clients see three: `nh3-scale` (100.64.0.1, NH3 residential
 egress 70.230.226.88), `esh-scale` (100.64.0.2, ESH 23.164.40.160), `ana-scale`
 (100.64.0.3, colo 38.120.12.42).
 ⚠ **Design reversal:** exit nodes REQUIRE SNAT, so `--snat-subnet-routes` was flipped back to
 **true** (NoSNAT=false) on all three routers — this undoes the earlier source-preserving
 choice. Consequence: cross-site subnet-route traffic now appears to come from the local
 router's IP, not the original host. Site-to-site re-verified OK after the flip. If per-host
 source visibility is later needed, the alternative is NoSNAT + a manual selective masquerade
 rule (`-s 100.64.0.0/10 ! -d 10.0.0.0/8 -o eth0 -j MASQUERADE`) on each router, made
 persistent — not done.
 **Use one from a client:** `tailscale set --exit-node=nh3-scale` (or `esh-scale` /
 `ana-scale`), `--exit-node=` to clear. A node that itself advertises an exit node cannot
 consume one (so the routers can't be used to test egress; use the laptop/iPad).
 Replaces the nh3-dev SOCKS5 residential-egress proxy — retire that once confirmed on real
 clients.
 ## 2026-09-06 — selective masquerade applied (supersedes the SNAT-reversal note above)
 Restored `--snat-subnet-routes=false` (NoSNAT=true) on all three routers AND added a
 persistent selective-masquerade rule, so BOTH properties now hold: exit nodes work (internet
 egress masqueraded) and site-to-site keeps the original host source.
 Per router: `/usr/local/sbin/mesh-exit-masq.sh` + `mesh-exit-masq.service` (oneshot, After
 tailscaled, enabled). Rule = a `MESH-EXIT` nat chain jumped from POSTROUTING for
 `-s 100.64.0.0/10 -o eth0`: RETURN for 10/8, 172.16/12, 192.168/16, 100.64/10 (source
 preserved), MASQUERADE the rest (real internet). ⚠ nf_tables iptables forbids multiple `-d`
 in one rule — hence the chain-with-RETURNs shape, not a single `! -d ... ! -d ...` rule.
 Verified: nh3-docker→ana-docker seen at colo as **10.100.50.40** (real host, preserved);
 nh3-dev via ana-scale exit → egress **38.120.12.42** (colo), via nh3-scale → 70.230.226.88
 (NH3). Persistent across CT reboot via the systemd unit. `tailscale set --operator` NOT set;
 drive nh3-dev's tailscale via `ssh infra-ops@10.100.10.50 sudo tailscale ...`.
 ## 2026-09-06 — throughput baseline (mesh vs the old IPsec)
 NH3↔colo over the direct WireGuard path (5ms), iperf3 between the 1-vCPU router CTs, medians
 of 3: **download colo→NH3 761 Mb/s (688–761), upload NH3→colo 464 Mb/s (391–507)**, parallel-8
 up ~530. Old IPsec ceiling was ~250 both ways (poor AES offload) — so ~3x down, ~1.8x up. CPU
 during upload sampled ~32% (single reading) → not clearly the cap; upload likely limited by
 userspace-WireGuard single-flow, the CT vNIC, or the WAN. Operator: the ANA AT&T line is
 SHARED fiber, so the upload figure may be the line, not the routers — left as-is (no CT core
 bump). Baseline for future comparison, not a hardware limit.
 ## 2026-09-06 — irv-ml1 folded in ADDITIVELY (operator inaccessible 5 days — lifeline preserved)
 Constraint: operator has NO physical access to the Irvine building for 5 days; irv-ml1 must
 not go dark unrecoverably. Its ONLY remote path is the bespoke WireGuard tunnel `wg0`
 (10.100.79.3 → nh3-dev, AllowedIPs 10.0/16+10.100/16+10.250/16); its LAN 10.6.110.50 is not
 reachable from the fleet.
 Enrolled irv-ml1 as headscale node **100.64.0.6** (`infra-ops@10.100.79.3` has NOPASSWD root;
 `tailscaled` was already running but logged out with STALE prefs → **public** Tailscale
 `controlplane.tailscale.com` + `RouteAll:true`). Used `tailscale up --reset --login-server
 https://headscale.phasefinal.com --authkey … --accept-routes=false --accept-dns=false
 --hostname irv-ml1`. ⚠ **accept-routes MUST stay OFF**: wg0's AllowedIPs are the exact /16s the
 mesh routers advertise, so accepting them would fight the lifeline over the prefix my SSH
 returns on = 5-day lockout. Armed an auto-revert dead-man switch (`tailscale down` in 10m
 unless disarmed) before the change; disarmed via sentinel after verifying.
 Result: **DUAL PATH** — wg0 (10.100.79.3, untouched, fleet routes still via it) AND mesh
 (100.64.0.6). Both verified from nh3-dev. Reboot-safe: `wg-quick@wg0` + `tailscaled` both
 enabled, prefs now headscale + accept-routes=false + WantRunning=true. Mesh peering currently
 rides over wg0 as transport (11ms).
 ⚠ **DEFERRED to the on-site window (do NOT do remotely):** retire wg0; enable accept-routes /
 subnet-routing over the mesh; migrate yt-voice-clipper egress off the dante SOCKS proxy. Each
 touches the exact routing that is the lifeline. When on-site (physical console as fallback):
 flip accept-routes on, confirm fleet reachability over the mesh, THEN down wg0.
 ⚠ **Lesson (repeat):** `pkill -f` / `pgrep -f <pattern>` over SSH self-matches when the pattern
 is in your own ssh argv — killed my session twice disarming the switch. Kill by explicit PID,
 or make the switch sentinel-based (which saved it here).
 ## 2026-09-06 — irv-ml1 CUTOVER COMPLETE + dante retired (done remotely, 5-day no-access)
 Operator wanted the cutover finished tonight despite no physical Irvine access for 5 days, and
 directed: dante proxy goes away (exit nodes instead), yt-voice-clipper may stay dead until
 scoped exit-node egress is wired.
 **Three independent recovery paths built BEFORE cutting wg0** (the discipline that made it safe):
 1. Mesh over DERP-LA (~12ms) — irv-ml1's own WAN (38.120.94.3, easy NAT), independent of wg0.
 2. **Reverse SSH tunnel** `revtun-nh3.service` on irv-ml1 → nh3-dev via NH3 public WAN
   (UDM forward `irv-revtun-ssh` tcp/47822 → 10.100.10.50:22, **src-restricted to 38.120.94.3**;
   key `/root/.ssh/revtun_ed25519`, authorized on nh3-dev lkraven with `restrict,port-forwarding`).
   Reach irv-ml1: `ssh -i ~/.ssh/infra-ops_ed25519 -p 2201 infra-ops@127.0.0.1` on nh3-dev.
   Independent of BOTH wg0 and the mesh/control-plane.
 3. **`wg0-watchdog.service`** on irv-ml1 — `wg-quick up wg0` if the fleet is unreachable over the
   mesh for ~5min (auto-failover). Plus wg-quick@wg0 stays enabled → reboot restores wg0.
 **Cut:** accept-routes=true on irv-ml1 (fleet /16s → tailscale0 table 52), then `wg-quick down
 wg0` over the reverse tunnel under a 12-min auto-revert dead-man (disarmed after verify).
 Verified: irv-ml1 reaches nh3-docker/colo/ESH/nh3-pve over pure mesh; unreachable ONLY 10.100.10.50
 (nh3-dev's own LAN IP — a mesh-node-reaching-a-mesh-node quirk; every other host fine; moot now
 dante is gone). wg0 dormant (down, enabled). Recovery via reverse tunnel + mesh IP both OK.
 **Dante retired** on nh3-dev: `danted` disabled+stopped, :1080 closed, `/etc/danted.conf` →
 `.retired-20260906`. Only live consumer was YTVC (operator-exempted). searxng ref was already
 commented out.
 **OPEN follow-ups (on-site or later):** wire yt-voice-clipper egress via a SCOPED exit-node path
 (do NOT set irv-ml1 `--exit-node` globally — it would route the reverse tunnel through the mesh and
 kill the independent lifeline; use tailscale `--socks5-server`+exit-node or a per-container netns);
 then bring YTVC back. On-site: decide whether to keep the reverse tunnel + UDM forward + revtun key
 or remove them (small WAN attack surface, src-restricted). Fully retire wg0 config if desired.
 ## 2026-09-06 — wg0 made non-reboot-restorable + Irvine subnet discovery (operator corrections)
 **wg0 no longer reboot-restorable** (supersedes the earlier "wg-quick@wg0 stays enabled" note):
 `systemctl disable wg-quick@wg0` on irv-ml1 — the cutover is real, wg0 does not silently return on
 reboot. Config `/etc/wireguard/wg0.conf` retained so the failover paths can still bring it up. The
 FAILOVER is now: `wg0-watchdog.service` (`wg-quick up wg0` if the fleet is unreachable over the mesh
 ~5min — works regardless of the unit's enable state) + the independent reverse tunnel. Rationale
 (operator): reboot-restore was redundant with the watchdog and made the cutover half-real; accept-routes
 table-52 would keep the mesh primary anyway, but a clean cutover shouldn't resurrect the retired tunnel.
 **Irvine subnet discovery LIVE (bidirectional):** irv-ml1 now `--advertise-routes=10.6.110.0/24`
 (approved on headscale, serving; ip_forward=1; default SNAT-on so mesh-unaware Irvine hosts reply to
 irv-ml1). Fleet-wide reach added via gateway static routes `10.6.110.0/24 → local mesh router`
 (NH3 UDM→nh3-scale, ESH UDM→esh-scale, FortiGate route 22→ana-scale). Verified: ana-docker→10.6.110.1
 and esh-docker-vm→10.6.110.50 both OK over the mesh. So irv-ml1 both REACHES the fleet /16s
 (accept-routes) and EXPOSES its Irvine LAN to the fleet (advertise-routes) — a full mesh subnet router.
 ## 2026-09-18 — NH3↔Anaheim was relayed, not direct; fixed at the Anaheim edge
 For months every NH3→Anaheim flow crossed Tailscale's LA DERP relay rather
 than a direct path. `tailscale ping` from nh3-scale reported `direct
 connection not established` after 8 attempts, all pongs via `DERP(lax)` at
 373–522 ms. DERP is a throttled fallback, not a data plane, so this cost both
 a fixed round-trip penalty and a hard bandwidth ceiling on everything
 crossing that hop — LiteLLM, Beszel, task-board, vor, the Henge.
 **Cause.** ana-scale advertised `38.120.12.42:41641`, but the Anaheim NAT was
 not preserving the port (netcheck mapped it to `:60798`) and `PortMapping` was
 empty — no UPnP/NAT-PMP — so inbound hole-punching always failed.
 `MappingVariesByDestIP: false`, so not symmetric NAT; simply no reachable
 inbound endpoint. ESH↔Anaheim and NH3↔ESH were already direct; only this pair
 failed.
 **Fix.** Four additive objects on ana-gw (10.250.0.1) giving ana-scale a
 stable inbound endpoint. The box runs `central-nat enable`, so a policy's
 `dstaddr` is the real internal address rather than the VIP — the existing
 `wg-to-ana-wg` VIP+policy pair is the house template.
 | object | value |
 |---|---|
 | `firewall address` | `ana-scale-ip` → 10.250.50.45/32, iface `servers` |
 | `firewall service custom` | `Tailscale-41641` → `udp-portrange 41641` |
 | `firewall vip` | `tailscale-to-ana-scale` → 38.120.12.42:41641 udp → 10.250.50.45:41641, extintf wan1 |
 | `firewall policy` id 75 | wan1→servers, all→`ana-scale-ip`, `Tailscale-41641`, accept |
 **Measured before → after:**
 | path | before | after |
 |---|---|---|
 | `tailscale ping` nh3-scale→ana-scale | 373–522 ms via DERP(lax) | **6 ms direct** |
 | STT via the Anaheim LiteLLM gateway (96 kB clip) | 1.399–1.449 s | **0.237–0.270 s** |
 | Beszel HTTP, nh3-dev→ana-docker | 0.94–1.29 s | **0.014–0.016 s** |
 ⚠ **This edge has no out-of-band access.** Back up before touching it —
 `ssh infra-ops@10.250.0.1 'show' > file` captures the full non-default config
 without needing a tftp server — and make additive changes only. Credentials
 are vaulted at `fortigate/ana-gw-infra-ops-password`.
 ⚠ **Still outstanding: irv-ml1 (100.64.0.6) remains `relay "lax"`.** Same class
 of problem at a different site; this change does not address it.
@@ -0,0 +1,174 @@
 # ESH IPv6 naming scheme
 Every ESH LAN carries an **eight-hex-digit phrase** as the first half of the
 interface identifier. Picked 2026-08-18/19. This file is the canonical record.
 > **Why this file exists.** The scheme originally lived as a single line in
 > `persistent-memory.md` and was silently deleted by a `memory: snapshot`
 > commit (`837fa36`). Recovering it took a hunt through session transcripts to
 > find the commit that had held it. A naming convention is not temporal state —
 > it belongs in a document, so it now is one.
 ## The names
 | network | hex | reads as |
 |---|---|---|
 | `Default` | **`4BA5:3417`** | A BASE FOR IT |
 | `esh-mgmt` | **`15DA:B055`** | IS DA BOSS |
 | `esh-server` | **`4411:B105`** | FOR ALL BIOS |
 | `esh-userland` | **`CAFE:4411`** | CAFE FOR ALL |
 | `esh-iot` | **`4DBA:D107`** | FOR DA BAD IOT |
 | `esh-cameras` | **`1533:FACE5`** | I SEE FACES |
 | *(reserved)* DMZ | **`4411:DBAD`** | FOR ALL DA BAD |
 The DMZ name is **claimed against a network that does not exist yet** — there is
 no DMZ on the ESH UDM. Do not reuse it.
 Substitutions are the standard hexspeak set: `0`→O, `1`→I/L, `5`→S, plus letters
 that are already native hex (`A`–`F`). Anything outside `0-9a-f` is not
 expressible — `b0ss` and `c00l` do **not** work, which is why the set above uses
 `B055` and avoids `c00l` entirely.
 House style, arrived at rather than designed: **eight digits, and a complete
 phrase rather than a single word.** Words are allowed to straddle the group
 boundary (`4DBA:D107` is `4·D·BAD·107`); the phrase reads through the colon.
 ## Address structure
 ```
 2607:73c0:402:1d02 : 4411:b105 : 50 : 45
 └──── ISP /64 ────┘ └ segment ─┘ └ 10.0.50.45 ┘
 ```
 - **Prefix** — Cityside's, not ours to name. ESH holds a `/56`
  (`2607:73c0:402:1d00::/56`, 256 × /64); the subnet id is assigned by UniFi's
  `ipv6_pd_prefixid`. `esh-cameras` is `1d00`, `esh-server` is `1d02`.
 - **Segment word pair** — 32 bits, from the table above.
 - **Host** — the last two IPv4 octets, written as literal digits so they read
  straight off the address. `10.0.50.45` → `:50:45`.
 The scheme lives entirely in the **interface identifier**, so it is
 **delegation-size independent**. It works identically on a `/56`, a `/48`, or
 NH3's single `/64`. It never competes with the subnet id, which is far too small
 to hold a word (8 bits at ESH — two hex digits).
 Note: `4411:b105:50:45` fills all four host groups, so there is **no `::`** in
 these addresses. Writing `...::4411:b105:50:45` is malformed and will be
 rejected.
 ## What can and cannot carry a name
 | slot | nameable? |
 |---|---|
 | `/64` subnet id (`ipv6_pd_prefixid`) | **No** — 8 bits at ESH, two hex digits, no room for a word |
 | the gateway's own address | **No** — fixed at `::1` by UniFi, no field for it |
 | a UniFi *client* reservation | **No** — UniFi has no IPv6 equivalent of `use_fixedip` |
 | **a host taking its own address** | **Yes** — this is the one that works |
 ⚠ **The original note concluded these names could never appear in a `dig` or
 `ip -6` output. That is wrong.** The first three rows are correct, but they only
 establish that *UniFi* cannot assign the address. A Linux host can simply take
 one within its own advertised prefix, and the router gets no vote. Appliances
 with no shell — cameras, most IoT — genuinely cannot, so `1533:FACE5` and
 `4DBA:D107` are likely to stay documentation-only.
 ## Applying it to a host
 Do **not** use an `iface … inet6 static` stanza: on Debian that sets
 `accept_ra=0`, killing SLAAC and the IPv6 default route — a good way to strand a
 headless box. Use an `if-up.d` hook that derives the live prefix instead.
 Live example, `/etc/network/if-up.d/ipv6-scheme-addr` on `esh-docker-vm`:
 ```sh
 #!/bin/sh
 [ "$IFACE" = ens18 ] || exit 0
 (
  i=0
  while [ $i -lt 30 ]; do
    PFX=$(ip -6 -o addr show dev "$IFACE" scope global 2>/dev/null \
          | awk '{print $4}' | cut -d/ -f1 | head -1 | cut -d: -f1-4)
    if [ -n "$PFX" ]; then
      ip -6 addr replace "${PFX}:4411:b105:50:45/64" dev "$IFACE" && exit 0
    fi
    sleep 2
    i=$((i + 1))
  done
 ) >/dev/null 2>&1 &
 exit 0
 ```
 Three deliberate properties:
 - **The prefix is derived, never hardcoded** — self-heals if Cityside
  re-delegates.
 - **Backgrounded with a retry** — SLAAC may not have landed when `if-up.d` runs,
  and a hook that blocks or fails would stall interface bring-up.
 - **Additive** — `/etc/network/interfaces` already sources `interfaces.d/`;
  nothing existing is edited, and removal is one `rm`.
 Remaining gap: a *mid-life* prefix change is only picked up at the next
 interface-up. A timer would close it; not worth building until the prefix is
 observed to actually move.
 ## Deployed
 All three Linux hosts on `esh-server` now carry the segment name, with the last
 two groups reading straight off their IPv4 address:
 | host | address | v4 | applied via |
 |---|---|---|---|
 | `esh-docker-vm` (AdGuard) | `2607:73c0:402:1d02:4411:b105:50:45` | 10.0.50.45 | `if-up.d` on `ens18` |
 | `esh-pve-nas` | `2607:73c0:402:1d02:4411:b105:50:55` | 10.0.50.55 | `if-up.d` on `vmbr0` |
 | `esh-vm-db` | `2607:73c0:402:1d02:4411:b105:50:60` | 10.0.50.60 | `if-up.d` on `ens18` |
 `esh-docker-vm`'s is load-bearing, not decorative: the ESH UDM advertises an
 IPv6 resolver to clients via RDNSS, macOS prefers it over the DHCPv4-supplied
 one, so whatever sits there is what resolves `*.internal` for every Mac on the
 network. It previously pointed at AdGuard's **MAC-derived SLAAC address**, which
 would have broken if that VM's NIC ever changed. Both `esh-userland` and
 `esh-server` now advertise the scheme address instead
 (`dhcpdv6_dns_auto=false` + `dhcpdv6_dns_1=<address>`), verified on the wire by
 soliciting an RA and parsing option type 25.
 ### ⚠ Proxmox bridges need `accept_ra=2` or SLAAC never runs
 `esh-pve-nas` had **link-local only** despite `accept_ra=1`, `autoconf=1` and
 IPv6 enabled — every sysctl looked correct. The cause: **`vmbr0.forwarding = 1`**
 (Proxmox sets per-interface forwarding on bridges), and the kernel ignores RAs on
 a forwarding interface unless `accept_ra` is explicitly **`2`**. `accept_ra=1`
 means "accept only if not forwarding", so it silently did nothing.
 Fixed in `/etc/sysctl.d/60-ipv6-accept-ra.conf` on that host:
 ```
 net.ipv6.conf.vmbr0.accept_ra = 2
 net.ipv6.conf.vmbr0.accept_ra_defrtr = 0
 ```
 `accept_ra_defrtr=0` is deliberate — it takes the advertised **prefix** (so
 SLAAC configures an address) while **declining the default route**, so a
 hypervisor gains an IPv6 identity with no change to its routing behaviour.
 Verified after: SLAAC address present, v6 default routes still **0**, v4 intact.
 Expect the same on any other Proxmox node when its LAN gets IPv6.
 ### Getting into a host with no direct root
 `esh-vm-db` refuses key auth for `root` and `infra-ops`, and `lkraven`'s sudo
 wants a password. It is VMID 101 on `esh-pve`, and the **QEMU guest agent** runs
 as uid 0 inside it, so the hook was installed with:
 ```
 qm guest exec 101 -- /bin/sh -c 'echo <base64> | base64 -d > /etc/network/if-up.d/... '
 ```
 base64 because quoting a multi-line script through two SSH layers mangles it.
 Worth remembering as the general path for guests whose credentials are not
 vaulted.
 Every Linux host on `esh-server` now carries its name. The remaining ESH
 segments have no eligible hosts: `esh-cameras` and `esh-iot` are appliances
 with no shell, and `esh-mgmt`, `esh-userland` and `Default` are still
 `ipv6_interface_type: none` pending the firewall-policy pass — enabling SLAAC
 there gives every client a globally reachable address.
@@ -0,0 +1,159 @@
 # LLM seat catalog — fv-ml1
 The durable, curated record of **what each LLM seat IS**: lineage, provenance,
 model-card facts, quantization, speculative decoding, and measured speed/depth.
 **This file is hand-curated and complements the generated inventory.** The two split
 by volatility:
 | file | owns | updated by |
 |---|---|---|
 | [`fv-ml1-gpu-seat-inventory.md`](fv-ml1-gpu-seat-inventory.md) | **live** placement, VRAM, KV tokens, concurrency, quant format, gateway aliases | `scripts/seat-inventory.py` (read-only, auto) |
 | **this file** | **durable** lineage, provenance, cards, licenses, measured tok/s + depth results, "why this seat" | **by hand**, on seat change or re-bench |
 Where they overlap (ctx, concurrency, VRAM), the **inventory is authoritative for the
 live number**; the values here are point-in-time and dated. If they disagree, the
 inventory won and this file is stale — fix it.
 ## Keeping this current
 Update this file whenever a seat changes — model swap, quant change, context/KV edit,
 or a new seat. Two commands regenerate the inputs:
 ```bash
 scripts/seat-inventory.py            # live placement/KV/concurrency (docs/pfi/fv-ml1-gpu-seat-inventory.md)
 scripts/seat-bench.py                # warm tok/s + deep-prefill OOM check (serial; prints the numbers below)
 ```
 ⚠ **Speed and depth numbers are measurements, not facts about the weights** — they
 carry a date and a harness. Re-run `seat-bench.py` after any context/KV/quant/spec
 change and update the table with the new date. A number without its harness is not a
 result (see the measurement-discipline note at the bottom).
 ---
 ## Summary (measured 2026-09-14, harness below)
 | seat | GPU | model | ctx | conc. | warm tok/s | VRAM | depth verified | OOM |
 |---|---|---|---|---|---|---|---|---|
 | **cyberprev** (sec) | 0 | Qwen3.8-27B cyber-SFT (abliterated base) | 262,144 | 1.37× | 136.6 | 47.1 GiB | 259,722 tok | none |
 | **gen-small** | 0 | Qwen3.6-35B-A3B Heretic | 262,144 | 2.56× | 254.8 | 36.1 GiB | 254,526 tok | none |
 | **gen** (flash-next) | 2 | Qwen3.8-Flash-Next (orcarouter) | 262,144 | 1.31× | 170.7 | 95.3 GiB | 254,273 tok | none |
 | **char-rp** | 1 | G4-MeroMero-v2-31B (Gemma4 dense) | 262,144 | 1.22× | 62.7 | 42.3 GiB | 254,858 tok | none |
 | **char-rp-fast** | 1 | G4-MeroMero-26B-A4B (Gemma4 MoE) | 262,144 | 2.04× | 225.2 | 27.0 GiB | 254,823 tok | none |
 | **coder** | 1 | Qwen2.5-Coder-1.5B (base) | 16,384 | 4.70× | 337.3 | 5.6 GiB | 15,905 tok | none |
 Support (non-generative): **reward** Skywork-Reward-V2-Llama-3.1-8B (9.2 GiB, GPU1),
 **embed** Qwen3-Embedding-0.6B (3.4 GiB, GPU1), **rerank** bge-reranker-v2-m3
 (2.1 GiB, GPU1). **gen (flash-next) is on GPU2 and off-limits to rearrangement;
 GPU3 is reserved scratch (empty).**
 ---
 ## Seats
 ### cyberprev — the `sec` / `sec-reasoning` seat (GPU 0, :8025)
 - **Serves gateway:** `sec`, `sec-reasoning`. Displaced mog-sec 2026-09-14.
 - **Lineage:** `Qwen/Qwen3.8-27B` → `hotdogs/Qwen3.8-27B-abliterated` (abliterated base)
  → **offensive-security tool-calling LoRA** trained on `hotdogs/cyber-sft-agent-qwen38`
  (8,400 rows, 22 pentest tools: nmap/sqlmap/metasploit/hydra/…), merged @ scale 1.0
  = `hotdogs/Qwen3.8-27B-abliterated-cyber-preview` → **in-house name-repair** (the raw
  export shipped 850/1199 body tensors with a tripled `language_model.` prefix — an
  unsloth export bug, see [[reference_unsloth_tripled_prefix_export_bug]]) → **house
  mixed NVFP4 quant** → `/tank/aimodels/cyberprev-nvfp4-mixed-v2`.
 - **It is a cyber SFT finetune**, NOT "an abliteration" — the abliteration is inherited
  from the base; the new capability is the cyber tool-calling SFT. Card: tool-call
  format 0/6→6/6, correct tool 0/6→6/6, general capability held (KL 0.04 general /
  0.81 tool = surgical re-target). "Will not refuse" per its card.
 - **Quant:** compressed-tensors mixed — NVFP4 W4A4 on MLP layers 0-55, FP8 W8A8 on
  attn/linear_attn/lm_head/MLP 56-63, FP8 KV. 15 MTP tensors grafted (bf16), vision
  tower + `re:^mtp.*` in ignore. Pipeline: `services/gen-seat-mixed-quant/`.
 - **Speculative:** dflash k=7 (drafter `qwen38-27b-dflash2-drafter`), ~2.77 mean accept len.
 - **Context:** native 262,144, **depth-probed clean to 259,722 tokens** (non-repeating
  prompt). Notable: same base arch as the retired mog-sec, which crashed above ~163,840
  on processing depth — cyberprev does not.
 - **License:** apache-2.0.
 ### gen-small — the fast A3B tier (GPU 0, :8026)
 - **Serves gateway:** `gen-small`, `gen-small-reasoning`, and backs `summarizer` +
  `classifier` (the bulk/triage aliases). Stood up 2026-09-14.
 - **Model:** `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only`
  → `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`.
 - **Lineage:** `Qwen/Qwen3.6-35B-A3B` (3B-active MoE, 256 experts, hybrid GDN+attn) →
  **Heretic v1.3.0** abliteration (MPOA — Magnitude-Preserving Orthogonal Ablation, the
  house-favorite method) → NVFP4 experts-only quant. **3.8 was requested but there is no
  general Qwen3.8 A3B** (the 3.8 MoEs are Flash-Next and the 2.4T), so this is the 3.6
  fallback.
 - **Card:** 88% fewer refusals (10/100 vs 83/100 original) at 0.0015 KL divergence
  (quality preserved). 19 MTP tensors preserved (native).
 - **Quant:** modelopt NVFP4, experts-only (256 experts NVFP4; attn/shared-expert/
  linear_attn bf16). Serves as-is, no re-quant.
 - **Speculative:** qwen3_5_mtp k=3, **measured 69.6% acceptance / 3.09 mean length** — MTP
  is a clear win here.
 - **Context:** native 262,144, depth-clean to 254,526. Cheap KV (A3B + fp8) → 2.56×
  concurrency on only 8 GiB KV.
 - **License:** apache-2.0 (per base `Qwen/Qwen3.6-35B-A3B`).
 - **Why:** high-volume, low-caliber work (summarization, classification, triage) belongs
  on a fast 3B-active seat, not the premium gen seat.
 ### gen — flash-next (GPU 2, :8022) — off-limits to rearrangement
 - **Serves gateway:** `gen`, `gen-large`, `gen-reasoning`, `summarizer-large`,
  `classifier-large`, `image-judge`, `chat-judge`, `qwen-image-bench`.
 - **Model:** `orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4` → **in-house PLE bf16→FP8
  conversion + `layer_types` rename** → `/tank/aimodels/qwen38-flash-next-orcarouter-nvfp4-plefp8`.
  See [[reference_qwen4exp_ple_loader_branch_order]].
 - **Lineage:** `Qwen/Qwen3.8-Flash-Next` (176B total: 125B main + 51B n-gram PLE table,
  ~6B active) → orcarouter uncensored NVFP4 → in-house PLE→FP8 so vLLM's
  `Qwen4ExpPLEEmbeddingMethod` selects the FP8 path.
 - **Architecture:** the only seat whose weights don't fit its card — the 51B PLE table
  lives in **pinned host RAM**, read over CUDA UVA. GDN linear-attn + QSA hybrid.
 - **Quant:** compressed-tensors mixed (W8A16 attn / W4A16 experts) + FP8 PLE.
 - **Speculative:** MTP k=3, ~60.4% acceptance.
 - **Context:** 262,144, depth-clean to 254,273. Fastest prefill of the fleet (27.7s/254K).
 ### char-rp — the char quality tier (GPU 1, :8016)
 - **Serves gateway:** `char-rp`, `char-rp-reasoning`.
 - **Model:** `/tank/aimodels/meromero-v2-nvfp4-work/G4-MeroMero-v2-31B-NVFP4A16` —
  in-house build, **Gemma4 dense, 60 layers**, heretic-abliterated, compressed-tensors
  NVFP4 W4A16.
 - **Speed:** 62.7 tok/s warm — the slowest seat, inherent to a dense 31B; this is the
  **quality** tier. Deep prefill 267s/254K (slowest). Gemma4 sliding-window attn
  (window 1024) keeps KV cheap at depth.
 - **Context:** 262,144 @ 1.22×, depth-clean to 254,858.
 ### char-rp-fast — the char speed tier (GPU 1, :8021)
 - **Serves gateway:** `char-rp-fast`.
 - **Model:** `/tank/aimodels/G4-MeroMero-26B-A4B-it-uncensored-heretic-NVFP4A16` —
  in-house build, **Gemma4 MoE (26B-A4B)**, heretic-abliterated, compressed-tensors
  NVFP4A16.
 - **Speed:** 225.2 tok/s warm — the throughput answer to char-rp's quality. 2.04×
  concurrency (KV pinned ~8.5 GiB). Depth-clean to 254,823.
 - **Context:** 262,144 @ 2.04×.
 ### coder — FIM code-completion (GPU 1, :8020)
 - **Serves gateway:** `coder-fast`. Backs Zed edit-predictions.
 - **Model:** `Qwen/Qwen2.5-Coder-1.5B` (base, unquantized), fp8 KV.
 - **Speed:** 337.3 tok/s (smallest model, fastest). Context 16,384 @ 4.70×.
 - **Note:** util-sized; the ~4.7× overshoots the "2-3×" intent because the 1.5B
  weight+overhead floor (~4.2 GiB) sits just under the util knob's resolution. Hitting
  ≤3× reliably needs a `--kv-cache-memory` byte pin (compose change), deferred.
 ---
 ## Benchmark harness (state it with any number above)
 - **warm decode tok/s:** greedy (temperature 0), **conc=1** (single stream), **n=3**
  reps, median, fixed ~40-word prompt → 300 output tokens. Decode throughput —
  generation is never prefix-cached, so reps are valid; spread was <1% on every seat.
 - **deep prefill / OOM:** one **non-repeating** random prompt at ~0.97× max-model-len,
  8 output tokens. PASS = returns AND the seat's allocator log shows **no OOM / CUBLAS /
  illegal-memory** across the probe window (verified 2026-09-14, 0 hits on all seats).
 - **Measured serially** (one seat at a time) — no cross-seat contention. These are clean,
  uncontended, single-stream **ceilings**; real aggregate throughput under concurrency is
  higher per-GPU and lower per-request. Re-derive with `scripts/seat-bench.py`.
@@ -0,0 +1,811 @@
 # Model quantization playbook — the lessons that keep costing us hours
 **Read this before starting any new quant.** Not the per-model runbooks — those are worked
 examples of a *specific* model at a *specific* point in time, and several carry claims that are
 now false (see §7). This file owns the **transferable** part: what recurs regardless of which
 model dropped this week.
 Written 2026-08-15, after the fourth quant in five weeks re-discovered the third-known instance
 of the same loader-class bug. Scope: NVFP4 / FP8 / mixed-precision on the Blackwell boxes
 (ana-ml2), vLLM-served. Ampere (irv-ml1) has no native FP4/FP8 — see §6.
 **Maintenance rule.** When a quant teaches you something *model-agnostic*, it lands here and the
 per-model README links up. When it's model-specific (this checkpoint's odd tensor names, this
 finetune's missing config), it stays in the per-model artifact. If you find yourself writing a
 "Gotchas" section that repeats §3, you are re-litigating — add the delta here instead.
 ---
 ## 1. The 60-second decision: which scheme
 On Blackwell + vLLM, for a dense-or-hybrid VL model you intend to serve at long context:
 | want | scheme | notes |
 |---|---|---|
 | **default, best speed/accuracy** | **mixed: NVFP4 W4A4 bulk MLPs + FP8 W8A8 attention/`lm_head`/last-8-layer MLPs** | the current answer. §2. |
 | max fidelity, don't care about prefill | NVFP4 **W4A16** (weight-only) | forces the **Marlin** kernel — ~half the prefill of native FP4 |
 | small model, VRAM is free | FP8 **W8A8** | safe and simple; 2× the weight bytes of 4-bit |
 | — | ~~"W4A8" = NVFP4 weights + FP8 activations~~ | **DOES NOT EXIST.** §3.1 |
 **Measured on Qwen3.8-27B (2026-08-15), W4A16 → mixed:** decode +18%, prefill **+78–98%**,
 MTP acceptance unchanged, perplexity +1.7%, weights −19%.
 Note the shape of that: **decode barely moves, prefill nearly doubles.** Decode at batch-1 is
 memory-bandwidth-bound and the weights are 4-bit under either scheme, so there is little to win;
 prefill is compute-bound, which is where native FP4 tensor cores replace the Marlin
 dequantize-to-BF16 path. If someone promises you a big *decode* win from a scheme change, be
 skeptical — and go measure §5 before believing it.
 **The accuracy cost is real and is paid on purpose.** Operator ruling 2026-08-15: the ~1.7%
 perplexity is an acceptable price for the speed. Settled — don't re-litigate. For correct
 attribution: it is the **activation**-quantization cost (A4/A8 vs BF16 activations), *not* an MTP
 cost. Turning MTP off does not recover it; only reverting the quant does.
 ---
 ## 2. The reference recipe (mixed-precision)
 Lifted from `unsloth/Qwen3.8-27B-NVFP4` and replicated in-house. **Prefer replicating a published
 recipe from a reputable quantizer over inventing one** — they have already paid for the
 sensitivity analysis.
 | group | scheme | targets |
 |---|---|---|
 | `group_0` | FP8 W8A8 — channel weights (static) + per-token dynamic activations | `self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`, `lm_head`, **the last 8 layers' MLPs** |
 | `group_1` | NVFP4 W4A4 — `tensor_group` gsize 16, fp8 scales, `imatrix_mse` weights, `dynamic:"local"` activations | **all remaining** MLP `{gate,up,down}_proj` |
 | kv cache | FP8 static tensor | |
 | ignore | vision tower, `linear_attn.{norm,in_proj_a,in_proj_b}`, `re:^mtp.*` | |
 Three things in there are load-bearing and easy to drop:
 - **Late layers stay FP8.** Holding the last ~8 layers' MLPs (and `lm_head`) at 8-bit is the
  accuracy-preservation trick — late layers are the sensitive ones. Uniform W4A4 is what collapses.
 - **`imatrix_mse` on the W4A4 weights**, not `memoryless_minmax`. Importance-weighted; needs
  calibration data.
 - **Group targets must be non-overlapping.** Do not let `group_1`'s `.*mlp\..*` also match the
  late layers and rely on group precedence to sort it out. Enumerate the early layers explicitly
  (`re:.*layers\.([0-9]|[1-4][0-9]|5[0-5])\.mlp\.…`) and **prove it** with a dry run (§4.1).
 **Toolchain:** `pip install llmcompressor` into stock `vllm/vllm-openai:latest` gives
 llmcompressor 0.13 + compressed-tensors 0.18 without disturbing torch/transformers.
 **Avoid nvidia-modelopt** — see §3.4.
 ---
 ## 3. The recurring landmines
 Ordered by how much time each has cost. Every one of these has bitten more than once.
 ### 3.1 "W4A8" is not a servable shape
 vLLM's compressed-tensors dispatcher (`compressed_tensors.py:704-713`) accepts NVFP4 weights with
 **exactly two** activation settings:
 | `input_activations` | result |
 |---|---|
 | `None` | W4A16 — and it **forces the Marlin kernel** (`kernels/linear/__init__.py:881-883`) |
 | NVFP4 | W4A4, native |
 Anything else — **FP8 included** — raises at load:
 ```
 ValueError: For NVFP4 weights, input quantization must also be NVFP4 format, None for NVFP4A16
 ```
 `CompressedTensorsW4A8Fp8` exists but is **INT4** weights (`W4A8_SUPPORTED_TYPES_MAP = {4: int4}`)
 gated on `_check_scheme_supported(90, match_exact=True)` — Hopper-exact, so on Blackwell (sm_120)
 it is closed twice over. **FP8 enters per-layer-group, never as activations on NVFP4 weights.**
 *Cost: one queued task written against an impossible scheme.*
 ### 3.2 Wrong loader class → silent weight-load failure
 **Rediscovered three times.** Load the model through the class vLLM actually serves — the
 `…ForConditionalGeneration` / `…ForImageTextToText` **wrapper**, never `AutoModelForCausalLM`.
 `AutoModelForCausalLM` resolves a VL config to the text-only inner class and saves a **flat**
 config with `model.layers.*` keys. vLLM's weight mapper wants `model.language_model.*` (+
 `model.visual.*`). The mismatch does not error — **every layer silently fails to load** and you
 get `!!!!` gibberish, or an engine that rejects the checkpoint outright.
 *Bit: heretic2 (gibberish), Dark-Scarlett (both vLLM and SGLang refused the checkpoint), and the
 2026-08 rounds.*
 ### 3.3 The MTP head — three separate ways to lose it
 Speculative decoding is a large fraction of the seat's throughput. It fails **silently**: the
 model serves fine, just at 0% acceptance.
 1. **The wrapper class does not instantiate `mtp.*`,** so the quant drops it. Post-quant you must
   graft the BF16 `model-mtp.safetensors` back and register its tensors in the output index.
 2. **`re:^mtp.*` must be in `quantization_config.ignore`** — else vLLM loads the grafted BF16 head
   as though quantized, it comes up **uninitialised**, and acceptance is 0%.
 3. **⭐ llm-compressor PRUNES `ignore` entries that matched no module at quant time.** Since the
   wrapper never loaded `mtp.*`, the entry matches nothing and is **silently deleted from the
   saved config — even though you put it in the recipe.** So it must be re-injected *after* the
   graft, and then **verified, not assumed.**
 *Cost: three rounds. The verify step caught it live on the third.*
 There is also a **modelopt-format-specific** version of this: vLLM 0.24 does not propagate
 modelopt `exclude_modules` to the spec-decode *draft* model, which no checkpoint config can fix
 (needs a `sitecustomize` runtime patch). Using compressed-tensors avoids it entirely — §3.4.
 ### 3.8 ⭐⭐ Multi-turn degeneration from TWO real compounding causes — how they masked each other
 The most expensive diagnosis this project has had, because there were **two real
 causes at once** and each partial fix moved the needle enough to look like *the*
 answer. Recorded precisely because the first write-up of this section
 over-attributed it to the quant alone; that was wrong.
 **Cause 1 (real, upstream): the vLLM `qwen3_5_mtp` × Gated-DeltaNet bug.**
 Confirmed by two cross-frontier peers and the tracker (vllm#47087 symptom-twin,
 #43559 fix lineage, #51113 fix): the GDN recurrent state cannot roll back on a
 partial draft-accept, so speculative decoding corrupts it, worse with context.
 Architectural — vLLM/SGLang/llama.cpp mainline all shared it. **Genuinely fixed
 enough** by moving to vLLM **nightly** (`v0.27.2rc1.dev150+`, carries #51113):
 the operator reported it "significantly better" — this was a real bug, not just
 an amplifier.
 **Cause 2 (real, quant): full W4A4 is mildly subpar, per the known gradient.**
 `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4` is **full** W4A4 — 4-bit
 *activations* on attention too, the bottom of the activation-precision ordering
 already in §1: **W4A4 (A4) < W4+FP8 (A8) < W4+bf16 (A16)**. Not "defective," just
 lowest-fidelity; on top of Cause 1 it degenerated ~15-20% of real multi-turn
 generations. The FP8-attention **mixed** build (`qwen38-27b-uncensored-nvfp4-mixed`,
 same base, same MTP, same nightly) sits a rung up that gradient and is coherent.
 AEON was purged 2026-08-17 (operator ruled it no-good; re-pullable from HF).
 **Why it cost days — and the process lessons that stand:**
 1. **Two real causes compound and mask each other.** Each mitigation (MTP-off,
   APC-off, the nightly #51113 fix) partially helped, so each looked like the fix
   and then failed in real use. When a mitigation "helps but doesn't fix," suspect
   a *second* cause rather than a wrong one.
 2. **Stochastic degeneration (~15-20%) is nearly invisible to a small synthetic
   probe** — a 7-turn run passes ~4 in 5. n=1 "clean" proves nothing; this class
   needs many runs or the operator's real high-volume use. Three non-fixes were
   "validated" by a single clean probe here.
 3. **Isolate the WEIGHTS in parallel with the serving flags, not after.** Swapping
   to a different quant of the same base (AEON→mixed) is what finally separated
   Cause 2 from Cause 1; doing it earlier would have shortened the hunt. But note
   it would NOT have found Cause 1 — the vLLM bug was real and needed the nightly.
 4. **Prefer FP8 attention (the §2 mixed recipe) over full W4A4** for a coherence-
   sensitive seat. AEON passed every static gate (abliteration 4/4, surface 6/6, a
   36k needle, 52% acceptance) and was still the lower-fidelity of the two.
 Current primary gen: the mixed FP8-attention build on pinned vLLM nightly with
 MTP, until the DavidAU Qwen3.8 lands. A W4+bf16 (W4A16) build would be higher
 fidelity still (§1) at a prefill cost — an option if the mixed build ever proves
 marginal.
 ### 3.7 ⭐ A LOADED MTP head can still corrupt output — Qwen3.8 multi-turn
 §3.3 is about *losing* the head (0% acceptance, silent). This is the opposite and
 worse failure: the head loads, acceptance looks healthy, single-turn output is
 perfect — and then it **corrupts multi-turn conversations** once cumulative context
 passes **~2,000 tokens**. The reply collapses in length *and* bleeds earlier turns
 into the current answer (a "describe durian" reply that contained the Krebs-cycle
 and winter answers from three turns back). Single-turn probes and the acceptance
 gate (§5) **do not catch it** — it only appears as accumulated context grows.
 Isolated 2026-08-16 (operator-confirmed), each step measured on a fixed 7-turn probe:
 - **Not the serving gateway, not sampling, not repetition/template.** Identical
  input gateway-vs-direct behaves the same; presence_penalty 1.5/0.5/0.0 all
  collapse; higher temperature collapses harder; a conversation of *unrelated*
  topics collapses at the same ~2k tokens as a repetitive one → it is context-
  length-driven, not template lock-in.
 - **Model-independent across every Qwen3.8-27B quant** (AEON W4A4, unsloth
  FP8-attn, our in-house mixed) — so not a quant-brand or scheme artifact.
 - **DECISIVE: same model + same conversation, MTP OFF → coherent through 4k+
  tokens, zero bleed.** Toggle it back on → collapse returns. MTP is the cause.
 **Qwen3.6-27B running the same `qwen3_5_mtp` method is CLEAN.** So the 3.6 MTP
 head/graft is fine and the 3.8 one is not — suspects: the bf16 graft being subtly
 wrong for the 3.8 head, or the vLLM `qwen3_5_mtp` impl diverging at `num_speculative_tokens=3`.
 Open upstream question (queried dvalin/bil-smithy 2026-08-17).
 **Rule: gate MTP on a MULTI-TURN coherence probe, not just single-shot acceptance.**
 Run a 7-turn varied-topic conversation and watch turns past ~2k cumulative tokens
 for length-collapse and cross-turn bleed.
 ⛔ **THE APC-OFF MITIGATION DID NOT HOLD — SUPERSEDED 2026-08-17, see §7.** What
 follows is kept for its history and **must not be applied**: *"disable prefix caching,
 keep MTP. The corruption is gated on MTP × prefix-caching together (vllm#43559 /
 #47194); with `--no-enable-prefix-caching` the buggy partial-accept align-path is
 inert. Confirmed on our stack: AEON W4A4, MTP on + prefix-caching off → the 7-turn
 varied series stays coherent through 3.9k tokens, zero bleed, at 104.6 tok/s / 53.6%
 acceptance."*
 **It passed that synthetic 7-turn probe and the operator still saw severe degeneration
 in real use.** Reverted the same day. The probe was structurally under-covering the
 real workload on both content distribution and depth — which is §3.7's own standing
 rule (*gate on a multi-turn coherence probe, not single-shot acceptance*) failing at
 one level up: the multi-turn probe was itself too small to gate on.
 **WHAT ACTUALLY RESOLVED IT.** The multi-day hunt root-caused to the **AEON W4A4 quant
 being defective** — ~15-20% of generations went degenerate — with MTP, prefix-caching
 and the gateway all merely *amplifying* it. That is why every partial mitigation
 "helped" without fixing anything (§3.8). The gen seat today runs the in-house
 Heretic **mixed NVFP4+FP8** build (FP8 attention, not W4A4) on vLLM nightly carrying
 #51113, with **MTP ON and prefix-caching ON**, and is coherent in real use.
 Verified against the live seat 2026-08-26: `vllm-gen` runs `--enable-prefix-caching`
 with `qwen3_5_mtp` / `num_speculative_tokens 3`. The compose file
 (`stacks/gen-seat/compose.yaml`) carries the full history inline and is the current
 authority; this section was stale against it for nine days.
 Things that do **not** work, ruled out: `num_speculative_tokens=1` (corruption is
 depth-independent — reproduces at n=1 and n=2, deterministically probed upstream);
 switching engine (vLLM / SGLang / llama.cpp mainline all share the GDN-rollback
 bug — it is architectural). The proper upstream fix (vllm#51113) is in `main` /
 `v0.27.2rc0` only — not in a stable release, so we hold at APC-off until it lands.
 Two cross-frontier peers (dvalin/bil-smithy) confirmed the bug class and pointed
 at the open symptom-twin issue #47087.
 ### 3.15 ⭐⭐ Fused 3-D MoE experts are INVISIBLE to a `targets=["Linear"]` recipe
 **Symptom: none.** The quant completes, the artifact loads, and 88.5% of the
 model is still BF16. Nothing warns you.
 Modern MoE checkpoints store each layer's experts as **two fused 3-D
 `nn.Parameter` tensors**, not as N `nn.Linear` modules. Gemma-4 26B-A4B:
    model.language_model.layers.N.experts.gate_up_proj   BF16 [128, 1408, 2816]
    model.language_model.layers.N.experts.down_proj      BF16 [128, 2816,  704]
 Note the **absent `.weight` suffix** — that is the tell. `mlp.down_proj.weight`
 is an `nn.Linear`; `experts.down_proj` is a bare parameter.
 Measured on that checkpoint, recipe targeting `["Linear"]`:
    Linear modules   427
    WILL quantize    205   (experts: 0)     <- 22.84 B params untouched
 **This is the same defect that killed QLoRA on this architecture** —
 `bitsandbytes` 4-bit replacement also walks `nn.Linear` modules and also
 silently skipped the experts. Two different tools, one blind spot, because the
 blind spot is in the *checkpoint layout*, not the tool.
 **The fix** (llm-compressor ≥ 0.12):
 ```python
 from llmcompressor.modeling.moe.linearize import linearize_moe
 model = SomeForConditionalGeneration.from_pretrained(...)
 linearize_moe(model)          # BEFORE building the recipe
 ```
    Linear modules   11947
    WILL quantize    11725  (experts: 11520)   # 30 layers x 128 x 3 proj
 `linearize_moe` unfuses the 3-D parameters into per-expert
 `experts.N.{gate,up,down}_proj` Linears. **No registration is needed** if the
 module satisfies `FusedExpertsProtocol` structurally — bare `down_proj` plus
 `gate_up_proj`/`up_proj` Parameters. `load_quantizable_moe(model_cls)` is the
 faster variant that linearizes during load rather than after.
 **Always assert the expert count before spending GPU time** (§4.1). The
 arithmetic is `layers × experts × projections`; if your target list does not
 hit it exactly, the recipe is wrong and the failure is silent.
 ⚠ **Keep routers in `ignore`.** A 4-bit router picks *different experts* — that
 error does not average out downstream, it changes which weights run at all.
 **And here is what that actually looks like when it ships — measured 2026-09-10 on the MeroMero
 26B-A4B, which reached a live seat before anyone noticed.** The A4B was quantized with the *dense*
 recipe (`services/meromero-quant/quant_a16_datafree.py`), whose IGNORE list has no `re:.*router.*`
 entry. All 30 routers went to NVFP4. Then:
 - the quant **completed cleanly**, 16 G, no warning;
 - the tensor table looked **plausible** — 11,755 quantized modules against 11,725 in the
  known-good build, a 0.26% difference nobody eyeballs;
 - vLLM **started, passed its healthcheck, and reported the correct KV pool**;
 - every request returned `finish_reason: "length"` with the **full completion_tokens count** —
  120 of 120, 600 of 600 — and `content: null`. The model was generating, and every token decoded
  to the empty string;
 - the give-away was **`logprobs` coming back NaN**, which surfaced only because a diagnostic asked
  for them and vLLM refused to serialize the response: `Out of range float values are not JSON
  compliant: nan`.
 ⚠⚠ **So the router mistake has no symptom you would catch by watching a seat come up.** It has
 exactly one cheap tell, and §4.4 now carries it: ask for `logprobs` once.
 **Two guards, both cheap, both would have caught this before the seat:**
 1. **Use the architecture-class-correct recipe and let its guard fire.**
   `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` refuses outright — *"⚠ REFUSING: a
   router/vision/audio Linear is in the quantize set"* — and asserts `layers × experts × 3` before
   any GPU time. Its `--dry-run` does the whole check with no GPU and no save. The dense recipe has
   neither guard and will happily eat a MoE.
 2. **Diff `quantization_config.ignore` against a known-good quant of the SAME ARCHITECTURE CLASS.**
   The broken build had **222** ignore entries against the good build's **252**; the 30 missing
   were exactly `model.language_model.layers.N.router.proj`. That diff is a two-line script and it
   names the defect precisely.
 ⚠ **The control has to match the architecture class, not just the family.** The broken A4B *was*
 structurally diffed before it shipped — against a verified-good **dense** 31B quant of the same
 Gemma-4 family. A dense model has no routers, so the one thing that was wrong was the one thing
 that control could not see, and the comparison came back clean. A positive control is only worth
 what it can distinguish; "same family" is not "same architecture class".
 ### 3.4 Toolchain version deadlocks
 Both directions have burned us, so the resolution is: **use llm-compressor / compressed-tensors,
 not nvidia-modelopt.**
 - modelopt **0.45** ↔ transformers 5.12: `mtq.quantize` dies `TypeError: issubclass() arg 2 must
  be a class` (modelopt registers transformers' `FusedMoE`, a *function* in 5.x, as an nn class).
 - modelopt **0.43** doesn't fix it — it drags transformers back to 4.57, which cannot load
  `qwen3_5` at all.
 - modelopt's config API also trails the current model families by a version.
 ### 3.5 Vision tower and its configs
 - Keep the **vision tower in `ignore`** (BF16). Only the LLM backbone gets quantized.
 - The wrapper-class save **drops `preprocessor_config.json`** (and the video one). Without it the
  seat crash-loops `Can't load image processor`. Restore from the source — and if the upstream repo
  omits it, **reconstruct it from `processor_config.json`'s `image_processor` sub-dict**.
 ### 3.6 Memory and device placement (large models)
 - **`device_map=None`/`"cpu"`, never `"auto"`.** `auto` fills GPU0 and OOMs during un-fusing;
  constraining with `max_memory` then offloads to the *meta* device, which cannot be `.copy_()`d.
  CPU-resident keeps every tensor real; the sequential pipeline still onloads per-layer to GPU.
 - **Avoid mmap on `/tank`.** `safetensors.safe_open()` mmaps a whole shard; on ZFS a 50 GB shard
  ENOMEMs regardless of free RAM (MAP_SHARED never consults the commit limit). Read with plain
  `read()` + `load(bytes)`, one shard cached at a time.
 - **`vm.overcommit_memory=1`** on ana-ml2 (durable via `playbooks/ana-ml2-overcommit-memory.yaml`).
 ### 3.9 ⭐⭐ A sharded forward can be silently WRONG — never trust `device_map="auto"` for activations
 Splitting **Qwen3.8-27B (Qwen3_5 hybrid)** across the two Blackwells with `device_map="auto"`
 produces a model that loads clean, reports no error, and computes **garbage**: the residual stream
 collapses to **exactly zero** a couple of layers past the GPU0→GPU1 boundary, and the logits decode
 to rubbish (`'8'`, `'�'`, `'b'`). Every layer *below* the boundary stays healthy, deterministic, and
 bit-identical to a single-GPU run — which is what makes it so dangerous. A capture that reads a
 low layer looks perfectly plausible and is fine; one that reads a high layer is reading zeros, and
 nothing in the pipeline says so. Measured 2026-08-20 (§9 Cold-Fusion).
 **Rule: any workload that reads activations — refusal-direction capture, calibration, activation
 statistics, PPL — must run on ONE device.** Sharding is for *storage*, and it is only safe when you
 consume the model's final output through an engine that was built for it (vLLM does TP correctly;
 `device_map="auto"` in transformers is not the same thing). If it does not fit on one card, shrink
 the model, not the guarantee: **truncating the decoder to N layers is exact** for any activation
 read at a layer < N (a causal stack's layer-N state cannot depend on layers above N), and it is
 cheap — verified by reproducing the full model's layers 18/20/22/26 bit-for-bit.
 **Gate it, don't remember it.** Assert single-device residency and zero offload before the forward:
 ```python
 dmap = getattr(model, "hf_device_map", {}) or {}
 gpus = {str(v) for v in dmap.values()} - {"cpu", "disk"}
 offloaded = [k for k, v in dmap.items() if str(v) in ("cpu", "disk")]
 if len(gpus) > 1 or offloaded:
    sys.exit("residency gate FAILED — sharded/offloaded forward reads garbage")
 ```
 ### 3.10 ⭐⭐ `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` corrupts retained tensors
 On torch 2.12+cu130 / Blackwell, tensors that **outlive their allocation** come back corrupted with
 this flag set: captured hidden states carried Inf / NaN / zeros that **moved between bit-identical
 forwards** (same input, same weights → a different layer corrupted each time). Unset, the identical
 forwards are exactly reproducible. Several runbooks recommend this flag for headroom on large
 loads; for anything that *keeps* activations it buys corruption.
 Two tells that distinguish this from a real numerical blowup, both worth knowing because they
 generalise: a genuine blowup **propagates** to later layers and is **deterministic**. Corruption
 does neither — downstream layers were finite and consistent, and the affected layer moved run to
 run. **If a "NaN" fails to propagate, stop debugging the math and start debugging memory.**
 Corollary: **do not read `output_hidden_states=True` off a returned object** on a large multi-device
 load. Take what you need *during* the forward with a `register_forward_pre_hook` that clones to CPU
 immediately — it closes the reuse window and never retains a `[B, seq, hidden]` tensor per layer, so
 it is cheaper than the thing it replaces.
 ### 3.11 Determinism is a necessary check, not a sufficient one
 Both defects above were found by the cheapest possible test — **run the same input twice and diff**
 — which no amount of eyeballing plausible-looking numbers would have caught. Add it to any
 activation-reading pipeline. But note the trap that followed: after fixing the allocator, the run
 went perfectly "deterministic" *because the corrupted layers were now stably zero*. Pair the
 determinism check with a **magnitude** check (residual norms should grow smoothly with depth; an
 exact 0.0 mid-stack is impossible) and, where you can, a **coherence** check (generate 40 tokens and
 read them).
 ### 3.12 ⭐⭐ You cannot free a 27B model in-process — give each model its own process
 Any A/B that loads two large checkpoints in sequence (KL, logit diffing, teacher-vs-student)
 will try to release the first before loading the second. **On this stack, it does not work.**
 Measured 2026-08-20 on Qwen3.8-27B bf16, free VRAM after each attempt:
 | teardown | free VRAM |
 |---|---|
 | `del model` + `gc.collect()` + `torch.cuda.empty_cache()` | 45,287 MiB |
 | same, with the model confined to an inner frame that exits | 45,287 MiB |
 | **the process exits** | **97,247 MiB** |
 The ~51,300 MiB of weights stayed resident through both in-process teardowns. The first
 run survived only because **PyTorch's allocator hit OOM on the second load, ran a collection
 itself, and retried** — the second model landed by rescue, not by design. That is not a
 release strategy: on an architecture where a silent CPU offload does not raise (§3.9), the
 day the retry does not fire you get confident garbage instead of an error.
 **Do this instead:** one process per model, hand results to disk between them
 (first-token log-probs for a 250k vocab are ~715 MiB per model — nothing), and gate each
 stage on free VRAM *before* the load. Reference implementation:
 `services/coldfusion-abliteration/kl_divergence.py` (`--stage ref|cand|score`).
 Two gate corollaries learned in the same session:
 - **⭐ A residency gate that reads `hf_device_map` cannot fail.** The map is **empty**
  whenever transformers puts the whole model on one device, so the check reports
  "unsharded" both when everything is fine and when there is nothing to inspect. Read
  `{p.device for p in model.parameters()}` — ground truth in every case. (Generalises
  [[feedback_assert_effective_value_not_substring]]: presence of a passing check is not
  evidence of a check that can fail.)
 - **⭐ Size VRAM from the checkpoint's own headers, never from a remembered figure.** A
  runbook carried "bf16 is 50 GB"; the real number was 50.10 **GiB** = 51,300 MiB of
  text-only weights. That 3.7 GB unit error is exactly the difference between "stop one
  co-tenant" and "stop both", and it cost an aborted window. Sum the safetensors header
  offsets (excluding tensors the loader class won't instantiate — vision, MTP); read only
  the 8-byte length prefix + JSON header, never `safe_open`, which mmaps the whole shard
  and ENOMEMs on ZFS (§ *Avoid mmap on `/tank`*).
 ### 3.13 ⭐⭐ The observer you ASKED for is not necessarily the observer you GOT
 `quant_mixed_nvfp4.py` sets `observer="imatrix_mse"` on the NVFP4 W4A4 group. It has
 **never once been used.** llm-compressor looks for importance data, finds none, and
 silently degrades:
 ```
 _get_validated_importance | WARNING - imatrix_mse: no importance data available.
                                      Falling back to uniform MSE.
 ```
 Confirmed on the 2026-08-20 09:59 incumbent quant **and** the 22:45 Heretic-300
 quant; `find /tank/aimodels -iname "*imatrix*" -o -iname "*importance*"` returns
 nothing. Every NVFP4 build in the fleet has run uniform MSE while the recipe claimed
 importance weighting.
 **Why it went unseen for months:** the warning scrolls past inside a tqdm progress
 bar during a ~20 minute quant. It is only visible if you read the log while it runs.
 **The generalisable rule, which is bigger than imatrix.** A quantizer, optimiser or
 observer that *silently falls back to a weaker default* is a whole class of invisible
 quality loss — the config is accepted, nothing errors, the artifact benchmarks
 plausibly, and you never learn you got the cheap path. So:
 - **Grep the quant log for `WARNING`, `Falling back`, `not available`, `ignoring`
  before trusting an artifact.** Make it a step, not a habit.
 - **Assert the effective setting, never the requested one** — the same rule as
  [[feedback_assert_effective_value_not_substring]], applied to quantizer internals
  rather than config files.
 - If the fallback turns out to be unavoidable in your toolchain version, **change the
  recipe to say what it actually does.** A recipe line that silently lies is worse
  than one that admits a limitation.
 ⚠️ **Do not "fix" this by assuming an imatrix would help.** Verify first that your
 llm-compressor version can consume an externally supplied importance matrix at all,
 and in what format. Parked as `park/nvfp4-recipe-asks-for-imatrix-mse-but-silently-2`
 (id 42) with the calibration corpus that would feed it.
 ✅ **Comparisons already made remain valid.** Because *every* build shares the
 fallback, the incumbent-vs-candidate A/Bs (47.2% acceptance, PPL 6.910, and the
 2026-08-20 Heretic-300 build) are apples-to-apples. This is unrealised upside, not a
 correction to past numbers.
 ### 3.16 Weight-only NVFP4A16 with a minmax observer is DATA-FREE — your calibration corpus is ignored, but its tokenizer side-effect is not
 Measured 2026-09-08 (Gemma-4 26B-A4B MoE, ERP run 6, llm-compressor 0.13): with
 `scheme="NVFP4A16"` (default `memoryless_minmax` weights, no activation quant) llm-compressor
 logs `Inferred DataFreePipeline for QuantizationModifier` and never touches the dataset — the
 whole 26B quant ran in ~90 s on one Blackwell. Two consequences: (1) do not budget calibration
 time or believe a corpus "shaped" the result — only `imatrix_mse`/activation observers consume
 data; (2) building the calibration set still calls the fast tokenizer with
 `truncation=True, max_length=N`, so §3.14's baked cap (`max_length: 8192` here) lands in the
 saved `tokenizer.json` **even though no calibration happened**. The §4.3 post-step caught it.
 Reference: `services/erp-seat-quant/quant_nvfp4a16_gemma4_moe.py` (linearize_moe + assert
 11,520 expert Linears + post-steps; the published `prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16`
 recipe replicated, 222→252 ignore entries with audio/norm/router regexes added).
 **So for any `*A16` scheme, do not pass a dataset at all** — not a shorter one, none. It removes
 the §3.14 tokenizer bake-in *and* llm-compressor's "initialize model processor ... required when a
 dataset is provided" demand, which is fatal on any upload that ships no `processor_config.json`.
 Both of those cost an attempt on MeroMero v2 (2026-09-10); dropping the corpus costs nothing,
 because a `DataFreePipeline` was never going to read it. Driver:
 `services/meromero-quant/quant_a16_datafree.py`. **Confirmed twice more the same day**: the A4B
 heretic quant, run *with* the corpus, shipped `max_length: 8192` in its `tokenizer.json`; the v2
 dense, run without it, came out `truncation: null`.
 ### 3.17 ⭐⭐ The transformers you measured is not the transformers that ran — llm-compressor pins it
 **Measured 2026-09-10, MeroMero v2-31B, and it cost a full misdiagnosis.** The quant died in
 `AutoTokenizer.from_pretrained` with
 `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is a per-layer attribute`. The obvious reading
 was that the source config carried a `per_layer_config` key the canonical one lacked, so that key
 was the defect. It was not.
 `pip install llmcompressor==0.13.0` **downgrades transformers underneath you** — 5.16.1 → 5.14.1 in
 the `vllm/vllm-openai` image. The config had been serialized *by* 5.16.1, which materializes
 `per_layer_config` from `global_head_dim` + `layer_types`; 5.14.1 carries the heterogeneity guard
 but not the gemma4 resolver, so it refuses the global read. Under the image's own 5.16.1 the very
 same config loads fine, which is exactly what makes this class expensive: **the version you print
 at the top of the script is not the version the quant runs on.**
 - **Print the version AFTER the install**, in the same container, and put it in the log.
  `python3 -c 'import transformers; print(transformers.__version__)'` as a pipeline step, not a
  thing you check by hand once.
 - **Pin the image by digest for the length of a campaign.** `:latest` was re-pulled between
  attempts 3 and 4 of this run and moved the toolchain mid-diagnosis, so the same command produced
  a different error for reasons that had nothing to do with the change under test.
 - **Reproduce the actual failing call, not a paraphrase.** A bare `AutoConfig.from_pretrained` did
  not reproduce this at all — the trigger was reached through `AutoTokenizer`. Testing the config
  in isolation would have "cleared" it.
 - **Keep a known-good tree as the positive control.** zerofata's canonical v2, quantized cleanly
  three weeks earlier, is what separated "this config is malformed" from "this toolchain moved".
  Without it, four green variants and one red one are just noise.
 Related but distinct from §3.4, which is about version deadlocks you can *see*. This one is a
 silent downgrade inside a line you already trusted. Instruments: `services/meromero-quant/`
 (`tok_repro.py`, `perlayer_test.py`).
 ### 3.14 ⭐⭐ Calibration BAKES a truncation cap into the shipped tokenizer
 **Symptom (on a newer transformers, at startup, on a vision model):**
 ```
 ValueError: Mismatch in `image` token count between text and `input_ids`.
 Got ids=[2047] and text=[16384]. Likely due to `truncation='max_length'`.
 ```
 The engine never serves a request. The number in `ids=[…]` is your **calibration seqlen minus
 one**, which is the tell.
 **Cause — an in-place mutation you never wrote.** Calibration tokenizes like this:
 ```python
 tok(b["text"], truncation=True, max_length=seqlen, add_special_tokens=False)
 ```
 For a **fast** tokenizer that call does not just return ids — it **mutates the Rust backend's
 truncation state in place**. A later `tok.save_pretrained(out)` then persists it:
 ```json
 "truncation": {"direction": "Right", "max_length": 2048, "strategy": "LongestFirst", "stride": 0}
 ```
 The source model has `"truncation": null`. **You shipped a tokenizer that clamps every prompt at
 the calibration length, permanently.**
 **Why it hid for months.** Older transformers does not enforce the text-vs-ids count check, so
 the cap sits latent — the model serves, gates pass, vision works, nothing logs. It only detonates
 when you bump the image, and then it presents as a *vision* bug at startup with no mention of
 tokenizers. It also caps the effective image resolution long before it kills the seat: at a 2048
 cap the largest servable image is ~1448×1448, because `(edge/patch)² / merge²` image tokens must
 fit under it.
 **The fix — never save the calibration tokenizer.** Re-read a pristine one from the source:
 ```python
 from transformers import AutoTokenizer as _AutoTokenizer
 _AutoTokenizer.from_pretrained(a.model, trust_remote_code=True).save_pretrained(a.out)
 ```
 then **assert** it, because this is exactly the class of defect that returns silently:
 ```python
 if json.load(open(f"{a.out}/tokenizer.json")).get("truncation"):
    raise SystemExit("FAILED CHECK: saved tokenizer carries a truncation cap")
 ```
 Both live in `quant_mixed_nvfp4.py` as of 2026-08-22.
 **Audit any build predating that.** One line per model:
 ```bash
 python3 -c 'import json,sys;print(json.load(open(sys.argv[1]+"/tokenizer.json")).get("truncation"))' <model_dir>
 ```
 Measured 2026-08-22 — every mixed-NVFP4 build from this pipeline was affected, and the two live
 ones were corrected in place (backup `tokenizer.json.bak-truncation-20260822`; only the
 `truncation` field changed, vocab and `added_tokens` byte-identical):
 | build | truncation as found |
 |---|---|
 | `qwen38-27b-orcarouter-nvfp4-mixed` (live `gen`) | **2048** → fixed |
 | `mog-sec-27b-nvfp4-mixed` (live `sec`) | **2048** → fixed |
 | `qwen38-27b-heresy-nvfp4-mixed` (retired) | 2048, left as-is |
 | `G4-MeroMero-v2-31B-NVFP4A16` (different pipeline) | `null` ✓ |
 | `mog-sec-27b-bf16` (source) | `null` ✓ |
 **Editing it is safe on a running seat** — vLLM reads the tokenizer at startup and holds its own
 copy, so the fix lands on the next restart with no disruption.
 **The general lesson, which is the transferable part:** this is the third defect in this playbook
 where *the artifact carries config authored against an older transformers and a newer one starts
 enforcing it* (see also the Gemma-4 heterogeneous `head_dim`). **Treat "we bumped the image" as a
 config-compatibility event, not just a version change** — and prefer saving artifacts re-read
 from the source over saving objects the pipeline has touched.
 ---
 ## 4. Pipeline shape
 ### 4.1 Prove the targets before spending GPU time
 Enumerate module names from the safetensors index and check your regexes against them: **zero
 overlap between groups, and the union covers every layer you intended.** This is free, takes
 seconds, and catches a mis-scoped regex that would otherwise surface as a mystery quality
 regression hours later. Reference: `services/gen-seat-mixed-quant/validate_targets.py`.
 ### 4.2 Quantize
 Calibration data matters for `imatrix_mse` + static activation observers. We use
 `/tank/aimodels/heretic2-nvfp4-work/production_calib_512.jsonl` (512 chat samples, RP/GM-flavoured
 — appropriate for our seats). 256 samples @ 2048 tokens ≈ 20 min for a 27B on one Blackwell.
 ### 4.3 The mandatory post-steps
 Never optional, always in this order, and the last one **verifies rather than assumes**:
 1. Graft `model-mtp.safetensors` + register its tensors in the output index.
 2. Restore `preprocessor_config.json` / `processor_config.json` / `video_preprocessor_config.json`.
 3. **Re-inject `re:^mtp.*` into `quantization_config.ignore` and confirm it is there** (§3.3).
 4. **Confirm the saved `tokenizer.json` has `truncation: null`** (§3.14) — calibration mutates the
   fast tokenizer in place and `save_pretrained` bakes the cap in. Latent on an older
   transformers, fatal on a newer one.
 Reference implementation: `services/gen-seat-mixed-quant/post_quant.py`.
 On Gemma-4 steps 1 and 3 are N/A — the family ships no MTP head at all — which leaves 2 and 4, and
 4 is the one that fires. `services/meromero-quant/post_quant_gemma4.py` runs them idempotently with
 a `--check` mode; point it at a tree you already trust before you trust its verdict on a new one.
 ### 4.4 Test on a temp port, never on the live seat
 Serve the candidate on an alt port with the live seat's **exact** flags, run the gate (§5), and
 only then flip `.env`. Keep the previous build on disk; rollback is one `.env` line.
 **Ask for `logprobs` once, on the temp port, before the alias moves.** A seat can pass its
 healthcheck, report the right KV pool, and answer every request with the full token count while
 every token decodes to the empty string — that is what a router-quantized MoE does (§3.15). NaN
 logits are invisible to `/health`, invisible to the token counts, and invisible to a tensor-table
 diff against the wrong control; a single `logprobs: 1` request surfaces them immediately, because
 vLLM cannot even serialize the response (`Out of range float values are not JSON compliant: nan`).
 Add it to the smoke set: **served name, one prose completion, one image if the model is
 multimodal, one tool call, and one `logprobs` request.**
 ⚠ **A co-resident temp port is not always reachable, and the fallback is reversibility, not
 skipping the test.** Measured 2026-09-10: with 19 GiB free on a shared card, a 16 G A4B refused
 admission at `gpu-memory-utilization 0.20` (18.26 free vs 18.99 requested), and at 0.185 it got
 past admission and past the KV reservation only to OOM in **multimodal encoder-cache profiling**
 (`profiled with 3 video items of the maximum feature size`) — a cost easy to forget when budgeting
 a vision model. When the card genuinely cannot hold both, the substitute is: back up the host
 `.env` to a named file first, prove the new seat on its real port **while no gateway alias points
 at it**, and move the alias last. That ordering is what kept a NaN-serving seat away from every
 consumer; the seat itself was down ~16 minutes and nothing downstream saw a broken alias.
 ---
 ## 5. The acceptance gate — and how measurement lies to you
 Speed alone does not justify cutting over a shared seat. Gate on **all** of: decode tok/s, MTP
 acceptance, perplexity, a behavioural surface test, and — for an abliterated model — that the
 abliteration survived.
 **Three ways the numbers have lied to us. All three produced confident, wrong results.**
 1. **Prefix caching fakes both speed metrics.** A fixed prompt returns byte-identical timings run
   after run; you are measuring cache, not compute. Worse for prefill: a *seeded* nonce
   regenerates the previous run's prompts verbatim and reads **~41k tok/s of cache-hit instead of
   ~5k of real prefill**. Use a fresh unseeded nonce per request; never seed a cache-buster.
 2. **`prompt_logprobs` are garbage while speculative decoding is on** — ~uniform over the vocab
   (median rank ~10⁵; " Paris" after "The capital of France is" ranked 69698). **Perplexity must be
   measured on a seat served without `--speculative-config`,** on both sides of the comparison.
 3. **A 0600 `.env` makes `docker compose` silently no-op.** Without `sudo` it fails
   `permission denied` reading `.env`, **leaves the old container running**, and reports success —
   producing a full page of "benchmark results" that were just the unchanged baseline.
   **Hard-verify the change landed against `docker inspect …Config.Cmd`.**
 **Re-measure the baseline before believing a target.** The 2026-08-15 handoff quoted ~68 tok/s;
 cache-busted, the incumbent was already doing 80.1 — essentially the *target* of the work queued
 against it. Had that not been re-measured, doing nothing would have looked like a 20% win.
 **Cheap shortcut worth taking first:** if a reputable published quant of the same architecture is
 already on-box (or is a small pull), **serve it as a probe and measure it** before committing
 hours to your own. It answers "is this gain even real?" in ten minutes *and* hands you the recipe.
 Harness: `services/gen-seat-mixed-quant/bench/` — `quickbench.py` (decode + acceptance),
 `prefill_bench.py`, `eval_quality.py` (PPL + abliteration), `surface_test.py` (chat, vision, tools,
 thinking split, long-context needle, streaming), `serve_probe.sh`.
 ---
 ### 5.1 ⭐⭐ Acceptance is not throughput — always run the DEPTH control
 **Measured 2026-08-22**, same instrument (vLLM's own `spec_decode` counters, delta over a fixed
 workload, temp 0), same target, same engine:
 | config | accepted tok/forward | throughput |
 |---|---|---|
 | MTP k=3 | 2.753 | 114.9 tok/s |
 | MTP k=7 | **3.041** ⬆ | **74.0 tok/s** ⬇ |
 **Raising `num_speculative_tokens` improved acceptance and destroyed throughput.** Reporting
 acceptance alone would have recommended a 36% regression.
 **Why:** a single-module MTP head (`mtp_num_hidden_layers: 1`, one `mtp.layers.0`) has no depth
 of its own — vLLM runs it **autoregressively**, so k draft tokens cost **k sequential forward
 passes**. Past a shallow depth the drafting cost exceeds what the extra accepted tokens save.
 Check `mtp_num_hidden_layers` before assuming depth is cheap.
 **The rule: when comparing two speculative methods, match k, or you are measuring depth rather
 than method.** A parallel-drafting drafter (DFlash2 and kin, which propose a whole block in one
 pass) at k=7 versus an autoregressive MTP at k=3 is not a method comparison — the depth control
 is what separates them. In our case the control showed most of the apparent acceptance win was
 depth, while the *throughput* win was real and came from parallel drafting, not better drafts:
 our MTP was **better at position 0** (79.6% vs 75.4%) and still lost overall.
 **Corollary — report both, always.** Acceptance rate, mean accepted length, and end-to-end
 tok/s. Any one of the three alone can point the wrong way.
 ---
 ## 6. Hardware and co-residency
 - **ana-ml2 = Blackwell (sm_120)**, 2× 96 GB. Native FP4 + FP8. Hopper-exact code paths
  (`match_exact=True` on sm90) are **closed** here — do not plan around them.
 - **irv-ml1 = Ampere (sm_86)**, 3090 + A6000. **No native FP8/FP4** — 4-bit there is a VRAM saving
  only, not a speed win. Don't port a Blackwell scheme over and expect the throughput.
 - **GPU co-residency is a zero-sum budget, and a *smaller* model can break its neighbour.**
  `gpu-memory-utilization` is a fraction of the *whole card*, so when new weights are smaller the
  seat absorbs the slack as extra KV rather than releasing it. That is exactly how a −5.2 GB
  requant left the co-resident seat **0.18 GiB** short and crash-looping. **After any requant,
  re-check both seats' budgets** and hand the space back explicitly.
 ---
 ## 7. Superseded claims — do not follow these
 Old docs stay for their history, but these specific claims are **false now** and will cost you a
 day if followed:
 | claim | where | status |
 |---|---|---|
 | "Use modelopt, NOT compressed-tensors — compressed-tensors can't load the BF16 MTP head, 0% acceptance" | `docs/runbooks/heretic2-nvfp4-mtp-seat.md` §landmine 2 | **SUPERSEDED 2026-08-14.** The 0% was the missing `re:^mtp.*` ignore (§3.3), not the format. compressed-tensors + the ignore gives 47.7–83.2% acceptance, live. Use compressed-tensors. |
 | "Abliteration desyncs the MTP head → uncensored models can't do MTP" | earlier auto-memory | **SUPERSEDED 2026-08-14.** A modest abliteration preserves MTP (83.7% at bf16). Test MTP on **bf16 first** to isolate abliteration from quant/graft confounds — and isolate before deleting a 50 GB source. |
 | "NVFP4 W4A4 is infeasible, no 4-bit wins both axes, FP8 is the Blackwell answer" | `reference_nvfp4_w4a4_granite_infeasible` | **NARROWED.** True for *uniform* W4A4 (measured on Granite-8B at 30k ctx). W4A4 on bulk MLPs **with FP8 on attention and late layers** is fine and is the current default (§2). |
 | "The Qwen3.8 MTP corruption is fixed by disabling prefix caching while keeping MTP; the gen seat runs APC-off" | this playbook §3.7 (now marked), earlier auto-memory | **SUPERSEDED 2026-08-17, and the staleness was only caught 2026-08-26.** APC-off passed a synthetic 7-turn probe and the operator still saw severe degeneration in real use; reverted the same day. The real cause was the **AEON W4A4 quant being defective** (~15-20% degenerate generations), with MTP / prefix-caching / gateway merely AMPLIFYING it (§3.8). The gen seat runs **MTP ON and prefix-caching ON** on the in-house mixed NVFP4+FP8 build — verified against the live container 2026-08-26. ⚠ The lesson inside the lesson: a *passing multi-turn probe* was not sufficient evidence either. |
 | "transformers' Qwen3.5 DeltaNet linear-attention NaNs in bf16 without causal-conv1d; it is precision-driven cancellation and fp32 resolves it" | `services/coldfusion-abliteration/README.md`, `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` | **SUPERSEDED 2026-08-20.** Precision was never the variable. The NaN came from **multi-GPU sharding** and **`expandable_segments`** (§3.9, §3.10); fp32 only made it rarer, which is worse than failing. On one GPU with a plain allocator, **bf16 is exactly deterministic through all 64 layers and generates coherent prose** — at 50 GB and 4.3× the throughput of the 111 GB fp32 it replaced. |
 | "Structural verification of a quant (layer split, MTP graft, ignore list, tensor dtypes all correct) means the quant is good" | general belief | **FALSE — cost a full cutover of a gibberish seat 2026-09-14.** The `cyberprev` quant passed every structural check AND matched the reference config exactly, yet served pure gibberish, because the SOURCE checkpoint (`hotdogs/Qwen3.8-27B-abliterated-cyber-preview`, an unsloth export) had 850/1199 body tensors carrying a **tripled `language_model.` prefix** — verified present in the UPSTREAM HF index, not our download. transformers silently loaded the unmapped tensors as random-init, quantized noise, and every structural check passed because it inspected the OUTPUT config, never that INPUT names mapped. **A plain greedy coherence smoke (spec off) is the load-bearing gate and caught it; structure never can.** Fix: rewrite tensor names to a known-good reference of the same arch (exact 1199-tensor match), re-quant. Add a name-set diff of the bf16 SOURCE against a reference BEFORE quantizing. |
 | "A checkpoint with an FP8 PLE table but no `ple_embedding_dtype` declaration cannot be loaded; compressed-tensors qwen4_exp needs a vLLM source patch" | `stacks/flash-next-seat/README.md`, `persistent-memory.d/2026-09-14-fv-seat-reorg-and-orca-blocker.md` | **SUPERSEDED 2026-09-14.** `from_quant_config` checks `ple_embedding_dtype` as **branch 1, before any quant-config type check**, and its `NotImplementedError` is scoped to the **PLE path only**. Declaring the FP8 PLE bypasses it on stock mainline — a **one-key config fix**, not a source build. Proven live: orcarouter (compressed-tensors) serves on gen-large after converting its bf16 PLE to FP8 + declaring it. ⚠ Declare only what is TRUE — declaring FP8 over a bf16 table is the gorbatjovy failure in reverse. |
 ---
 ## 8. Measured negatives — don't re-chase
 - **`num_speculative_tokens` = 3 is optimal** on the Qwen3.8-27B seat. Swept: n=2 → 77.1,
  **n=3 → 80.1**, n=4 → 78.7, n=5 → 75.9 tok/s. Higher n trades acceptance for draft width and
  loses. Re-sweep only if the drafter architecture changes.
 - **Uniform W4A4** — see §7 row 3.
 - **Dense-VL as the anatomy judge** — A/B'd, MoE retained. Don't re-propose.
 ---
 ## 9. Worked examples
 Per-model artifacts. Read for *how a specific model went*, not for the general lessons — those are
 above, and where the two disagree, **this file wins**.
 | artifact | what it is |
 |---|---|
 | `services/gen-seat-mixed-quant/` | **current reference.** Mixed NVFP4+FP8 on Qwen3.8-27B-Uncensored: scripts, acceptance harness, raw measurements. |
 | `stacks/gen-seat/README.md` | the live `gen` seat (7 LiteLLM aliases) |
 | `services/meromero-quant/` | NVFP4A16 on Gemma-4 (MeroMero A4B + v2-31B ablits): the five-attempt failure chain, the pinned-transformers trap (§3.17), and the GPU-free verification instruments. |
 | `stacks/meromero-charrp/README.md` | Gemma-4 seat — the **tool-call/reasoning-parser** trap (a parser default that returns null `content` for all prose) |
 | `services/heretic2-nvfp4-quant/` | modelopt-format MTP seat — historical; see §7 before following it |
 | `tools/mistral-small4-nvfp4/` | MoE + native-convert path; source of §3.6 |
 | `docs/pfi/recommended-model-settings.md` | serve-time sampler/flag defaults (not quant) |
 **A new model just dropped and needs requanting?** §1 → §2 → §4 → §5. Skim §3 first; it is the
 part that costs hours.
@@ -0,0 +1,148 @@
 # OPNsense API — vendored reference + field-shape notes
 **Source:** <https://docs.opnsense.org/development/api.html> (fetched 2026-09-13)
 **Why vendored:** the upstream page documents the REST conventions but explicitly
 *not* parameter shapes — "the auto-generated API documentation captures endpoints
 and HTTP methods but not all parameter details." Those shapes are where the time
 goes, so the hard-won ones are recorded in § Field shapes below.
 ## Authentication
 HTTP Basic auth: **key = username, secret = password**.
 ```sh
 curl -s -u "$KEY:$SECRET" "http://<gw>/api/<module>/<controller>/<command>"
 ```
 Fleet creds are vaulted: `fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`
 (see `secret get`). Authorization is per-user **Effective Privileges** — a key only
 reaches endpoints its owner is authorized for.
 ## Conventions
 - **Endpoint pattern:** `/api/<module>/<controller>/<command>/[<param>/...]`
 - **GET** retrieves; **POST** creates, updates, or executes an action.
 - Request bodies and responses are `application/json`.
 Search endpoints take a paging body and return a rows envelope:
 ```json
 {"current":1,"rowCount":7,"sort":{},"searchPhrase":""}
 ```
 ```json
 {"total":10,"rowCount":7,"current":1,
 "rows":[{"id":"configd","locked":1,"running":1,"description":"System Configuration Daemon","name":"configd"}]}
 ```
 ## Service control
 `/api/<module>/service/{status,start,stop,restart,reconfigure}` — `reconfigure`
 writes config **and** applies it, which is normally the one you want after a
 `settings/set`. Observed status values include `disabled`, `stopped`, `running`
 — note **`disabled` ≠ `stopped`**: a disabled service will not start until its
 model's `enabled` field is set to `"1"`.
 ## ⚠ Endpoints are ACTIONS — never probe for existence by calling them
 A 404 tells you an endpoint is absent; a 200 tells you it **ran**. There is no
 safe "does this exist?" POST against a live firewall.
 **2026-09-15:** looking for the call that applies a user change, a loop POSTed an
 empty body at four guessed endpoints to see which returned 404. One of them was
 `/api/core/system/reboot`. It returned 200 because it rebooted the FV edge
 firewall, taking the whole site — including the BMC, which sits behind it —
 dark for 3.5 minutes. The call it was actually looking for is documented
 directly above, in § Service control, in this file.
 - Read this reference and the upstream endpoint list first.
 - If you must discover, use **GET** on a `get`/`search`/`status` command, never
  POST on an unknown name.
 - Take `/api/core/backup/download/this` **before** any write. That part went
  right and is the only reason the change was reversible.
 ### The one useful thing that fell out of it
 `POST /api/core/system/reboot` with `{}` is a **reliable remote reboot** for the
 FV gateway — it came back cleanly on its own in ~3.5 min from an API-initiated
 restart, which is a capability worth knowing deliberately rather than by
 accident. `/api/core/service/restart/<id>` (e.g. `openssh`) restarts one service
 without the site outage, and is almost always what you want instead.
 ## Applying a change — `service/reconfigure`, not a reboot
 `settings/set` (and `auth/user/set`) write config.xml. They do **not** sync the
 change to the running system. The apply step is the module's service endpoint:
 ```sh
 POST /api/<module>/service/reconfigure   {}
 ```
 ⚠ Some modules have no `reconfigure` and return `{"errorMessage":"Endpoint not
 found"}` — `auth/user` is one. For those the OS-level sync happens on the UI's
 own save path or at boot, so an API-only key edit sits in config.xml and does
 nothing until then. Verified 2026-09-15: `authorizedkeys` + `shell` for
 `infra-ops` persisted immediately but SSH kept refusing, and started working
 after a reboot completed the user sync.
 ⚠ `POST` with **no body at all** returns `411 Length Required`. Send `{}`.
 ## Field shapes — the part upstream does not document
 ⚠ **A `settings/get` response is NOT a valid `settings/set` body.** They are
 different shapes, and mixing them returns a bare
 `{"errorMessage":"Unexpected error, check log for details"}` with no indication
 of which field was wrong.
 **Selection / option fields.** `get` returns every option with a `selected` flag;
 `set` wants only the selected key.
 ```jsonc
 // get
 "useExitNode": {"": {"value": "None", "selected": 1}, "abc": {"value":"node-a","selected":0}}
 // set
 "useExitNode": ""
 ```
 **Array / list fields are UUID-keyed objects, not lists.** This is the one that
 costs an afternoon. `get` renders an empty array as `[]`, which misleads you into
 POSTing a list.
 ```jsonc
 // get, when empty  -> looks like a plain list
 "subnets": {"subnet4": []}
 // set  -> object keyed by a UUID you generate, values are the item's own fields
 "subnets": {"subnet4": {"<uuid4>": {"subnet": "10.251.0.0/16", "description": "..."}}}
 ```
 **Discovering an array item's inner fields — two reliable tricks:**
 1. POST the wrong shape on purpose. Validation names the full path:
   `{"result":"failed","validations":{"settings.subnets.subnet4.<uuid>.subnet":"A value is required."}}`
 2. Many models expose a template getter: `/api/<module>/settings/get_<item>`
   returns the blank item, e.g. `get_subnet` → `{"subnet4":{"subnet":"","description":""}}`
 **Partial `set` is supported** — POST only the fields you are changing rather than
 round-tripping the whole model. This sidesteps every other field's shape problem
 and is the recommended approach for a surgical change.
 **Validation error format:** `{"result":"failed","validations":{"<dotted.path>":"<message>"}}`.
 A successful save is `{"result":"saved"}`.
 ## Working example — enabling a subnet router (2026-09-13, FV gateway)
 ```jsonc
 POST /api/tailscale/settings/set
 {"settings":{"enabled":"1",
  "subnets":{"subnet4":{"<uuid4>":{"subnet":"10.251.0.0/16",
                                   "description":"FV site subnet router"}}}}}
 POST /api/tailscale/service/reconfigure   {}
 ```
 **Always read back.** `settings/get` after the write, and re-check anything the
 change could disturb (here: `/api/firewall/source_nat/search_rule` total stayed
 at 2). A write that reports success is not evidence the value landed.
 ## Related
 - `docs/runbooks/fv-to-ana-nat.md` — the FV gateway's manual SNAT rule
 - `docs/runbooks/fv-site-dark-20260913.md` — the outage this reference came out of
@@ -0,0 +1,416 @@
 # Ops lessons playbook — the transferable ones
 The operational sibling to `model-quantization-playbook.md`, and it exists for the
 same reason: hard-won lessons kept dying inside per-host runbooks where nobody
 finds them until they have already repeated the mistake.
 **What belongs here:** a lesson that would bite identically on a different host.
 **What does not:** anything true only of one machine — that stays in
 `servers/<host>/README.md` or the relevant runbook.
 Each entry states the rule, what it cost, and how to recognise the situation.
 When an entry turns out to be wrong, add a dated row to § Superseded rather than
 quietly editing it, so older references stop misleading people.
 ---
 ## 1. `mount --rbind` into a chroot needs `--make-rslave`
 **Rule:** after every `mount --rbind /x /target/x`, immediately
 `mount --make-rslave /target/x`. Guard on it — refuse to proceed while
 `findmnt -o PROPAGATION` reports `shared` for any chroot bind.
 **Why:** on a systemd host `/` has *shared* mount propagation, so an `--rbind`
 shares propagation with the original. A later `umount -R` of the chroot copy
 **propagates back into the live system** and unmounts the real `/sys/fs/cgroup`,
 `/dev/pts`, `/dev/shm`. `--make-rslave` makes propagation one-way (host → chroot),
 so teardown cannot reach back.
 **Cost:** an unplanned production outage on esh-pve-nas, 2026-08-18.
 **Recognising it — and this is the valuable part, because it does not look like
 what it is.** With cgroup2 gone, `systemd-logind` cannot create sessions, which
 produces a host that:
 - answers ping and accepts TCP
 - **completes SSH authentication**
 - keeps serving from daemons already resident in memory (a PVE box returned clean
  HTTP 401s from `pveproxy` throughout)
 - **hangs on every new `exec`** — including `/sbin/reboot`, so a reboot issued to
  fix it never runs
 That is an almost perfect impostor of **failing root-disk I/O**, and it was
 misdiagnosed as exactly that. If you see "daemons answer but nothing new can
 start," check `findmnt /sys/fs/cgroup /dev/pts /dev/shm` before you suspect the
 disk.
 **Recovery needs no console.** Exec succeeds in brief windows; loop an idempotent
 remount until one lands:
 ```sh
 mountpoint -q /sys/fs/cgroup || mount -t cgroup2 none /sys/fs/cgroup
 mountpoint -q /dev/pts       || mount -t devpts devpts /dev/pts -o gid=5,mode=620,ptmxmode=666
 mountpoint -q /dev/shm       || mount -t tmpfs tmpfs /dev/shm -o mode=1777,nosuid,nodev
 ```
 Then `systemctl reset-failed`. Full narrative:
 `docs/runbooks/esh-pve-nas-boot-migration.md` § The mount-propagation incident.
 ---
 ## 2. A reboot is not confirmed until the host is observed DOWN
 **Rule:** poll for the host's *disappearance* first, then for its return. Never
 infer a reboot happened because the host answers.
 **Why:** "never went down" and "went down and came back quickly" are
 indistinguishable if you only watch for it to answer. On 2026-08-18 a
 down-detector never once reported the host down; that was read as a fast reboot
 when in fact `/sbin/reboot` could not exec and the machine never rebooted at all.
 Everything diagnosed afterwards was built on that false premise.
 **The cheap confirmation** is the boot timestamp — `uptime -p`, or the last
 `dmesg` timestamp. A `dmesg` tail whose last entry sits at `[12114881]` seconds
 is telling you the machine has been up 140 days, whatever else you believe.
 ```sh
 down=0
 while :; do
  if ping -c1 -W1 "$H" >/dev/null 2>&1; then
    [ $down -eq 1 ] && break || echo "up (not yet down)"
  else down=1; echo "DOWN confirmed"; fi
  sleep 2
 done
 ```
 ---
 ## 3. Assert the effective value, not the presence of a substring
 **Rule:** a verification step must check what the system will actually *use*, not
 that the correct-looking string appears somewhere in a file.
 **Why:** the check "does `root=ZFS=nvme/ROOT/pve-1` appear in `grub.cfg`?" passed
 happily while **every menu entry was still broken** — the correct value had been
 appended by a drop-in, and the broken pool-less value was still first on the line.
 Since the kernel takes the *last* `root=`, only a check that extracts the last one
 per entry and compares it against a known-good set proves anything.
 ```awk
 /^[[:space:]]*linux[[:space:]]/ {
  r=""; for (i=1;i<=NF;i++) if ($i ~ /^root=/) r=$i;
  if (r != "root=ZFS=pool/dataset" && r != "root=/dev/mapper/x") { print "BAD: " r; bad=1 }
 } END { exit bad?1:0 }
 ```
 Generalises well beyond GRUB: last-wins config keys, layered drop-ins, anything
 with override semantics. **Grep proves presence; only evaluation proves effect.**
 ---
 ## 4. Ask the server who its clients are
 **Rule:** before taking a service down, enumerate its dependents **from the
 service**, not from documentation.
 **Why:** a runbook named two NFS dependents. `ss` on the NFS server found five —
 including a database VM with a `hard` mount and no SSH access. Documented
 dependent lists rot silently because nothing forces them to be updated when a new
 client mounts.
 ```sh
 # NFS server: who is actually connected right now
 ss -tnH state established '( sport = :2049 )' | awk '{print $4}' | sed 's/:[0-9]*$//' | sort | uniq -c
 ```
 Equivalents worth reaching for: `ss -tnp` by port for any service, `docker ps`
 plus mount inspection for bind-mount consumers, `pvesm status` for storage.
 **Corollary on `hard` NFS mounts:** a `hard` mount with **no active user** blocks
 and then resumes when the server returns — that is what `hard` is for, and it came
 through read-write across two server reboots. The disaster case is a *process
 actively using* the mount. So quiescing means stopping the consumers, not
 necessarily unmounting; and when unmounting is expensive or risky (a host you
 cannot SSH to), leaving an idle hard mount is often the lower-risk branch.
 ---
 ## 5. The scoped-looking command can be the dangerous one
 **Rule:** when a command names one member of a set, ask what happens to the
 members it does not name.
 **Why:** `zpool set cachefile=/etc/zfs/zpool.cache nvme` looks careful and
 narrow. It is not: populating a cachefile flips the host from import-by-scan to
 import-by-**cache**, so a cache containing only `nvme` leaves `ssd` and `tank`
 unimported at boot. On a host whose NAS container had twelve bind mounts spanning
 all three pools, that empties every export. The broad form — setting it on all
 three — is the safe one.
 ---
 ## 6. Long uptime hides breakage; a forced look is worth more than it seems
 Not a rule so much as a calibration. One migration on a pair of hosts with 20
 weeks of uptime surfaced, none of it caused by the work:
 | found | dead for |
 |---|---|
 | `pvestatd` SEGV'd (node rendered dark in the UI, otherwise healthy) | 82 days |
 | a `vzdump` hung at 0% of 256 GiB, holding `lock: backup` | 126 days |
 | a VM stuck in QEMU `prelaunch` behind that lock | ~4 months |
 | a VM silently missing `sshd`, `mongod` and its guest agent | unknown |
 | an undocumented 2-node cluster, and 3 undocumented NFS clients | always |
 **When a host has not been rebooted or audited in months, budget for finding
 unrelated breakage, and treat that as part of the value rather than as scope
 creep.** Several of these were invisible precisely because nothing had forced
 anyone to look.
 Corollary: **a cosmetic-only symptom can hide for a very long time.** Nothing
 alerted on `pvestatd`; its sole symptom was a grey tile in a UI nobody had reason
 to stare at. Worth a watchdog on anything whose failure mode is "the dashboard
 quietly stops being true."
 ---
 ## 7. Verify a "this will break X" premise before building around it
 **Rule:** when a risk is asserted but never tested, test it — especially before it
 justifies a body of work.
 **Why:** fleet IPv6 work was justified largely by "ESH fiber behind CGNAT will
 break Site Magic on IPv4." The fiber cutover tested it for free: Cox was
 unplugged, ESH failed over to 5G on `192.168.200.111` — **RFC1918, double-NAT,
 no inbound path, strictly worse than CGNAT** — and the tunnel held, carrying real
 traffic to all four ESH hosts.
 The mechanism was discoverable in advance and made the outcome predictable:
 Site Magic is **WireGuard**, and the far side (NH3) has a public endpoint, so the
 NAT'd side dials out and never needs reachability. Ten minutes of reading the
 device config would have graded the risk correctly.
 **How to apply:** for any "X will break Y" belief, ask what protocol Y actually
 uses and which side must be reachable. NAT breaks *inbound* reachability; it does
 not break outbound-initiated tunnels with keepalives. Beliefs that gate real work
 deserve a test or an explicit "untested" label — and when they do get tested,
 record the result where the belief lived, not only where the test happened.
 **Related:** Site Magic has **no WAN binding** — `magic_site_to_site_vpn` on the
 gateway is just `enabled` plus a keypair, peers orchestrated in the UniFi cloud.
 It rides whichever uplink is active, so the only lever is failover priority, and
 that moves *all* site traffic rather than just the tunnel.
 ---
 ## 8. A result proven for one protocol does not transfer to another
 **Rule:** when a test clears a risk, state **which mechanism** it cleared it for,
 and check whether every affected system shares that mechanism.
 **Why:** proving that NAT does not break **Site Magic** (WireGuard, outbound-dialed
 to a public peer) I wrote up as "no addressing outcome threatens the inter-site
 tunnel." But the fleet has *two* inter-site links with opposite NAT behaviour, and
 the other one — **IPsec** to the colo FortiGate — was **already broken at that
 exact moment**, traffic leaking unencapsulated to the carrier. The operator caught
 it; the test I had just run would have caught it too, had I run it against both
 links instead of one.
 **How to apply:** ask what property made the test pass — here, "outbound-initiated,
 peer needs no inbound reachability" — and then ask which systems *lack* it. IPsec
 site-to-site pins a peer IP and expects a routable address; WireGuard does not.
 Same NAT, opposite outcome. Enumerate the affected set before generalising, and
 name the mechanism in the conclusion so the scope is visible to the next reader.
 ---
 ## 9. IPsec to a NAT'd site: dialup peer + NAT-T, and you cannot convert in place
 **Rule:** a site-to-site IPsec tunnel to any endpoint that might sit behind NAT
 needs **`type dynamic`** (dialup responder) **and `nattraversal enable`**. Both.
 Neither alone is sufficient.
 **Why:** ESH↔colo died the moment ESH stopped having a public IP. Two independent
 causes, and the second was invisible until the first was investigated:
 | setting | broken tunnel | working tunnel |
 |---|---|---|
 | `type` | `static`, `remote-gw 70.181.90.232` (a dead address) | `ddns` |
 | `nattraversal` | `disable` | `disable` — but NH3 is **publicly addressed**, so it never mattered |
 The static peer IP is the obvious failure. The subtle one is that **`nattraversal
 disable` would have kept the tunnel down even with the correct peer IP**, because
 ESP cannot traverse NAT without UDP-4500 encapsulation. A "just re-pin the IP"
 fix would have failed and looked mysterious.
 ⚠ **FortiOS refuses `set type dynamic` on an existing tunnel** — *"Cannot change
 tunnel type once configured"*, with a clean rollback. So the fix is not an edit.
 **Prefer building the replacement ALONGSIDE the broken one, not recreating it.**
 Deleting a phase1 cascades into its phase2, its static routes and every policy
 referencing the interface — on the affected box that was 1 + 2 + 10 objects.
 A new `phase1` + `phase2` + one route + two consolidated policies is additive,
 leaves the old config intact as rollback, and cannot break what still works.
 **Confirming it worked** — the tunnel summary line says everything:
 ```
 'ana-eshudm-dyn_0'  97.170.236.56:4500  selectors(total,up): 1/1
 ^^^ _0 = dialup child      ^^^ carrier IP    ^^^ :4500 = NAT-T
 ```
 `_0` means the peer was accepted without being known in advance; `:4500` means
 NAT-T is carrying ESP; the address is the carrier's, which could never have been
 pinned. And traceroute drops from "8 hops wandering the carrier" to "gateway →
 peer → destination".
 ⚠ **Residual fragility on the UniFi end.** The UDM's `ipsec_local_ip` must hold a
 literal address — `""` is rejected with `api.err.InvalidPayload` — so it still
 needs updating whenever that site's WAN address changes. The gateway end is now
 address-agnostic; the UniFi end is not.
 ---
 ## 10. IPv6 collapses two independent exposure controls into one, and it fails open
 **Rule:** before enabling IPv6 on any segment carrying real hosts, write explicit
 default-deny inbound policy for that segment **and verify it from off-net**.
 Reading the ruleset is not verification.
 **Why — the asymmetry, which is the part worth internalising.** Under IPv4 with
 NAT, exposing an internal host required **two** affirmative acts: a DNAT/port
 forward *and* an accept rule. Miss either and the host stays dark. There is no
 v4 misconfiguration that accidentally exposes an internal host, because without
 the translation there is no path at all. NAT was load-bearing security whether or
 not it was designed as such.
 Under IPv6 the path exists inherently — the address is routable from birth. The
 firewall is now the *only* control, so two independent things that both had to
 succeed become one thing that must not fail. **The failure mode inverts from
 fail-closed to fail-open.**
 **Concrete ways it bites:**
 | failure | v4 consequence | v6 consequence |
 |---|---|---|
 | permissive rule ordered above the deny | harmless, no forward exists | immediate exposure |
 | ruleset silently only matches one address family | v4 covered, v6 ungoverned | whole segment on default |
 | new VLAN added, firewall not updated | just a VLAN | live on the internet at first RA |
 | ISP re-delegates a different prefix | n/a | address-literal rules stop matching |
 **How to apply:**
 - Key rules on **interface/zone, not address literals** — a re-delegated prefix
  must not be able to silently unmatch a rule.
 - Treat "enable v6 on a segment" as a change requiring the policy to exist
  *first*, not as a networking toggle followed by cleanup.
 - **Verify from outside.** Probe the segment's v6 addresses from off-net and
  confirm the denies hold. This is §3's "assert the effective value, not the
  presence of a substring" applied to firewall policy: a ruleset that *says*
  deny is not evidence that packets are dropped.
 Operator position on the ESH fleet (2026-08-19): **no 1:1 inbound pass-through.**
 The policy work is writing and proving default-deny, not deciding what to expose.
 ---
 ## 11. Check the writer and the reader together — and name the failure's DIRECTION
 A guard's predicate has to be read against what its writer actually produces.
 Each line is individually reasonable; the mismatch only exists when you hold
 both. **A guard whose test disagrees with its writer's contract has quietly
 stopped guarding, in whichever direction the mismatch runs.**
 Two of these surfaced within an hour on 2026-09-09, on opposite sides of the same
 gate.
 **The dangerous half (ours).** `scripts/refresh-server-info.sh` and its Proxmox
 sibling promoted the SSH capture with an unconditional `mv` whenever ssh exited
 0. Every reader downstream tests that snapshot with `-s`. So a host that
 connected fine and emitted nothing — inspect script dying before its first write,
 output swallowed by a remote wrapper — would **replace a good snapshot with an
 empty file and report `ok (0 bytes)`.** The header claimed "a failed run never
 clobbers the previous good snapshot", which was true only for a failed
 *connection*; the succeeded-but-empty case is the half nobody inspects, because
 the line reads reassuring. Fixed `9b9f062`: empty capture refused, previous
 snapshot kept, host counted as failed and the exit code carries it. A capture
 that collapses to under a quarter of the previous one still **promotes but is
 flagged** — a host really can shed services, and a script that guesses there will
 eventually guess wrong on a real one.
 **The annoying half (brokkr-smithy-dev's, same day).** A launch guard tested a
 completion sentinel with `-s`; the writer creates it with `touch`, so it is zero
 bytes. The precondition could never pass, whatever the upstream job did. Worse
 *shape* than a regression: from the outside it is indistinguishable from a
 legitimate refusal, so it sends you hunting a problem that does not exist.
 ### The three directions, because the mitigations differ
 Same root — the instrument observed something *adjacent* to what it was named
 after — but these do not collapse into "the instrument was wrong":
 | direction | specimen (all 2026-09-09) | what it wants |
 |---|---|---|
 | **False reassurance** | the empty-snapshot promote; `pgrep -f base_window_r7` over ssh matching its own argv, reporting a peer's job "alive" for 2.5 h while blind to it | an **independent observation of the object** — the seat's own request log (`Running: N reqs`), the artifact itself |
 | **False refusal** | `-s` on a `touch`ed sentinel | a **predicate that matches its writer's contract** |
 | **False alarm** | an error scan reporting 2 hits by matching the word "refusal" in a log | a **pattern that matches the thing**, not a word appearing near it |
 **False reassurance is the one that kills you quietly.** The other two announce
 themselves: they waste attention and misdirect, but they cannot silently destroy
 a good artifact. A post-mortem that lumps all three together loses the half that
 decides what to do about it.
 ### Filter on the ARTIFACT, not on the name pointing at it
 Measured 2026-09-09 while quantifying how much traffic had reached a tune that
 failed a safety gate. The gateway alias `trial` had pointed at three different
 artifacts across the day, so the obvious query — *"how many calls to `trial`?"* —
 answers a question about a **name**, not about the **thing**:
 | filtered on | rows |
 |---|---|
 | `model_group = 'trial'` (the alias) | **363** |
 | `model = 'hosted_vllm/erp-tune-v7-nvfp4a16'` (the artifact) | **77** |
 **Wrong by 4.7x, in the direction that looks careful.** Reporting 363 would have
 overstated the operator's own exposure nearly fivefold, and nothing about the
 query would have looked sloppy — an alias is what a caller types, so counting it
 feels like counting usage.
 **The rule:** when the question is *"what did this artifact do"*, filter on the
 artifact's identity, never on a mutable pointer to it. A name that has been
 repointed carries the history of everything it ever pointed at. Same family as §3
 (identity, not resemblance): an alias resembles the thing and is not it.
 ### A fourth variant: the instrument read a surface MID-TRANSITION
 Added 2026-09-09 from a near-miss brokkr-smithy-dev caught and did not send.
 Verifying the `trial` alias removal, its first read returned 34 aliases with the
 alias **still present** — which looked exactly like the fix had not taken. It
 had. The read had raced the gateway restart. The tell was that the next three
 reads came back non-JSON, because the service was mid-restart; waiting for it to
 settle returned 33 and no alias.
 Had that first read been sent, it would have been a **false alarm during an
 incident**, and the cost is specific: the other party goes back to re-verify a
 fix that was already correct, on the word of an observer who sampled a surface at
 a moment nobody meant to ask about. The instrument answered honestly about the
 wrong instant.
 **The rule: a disagreement between two observers is not a finding until the
 boring explanation is ruled out** — a race, a restart, a cache, a stale read.
 During an incident the pressure runs the other way, because a discrepancy feels
 urgent and urgency argues for sending it immediately. Read twice, let the surface
 settle, and prefer the explanation that requires nothing to be wrong.
 **How to apply.** When you write or review a guard, open its writer in the same
 pass and state the contract out loud — `touch` → exists-but-empty; `mv` on
 exit-0 → may be empty; `>` → may be truncated; `pgrep -f <literal>` → matches
 your own argv. Then ask which direction this predicate fails toward. Sibling of
 §3 (identity, not resemblance) and §2 (observe the state, don't infer it).
 ## Superseded claims
 | date | claim | correction |
 |---|---|---|
 | 2026-08-18 | "ESH behind CGNAT will break the inter-site tunnels, so IPv6 is the escape hatch" | **Half true, and the halves matter.** Tested live on RFC1918 double-NAT (`192.168.200.111`): **Site Magic (NH3↔ESH, WireGuard) HELD** — it dials out to NH3's public edge and never needs inbound reachability. **IPsec (colo↔ESH, ana-gw FortiGate) BROKE** — traceroute showed traffic unencapsulated, leaking to the carrier. IPv6 keeps its justification on the IPsec link only. |
 | 2026-08-18 | *(my own, same day)* "no addressing outcome on the fiber threatens the inter-site tunnel" | **Over-generalised.** I proved it for WireGuard and wrote it as if it covered every link. Operator caught it. See lesson 8. |
@@ -0,0 +1,183 @@
 # The fleet ops log
 **What it is:** one appended line per host-changing action, plus a lightweight
 claim so two agents do not deploy the same stack at the same time.
 **Tool:** [`scripts/ops-log`](../../scripts/ops-log).
 **Storage:** `<repo>/.ops-log/` on nh3-dev — gitignored, override with `OPS_LOG_DIR`.
 ---
 ## Why it exists
 `infra-ops` and `infra-hermes` are two different agents that act as the **same
 OS identity** (`ssh infra-ops@<host>`), and dockerd does not log `exec` per
 caller. Host-side changes are therefore fingerprint-less: when a host differs
 from expectation, neither agent can tell whether the other did it, a prior
 session did, or something broke on its own. Git does not close the gap either —
 every commit in this repo is attributed to Vuong Hoang by convention, which is
 correct for authorship and useless for attribution.
 With one agent this was theoretical. With two doing routine ops daily it is
 operational.
 **The precipitating incident (2026-09-18).** A second session edited the
 searxng stack while another was deploying it. Fleet search crash-looped for
 ~4 minutes, and afterwards the author was unidentifiable: the commit said
 Vuong Hoang and the on-host file carried no provenance at all.
 ---
 ## Design decisions, and why
 Four questions had to be settled. Each one had a defensible alternative; the
 reasoning matters more than the answer, because the alternatives will look
 attractive again the next time someone extends this.
 ### 1. The log lives centrally on nh3-dev — not per-host, not on the post office
 Both agents run as the **same unix user on nh3-dev** (`infra-hermes` is
 `althing-pump-infra-hermes.service`, a *user* unit under `lkraven`). So a single
 file is shared instantly, with zero provisioning, zero permissions story, and no
 network hop.
 - **Not per-host.** Provenance next to the thing changed is appealing, but it
  needs a writable log path created on ~25 heterogeneous hosts (Debian, PVE,
  DSM) with different sudo situations — and an instrument that is only deployed
  on some of them lies by omission. It also puts the record of "we changed X on
  host Y" *on host Y*, which is exactly where it is least available when Y is
  the thing that broke.
 - **Not syslog/journald.** Tempting (no provisioning at all), but journald only
  shows an unprivileged reader entries matching their own `_UID`, and the fleet
  acts as `infra-ops` on some hosts and `lkraven` on others. The log would have
  silently split in half along an axis nobody would think to check.
 - **Not the post office.** A message bus is not a log: ordering and query are
  poor, it becomes inbox noise, and a post-office outage would block ops during
  precisely the incident you are trying to reconstruct.
 **The known hole, stated rather than papered over:** an actor operating from a
 box other than nh3-dev is not covered. Today that is only the operator on his
 laptop. If a third agent ever runs elsewhere, this decision is the one to revisit.
 ### 2. The claim is advisory, and enforced in the tooling
 `deploy-stack.sh` **refuses** (exit 3) a stack another agent holds. Nothing
 stops a raw `ssh` — the point is to make the sanctioned path safe, not to build
 a cage that people route around.
 Acquisition is `mkdir` of a claim directory, which is atomic, so the check is
 genuinely race-free rather than "probably fine". Claims carry a TTL (default
 30m) and a stale claim is auto-broken — **and the break is logged**, so a claim
 that keeps getting broken is visible rather than silently ineffective.
 ### 3. The writers are automatic
 `deploy-stack.sh` and `elway` record themselves. This was the question that
 mattered most: **a log that depends on remembering is the same class of
 instrument as a health check that passes in both states.**
 ### 4. There is a detector, not just a rule
 Automatic writers cannot cover the manual path — someone ssh'ing in and editing
 a compose file by hand. The answer is not a louder rule; it is
 `ops-log audit`, which asks each host what changed on disk and compares it to
 the newest log line for that stack. Anything that changed with no log line at or
 after it is reported.
 Per the same discipline, `audit` refuses to say "clean" for a host it could not
 reach — an unreachable host reports `INCOMPLETE` and exit 5, never 0.
 ---
 ## Using it
 ```bash
 # Before hand-editing a stack on a host, so other agents' tooling refuses it:
 scripts/ops-log claim nh3-docker searxng --why "raising the engine timeout"
 scripts/ops-log release nh3-docker searxng          # when done
 # After any change made OUTSIDE deploy-stack.sh / elway:
 scripts/ops-log record --host nh3-docker --action restart --target searxng \
  --detail "docker restart searxng after the conf edit"
 # Reading:
 scripts/ops-log tail --since 24h                    # everything today
 scripts/ops-log tail --host nh3-docker -n 20        # one host
 scripts/ops-log claims                              # who is holding what
 # The detector — run it when a host surprises you, or periodically:
 scripts/ops-log audit                               # the six stack hosts
 scripts/ops-log audit nh3-docker --since 7d
 scripts/ops-log audit all                           # every non-tenant host
 ```
 `deploy-stack.sh` claims and records on its own. Escape hatches:
 `DEPLOY_NO_CLAIM=1` skips the claim, `DEPLOY_CLAIM_TTL=<dur>` lengthens it.
 ops-log being absent or broken never blocks a deploy — only a live foreign
 claim does.
 ### Exit codes
 | code | meaning |
 |---|---|
 | 0 | success / claim acquired / audit clean |
 | 2 | usage error |
 | 3 | claim refused — held by another agent |
 | 4 | audit found unlogged changes |
 | 5 | audit could not reach every host (**incomplete, not clean**) |
 ---
 ## How this instrument has failed — the pattern, not just the fixes
 **All four failures below shipped on the day it was built, 2026-09-19, and
 every one of them RECORDED SOMETHING.** None of them recorded something
 *findable* or *attributable*. That is the shape of the failure class here: this
 tool does not fall over loudly, it answers the wrong question quietly. Read
 this table before assuming a missing entry means the event did not happen.
 | # | Failure | Symptom | Cause | Fixed |
 |---|---|---|---|---|
 | 1 | **Claim released by a sub-tool** | A 45-min operation claim vanished mid-rollout; nothing refused anything | `deploy-stack.sh` claims and releases around its own work, and released a claim it had only refreshed | `3e7d3a3` — `claim` exits 10 when already yours and leaves the holder file untouched |
 | 2 | **Wrong order in the hook chain** | Some commits logged, some not; looked flaky | The commit hook was APPENDED behind graphify's eight `exit 0` paths, so a `graphify-out/`-only or empty commit could never reach it | `9141a41` — prepend after the shebang; attribution runs first, unconditionally |
 | 3 | **No handle in the environment** | A peer's commit logged as `lkraven@nh3-dev` | `ALTHING_HANDLE` was exported only by `althing-infra-hermes-seat-run.sh`, not by `hermes-gateway.service` or the pump unit | Drop-in overrides (infra-hermes) + `4e778ae` — the fallback now says `unattributed(login)` instead of a bare login that reads like an answer |
 | 4 | **Wrong host key on write** | Five records of a real `jobs.cfg` change on esh-pve were invisible to `--host esh-pve` | elway passed its ssh TARGET through as the host, so they landed under `infra-ops@esh-pve` | `f3b68e2` — `normalize_host` strips `user@` on write AND on query, so pre-fix records are findable too |
 **The rules that fell out of it, in descending order of how much they cost:**
 1. **A log you cannot query under the obvious name is not a log.** #4 was not
   cosmetic — it produced a confident "this change is unattributed" report
   about a change that had five records.
 2. **Attribution must run FIRST**, never behind another tool's opinion about
   whether an event is interesting.
 3. **An instrument that cannot answer must say so.** A bare unix login where a
   handle belongs is answer-shaped; `unattributed(...)` is not.
 4. **A sub-tool must not drop the claim wrapping its caller.**
 5. **Two samples that look flaky deserve a control-flow audit before a
   concurrency hypothesis.** #2 was diagnosed as an `fcntl` race and was not;
   it was deterministic and five lines up.
 ⚠ **Absence of an entry is not evidence of absence of the event** until you
 have checked that this tool could have recorded it in the first place.
 ## What it deliberately does not do
 - **It does not cover raw `ssh`.** It cannot. `audit` is the backstop.
 - **It does not claim per-host for elway.** elway records but does not claim:
  two unrelated playbooks on one host are not a collision, and a host-wide lock
  would be ignored within a week.
 - **It does not audit `corviduo-dev`.** Worldtree deploys there are CI/CD-driven
  and rewrite the tree constantly, so it would report unlogged changes forever
  and train us to ignore the output.
 - **It does not audit the SureFire tenant hosts** (`sf-*`, `sfsrv-ana`), which
  are client property under the hosting agreement.
 - **It does not cover DNS, Cloudflare, UniFi or FortiGate changes automatically.**
  Those have no host-side compose tree to diff. Record them by hand with
  `--host <appliance>` until `dns-sync.py` and friends grow the same hook.
 ## Open follow-ons
 - Hook `scripts/dns-sync.py` (and the UniFi/FortiGate helpers) so
  control-plane changes record themselves too.
 - Run `audit` on a timer and alert on non-zero, rather than only when someone
  remembers to look.
@@ -1,262 +1,77 @@
-# PFI-ANA Proxmox VM Inventory
+# PFI-ANA Proxmox guest inventory (pfi-pve)
-**Hypervisor**: Proxmox VE at `10.250.250.31:8006`
+_Refreshed 2026-09-06 04:47Z from live `qm list`, `pct list`, `/etc/pve/*/N.conf` and
-**Storage Pool**: `ospool` (CEPH/zfs — all VM disks reside here)
+guest-agent IPs. Earlier versions of this file described guests that no longer exist
-**Network Bridge**: `vmbr0` with VLAN tag `50` on all VMs
+(VM 100 as the TrueNAS box, VM 106 as "PFI-Tailscale", VM 108 "PFI-ANA--DEV"); this
-**QEMU Version**: 7.2.0 (primary), VM 106 on 8.1.5
+version is the live state._
-## Virtual Machines
+**Hypervisor:** `pfi-pve` — Dell PowerEdge R750xs, Xeon Silver 4310 (48 threads), 188 GB,
 Proxmox VE 8.3.5, `https://10.250.250.31:8006`, `ssh infra-ops@10.250.250.31` (NOPASSWD sudo, since 2026-09-06) or `root@`. iDRAC `https://10.250.250.30`.
-### VM 100 — PFI-ANA-TRUENAS
+**Storage:**
-| Property | Value |
+| id | type | backs | notes |
 |---|---|
 | **VMID** | 100 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 2 cores = 4 vCPU (host passthrough) |
 | **Memory** | 8,196 MB |
 | **Disk** | `scsi0`: 80G on ospool |
 | **CDROM** | `ide2`: TrueNAS-SCALE-22.12.1.iso _(original install media; current OS is Debian 12)_ |
 | **Network** | `net0`: virtio, MAC `9A:90:79:7A:86:87`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 2, delay 120s |
 | **Onboot** | No |
 **Purpose**: NAS/NFS/SMB storage for the colo environment.
 **Current OS**: Debian 12 (bookworm), reprovisioned from the original TrueNAS SCALE install — SSH banner `OpenSSH_9.2p1 Debian-2+deb12u9`, no web UI on 80/443, no TrueNAS middleware. Shares are hand-managed via `/etc/exports` and `/etc/samba/smb.conf`. Mount paths kept the `/mnt/<pool>/…` shape of the original ZFS layout for host compatibility.
 ---
 ### VM 101 — PFI-ANA-DC
 | Property | Value |
 |---|---|
 | **VMID** | 101 |
 | **OS Type** | Windows 11 |
 | **BIOS** | OVMF (UEFI) with TPM 2.0 |
 | **Machine** | pc-q35-7.2 |
 | **CPU** | 2 sockets × 6 cores = 12 vCPU (host passthrough) |
 | **Memory** | 24,576 MB |
 | **Disk** | `scsi0`: 240G on ospool |
 | **EFI Disk** | `efidisk0`: 1M on ospool |
 | **TPM** | `tpmstate0`: 4M, v2.0 on ospool |
 | **CDROM** | `scsi1`: virtio-win-0.1.229.iso (VirtIO drivers) |
 | **Network** | `net0`: e1000, MAC `CE:C8:D7:FE:32:40`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → net0 → ide0 → scsi1 |
 | **Startup** | Order 5, delay 120s |
 | **Onboot** | Yes |
 **Purpose**: Windows Domain Controller for the Anaheim environment. UEFI with TPM 2.0 suggests Active Directory / Group Policy services.
 ---
 ### VM 102 — PFI-ANA-Docker
 | Property | Value |
 |---|---|
 | **VMID** | 102 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) |
 | **Memory** | 16,384 MB |
 | **Disk** | `scsi0`: 250G on ospool |
 | **Network** | `net0`: virtio, MAC `BA:AF:E7:E9:79:23`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → net0 → scsi1 |
 | **Startup** | Order 4 |
 | **Onboot** | Yes |
 **Purpose**: Primary Docker host for the colo. Runs Dockge for compose management and Traefik for reverse proxy. All Docker services documented in [docker-stack.md](docker-stack.md) run here.
 ---
 ### VM 103 — PFI-SlaveBot
 | Property | Value |
 |---|---|
 | **VMID** | 103 |
 | **OS Type** | Windows 10 |
 | **Machine** | pc-i440fx-7.2 |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
 | **Memory** | 8,192 MB |
 | **Disk** | `ide0`: 256G on ospool |
 | **Network** | `net0`: e1000, MAC `CE:F0:49:C9:03:70`, vmbr0, VLAN 50 |
 | **Boot** | ide0 → net0 → scsi0 |
 | **Startup** | Not configured |
 | **Onboot** | Yes |
 **Purpose**: Windows 10 workstation/bot. Likely a task automation or RDP-accessible machine.
 ---
 ### VM 104 — PFI-Mongo
 | Property | Value |
 |---|---|
 | **VMID** | 104 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
 | **Memory** | 8,196 MB |
 | **Disk** | `scsi0`: 256G on ospool |
 | **Network** | `net0`: virtio, MAC `32:57:90:B2:66:61`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 3, delay 60s |
 | **Onboot** | Yes |
 **Purpose**: MongoDB server. Config file contains a connection string reference: `mongodb://10.250.50.81:27017/`.
 ---
 ### VM 105 — PFI-Postgres
 | Property | Value |
 |---|---|
 | **VMID** | 105 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 4 sockets × 4 cores = 16 vCPU |
 | **Memory** | 8,196 MB |
 | **Disk** | `scsi0`: 80G on ospool |
 | **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
 | **Network** | `net0`: virtio, MAC `C2:1F:CC:71:66:D0`, vmbr0, VLAN 50 |
 | **Serial** | `serial0`: socket (IPMI/serial console) |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 3, delay 60s |
 | **Onboot** | Yes |
 **Purpose**: PostgreSQL database server running Debian 11.
 ---
 ### VM 106 — PFI-Tailscale
 | Property | Value |
 |---|---|
 | **VMID** | 106 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU (x86-64-v2-AES) |
 | **Memory** | 2,048 MB |
 | **Disk** | `scsi0`: 256G on ospool |
 | **CDROM** | `ide2`: debian-12.2.0-amd64-netinst.iso |
 | **Network** | `net0`: virtio, MAC `BC:24:11:D7:E9:52`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Not configured |
 | **Onboot** | Yes |
 **Purpose**: Tailscale VPN node for mesh connectivity. Provides the VPN tunnel endpoints that link the three PFI sites together. Running Debian 12 (newer than most other VMs). Lightweight at 2G RAM.
 ---
 ### VM 107 — PFI-Pteradactyl
 | Property | Value |
 |---|---|
 | **VMID** | 107 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU (host passthrough) |
 | **Memory** | 8,192 MB |
 | **Disk** | `scsi0`: 256G on ospool |
 | **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
 | **Network** | `net0`: virtio, MAC `CA:44:37:8A:BF:E0`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Not configured |
 | **Onboot** | Yes |
 **Purpose**: Pterodactyl game server panel. Manages game server instances.
 ---
 ### VM 108 — PFI-ANA--DEV
 | Property | Value |
 |---|---|
 | **VMID** | 108 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 2 sockets × 4 cores = 8 vCPU |
 | **Memory** | 8,192 MB |
 | **Disk** | `scsi0`: 120G on ospool |
 | **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
 | **Network** | `net0`: virtio, MAC `F2:EF:82:2C:AF:90`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 10, delay 60s |
 | **Onboot** | No |
 **Purpose**: Development environment. Not set to auto-boot, starts after core infrastructure (order 10).
 ---
 ### VM 110 — PFI-ANA-Webhost
 | Property | Value |
 |---|---|
 | **VMID** | 110 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 4 sockets × 4 cores = 16 vCPU |
 | **Memory** | 4,096 MB (balloon: 1024 MB minimum) |
 | **Disk** | `scsi0`: 250G on ospool |
 | **CDROM** | `ide2`: debian-11.6.0-amd64-netinst.iso |
 | **Network** | `net0`: virtio, MAC `E6:F9:3A:C9:61:2A`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 30, up 120s, down 120s |
 | **Onboot** | Yes |
 **Purpose**: Web hosting server. Highest startup order (30) — starts last. Also has the longest graceful shutdown timeout (120s). Memory ballooning enabled for dynamic allocation.
 ---
 ### VM 111 — pfi-tacticalrmm
 | Property | Value |
 |---|---|
 | **VMID** | 111 |
 | **OS Type** | Linux (l26) |
 | **CPU** | 4 sockets × 4 cores = 16 vCPU |
 | **Memory** | 8,192 MB |
 | **Disk** | `scsi0`: 256G on ospool |
 | **Network** | `net0`: virtio, MAC `BA:FA:65:F6:46:25`, vmbr0, VLAN 50 |
 | **Boot** | scsi0 → ide2 → net0 |
 | **Startup** | Order 20 |
 | **Onboot** | Yes |
 **Purpose**: Tactical RMM (Remote Monitoring and Management) server. Provides IT management, remote access, and monitoring capabilities.
 ---
 ## Startup Order Summary
 VMs are brought up in the following order on host boot:
 | Order | VMID | Name | Delay |
 |---|---|---|---|
-| 2 | 100 | PFI-ANA-TRUENAS | 120s |
+| `ospool` | zfspool, 2×mirror (bays 0–3), 10.9T raw | **every VM boot disk and every CT rootfs** except pbs-ana | the pool that boots the site; also holds `ospool/naspool-evac` (parked copy from the 2026-09-06 rebuild, to be destroyed) |
-| 3 | 104 | PFI-Mongo | 60s |
+| `NASPool` | zfspool, **raidz2 ×6** (bays 4–8, 11), 21.7T usable | backup / bulk tier: `backupStore` (PBS-ANA datastore, restic repos), `pve-VMStorage`, `db`, `docker`, `webdav` | rebuilt 2026-09-06 (`docs/runbooks/pfi-pve-naspool-rebuild.md`); bays 9/10 dead (backplane), drives in them = cold spares |
-| 3 | 105 | PFI-Postgres | 60s |
+| `naspool-vmstorage` | dir on `/NASPool/pve-VMStorage` | ISOs, templates, legacy vzdump (newest-per-guest only) | was `pve-truenas` until 2026-09-06 |
-| 4 | 102 | PFI-ANA-Docker | — |
+| `local-lvm` | lvmthin | pbs-ana's boot disk | |
-| 5 | 101 | PFI-ANA-DC | 120s |
+| `local` | dir | ISOs | |
-| 10 | 108 | PFI-ANA--DEV | 60s |
+| `pbs-ana` | pbs at 10.250.50.90 | vzdump jobs (03:00 all guests except 100,109; 22:00 CT 109) | datastore lives on `NASPool/backupStore` via ana-nas NFS |
 | 20 | 111 | pfi-tacticalrmm | — |
 | 30 | 110 | PFI-ANA-Webhost | 120s |
-VMs without a startup order (103, 106, 107) will start based on their `onboot` setting but without a specific sequencing delay.
+**Network:** all guests on `vmbr0` VLAN tag 50 (`10.250.50.0/24`), firewall enabled on
 every NIC. Linux = virtio, Windows = e1000. CTs take DHCP (reserved).
-## Resource Summary
+## Virtual machines
-| VMID | Name | vCPU | RAM (MB) | Disk | OS |
+| VMID | name | IP | vCPU (sockets×cores) | RAM | disk | OS | onboot / startup | role |
-|---|---|---|---|---|---|
+|---|---|---|---|---|---|---|---|---|
-| 100 | PFI-ANA-TRUENAS | 4 | 8,196 | 80G | Debian 12 (reprovisioned from TrueNAS SCALE) |
+| 100 | pbs-ana | 10.250.50.90 | 4 (1×4, x86-64-v2-AES) | 8 G | 32G local-lvm | Debian 12 | yes / — | Proxmox Backup Server, fleet primary. `servers/pbs-ana/` |
-| 101 | PFI-ANA-DC | 12 | 24,576 | 240G | Windows 11 |
+| 101 | PFI-ANA-DC | 10.250.50.60 | 12 (2×6, host) | 24 G | 240G ospool + EFI + TPM2 | Windows 11, q35, OVMF | yes / order 5, up 120s | Active Directory domain controller |
-| 102 | PFI-ANA-Docker | 8 | 16,384 | 250G | Linux |
+| 102 | PFI-ANA-Docker | 10.250.50.70 | 8 (2×4, x86-64-v2-AES) | 16 G | 250G ospool | Debian | yes / order 4 | `ana-docker`: LiteLLM gateway, gitea, Langfuse, AdGuard, Beszel/Dozzle hubs, rest-server-ana, backrest. `servers/ana-docker/` |
-| 103 | PFI-SlaveBot | 8 | 8,192 | 256G | Windows 10 |
+| 103 | PFI-SlaveBot | 10.250.50.151 | 8 (2×4, host) | 8 G | 256G ospool (ide0) | Windows 10, i440fx | yes / — | purpose undocumented; **not in `servers/`** |
-| 104 | PFI-Mongo | 8 | 8,196 | 256G | Linux |
+| 104 | PFI-Mongo | 10.250.50.81 | 8 (2×4, host) | 8 G | 256G ospool | Linux | yes / order 3, up 60s | MongoDB; **not in `servers/`** |
-| 105 | PFI-Postgres | 16 | 8,196 | 80G | Debian 11 |
+| 105 | PFI-Postgres | 10.250.50.80 | 16 (4×4) | 8 G | 80G ospool | Debian 11 | yes / order 3, up 60s | shared Postgres (vaultwarden/gitea/paperless); PGDATA local since 2026-04-23. `servers/pfi-postgres/` |
-| 106 | PFI-Tailscale | 8 | 2,048 | 256G | Debian 12 |
+| 106 | corviduo-dev | 10.250.50.152 | 8 (2×4, x86-64-v2-AES) | 8 G | 80G ospool | Debian 13 | yes / order 10 | Worldtree-team dev VM, PFI-hosted. `servers/corviduo-dev/` |
-| 107 | PFI-Pteradactyl | 8 | 8,192 | 256G | Debian 11 |
+| 107 | PFI-Pteradactyl | 10.250.50.55 | 8 (2×4, host) | 8 G | 256G ospool | Debian 11 | yes / — | Pterodactyl game panel. `servers/pfi-pteradactyl/` |
-| 108 | PFI-ANA--DEV | 8 | 8,192 | 120G | Debian 11 |
+| 110 | PFI-ANA-Webhost | 10.250.50.52 | 16 (4×4) | 4 G (balloon 1 G) | 250G ospool | Debian 11 | yes / order 30, up/down 120s | web workload. `servers/pfi-ana-webhost/` |
-| 110 | PFI-ANA-Webhost | 16 | 4,096 | 250G | Debian 11 |
+| 111 | pfi-tacticalrmm | 10.250.50.57 | 16 (4×4) | 8 G | 256G ospool | Linux | yes / order 20 | TacticalRMM. `servers/pfi-tacticalrmm/` |
 | 111 | pfi-tacticalrmm | 16 | 8,192 | 256G | Linux |
 | | **Totals** | **112** | **105,348** | **2,300G** | |
-## Network Notes
+All VMs have the QEMU guest agent enabled.
- All VMs are on **VLAN 50** via `vmbr0`.
+## Containers
- All VMs have **firewall enabled** on the network interface.
+
- Linux VMs use `virtio` network adapters; Windows VMs use `e1000`.
+| CTID | hostname | IP | cores | RAM | rootfs | privileged | onboot / startup | role |
- The MongoDB connection string embedded in VM 104's config references IP `10.250.50.81`, suggesting VLAN 50 maps to the `10.250.50.0/24` subnet within the `10.250.0.0/16` range.
+|---|---|---|---|---|---|---|---|---|
 | 109 | ana-nas | 10.250.50.50 | 4 | 2 G | 80G ospool | **yes** (apparmor unconfined, all devices) | yes / order 1, up 45s | NFS/SMB server straight off the ZFS pools — bind-mounts `/ospool/backup` and every `NASPool/*` dataset under `/mnt/…`. Data-serving SPOF. `servers/ana-nas/` |
 | 112 | ana-filebot | 10.250.50.53 | 4 | 2 G | 80G ospool | **yes** (same binds as 109) | yes / order 10 | file-task automation. `servers/ana-filebot/` |
 | 113 | ana-wg | 10.250.50.252 (+ wg0 10.30.10.1) | 4 | 2 G | 8G ospool | no (nesting) | yes / order 2 | WireGuard remote-access endpoint, UDP 31337 via FortiGate VIP. `servers/ana-wg/` |
 ## Startup order (host boot)
 | order | guest | delay |
 |---|---|---|
 | 1 | CT 109 ana-nas | up 45s |
 | 2 | CT 113 ana-wg | — |
 | 3 | VM 104 PFI-Mongo, VM 105 PFI-Postgres | up 60s |
 | 4 | VM 102 PFI-ANA-Docker | — |
 | 5 | VM 101 PFI-ANA-DC | up 120s |
 | 10 | VM 106 corviduo-dev, CT 112 ana-filebot | — |
 | 20 | VM 111 pfi-tacticalrmm | — |
 | 30 | VM 110 PFI-ANA-Webhost | up/down 120s |
 | unordered | VM 100 pbs-ana, VM 103, VM 107 | onboot=1, no sequencing |
 Reverse this for a graceful host shutdown (done 2026-09-05 for the PERC swap).
 ## Resource summary
 | | vCPU | RAM | disk |
 |---|---|---|---|
 | 10 VMs | 108 | 100 G | 2,180 G |
 | 3 CTs | 12 | 6 G | 168 G |
 | **total** | **120** on 48 threads (2.5× overcommit) | **106 G** of 188 G | on ospool 547 G actually allocated (thin) |
 ## Not on this hypervisor
 `fv-ml1` (10.251.50.54) is bare-metal Supermicro; `sfsrv-ana` (10.250.250.115) is the
 SureFire client's own Dell R630; `pfi-gx10` is at NH3.
@@ -560,7 +560,8 @@ override the config default.) Values set per the `dvalin-smithy-dev` research pa
 | `gen`, `summarizer-large`, `qwen-large`, `qwen3.5-122-a10b` (non-thinking) | **0.7** | 0.8 | 20 | **1.0** | — | Qwen3 non-thinking + operator anti-repetition |
 | `gen-reasoning`, `qwen-large-reasoning`, `qwen3.5-122-a10b-reasoning` (thinking) | **0.6** | 0.95 | 20 | **1.0** | — | Qwen3 thinking |
 | `qwen-image-bench`, `image-judge` | **0** | 1.0 | 1 | — | 1.05 | Qwen-Image-Bench judge reproducibility table |
-| `selene-1-mini-8b`, `chat-judge` | **0.6** | 0.9 | — | — | — | Selene `generation_config` |
+| ~~`selene-1-mini-8b`~~ | — | — | — | — | — | **RETIRED 2026-08-23**; name 404s by design, not aliased |
 | `chat-judge` | **0** | 1.0 | 1 | — | 1.05 | Repointed to `gen` 2026-08-23; deterministic judge profile copied from `image-judge`. The benchmark that selected `gen` ran at temperature 0 — match it. |
 | `glm-5.1`, `glm-5.2`, `glm-5-turbo`, `glm-4.7`, `gen-frontier` | **1.0** | 0.95 | — | — | — | z.ai API defaults (5.x / 4.7 series) |
 | `glm-4.5-air` | **0.6** | 0.95 | — | — | — | z.ai API default (4.5 series) |
 | `qwen3-embedding`, `qwen3-reranker`, `reranker` | — | — | — | — | — | no sampling (embedding / rerank) |
@@ -0,0 +1,254 @@
 # Fleet reranker selection — process ledger
 Running record of the Brokkr-driven fleet-reranker selection, and every
 assumption / autonomous decision infra-ops makes on the operator's behalf
 during it. The operator (Vuong) will review this at the end and reverse
 anything he wants. **This is the audit trail for unattended operation.**
 Started: 2026-08-06. Driver: **brokkr-smithy-dev**. Executor: **infra-ops** (this session).
 ---
 ## Operator authorization envelope (2026-08-06)
 Brokkr drives a reranker-selection process; infra-ops is cleared to proceed on
 Brokkr's recommendations **unattended** (no per-step operator check-in), with
 authority to do whatever is necessary to reach a recommendation **or**
 implementation.
 **CLEARED (green):**
 - Execute Brokkr's reranker-selection recommendations unattended.
 - Bring **down the prod reranker** at `fv-ml1:8002` (qwen3-reranker-0.6B) —
  **temporarily OR permanently**.
 - Down **ONE** of the RP (roleplay) seats on fv-ml1 **temporarily** to free
  GPU/VRAM for testing.
 - Temporarily clear space for the smoke/bench.
 - Pull models, stand up side-port vLLM benches, run the harness — whatever the
  eval needs.
 **RED LINES (hard NO — stop + surface even under standing auth):**
 - **NO permanent deletion of anything** (no `rm`/`docker volume rm`/model-weight
  deletion/data destruction). Downing ≠ deleting.
 - **NO taking anything else offline** beyond (a) the prod reranker and (b) ONE
  fv-ml1 RP seat. (Not granite/embed/reward/coder/gen/a second RP seat/muninn/etc.)
 - **NO rebooting machines.**
 **Process:** accumulate assumptions here; operator reverses at the end.
 ---
 ## Standing assumptions / autonomous-decision log
 - **A1 — Coordinated-change notify still applies.** Even under unattended auth,
  every `:8002` state change gets a timestamped announcement to worldtree-dev +
  brokkr-smithy-dev (their standing coordinated-change ask; the operator waived
  per-step *operator* approval, not the peer *notify* courtesy). No silent flip.
 - **A2 — Weights are never deleted, only unserved.** "Permanently down the qwen
  reranker" = stop serving + (optionally) repoint the gateway alias; the 0.6B
  model weights stay on disk (deletion is a red line).
 - **A3 — RP-seat pick = lowest-impact, temporary, restored after.** When a seat
  must come down for VRAM, I pick the lowest-impact RP seat, log which + its
  exact restore command, and bring it back when the bench frees the GPU.
 ---
 ## Current board at handoff
 - **Prod reranker:** `fv-ml1:8002` = `vllm-rerank` (Qwen/Qwen3-Reranker-0.6B),
  reverted to baseline `classifier_from_token:["no","yes"]`, healthy. Compose:
  `/opt/docker/compose/vllm/compose.yaml` (canonical mirror
  `stacks/vllm/compose.yaml`). Gateway alias `reranker`/`qwen3-reranker` →
  litellm → :8002.
 - **Root cause (converged, both sides):** 0.6B is capacity-bound on bare-name
  queries over a real candidate pool; NOT misconfigured. Fix = larger model.
 - **Verified on-prem candidate shortlist (all HF-real, ungated):**
  Qwen/Qwen3-Reranker-4B, Qwen/Qwen3-Reranker-8B, mixedbread-ai/mxbai-rerank-large-v2,
  mixedbread-ai/mxbai-rerank-base-v2, BAAI/bge-reranker-v2-gemma,
  Alibaba-NLP/gte-reranker-modernbert-base, jinaai/jina-reranker-v2-base-multilingual.
  (BAAI/bge-reranker-v2-m3 exists but the fleet already moved off it.)
 - **Eval assets (all on nh3-dev):**
  - Scorer: `scripts/probe_389_rank_decomposition.py` (Worldtree repo, main) —
    rank-recovery = `rrf_rerank` column climbing back toward `rrf`.
  - worldtree-dev grids: `~/snapshots/r42-gate-snapshot/` (probe_389_run3.json,
    probe_389_question_shaped.json, probe_389_bigboi_control.json).
  - Frozen gate Chroma snapshot: `~/snapshots/r42-gate-index/` (retained until
    worldtree-dev signals the lever run is done).
  - **Dual query-set requirement (hard):** score bare-name anchor queries AND
    question-shaped; bar = recovering the name-lookup class.
 - **VRAM:** 4B ≈ 4–5 GB fp8, 8B ≈ 9 GB; fv-ml1 Blackwell has headroom.
 ---
 ## Progress log
 ### 2026-08-06 — A2 brought up (Brokkr thread 01KZBSTSJA…)
 - **Backend:** `vllm-rerank-a2` — standalone `docker run` (NOT in the vllm compose
  stack), on fv-ml1 **GPU1**, host port **:8012** → container 8000. Image
  `vllm/vllm-openai:latest` (=0.24.0). Args: model
  `tomaarsen/Qwen3-Reranker-0.6B-seq-cls`, `--runner pooling`, `--gpu-memory-utilization
  0.03`, `--max-model-len 8192`, `--dtype auto`, `--restart no`. Native
  `Qwen3ForSequenceClassification` — NO hf-overrides. Routes /rerank /score /classify.
 - **Gateway alias:** `reranker-a2-qwen3-seqcls` → `http://10.251.50.54:8012/v1`,
  mode rerank. Added via LiteLLM **`/model/new`** (DB-backed, `store_model_in_db:true`)
  — **no gateway restart** (respects the "nothing else offline" line). Verified 200
  through the gateway.
 - **Metrics:** VRAM ≈ **3.5 GB** (GPU1 free 14167→10616 MiB). Latency (20-doc pool,
  ~1500-char docs, shared GPU1): single p50 **87 ms**; 8-concurrent p50 **140 ms**,
  ~**55 req/s**.
 - **Correctness (3-probe smoke, not the grid):** tracks the incumbent within noise →
  early signal the failure is the **training prior, not the inference head**.
 **Autonomous decisions this step (reversible):**
 - D1 — port :8012, GPU1, util 0.03 to mirror the incumbent's exact footprint (clean control).
 - D2 — standalone `docker run` (not compose) so bench arms are throwaway; no canonical churn to revert.
 - D3 — gateway wired via `/model/new` (runtime, DB-persisted) rather than config-edit + restart.
 - D4 — did NOT down any RP seat (A2 is 0.6B / 3.5 GB; no VRAM pressure).
 **Cleanup for A2 (run at end / on reversal):**
 - `ssh infra-ops@10.251.50.54 'sudo docker stop vllm-rerank-a2 && sudo docker rm vllm-rerank-a2'`
 - Delete gateway alias: `POST /model/delete {"id": <model_id>}` (id via `/model/info?model_name=reranker-a2-qwen3-seqcls`), infra-ops admin key. (DB-persisted, so it survives a restart — must be explicitly deleted.)
 - No weights deleted (red line); HF cache under /tank/aimodels/huggingface retains the 0.6B-seq-cls download.
 **Ports reserved for the bench:** :8012 (A2), :8013 (A3), :8014 (A4), :8019 (A5).
 ### 2026-08-06 — A3 + A4 pre-staged (Brokkr said pre-stage in parallel, hold A5)
 - **A3** `vllm-rerank-a3` — fv-ml1 GPU1 :8013, `BAAI/bge-reranker-v2-m3`
  (XLMRobertaForSequenceClassification), same run pattern, util 0.03. VRAM ≈ **2.3 GB**.
  Latency (20-doc, ~1500-char, shared GPU1): single p50 **105 ms**; 8-conc p50 214 ms, ~34 req/s.
  Gateway alias `reranker-a3-bge-v2-m3` via /model/new (200, verified).
 - **A4** `vllm-rerank-a4` — fv-ml1 GPU1 :8014, `Alibaba-NLP/gte-reranker-modernbert-base`
  (ModernBertForSequenceClassification), util 0.02. VRAM ≈ **1.4 GB**. Latency: single
  p50 **102 ms**; 8-conc p50 153 ms, ~51 req/s. Gateway alias `reranker-a4-gte-modernbert`
  via /model/new (200, verified).
 - **Smoke (2-doc, NOT authoritative):** BOTH decisively rank the bare-name Hobgoblin doc top
  (A3 0.999, A4 0.982) where A2/incumbent FAIL (0.33). Cross-encoder / different-lineage.
  Caveat: Brokkr warned isolated tests overstate; his 20-pool grid is the real call.
 - **GPU1 state:** A2+A3+A4 ≈ 7.2 GB resident; GPU1 free ≈ **6.9 GB**. No RP seat downed.
  If A5 (4B, ~4–5 GB) is greenlit: fits GPU1 tight or GPU0 (~9 GB free) — no RP-seat downing expected.
 **Cleanup for A3/A4 (same pattern as A2):** `docker stop/rm vllm-rerank-a3 vllm-rerank-a4`
 on fv-ml1; `/model/delete` the two aliases (DB-persisted); weights retained in HF cache.
 ### 2026-08-06 — A2 verdict (Brokkr full grid): training-prior confirmed
 - **A2 ≡ incumbent, statistically indistinguishable** (identical gold-rank on 7/8 probes,
  max 1-rank divergence; n=14: A2 7/14 top-10 @ mean rank 9.71 = incumbent to 2 dp;
  no-reranker 13/14 @ mean 2.79). The seq-cls head changes nothing → the fault is a
  **training prior in the weights**, not the scoring head. (Smoke called it pre-grid.)
 - **A5 (Qwen3-Reranker-4B): HELD INDEFINITELY, not staged** per Brokkr — A2 voided its
  rationale (scale can't fix a prior the head wasn't causing). *Decision: the one expensive
  bring-up is avoided unless Brokkr formally revisits.*
 - **A3/A4:** proceed — already live for Brokkr's grid; now a training-corpus test (BGE vs
  GTE vs Qwen data), lower EV, cost sunk. Awaiting his scoring.
 - **Likely endgame:** NO model swap. Recommendation trending to a **policy change** —
  wing-scoped rerank bypass or `rrf:60` fusion — landing as Worldtree core code behind
  config, NOT a new serving commitment. Would FREE a GPU seat, not allocate one; prod
  `reranker` eventually retired for the fiction path (never silently repointed; Brokkr
  flags before anything touches the prod alias). *Plan: if confirmed, tear the whole bench
  down (A2/A3/A4 containers + 3 aliases) and hand back the GPU.*
 ### 2026-08-06 — FINAL verdict (Brokkr R43.1): A3 wins; cutover HELD for operator
 - **Winner: A3 = `BAAI/bge-reranker-v2-m3`.** Write-up:
  `research/R43-fleet-reranker-selection/RECOMMENDATION.md` (Brokkr repo, tag R43.1).
 - **The incumbent harms the fleet, not just fiction.** n=90 over main + knowledge_base:
  | arm | top-10 | mean rank | harmed vs no-rerank |
  |---|:--:|:--:|:--:|
  | A0 no-reranker | 89/90 | 0.54 | — |
  | A1 incumbent | 56/90 | 7.78 | **80/90 (worst −19)** |
  | **A3 bge-v2-m3** | 90/90 | 0.19 | 7/90 (worst −3) |
  | A4 gte-modernbert | 90/90 | 0.08 | 1/90 (worst −1) |
 - **A3 over A4:** A4 edges A3 on main/kb + is smaller/faster, BUT A4 is **English-only
  (ModernBERT)** → silent degradation on non-English fleet content; A3 is **multilingual
  (XLM-R)** and decisively better on the bare-name regime that started this. A3 also ~1.2 GB
  *cheaper* than the incumbent. A4 kept as documented throughput fallback.
 - **CUTOVER = OPERATOR DECISION (pending).** Brokkr drafted then PULLED the repoint: a
  fleet-wide alias change affecting consumers he doesn't own shouldn't ship on a relayed
  blanket auth while the operator is away. → Surfaced to Vuong. Proceeding-on-Brokkr's-rec
  now literally = HOLD. **Nothing torn down (incl. A2); prod `reranker` :8002 stays incumbent.**
 - **Cutover conditions (when operator says yes):** repoint gateway `reranker` alias
  incumbent→A3; keep incumbent :8002 warm (rollback = one alias edit); keep
  `reranker-a3-bge-v2-m3` as its own distinct alias; keep A4 up as fallback; **announce the
  boundary timestamp on-bus** (worldtree probe re-run + Brokkr v13 gate render need it).
 - **Flag (worldtree-side, not infra):** `rerank_hybrid_floor` should be **dropped, not
  re-tuned** — it compensates for the scorer being replaced. No serving work to stage for it.
 ### 2026-08-06 — CUTOVER SHIPPED (operator authorized directly + to Brokkr)
 - **Operator authorized** the fleet repoint (to me: "go a/3"; to Brokkr directly: "go ahead
  with the cutover") and explicitly cleared the litellm restart blip ("authorized to blip litellm").
 - **BOUNDARY: 2026-08-06T17:37:48Z.** Gateway `reranker` alias now resolves 100% to A3
  (`BAAI/bge-reranker-v2-m3` @ :8013). Verified through gateway: "Hobgoblin Pus" relevant
  doc top @ 0.9989 (BGE signature; incumbent was ~0.33).
 - **Mechanism:** `reranker` was config-defined (not DB), config mounted `:ro`, no hot-reload →
  edited `/opt/docker/conf/litellm/config.yaml` reranker block (block-scoped script, asserted
  1+1 change) + `docker restart litellm`. **Blip was ~52s** (litellm reloads all 28 models on
  boot), not the ~15s estimated — reported honestly to operator + Brokkr + worldtree-dev.
 - **`qwen3-reranker` alias LEFT UNTOUCHED** → incumbent still served at :8002 (rollback path;
  also avoids a false alias — the Qwen name still names the Qwen model).
 - **Canonical synced:** `stacks/litellm/conf/config.yaml` reranker block updated to match live.
  (Live config had pre-existing drift from canonical — only the reranker block was reconciled.)
 **ROLLBACK (one-liner, ~1 min):** revert the `reranker` block in
 `/opt/docker/conf/litellm/config.yaml` to `model: hosted_vllm/Qwen/Qwen3-Reranker-0.6B` +
 `api_base: …:8002/v1` (backup at `config.yaml.bak-pre-rerank-cutover-*`), then
 `sudo docker restart litellm`. Incumbent backend (`vllm-rerank` :8002) is up and untouched.
 ### OPEN / cleanup owed at process end (operator reverses/approves)
 - **A5** never staged (Brokkr cancelled) — nothing to clean.
 - **A2 (`vllm-rerank-a2` :8012)** + alias `reranker-a2-qwen3-seqcls` — bench-only, tear down when
  Brokkr signals the bake-off is closed (`docker stop/rm` + `/model/delete`).
 - **A4 (`vllm-rerank-a4` :8014)** + alias — KEEP for now (Brokkr's documented throughput fallback).
 - **A3 (`vllm-rerank-a3` :8013)** — now PRODUCTION (backs the `reranker` alias). Hardened
  2026-08-06: `docker update --restart unless-stopped` (survives fv-ml1 reboot, no recreate).
  A4 given the same. **Remaining follow-up (not urgent): promote A3 from throwaway `docker run`
  to a canonical compose service** (`stacks/vllm/`) for config-managed consistency — a recreate,
  so do it in a window since it briefly drops `reranker`.
 - **Incumbent (`vllm-rerank` :8002)** — keep up as rollback until Brokkr/worldtree close the
  post-cutover watch; retire (not delete) only on explicit sign-off.
 - **`rerank_hybrid_floor`** — Brokkr routing to worldtree-dev directly (drop, don't re-tune).
 ### 2026-08-06 — VERIFIED + A2 torn down + throughput characterized
 - **Brokkr independent verify: CUTOVER VERIFIED** — prod `reranker` == `reranker-a3-bge-v2-m3`
  at maxdiff 0.000000 (5 samples, spread 0.000009), single backend, no split routing.
 - **R42 v13 acceptance gate PASSES** — anchors_flip 4/4, no_regression 8/8, no_distractor_rise
  TRUE, zero aborts. **First PASS in R42 history after 4 failed verdicts.** Production main+kb:
  56/90 → 90/90 top-10; evictions 33 → 0.
 - **A2 torn down** (Brokkr signalled done): gateway alias `reranker-a2-qwen3-seqcls` deleted
  (/model/delete 200) + container removed. ~3.5 GB freed on GPU1. Remaining: `vllm-rerank`
  (incumbent, rollback), `vllm-rerank-a3` (prod), `vllm-rerank-a4` (fallback).
 - **Throughput characterized (the one open risk):** A3 caps ~34 req/s — flat from 8→16
  concurrent while latency climbs gracefully (p50 214→332→456 ms; p99 525 ms @16-conc). It
  QUEUES, doesn't cliff. ~40% below the incumbent's ~55 req/s. Likely fine for fleet rerank
  QPS (internal, per-search), but if real p99/queue-depth bites: levers are (a) swap to A4
  (~51 req/s, but English-only), (b) raise A3 `--gpu-memory-utilization` for bigger batching
  (recreate = brief blip), (c) run a 2nd A3 replica load-balanced behind `reranker` (~2× tput,
  identical replicas so no split-measurement issue now the bake-off is closed). Brokkr will
  re-run the grid against A4 if it bites — no intuition swaps.
 ## 2026-09-24 — second site + a dead alias repaired (infra-ops)
 - **`reranker-a3-bge-v2-m3` was dead from the fv-ml1 relocation (2026-09-12) until
  2026-09-24.** It is a DB-only alias (added via /model/new, so not in `config.yaml`) and
  still named ana-ml2's old address `10.250.50.54:8013`; every call returned 500 after ~23 s.
  Spend logs show **no callers in the 7-day retention window** besides the probe that found
  it. Repointed with `PATCH /model/{id}/update` to `10.251.50.54:8013` (3/3 × 200). Lesson:
  a DB-only alias is invisible to a grep of the config, so an IP migration misses it —
  sweep `/model/info` too.
 - **esh-ml1 added as an order-2 failover behind `reranker`** (and `qwen3-embedding`) — same
  model and vLLM version on an RTX 2000E Ada at ESH. This is lever (c) above in failover
  form, not load-balancing: fv-ml1 serves every request while it is up. Parity: |score|
  FV-vs-ESH max 0.000145 vs FV-vs-FV floor 0.000181, identical ranking over 11 documents.
  See `servers/esh-ml1/README.md`.
 ## 2026-09-25 — the reranker moved to TEI on esh-ml1 (Prime)
 - TEI adopted as the fleet embed/rerank engine after a bake-off
  (`docs/pfi/embed-rerank-tei-vs-vllm-bakeoff.md`). `vllm-rerank-a3` on fv-ml1 was
  **retired**; `reranker` and the DB alias `reranker-a3-bge-v2-m3` now both route to
  `tei-rerank` on esh-ml1 :8013 via LiteLLM's `huggingface/` provider. Same model
  (bge-reranker-v2-m3): top-1/top-3 identical to the retired seat, and scores move
  up to 0.019 in the near-zero tail.
@@ -11,6 +11,170 @@ human-to-agent communication channel for AIPA. The stack consists of:
 Federation is disabled (internal-only deployment). Registration is disabled (admin-created accounts only).
 > ⚠ **The build steps below date from the AIPA era and describe how this was
 > originally stood up.** They are kept for provenance. For what is actually
 > running, read *Current state* immediately below — the versions, the appservice
 > owner, and the bridge have all moved since.
 ---
 ## Current state — 2026-09-01
 | | |
 |---|---|
 | Synapse | **v1.159.0** (upgraded from v1.120.0, 2026-09-01 — 39 releases, 21 months) |
 | Element Web | **v1.12.27** (from v1.11.80) |
 | PostgreSQL | **16-alpine** — deliberately NOT upgraded alongside Synapse |
 | Appservice | registration `id: aipa-bridge`, but the service behind it is **`wt-matrix-bridge`**, the Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. The `aipa` name is legacy. |
 | Accounts | 15 — 13 appservice-owned, plus `@vhoang` (admin) and `@miranda` |
 | Encryption | **zero encrypted rooms**, server-wide |
 | Federation | `federation_domain_whitelist: []` — present but EMPTY, so federation is effectively OFF. The federation endpoint still answers, which is misleading. |
 **Ops ownership:** worldtree-dev writes the bridge code; **infra-ops operates this
 instance** and has full authority to change it (operator, 2026-09-01). Do not route
 provisioning or config requests to worldtree-dev.
 ### Upgrade procedure (learned 2026-09-01)
 **Synapse schema migrations are one-way.** v1.120 cannot start against a v1.159
 database. Rollback is restore-from-dump, not revert-the-tag. Therefore:
 ```bash
 D=/opt/docker/backups/synapse-preupgrade-$(date -u +%Y%m%dT%H%M%SZ)
 docker exec synapse-db pg_dump -U synapse -d synapse --format=custom --compress=6 > $D/synapse.dump
 docker exec -i synapse-db pg_restore --list < $D/synapse.dump | wc -l   # verify it reads back
 ```
 Snapshot `compose.yaml`, `homeserver.yaml`, `aipa_appservice.yaml` and
 `element-config.json` alongside it. Then bump tags and `up -d`; watch for
 `Schema now up to date` in the logs.
 A benign startup warning — `Failed to listen on 0.0.0.0 ... Address already in use`
 — is normal Linux dual-stack behaviour after it binds `[::]:8008`. Not a fault.
 ### Appservice namespace — why `exclusive` is false
 The `aipa-bridge` registration claims a user namespace of
 `@[a-z][a-z0-9_-]*:matrix.phasefinal.com` — effectively every localpart on the
 server. It was `exclusive: true` until 2026-09-01.
 **`exclusive` governs who ELSE may act, not what the appservice may do.** On a
 homeserver with registration disabled, one admin, and no competing actor, it bought
 anti-squatting protection against a threat that cannot occur, while locking out
 every other means of account creation — admin shared-secret registration returned
 `M_EXCLUSIVE` for any conventional name, with no explanation of why.
 Now `exclusive: false`, regex unchanged. The bridge is unaffected: it creates and
 puppets exactly as before. The narrow **aliases** namespace (`#aipa-debug-*`) is
 left exclusive — it is specific and costs nothing.
 ⚠ **Do not narrow the users regex to a prefix.** All 13 appservice accounts fall
 inside it; narrowing would orphan them. ⚠ **Do not rename the `id`** — Synapse keys
 account ownership on `aipa-bridge` in the `users` table. The FILE may be renamed.
 ### Shared-secret registration — the HMAC has no trailing null
 `POST /_synapse/admin/v1/register`. The MAC is
 `HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**,
 it does not **terminate**. A trailing `\x00` after `notadmin` yields `HMAC incorrect`.
 Run it inside the container against `localhost:8008`; port 8008 is not published to
 the host.
 ---
 ## Conventions
 ### User IDs
 `@<agent>:matrix.phasefinal.com` — one account per agent or persona.
 ### Room aliases — ratified by the operator 2026-09-01
 ```
 #<agent>-<purpose>:matrix.phasefinal.com
 ```
 Proposed by ledger-dev, ratified by the operator. Mirrors the user-ID convention
 rather than introducing a second, independent axis.
 - **Ownership is legible from the alias alone** — `#miranda-summons` is
  unambiguously Miranda's without opening the room.
 - **Namespaced on the AGENT, not a product**, so it survives a consuming project's
  rename.
 - **`#aipa-debug-*` is avoided by construction** — no agent is named `aipa-debug`,
  and that range is the bridge's exclusive alias namespace.
 - **Load-bearing rationale, not tidiness:** purpose-as-tier is what makes a summons
  triageable **so the room IDENTITY carries the tier**. (Stated carefully: it is
  *not* that the push payload carries the room name — see the pusher note below.)
 **Known cost, accepted deliberately:** a genuinely shared room with no single owner
 has no natural slot. Preferred over pre-building a namespace for a room that may
 never exist; name that case deliberately if it arrives.
 **Pre-existing rooms are not renamed.** "The High Seat"
 (`!NiVVoMsyoHCBRPrrrn`) predates the convention, has no canonical alias, and works.
 Renaming a live proven channel to satisfy a scheme it did not exist under is the
 churn the convention exists to prevent. New rooms follow it.
 ---
 ## Push notifications — what actually reaches a phone
 Established 2026-09-01 while proving Ledger's summons channel.
 The registered pusher (`@vhoang`, Element X iOS) uses
 **`"format": "event_id_only"`** via `https://matrix.org/_matrix/push/v1/notify`.
 That payload carries `event_id`, `room_id` and counts — **no room name, no sender,
 no content.**
 It still produces a useful notification because `mutable-content: 1` means Element X
 runs a **Notification Service Extension**: iOS wakes it with the near-empty payload,
 and the extension then **fetches the event from the homeserver and renders the
 notification on the device.**
 Three consequences:
 1. **The tier-in-room-identity scheme works, but via the client fetch** — not
   because the push carried the name. `m.room.name` must still be set at creation.
 2. **`push: include_content: false` is irrelevant for clients with an NSE.** The
   extension reads the event directly. It *does* bite clients without one.
 3. **Server-invisible failure mode:** if the phone cannot reach
   `matrix.phasefinal.com` at wake time, the fetch fails and iOS shows the fallback
   — the bare word "Notification". **Synapse records `last_success` and sees a
   delivered push.** The degradation happens entirely on the device.
 Because the payload carries nothing meaningful, **self-hosted sygnal was considered
 and rejected** — it would remove matrix.org from a path that already carries no
 content, room name or sender.
 ⚠ Sygnal is a **relay** to FCM/APNs, not a replacement for them. Self-hosting it
 removes matrix.org and nothing else. "Notifications must not leave the fleet" is not
 satisfiable with mobile push at any price.
 ---
 ## Sign in with QR code — requires Matrix Authentication Service
 Asked 2026-09-01; **not enabled.** From Synapse 1.159.0's own config validator:
 ```python
 if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) \
        and not auth_delegated:
    raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled")
 ```
 QR sign-in is **MSC4108**, an OIDC flow that hard-requires **MAS**. Synapse refuses
 to start with the flag set and no MAS.
 **MSC4388** (`msc4388_mode: off|open|authenticated`) enables independently, but it is
 only the **rendezvous channel** — the transport, not a login flow.
 Deferred because MAS is a service, a database, and a migration of every account off
 built-in auth — and the v1.139.0 upgrade note warns that `/register` from **old
 application service implementations may break under MAS**, which is precisely the
 bridge owning 13 of the 15 accounts. Revisit when MAS becomes mandatory rather than
 optional, in its own change window, with the bridge regression-tested either side.
 ---
 ## Architecture
@@ -0,0 +1,377 @@
 # althing deploy — six surfaces
 **`scripts/deploy-althing.sh`** does all four. `--check` reports drift without
 touching anything. This page is the why.
 Deploying althing means updating **four independent surfaces on nh3-dev**. Three
 were known; the fourth had no step in any runbook and drifted for five days
 before anyone noticed, on 2026-09-01.
 | # | surface | what it is | how it drifts |
 |---|---|---|---|
 | 1 | **binaries** | `uv tool install --force .` — the 7 console scripts | silently, see below |
 | 2 | **herald** | `systemctl --user restart althing-po-herald` | new guards live here; nothing takes effect until it restarts |
 | 3 | **skill** | `scripts/sync_skill.sh` → `~/.agents/skills/althing/SKILL.md` | covered by its own `--check` |
 | 4 | **plugin** | repo `plugin/` → marketplace dir → Claude Code cache | **two hops, neither was automated** |
 | 5 | **per-seat route** | `althing-route declare` on each seat | a channel change needs a re-declare; the plugin's SessionStart hook does it for CC seats |
 | 6 | **`~/.claude/settings.json`** | `crossSessionInbound: "accept"` | **outside the althing repo entirely — no version can carry it** |
 ## ⚠ `uv tool install .` without `--force` is a silent no-op
    $ uv tool install .
    `althing-core @ file:///home/lkraven/development/althing` is already installed
    $ echo $?
    0
 uv matches on the source **spec**, not its **contents**. On a box that already
 installed from that path it declines and reports success. An operator following
 a runbook literally would restart the herald, see everything green, and wonder
 why the new binary was missing — with every command exiting 0.
 ## ⚠ Surface 4 is the one that bites, and it ate a hook
 The chain is:
    repo plugin/  →  ~/.local/share/althing-plugin/  →  ~/.claude/plugins/cache/althing/althing/<ver>/
                  ^^^ nothing synced this hop
 On 2026-09-01 the marketplace directory was a frozen copy from **Aug 28**:
    deployed 0.0.1   hooks: ['UserPromptSubmit']
    repo     0.1.1   hooks: ['UserPromptSubmit', 'SessionStart', 'SessionEnd']
                     + scripts/pane-route.sh   (absent from the deployment entirely)
 So **"CC seats re-declare their pane route automatically at next SessionStart"
 was never true on this box.** The hook existed and worked upstream; it was never
 deployed. That is why every seat — including infra-ops' own Claude Code seat —
 had to be hand-declared with a pid someone measured by hand, and why the
 `idle_cursor` pin from 3.2.1 would not have self-applied either.
 ### ⚠ Compare the HOOK LIST, not the version string
 A version number cannot tell you what a stale plugin actually cost you. 0.0.1 and
 0.1.1 differ by **two hooks and a script**:
 ```bash
 for d in ~/.claude/plugins/cache/althing/althing/*/; do
  printf "  %-8s hooks: %s  pane-route.sh: %s\n" "$(basename "$d")" \
    "$(python3 -c "import json;print(list(json.load(open('$d/.claude-plugin/plugin.json'))['hooks'].keys()))")" \
    "$(test -f "$d/scripts/pane-route.sh" && echo yes || echo NO)"
 done
 ```
 Expect `['UserPromptSubmit', 'SessionStart', 'SessionEnd']` and `pane-route.sh: yes`.
 Anything less and pane routes are not being declared or released automatically.
 ### ⚠ WHY THIS HID FOR FIVE DAYS — the shape to recognise
 **A missing deploy surface does not present as an error. It presents as
 "the migration needs manual work"** — and we had a ready explanation for that,
 because four of five seats were non-Claude and genuinely did need hand-holding.
 The seat that falsified the story was **infra-ops' own**: a Claude Code seat that
 should have self-declared and didn't, and it looked exactly like the other four.
 Neither infra-ops nor forseti asked why the automatic path had not fired for the
 one seat it was supposed to work on.
 ⚠ **When a migration needs manual intervention, check whether the automatic path
 was ever deployed before concluding it does not apply to your case.**
 **Use the supported CLI for the second hop.** `claude plugin update althing`
 (also `install` / `uninstall` / `list` / `details` / `validate` / `marketplace`).
 ⚠ Do **not** hand-edit `~/.claude/plugins/installed_plugins.json` or fabricate a
 cache directory — that is Claude Code's own bookkeeping, and a subtle mistake
 there breaks the plugin in a way that looks like an upstream bug.
 ⚠ **A Claude Code restart is required** for new plugin hooks. They are read at
 session start; a running session keeps the old ones.
 ## Verifying a seat is actually reachable
 `postbox status` reports `mode: push`, but until 3.2.4 a **failed** push
 declaration was silent — `_declare_push` devnulled both streams with
 `check=False`, so a missing binary, a non-zero exit and a server-side rejection
 rendered identically as nothing. A seat could have been silently pull-only since
 3.1.2.
 Audit for "looks armed but is not", cross-referencing local waiter locks against
 what the post office believes:
 ```bash
 for f in ~/.althing/wake-listener-*.lock; do
  h=$(basename "$f" .lock); h=${h#wake-listener-}
  pid=$(cat "$f" 2>/dev/null)
  [ -n "$pid" ] && [ -r "/proc/$pid/cmdline" ] || continue
  # IDENTITY, not liveness: a lock left by a reaped listener names a pid the
  # kernel is free to hand to anything, so `kill -0` alone reports a stranger
  # as a live waiter. Match the same cmdline segment `althing-listen --stop`
  # requires before it will signal anything.
  ours=0
  while IFS= read -r -d '' seg; do [ "$seg" = "--_route=$h" ] && ours=1; done \
    < "/proc/$pid/cmdline"
  [ "$ours" = 1 ] || { printf "  %-24s pid %-8s STRANGER (recycled pid)\n" "$h" "$pid"; continue; }
  printf "  %-24s waiter %-8s mode: %s\n" "$h" "$pid" \
    "$(postbox status --handle "$h" 2>/dev/null | grep -oP 'mode: \K\w+')"
 done
 ```
 A live waiter reporting `mode: pull` is a seat that will never be poked. From
 3.2.4 onward `$ALTHING_ROOT/listen.log` records failed declarations directly.
 ⚠ **Liveness is not identity, and this loop is the place that gets it wrong.**
 The earlier `kill -0` form would print a phantom waiter for any handle whose
 stale lock happens to name a recycled pid — and a phantom waiter is exactly what
 sends a false "you are unreachable" notice to a seat that is fine.
 `session_listener.sh` refuses to SIGTERM on liveness alone for precisely this
 reason; an audit that only *reads* has no excuse for a weaker standard than the
 one that *kills*. (Adopted 2026-09-02 after the regin-smithy-dev
 cross-reference, where the pid happened to be genuine and the weaker check
 happened to be right.)
 ⚠ **A stale `wake-listener-*.lock` is NOT a fault.** The gate is `flock -n` on an
 open fd, which the kernel releases when the holder dies, so a lock file left by a
 reaped listener is inert and exit 3 only fires against a genuinely live holder.
 The pid in the file is read by `--stop` alone. (Recorded because infra-ops
 claimed the opposite, untested, on 2026-09-01; forseti measured it.)
 ## ⚠ The plugin cache does not take a content-only change
 `claude plugin update althing` matches on the version in `plugin.json` and
 declines when it is unchanged. A release that edits hook or script CONTENT
 without bumping the plugin version therefore leaves the Claude Code cache
 stale while **every version check reports success** — the marketplace dir gets
 the new bytes, the cache does not, and `update` says "already at the latest
 version".
 Measured on the 3.3.0 deploy (2026-09-02): marketplace and live cache both read
 `0.1.1`, and `pane-route.sh` + `README.md` differed. That delta was
 documentation-only and harmless. The next one need not be.
 `scripts/deploy-althing.sh` now diffs the marketplace tree against the live
 cache dir on every run and on `--check`, so the drift is reported rather than
 inferred. **The fix is a plugin version bump upstream, not a hand-edit** —
 `~/.claude/plugins/installed_plugins.json` and the cache directory are Claude
 Code's own bookkeeping.
 The 2026-09-01 lesson was *compare the hook list, not the version string*. This
 is the same lesson one turn deeper: here the hook list was identical too, and
 only the file contents moved.
 ## A seat already running when the plugin lands never auto-declares — leave it
 The SessionStart hook fires at session start and nowhere else, so any session
 that was already running when a plugin deploy landed keeps its old channel
 until it restarts. After the 3.3.1 deploy that was **eight of seventeen live
 seats**, all still on `fifo`.
 **Operator ruling 2026-09-02: leave them.** They are reachable now, the post
 office holds mail regardless, and each self-corrects at its next session start.
 The residual exposure is that a fifo waiter is what Claude Code 2.1.257 reaps —
 one seat was reaped three times in a morning — and a reap converts a healthy
 seat into a silent one until the next restart. That costs timeliness, never
 mail.
 ⚠ **Do not mass-message the affected seats.** Standing operator directive: no
 unsolicited fleet notification, ever, unless asked. Direct correspondence with
 a counterparty about work in hand is fine; sweeping the handle list is not.
 Audit it, report the count, and let him decide — as he did here.
 The one-liner, for a seat that asks or an operator who wants one moved:
    althing-route declare --discover-pid
 ## ⚠ The `cc` channel is an undocumented interface, taken deliberately
 From 3.3.0 a Claude Code seat is poked over its own message socket
 (`$XDG_RUNTIME_DIR/cc-socks/<pid>.sock`) rather than by typing into its pane.
 No process to reap, nothing near the input line, and delivery lands at the
 receiver's next turn boundary.
 It is not a published interface and **is expected to break on some future
 Claude Code release**. Accepted on the operator's reasoning: the FIFO poker it
 replaces was also an unsanctioned hack of the background-watcher system, so
 this is a better instance of a dependency class we already had, not a new one.
 When it breaks the failure mode is a seat going **pull-only with a logged
 reason** — not lost mail; the post office holds it either way. Recovery is one
 command:
    althing-route declare --pid <pid>     # prefers cc, falls back to pane
 **Why cc is worth an undocumented dependency, in one measurement.** A pane poke
 is typed into the session and submitted, so Claude Code records it in the
 operator's prompt history exactly as if he had written it — 8 herald-written
 entries accumulated across three projects, permanently in up-arrow recall. A cc
 poke enters none: 2 delivered, 0 entries. Verified from the herald log against
 the history on 2026-09-02. Channel sequence on one seat that day, all three in
 order:
    09-01 21:44  poked via fifo            the old way, later reaped
    09-02 08:34  could NOT reach via pane  guard 4 refusing: he was typing
    09-02 09:07  poked via pane            typed into his input line
    09-02 09:14  poked via cc              arrived as a peer message
    09-02 09:31  poked via cc              after crossSessionInbound accept
 ⚠ **Count herald pokes by the whole display string, not by grepping
 `ALTHING-HERALD`.** The substring matches the operator's own messages when he
 pastes a hold notice into one, so the count runs high and the excess reads as
 new activity. See the identity-not-resemblance note in the audit section.
 ⚠ **Ordering constraint at any upgrade that changes the channel set:** a
 pre-3.3.0 herald refuses `channel=cc` at parse, so a seat that declares the new
 channel before the herald restarts goes silently pull-only. `uv tool install
 --force` then the herald restart, and only then let anything re-declare.
 `deploy-althing.sh` already runs them in that order and updates the plugin
 (whose SessionStart hook declares the route) last, which is what keeps the
 window to a couple of seconds.
 ## ⚠ Surface 6 — `crossSessionInbound`, and it is not in the althing repo
 **A cc poke to a default-configured seat is HELD, not delivered.** Claude Code
 auto-delivers an inbound cross-session message only when the sender's
 permission-mode class matches the receiver's, and **a sender that asserts no
 class is held**. The herald is a daemon and asserts none, deliberately. What
 the operator sees instead of a delivery:
    Held peer message — from an unidentified session [verified pid <herald>];
    preview: «ALTHING-HERALD althing: you have mail ...» — not delivered to
    Claude (1 held). The sender did not attest its permission mode and this
    session bypasses prompts.
 Claude Code verified the herald through `SO_PEERCRED` and then correctly
 declined to let it speak, because it would not say what it was.
 **The fix is one key in `~/.claude/settings.json`:**
    "crossSessionInbound": "accept"
 Set on nh3-dev 2026-09-02. The operator's reasoning is the part to keep: *the
 herald can only reach local seats, so bypass is the correct authorization type;
 when the guard was not there, that was our default posture, including the pane
 poke.* A pane poke types into a session and presses Enter — bypass-level access
 by any measure, and what we had been doing all along. **The socket channel is
 strictly narrower than what it replaces**, so accepting here states the existing
 trust boundary rather than widening it.
 ⚠ **There is no attestation the herald could send instead.** Four probes
 established that Claude Code identifies a sender by verified pid against the
 session registry and reads that session's *live runtime* permission mode; a
 daemon has none and the registry has no field for one. A `from_mode` field on a
 `type:"user"` frame is not consulted at all — it belongs to the control actions.
 Adding it would be shipping a field nothing reads.
 **Cost without the setting is smaller than it looks:** the correspondent record
 is in-memory session state, so the hold is **first-contact, not per-message** —
 one approval per seat per session lifetime, and a long-lived pane pays it once.
 ⚠ **Why this surface is the dangerous one.** A seat without it is declared,
 reachable and green, and its pokes go to a human instead of to the session.
 That is the same shape as the SessionStart hook that was never deployed:
 everything reports success, nothing arrives. `deploy-althing.sh` reads the key
 and reports it on every run and on `--check` — and **deliberately never sets
 it.** That file is the operator's permission configuration; a deploy script
 that edits it is a deploy script granting itself trust.
 ## The status-line segment belongs to althing, not to this repo
 From althing **3.4.0** the segment is a console script, `althing-statusline`,
 installed on PATH by the same `uv tool install --force` as the other seven
 executables. `scripts/claude-statusline-command.sh` calls it and does not
 reimplement it:
    althing=$(printf '%s' "$input" | timeout 5 althing-statusline 2>/dev/null)
 ⚠ **Keep that outer timeout LOOSER than the program's own 2 s budget.** If the
 outer one fires first the segment comes back empty, which renders as "not an
 althing directory" — the outage conflation, reintroduced by the guard meant to
 prevent a hang.
 **Why a console script rather than the `sync_skill.sh` canonical-file pattern:**
 a script, unlike a document, has somewhere to be *installed*. Installing it makes
 drift impossible instead of merely visible.
 **Why it needed solving at all.** On 2026-09-02 the same twelve lines existed in
 three places — althing's `plugin/scripts/statusline.sh`, the operator's wired
 `~/.claude/statusline-command.sh`, and this repo's copy — and one audit found a
 **different defect in each**. All three were fixed separately, by hand, the same
 day. Two of the defects were in this repo's copy and are worth knowing because
 they are both silent:
 - the handle was resolved as the most recent launch in the directory, so a
  directory hosting both a claude and a codex seat (`~/development/Worldtree`)
  reports the **codex** handle's unread count and reachability to a Claude
  session the moment the codex pane relaunches last;
 - the post-office address was hardcoded, so it survives exactly until the post
  office moves and then reports an outage that is really a stale constant.
 ### Deploying it
 `scripts/claude-statusline-command.sh` is the tracked copy of what Claude Code
 actually runs at `~/.claude/statusline-command.sh` (`settings.json` →
 `statusLine.command`). Copies, not symlinks — same rule as `stacks/`.
    diff -u scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh
    cp scripts/claude-statusline-command.sh ~/.claude/statusline-command.sh
 ⚠ **Diff before you copy, in both directions.** On 2026-09-02 the live path was
 edited directly and was *ahead* of the tracked copy; a reflexive `cp` from the
 repo would have destroyed the better version. The tracked copy is intent, the
 live path is reality, and reality is sometimes right.
 Smoke test — **strip the althing env vars, or you will test your own session's
 identity instead of the resolution logic.** A hand-run inherits the exported
 `ALTHING_HANDLE` from the session you run it in, so the seat's own directory and
 a directory with no althing identity render identically and both look right.
 That false pass caught two agents within one hour on 2026-09-02.
 ⚠ **Open question — whether Claude Code's own invocation inherits the
 environment is NOT established, and nothing here depends on it.** An earlier
 version of this page asserted a "bare shell"; that claim was never verified and
 the evidence points the other way. Measured 2026-09-02: this box's Claude Code
 process carries `ALTHING_HANDLE` and `ALTHING_POST_OFFICE` in its own `environ`
 (inherited from the `dev-launch` shell), and a spawned child would normally
 inherit both — observed independently on two seats. Whether Claude Code scrubs
 the environment before spawning the status-line command was not observed either
 way. **Write code that does not depend on the environment being present**, which
 `althing-statusline` already does: environment first, launch history second, and
 a seat started outside `dev-launch` has no handle in either place. Test with the
 variables stripped regardless, because that is the harder case:
    probe() {  # $1 = cwd, $2 = what you expect
      got=$(printf '{"workspace":{"current_dir":"%s"}}' "$1" \
            | env -u ALTHING_HANDLE -u ALTHING_POST_OFFICE \
              bash scripts/claude-statusline-command.sh)
      printf 'got [%s]  wanted [%s]\n' "$got" "$2"
    }
    probe /tmp                                    ''      # no althing identity
    probe /home/lkraven/development/eshpfi-management '🔔 …'
 ⚠ **Print what you got beside what you wanted, on the same line.** Expected
 values in trailing comments are what produced the false pass that caught two
 agents inside an hour on 2026-09-02: `/tmp` rendered `🔔` and both of us read it
 as "fine" rather than "that directory has no identity and should be empty." A
 plausible value in the right shape is the hardest kind of wrong to see, and the
 only thing that reliably catches it is the wanted value sitting on the same line
 as the got value.
 ### ⚠ The outage probe: a refused port does NOT test the timeout
 `📵` has two causes and they take different paths. Measured 2026-09-02 on
 nh3-dev:
 | address | elapsed | what it proves |
 |---|---|---|
 | `http://10.100.50.40:9999` — refused port on a **live** host | **0.079 s** | the glyph, and nothing about the deadline |
 | `http://10.100.50.199:8390` — **unrouted** host, black-holes | **2.065 s** | the 2 s budget actually fires |
 A refused port answers instantly with an RST, so the fast probe returns `📵`
 without ever approaching the timeout. Use **both**, and know which is which —
 letting the fast case imply the slow one is how a status line that hangs the
 prompt ships with a green test beside it.
 ## Rollback
    uv tool install althing-core==3.1.2
 Routes written by later versions stay parseable — the old reader ignores unknown
 keys — so nothing is stranded.
@@ -82,6 +82,33 @@ auth, append-only, private repos). The split is by site:
 ---
 ## ⚠ Answering "is X backed up?" — ask the repo, never the job
 **Coverage is a property of the backup SYSTEM, not of any one job's configured
 scope.** Establish it by querying the repository for the path and seeing it in a
 real snapshot. Never by reading a job's `SRC=`.
 ```sh
 # as infra-ops on the host, against its own profile
 sudo -n bash -c 'set -a; . /etc/restic/restic.env; set +a
  export RESTIC_PASSWORD_FILE=/etc/restic/password
  id=$(restic snapshots --latest 1 --json | jq -r ".[0].short_id")
  restic ls "$id" /path/in/question | head'
 ```
 **The failure this rule exists to prevent (2026-09-22).** A peer agent checked
 `dev-backup.sh`, found `SRC="$HOME/development/"`, and reported — with
 specifics, unhedged, to the operator — that five home-directory paths including
 Miranda's entire conversation had *"never been backed up anywhere"*. All five
 were in that night's restic snapshot. `dev-backup` is the **hourly** job for one
 directory; **resticprofile** is the daily job covering all of `/home/lkraven`,
 and it is what actually protects home. Checking one job and generalising to the
 system produced a confident, false, escalated claim.
 The true finding underneath was much smaller and is a **cadence** question, not
 a coverage one: live conversation state has a 24-hour RPO because it rides the
 daily job rather than the hourly one.
 ## The 2-minute freshness check
 Run these any time you need to answer "are we backed up?"
@@ -150,6 +177,77 @@ the stop→umount→rm-ghost→remount→start variant.
 ## Known gaps / TODO
 ### ✅ restic content assertion (2026-09-22, CLOSED)
 ⚠ **I reported this gap wrongly first.** I ran `grep -ic restic` against
 `backup-freshness-alert.sh` — the **wrapper** — got 0, and told the operator
 nothing on the fleet verified restic. The actual checking lives in
 `check-backup-freshness.sh`, which had always checked restic snapshot ages for
 seven hosts. Grepping one file of a two-file tool and generalising is the same
 error a peer had just made with `dev-backup`, made while correcting them.
 **The real gap was narrower and still real:** the host list read
 `irv-ml1 nh3-docker` and **omitted nh3-dev** — the repo holding every Claude
 Code transcript, althing routes, hermes history and Miranda's conversation.
 `/volume1/Backup/restic/nh3-dev/` had always existed and always been written;
 it was simply never enumerated, so a total failure of the one repo nobody could
 reconstruct would have gone unreported indefinitely.
 Both are fixed. nh3-dev is in the list, and a **content assertion** now runs per
 the operator ruling of 2026-09-22 (assert snapshot content, not REST-server
 reachability):
 - **Conjunctive**: latest snapshot newer than `MAX_AGE_H` **AND** the probe path
  present **AND** a restore returns non-zero bytes. Age alone was the old
  check's problem; content alone is the same problem rotated.
 - **It restores.** Listing proves the *index*; only a restore proves the bytes
  come back, and metadata is exactly what survives a pruned or partially-written
  repo. ~493 KB from `~/.local/state/svos`, sub-second. The large paths are not
  restored: a repo that returns one file will return others, one that cannot is
  broken for everything.
 - **Identity**: the timer runs as `lkraven` (no NOPASSWD sudo here), so the probe
  hops through `infra-ops@localhost`, which has it. The credentials under
  `/etc/restic` are deliberately root-only.
 ### Repos are DISCOVERED, not enumerated
 The host list is no longer hand-written. `check-backup-freshness.sh` asks each
 NAS which repositories exist — a directory counts as one when it has a
 `snapshots/` child, which distinguishes a real repo from a container dir
 (`/volume1/Backup/restic/repo/` holds `ana|esh|nh3` namespaces and no snapshots
 of its own).
 **Why (svos-dev's framing, 2026-09-22):** adding nh3-dev fixed the *instance*;
 the *class* was that a hand-maintained list of things to watch, sitting beside a
 NAS that already knows which repos exist, means **the next repo added is
 unwatched by default and nothing says so.** The list of what to check can
 silently disagree with the set of what exists.
 The hand-written list survives **demoted**, as `EXPECTED_RESTIC_ANA` /
 `EXPECTED_RESTIC_NH3`, used only to report a repo that has *vanished*. That
 keeps two facts distinct that would otherwise both read as silence:
 | fact | now |
 |---|---|
 | a repo exists that nobody watches | **impossible** — it is discovered |
 | a repo we expected is gone | reported as `EXPECTED REPO NOT FOUND` |
 Currently 8 repos: 5 on ana-nas (`repo/ana/*`), 3 on nh3-nas. The `esh` and
 `nh3` namespaces under `repo/` on ana-nas are empty.
 ⚠ **`restic ls` always prints a header line**, matched or not — a path absent
 from the repo returns 1 line, a real one returned 6. Counting with `grep -c .`
 read the header as a hit and made the ABSENT case fall through to *"blobs
 gone"*, telling an operator the repository was corrupt when the truth was a
 mistyped path. Count `grep -c '^/'`.
 **Tested by making it fail**, because a check only ever seen passing is
 untested: a bogus probe path reports *absent*; `BACKUP_MAX_AGE_HOURS=1` reports
 *too old*; the healthy path reports the snapshot id, age, entry count and bytes
 restored.
 - [x] **Backup-freshness alerting — DONE (2026-06-20).**
      `scripts/check-backup-freshness.sh` (the 2-min check, exit 1 on
      stale/down) + a daily **systemd user timer on nh3-dev** at 08:00
@@ -37,7 +37,7 @@ pfi-pve during morning backup window. Backup suspected as trigger
 **Blast radius:**
 - ~~pfi-postgres (VM 105) — PGDATA on `/mnt/db`~~ — **migrated to local disk 2026-04-23**. vaultwarden, gitea, paperless-ng, zammad no longer cascade on ana-nas outage. Left in history for the recovery pre-migration.
- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, ana-ml2, esh-docker-vm, vm-esh-nas)
+- ana-docker rest-server-ana — repo data on `/mnt/backup` → all ana-side restic clients fail (ana-docker, fv-ml1, esh-docker-vm, vm-esh-nas)
 - PBS-ANA datastore — NFS-backed on `/mnt/backup/pbs-ana` → fleet vzdumps fail, PBS-NH3 sync fails
 - ana-docker NFS mounts for `/mnt/docker`, `/mnt/compose`, `/mnt/pve-VMStorage` if used → various stack misbehavior
@@ -207,12 +207,12 @@ Essentially **all Anaheim primary services** go offline. Because ana-nas lives h
 ## Tier 4 — Specialty workloads
-### ana-ml2 (bare metal Supermicro, 10.250.50.54, BMC 10.250.250.50)
+### fv-ml1 (bare metal Supermicro, 10.251.50.54, BMC 10.251.250.50)
 **Blast radius:** AI inference services (llama-swap, vllm). Consumer-facing chat/embedding/reward-scoring endpoints fail.
 **Recovery:**
-1. Check OS via SSH. If unresponsive, BMC console at <https://10.250.250.50>.
+1. Check OS via SSH. If unresponsive, BMC console at <https://10.251.250.50>.
 2. If hardware issue: BMC logs, power cycle via IPMI, check GPU health (`nvidia-smi`).
 3. Docker stacks auto-start via compose `restart: unless-stopped`.
@@ -0,0 +1,117 @@
 # ESH-Media 10G uplink — the DAC that negotiated 1G, and the fix
 2026-09-04. The link runs at 10G by an explicit rate override on the UDM side.
 **Read the pinned-port warning below before touching that port.**
 > ## CURRENT STATE: REVERTED to autoneg / 1G. The force is gone.
 >
 > **2026-09-04 14:38 — the operator reported the 10G link "flaking hard" and I
 > reverted it.** Both ends are back at 1000 Mbps, UDM SFP+2 is `autoneg True /
 > setting_preference auto`, ESH-Media is reachable and stable. The revert took
 > about 50 s of renegotiation and nothing rebooted.
 >
 > **The forced-10G experiment failed, and the failure was progressive.** Error
 > counts on the ESH-Media side over the life of the force:
 >
 >     10:03   200      link-up burst
 >     10:48   221      +21 in 42 min  -- the first sign it was not plateauing
 >     14:38   416      +195 over the next 4 h, plus user-visible flapping
 >
 > ⚠ **The lesson, and it is mine to own: I read a two-minute flat window as a
 > plateau and reported it as one.** The counter simply had not moved yet. A
 > marginal link declares itself over hours, not minutes — and the operator felt
 > the flapping before the soak I left running had accumulated enough to call it.
 >
 > **Do not re-force this port.** The cable is being replaced at the weekend; a
 > correctly-coded DAC negotiates 10G on its own, which is the actual fix.
 ## The diagnosis, and what made it certain
 The cable was linking *cleanly* at the wrong speed — **zero rx/tx errors, no
 LOS, no tx/rx fault, satisfaction 100**. That rules out a damaged or marginal
 cable, which links at 10G and throws CRCs rather than dropping a rate. Both ends
 were *choosing* 1G.
 The reason showed on the switch side:
    sfp_compliance:  Unknown
 The switch reads the cable's EEPROM but cannot parse its compliance codes.
 `SFP-H10GB-CU3M` is a Cisco part number and the vendor string is literally
 `OEM` — a third-party cable wearing Cisco coding. With no recognised 10G
 compliance byte, autonegotiation falls back to the safe rate.
 **The control case is what settled it.** On the *same UDM*, one port over:
    SFP+1 (eth9)   TP-Link TL-SM5220-1M    ->  10000
    SFP+2 (eth10)  OEM SFP-H10GB-CU3M      ->   1000
 Same `speed_caps`, autoneg on both. So the port hardware, the firmware and
 autoneg were all exonerated before anything was changed, and the cable was the
 only remaining variable.
 ## The fix
 An explicit rate override on the **UDM** end only:
    autoneg False   speed 10000   full_duplex True   setting_preference manual
 ESH-Media's end was left on autoneg and **followed to 10000 by itself** — which
 is the proof the cable was always electrically capable of 10G and only the
 EEPROM-driven negotiation was wrong.
 ## ⚠ Why the UDM end and not the switch end
 The switch end is the unrecoverable one. ESH-Media reaches the controller
 *through* this link, so a forced rate that fails to link leaves the switch with
 no uplink and therefore no way to receive the revert — a physical visit.
 The UDM end is safe because the path to the controller does not cross this link:
    nh3-dev -> 10.100.10.1 -> 10.0.0.1     two hops, site VPN, terminates on the UDM
 Verified with `ip route get` / `traceroute` **before** the change, and the exact
 pre-change `port_overrides` array was written to a revert payload before the
 forward one was sent. If a future attempt needs *both* ends forced, that is a
 different risk class and needs the operator's sign-off, not an agent's.
 ## The port is NOT pinned any more — the override was removed 2026-09-04
 While the force was in place, UDM SFP+2 was `autoneg False, speed 10000` and a
 non-10G module would not have linked there. That is no longer true: the override
 is gone and the port is back on autoneg. Recorded because the warning stood in
 this runbook for four hours and someone may remember it.
 ## What the link bounce cost
 ~60 seconds. ESH-Media, the AP and the office switch all returned `state=1` at
 `uplink_speed=10000` with their **uptimes unbroken** (the switch did not reboot,
 only the link renegotiated). The Zigbee coordinator and Home Assistant's path to
 it were re-verified afterwards.
 ## Errors: 200 at link-up, then flat
 The switch end logged ~200 rx / ~196 tx errors **during the renegotiation
 window** and none afterwards. Counters were flat across the following two
 minutes while real traffic flowed, and a longer soak was run to confirm.
 ⚠ **This is the number to watch, and it went the wrong way.** A two-hour soak
 showed the errors do **not** plateau: 200 → 221 rx and 196 → 199 tx over 42
 minutes, monotonic, about 1 rx error per 23.2 MB — plus one link drop at 10:12
 that recovered on its own with no device reboot.
 ⚠⚠ **So an earlier claim in this runbook's own history was wrong and is
 corrected here: a clean, zero-error link at 1G does NOT rule out a marginal
 cable.** It only proves the cable is clean *at 1G*. Clean at 1G and marginal at
 10G is exactly what a 3 m OEM twinax with unparseable compliance codes turns out
 to be — which means the platform's autoneg fallback was protecting something
 real, not being fussy about vendor coding. For comparison, the TP-Link
 TL-SM5220 on the adjacent UDM port carries 780 GB with 4 errors.
 The fix is the cable, not the configuration.
 ## Rollback
 Set UDM SFP+2 back to `autoneg True, setting_preference auto`, clearing `speed`
 and `full_duplex`. Remember `port_overrides` is a **whole-array PUT**: the UDM
 carries 2 overrides and anything omitted is deleted.
@@ -0,0 +1,434 @@
 # esh-pve-nas — moving PVE root off the USB DOM
 **Status: DONE — cut over 2026-08-18.** Root is `nvme/ROOT/pve-1` on the mirrored
 NVMe; `/boot` is ext4 on the DOM; the DOM is out of the runtime I/O path. All five
 guests healthy, all three pools ONLINE, `systemctl is-system-running` = `running`.
 The ext4 root (`pve-root`) is intact, unmounted, and still carries its own kernel
 and initrd as the rollback.
 Post-cutover boot config: `saved_entry=pve-zfs-root`, no `next_entry`. If grubenv
 were ever unreadable GRUB falls through to menu entry 0, which the
 `/etc/default/grub.d/zfs-root.cfg` drop-in also points at `root=ZFS=nvme/ROOT/pve-1`
 — so every path boots ZFS.
 ⚠ **The window cost an unplanned outage, caused by a bug in this runbook's own
 tooling, not by the migration.** Read § The mount-propagation incident before
 running anything like this again. Two other findings — the blast radius being
 more than double what was documented, and the one-shot rollback not actually
 working — are recorded in § The pool-name bug's neighbours below.
 Staging is two playbooks, both rerunnable:
 | phase | playbook | what it did |
 |---|---|---|
 | 1 | `playbooks/esh-pve-nas-stage-zfs-root.yaml` | carved the `/boot` LV out of swap, populated it, rsynced the root into `nvme/ROOT/pve-1`, wrote the copy's fstab |
 | 2 | `playbooks/esh-pve-nas-stage-bootloader.yaml` | ZFS initramfs, grub.cfg, rollback entry, grubenv — **without** `grub-install` |
 **Plan revised 2026-08-17** from "reinstall to a mirrored-NVMe ZFS root" to
 **"split the boot chain from the root filesystem"** — operator's proposal, and it
 is strictly better. The original reinstall plan is kept at the bottom as the
 fallback.
 ## Why
 PVE root lives on a **USB Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`,
 `ID_VENDOR=NORELSYS` — as a 6 GB ext4 root plus 768 MB swap and a 512 MB ESP.
 A DOM is SLC/pSLC with a real controller, so the 284 GB written since boot is
 unremarkable and **wear is not the driver**. The actual problems:
 1. **It is on the USB bus.** A bus reset or re-enumeration drops the *root
   filesystem* out from under a running hypervisor while its guests keep going.
 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
 4. **It has blocked patching for months.** This is the operator-visible symptom
   and the real urgency: `apt-get -s dist-upgrade` shows **225 packages pending,
   161 of them carrying `deb12uN` / Debian-Security bumps** — including `ssh
   1:9.2p1-2+deb12u10`. The host sits on `pve-manager/8.4.11` while its sibling
   esh-pve is on 8.4.14, and it has 20 weeks of uptime because it cannot take a
   kernel.
   ⚠ **Do not attempt the upgrade before the migration.** The pending set
   includes `proxmox-kernel-6.8.12-42-pve-signed` (from -13) — a signed kernel
   plus initramfs is ~250 MB, and **`/boot` is on root**, which has 1.3 GB free.
   225 packages unpacking (dpkg, perl, glibc-adjacent) into that headroom risks
   filling the disk mid-transaction and leaving a broken dpkg state on a
   hypervisor running five guests. Recovering a wedged dpkg on a full root is
   far worse than waiting for the reboot.
   If patching genuinely cannot wait, the escape hatch is to keep downloads off
   root — `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives dist-upgrade`
   — but the kernel still lands in `/boot` on root, so this reduces the risk
   rather than removing it. Migrating first is the shorter path to safety.
 ## The design: boot on the DOM, root on ZFS
 Boot and root do not have to live on the same device. Split them:
 | | device | contents | written when |
 |---|---|---|---|
 | **boot** | DOM `sdq` | ESP + `/boot` (ext4): GRUB, kernels, initramfs | **only on kernel/GRUB updates** |
 | **root** | `nvme` pool | `nvme/ROOT/pve-1` — everything else | constantly, on mirrored NVMe |
 GRUB reads the kernel and initrd from **ext4 on the DOM**, so GRUB never has to
 read ZFS — which matters, because the `nvme` pool has `encryption`,
 `large_dnode` and `zstd_compress` enabled and GRUB cannot read those. The
 initramfs then imports the pool and pivots to `root=ZFS=nvme/ROOT/pve-1`.
 ### Why this beats the reinstall
 - **The `nvme` pool is not destroyed.** The root dataset is created *inside* the
  existing pool. No guest migration, no `zpool export/import` of `ssd`/`tank`,
  no reinstall.
 - **Downtime is one reboot**, not half a day.
 - **Rollback is a GRUB menu entry.** The existing ext4 root stays on the DOM,
  untouched. If ZFS root fails to come up, pick the old entry and you are back in
  a minute. That is a far better rollback than "reinstall and restore."
 - **The #1 risk is actually retired.** Once booted, root is on NVMe — a USB bus
  reset mid-run no longer takes the running system down. The DOM becomes
  read-mostly.
 - **Free upside: boot environments.** `zfs snapshot nvme/ROOT/pve-1@pre-upgrade`
  before an apt run, roll back if it breaks.
 ### What it does NOT fix
 The DOM remains the **only boot path**. If it dies, the machine will not boot
 until the image is restored — though the ZFS root, with all config and guests,
 stays intact. Mitigation is a **cloned fallback image** (`dd` of `sdq`, ~7 GB,
 refreshed after kernel updates), kept off-box next to the config snapshot.
 ## Preconditions — all already satisfied
 Verified on the host 2026-08-17:
 - **UEFI** firmware, `grub-efi-amd64 2.06-13+pmx7` installed
 - **`zfs-initramfs 2.2.8-pve1` is already installed**, and the running initrd
  already carries **76 ZFS files** — the pivot capability exists today, no new
  packages. (The pending upgrade would take ZFS to 2.2.10-pve1; 2.2.8 is fully
  capable of root-on-ZFS, so migrate on what is installed and upgrade after.)
 - `/boot` is currently *part of* root (108 MB), so it must be split out onto its
  own ext4 filesystem on the DOM as part of this work
 - root is only **4.3 GB** to copy
 - swap is 767 MB with 123 MB used against 125 GB of RAM — irrelevant; leave it
  on the DOM LV. **Do not put swap on a zvol** (deadlock risk)
 ⚠ **`cachefile` is `none` and `/etc/zfs/zpool.cache` is 0 bytes** — pools import
 by scan today (verified: `zfs-import-scan.service` active,
 `zfs-import-cache.service` inactive). For root-on-ZFS this must be
 deterministic, or the pool may not be imported early enough to find root.
 ⚠⚠ **Set the cachefile on ALL THREE pools, not just `nvme`.** An earlier draft
 of this runbook said `zpool set cachefile=/etc/zfs/zpool.cache nvme`, and that
 one-pool form is a trap. Populating a cachefile flips the host from
 import-by-scan to import-by-cache — so a cache containing only `nvme` means
 **`ssd` and `tank` never get imported at boot.** CT 103 `esh-nas` has twelve
 bind mounts spanning all three pools (`/tank/media`, `/ssd/compose`,
 `/nvme/nvme-pvestore`, …), so the NAS would come up with every export empty and
 both NFS clients would hang on `hard` mounts. The scoped-looking command is more
 dangerous than the broad one.
 Done 2026-08-18 for `nvme`, `ssd` and `tank`; verified all three present in the
 resulting 11,976-byte cache via `zdb -C -U /etc/zfs/zpool.cache`. Phase 1's
 third guard step re-asserts this on every run.
 ## ⚠ Blast radius — unchanged, and still the gating constraint
 **CT 103 `esh-nas` (10.0.50.50) is the NAS, and it runs on this host.** Two
 dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
 | client | mounts |
 |---|---|
 | **esh-docker-vm** (10.0.50.45) | `/mnt/books`, `/mnt/backup` |
 | **esh-pve** (10.0.250.35) | `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` |
 This is a known incident shape: the only remedy for esh-docker-vm's D-state is a
 host reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's
 SQLite risks corruption under `soft`.
 The reboot in this plan is brief, but it is still a reboot — quiesce the clients
 first.
 ## Where the `/boot` LV came from — the VG was full
 The original step 7 said `/boot` could "stay inside the DOM's existing LVM as
 its own small ext4 LV, or reuse the freed space once root moves off." Neither
 was available: **VG `pve` had 4 MB free**, and the 6 GB root is mounted ext4,
 which cannot shrink online — freeing space from it needs a rescue boot, which
 would have cost the "one reboot" property the whole design rests on.
 The only space reclaimable live was the **768 MB swap LV** (123 MB in use
 against 125 GB of RAM). Operator's call 2026-08-17: **shrink swap rather than
 drop it.** Final layout:
 | LV | size | role |
 |---|---|---|
 | `pve-root` | 6.04 G | ext4 — **untouched**, the rollback root |
 | `pve-boot` | 512 M | ext4 — the new `/boot` (NEW) |
 | `pve-swap` | 256 M | swap (was 768 M) |
 Rejected alternatives: dropping swap outright (more kernel headroom, no OOM
 cushion); `proxmox-boot-tool` on the 512 MB ESP (PVE-native and no LVM surgery,
 but it reformats the ESP and downgrades rollback from "pick a menu entry" to
 "restore the DOM image"); rescue-boot to shrink root (keeps swap whole, costs a
 second reboot and an offline resize of the filesystem we are fleeing).
 ## Sequence
 **Pre-flight (no downtime)** — done 2026-08-18
 1. `dd` the DOM to an off-box image. **Crash-consistent, not clean** — the root
   LV is live during the read, so a restore replays the ext4 journal. That is
   fine for its purpose (boot-chain insurance) and is what a snapshot backup
   does anyway. Not fixable with an LVM snapshot: the VG has no free extents.
 2. Refresh the config snapshot (`nh3-dev:~/backups/esh-pve-nas/`).
 3. `zpool set cachefile=/etc/zfs/zpool.cache` on **`nvme`, `ssd` AND `tank`**
   (see the precondition warning above — the one-pool form breaks the NAS).
 **Phase 1 — `playbooks/esh-pve-nas-stage-zfs-root.yaml`** (live, no disruption)
 4. `zfs create -o mountpoint=none nvme/ROOT`, then `nvme/ROOT/pve-1` with
   `canmount=noauto`, `compression=zstd`, `xattr=sa`, `acltype=posixacl`.
   Create it with `mountpoint=none` and only set `/` at the very end —
   `canmount=noauto` alone is the documented guard, but never having a dataset
   that claims `/` while the ext4 root is live is the guard that cannot misfire.
 5. Reclaim the swap LV into `pve-boot`, mkfs, populate from `/boot`.
 6. Mount the dataset at `/mnt/newroot` and rsync the live root in.
   `--one-file-system` does the exclusion work: every path the old plan listed
   by hand (`/proc /sys /dev /run /nvme /ssd /tank /var/log/journal /boot`) is
   already a separate mount, so it is skipped structurally rather than by a
   list that can drift.
 7. Write the copy's `/etc/fstab`: no root line (the initramfs mounts it), plus
   `/dev/pve/boot /boot ext4`, the ESP, and swap.
 **Phase 2 — `playbooks/esh-pve-nas-stage-bootloader.yaml`** (live, no disruption)
 8. Chroot into the copy with the boot LV and ESP mounted, then
   `update-initramfs -u -k all` + `update-grub`.
 9. ⚠⚠ **`grub-mkconfig` gets the ZFS root WRONG here, silently. Override it.**
   See § The pool-name bug below — this is the single most dangerous thing
   found during staging.
 10. `GRUB_DEFAULT=saved`, plus `40_custom` carrying **both** boot paths as
    hand-authored entries with stable ids (`pve-zfs-root`, `pve-ext4-rollback`),
    with grubenv pinned to the **rollback**, not to ZFS (see § Cutover for why).
 ## ⚠ The pool-name bug — the near-miss worth reading
 Left to itself, `update-grub` on this host produces:
 ```
 linux /vmlinuz-6.8.12-13-pve root=ZFS=/ROOT/pve-1 ro quiet intel_iommu=on
 ```
 **The pool name is missing.** It should be `root=ZFS=nvme/ROOT/pve-1`. That
 boots to an initramfs prompt — with CT 103 `esh-nas` down and both NFS clients
 hanging on `hard` mounts, at whatever hour the window happens to be.
 It is not a typo, and it is not random. Debian's `/etc/grub.d/10_linux` builds
 the ZFS root as `${rpool}${bootfs}`:
 | part | from | value here |
 |---|---|---|
 | `rpool` | `grub-probe --device <dev> --target=fs_label` | **empty** |
 | `bootfs` | `make_system_path_relative_to_its_root /` | `/ROOT/pve-1` |
 `grub-probe --target=fs /` fails outright on this pool — `grub-probe: error:
 unknown filesystem` — because **GRUB's own ZFS reader cannot open a pool with
 `encryption`, `large_dnode` and `zstd_compress` enabled.** So `rpool` comes back
 empty and concatenates to nothing.
 That is the *same* feature set that forced `/boot` to stay ext4 on the DOM. The
 design already accounted for GRUB being unable to read the pool; what was missed
 is that the same limitation also corrupts the kernel command line — and does it
 **without an error**, because `grub-probe`'s failure is swallowed by
 `2>/dev/null || true`.
 **The fix, in two layers:**
 1. `/etc/default/grub.d/zfs-root.cfg` sets
   `GRUB_CMDLINE_LINUX="root=ZFS=nvme/ROOT/pve-1 boot=zfs"`. This is appended
   *after* the bogus value, and both the kernel and the zfs initramfs script
   take the **last** `root=` on the line — so every auto-generated entry becomes
   correct. A drop-in, not an edit to `/etc/default/grub`, so a grub package
   upgrade cannot revert it in a conffile merge.
 2. `40_custom` carries an explicit `pve-zfs-root` entry with a single clean
   `root=` and a stable id. That is what cutover's `grub-reboot` targets — the
   auto-generated ids are derived from pool member device paths
   (`gnulinux-simple-/dev/nvme0n1p1_/dev/nvme1n1p1`) and would shift if the
   mirror ever changed.
 **The general lesson, which is the transferable part:** the phase-2 verify step
 originally grepped for `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in
 grub.cfg. Once the drop-in was added that grep passes — while pool-less entries
 sit in the menu untouched. The check that actually holds walks every `linux`
 line, takes the **last** `root=` on it, and asserts it against a known-good set.
 Assert the effective value, not the presence of a substring.
 10. **`grub-install` is deliberately NOT run during staging.** The ESP stub
    still points at the old `/boot` inside the ext4 root, so the host's boot
    path stays byte-identical to what it has been for 140 days. Everything
    error-prone is built and verified in advance; the ESP rewrite is a
    two-second idempotent command held back to the window.
 **Cutover** — the remaining work, § Cutover below.
 > **The transferable lessons from this migration live in**
 > [`docs/pfi/ops-lessons-playbook.md`](../pfi/ops-lessons-playbook.md) — the ops sibling
 > to the quantization playbook. Everything below is the ESH-specific narrative;
 > the rules that would bite on any host are collected there.
 ## ⚠ The mount-propagation incident — the expensive lesson of 2026-08-18
 **What broke.** The staging chroot was built with `mount --rbind /dev` and `/sys`
 and **no `--make-rslave`**. On a systemd host `/` has *shared* propagation, so
 those binds propagate in both directions. When the cutover tore the chroot down
 with `umount -R`, the unmounts **propagated back into the live host** and removed
 the real `/sys/fs/cgroup`, `/dev/pts` and `/dev/shm`.
 With cgroup2 gone, `systemd-logind` could no longer create a session. The result
 is a host that:
 - answers ping, accepts TCP, and **completes SSH authentication**
 - keeps serving from daemons already resident in memory (`pveproxy` returned a
  clean HTTP 401 throughout)
 - **hangs on every new `exec`**, including `/sbin/reboot` — so the reboot that was
  supposed to end the window never ran
 **Why it cost so much time: it is a near-perfect impostor of failing root-disk
 I/O.** Both present as "host is up, daemons answer, nothing new can start." The
 session diagnosed it as the USB DOM dying and told the operator to walk to the
 machine. That was wrong, and the operator caught it: the DOM had been reliable
 for years and the wedge began immediately after a change.
 **The evidence that settles it, and was available the whole time** — from
 `dmesg`, obtainable in the brief windows when exec did succeed:
 | line | says |
 |---|---|
 | `[16.00] sd 56:0:0:0: [sdq] Attached SCSI removable disk` | DOM enumerated **cleanly, no errors** |
 | `[12114881.98] systemd[1]: nvme-varlog-stage.mount: Deactivated` | timestamp is **140 days** — this is the ORIGINAL boot |
 That second line is the whole answer: **the machine never rebooted.** A
 down-detector loop had also never once reported the host down; that was read as a
 fast reboot rather than as no reboot at all.
 **Rules that follow:**
 1. **Always `mount --make-rslave` after `mount --rbind` into a chroot.** Phase 2
   now does this and carries a guard that refuses to continue if any bind still
   reports `shared` propagation.
 2. **A reboot is not confirmed until the host is observed DOWN.** Poll for
   disappearance, not just for reappearance. "Never went down" and "went down and
   came back fast" are indistinguishable if you only watch for the host to answer.
 3. **Before blaming hardware for a wedge that began right after a change, get
   `dmesg` and check the boot timestamp.** Diagnose the change first; hardware is
   the explanation of last resort, not first.
 **Recovery took no console access.** Windows where `exec` briefly succeeded were
 enough to land an idempotent remount of cgroup2 / devpts / shm, after which
 `systemctl reset-failed` returned the host to `running`. Total data loss: none.
 The root filesystem, the DOM and all three pools were never at risk — this was a
 mount-namespace fault, not a storage one.
 ## The pool-name bug's neighbours — two more corrections
 **The blast radius was more than double what was documented.** The runbook named
 two NFS dependents. `ss -tn '( sport = :2049 )'` inside CT 103 showed **five**:
 | client | mount | disposition |
 |---|---|---|
 | `10.0.50.45` esh-docker-vm | `/mnt/books`, `/mnt/backup` — **hard** | quiesced |
 | `10.0.250.35` esh-pve | `esh-nas`, `tank-vmbu` — **hard** | quiesced |
 | `10.0.50.60` **esh-vm-db** | `/mnt/backup` — **hard** | **left mounted deliberately** |
 | `10.0.50.154` vm-esh-nas | — | is VM 104 *on this host*; stops with it |
 | `10.100.10.50` nh3-dev | `/mnt/books` — **soft,ro** | safe, errors instead of blocking |
 Ask the *server* who its clients are. A runbook's list of dependents is a snapshot
 that rots; `ss` on the NFS server is ground truth.
 esh-vm-db was left mounted on purpose and **came through read-write** — a `hard`
 mount with no active user blocks and resumes, which is what `hard` is for. Its
 backup timers were ~19h out, and unmounting would have meant an unmount/remount
 cycle over the qemu guest agent on a host with no ssh access.
 **The one-shot rollback does not work, and the warning was right.**
 `grub-reboot` printed *"Detected GRUB environment block on lvm device — will
 remain the default boot entry until manually cleared."* Confirmed empirically:
 after the successful ZFS boot, `next_entry=pve-zfs-root` was **still set**. GRUB
 can read grubenv on LVM but cannot write it, so `boot_once` degrades to a sticky
 default. **There is no auto-fallback on this host.** A failed boot must be
 corrected at the console.
 The steady-state config therefore does not rely on it: `saved_entry=pve-zfs-root`
 with `next_entry` cleared. Restoring a real one-shot would mean relocating grubenv
 onto the ESP (vfat on a plain partition, which GRUB *can* write) — parked, not
 required.
 ## Cutover
 The only remaining work. Everything below the quiesce is minutes.
 1. **Quiesce the NFS clients** (see § Blast radius). On **esh-docker-vm**
   (10.0.50.45) stop whatever holds `/mnt/books` and `/mnt/backup` and unmount
   them; on **esh-pve** (10.0.250.35) disable the `esh-nas` and `tank-vmbu`
   storages. Do this first and confirm it — a `hard` mount left live turns a
   brief reboot into an unkillable D-state needing a reboot of *that* host too.
 2. Shut down the five guests.
 3. Point the ESP at the new `/boot` and arm the one-shot:
   ```
   chroot /mnt/newroot grub-install --target=x86_64-efi \
       --efi-directory=/boot/efi --bootloader-id=proxmox
   chroot /mnt/newroot grub-reboot '<zfs entry id — phase 2's verify prints it>'
   ```
 4. Set the dataset's final mountpoint, then reboot:
   ```
   zfs set mountpoint=/ nvme/ROOT/pve-1     # canmount stays noauto
   reboot
   ```
 **Why `grub-reboot` and not a new default.** `GRUB_DEFAULT=saved` with grubenv
 pinned to the ext4 rollback means the ZFS entry is tried **exactly once**. If it
 fails, the next reboot returns to ext4 *by itself* — no console, no hands. That
 matters more here than on a normal host: a boot that hangs at an initramfs
 prompt takes CT 103 `esh-nas` down with it, and the NFS clients hang rather than
 fail. Only after the second successful ZFS boot (§ Verification) should the
 saved default move to the ZFS entry with `grub-set-default`.
 ## Verification
 - `findmnt -no SOURCE,FSTYPE /` → `nvme/ROOT/pve-1  zfs`
 - `df -h /` shows hundreds of GB, not 5.9
 - `findmnt /boot` → ext4 on the DOM; `/boot/efi` mounted
 - all five guests running; CT 103 serving NFS (`pct exec 103 -- exportfs -v`)
 - esh-docker-vm remounted and healthy; esh-pve storages green
 - **a second reboot** to prove it was not a one-off
 - only then: refresh the DOM image, since `/boot` has changed
 ## Rollback
 Instant and cheap at every stage: the ext4 root on the DOM is never modified, and
 its GRUB entry stays in the menu. Worst case is a boot to initramfs → reboot →
 pick the old entry. Keep the ext4 root for at least a few weeks of normal
 operation before reclaiming it.
 ## Open decisions
 - **Second boot device?** The split fixes runtime fragility but not boot-time
  single-point-of-failure. A cloned DOM/USB as a cold spare is the cheap answer.
 - **`esh-filebot` (CT 106)** is an empty container — 80 GB quota, six passthrough
  mounts, nothing running since March. Retire rather than carry it.
 - **Reclaiming the old ext4 root** once the ZFS root has proven itself.
 ---
 ## Fallback plan: full reinstall to a mirrored-NVMe ZFS root
 Only if the split above proves unworkable. Fresh PVE install to ZFS RAID1 across
 both NVMes — mirrored boot with proper ESPs under `proxmox-boot-tool`, no USB in
 the path at all.
 Costs: the `nvme` pool must be destroyed, so its **32 GB of guest rootfs** moves
 to `ssd` (1.42 T free) first; `ssd` and `tank` must be cleanly exported so the
 installer cannot touch them; guest configs restore from the snapshot plus the 8
 PBS backups per guest. Half a day, and rollback after the install step is
 "reinstall and restore".
 Note both NVMes are *whole-disk* ZFS members (partition 1 spans all 931.5 GiB,
 1.7 MiB free), so adding an ESP to them without destroying the pool is
 impossible — which is what forces the reinstall in this variant, and what the
 split plan avoids entirely.
@@ -0,0 +1,92 @@
 # Cutover: ana-ml2 → fv-ml1 (Anaheim → Fountain Valley colo)
 **Why:** ana-ml2's power draw (dual Blackwell PRO 6000, ~1.5 kW peak) repeatedly
 trips the Anaheim rack breaker — the root cause of the 2026-08-26 and 2026-09-11
 whole-site outages. Moving it to the Fountain Valley colo on its own circuit.
 **Clean-cutover property:** the box is already down (BMC dark, no power) as of the
 2026-09-11 outage, so this is a relocation, not a take-down. `/tank` is LOCAL ZFS
 (no NFS from ana-nas — verified), so the data travels with the chassis; nothing
 copies over the wire.
 ## Address map (same shape as Anaheim)
 | thing | Anaheim (old) | Fountain Valley (new) |
 |---|---|---|
 | server subnet | 10.250.0.0/16 | **10.251.0.0/16** |
 | box | `ana-ml2` 10.250.50.54 | **`fv-ml1` 10.251.50.54** (confirmed) |
 | BMC | `ana-ml2-bmc` 10.250.250.50 | **`fv-ml1-bmc` 10.251.250.50** |
 | site gateway / OPNsense | ana-gw 10.250.0.1 | **OPNsense = `.1` in EVERY FV VLAN** (server 10.251.50.1, mgmt 10.251.250.1, …) — multi-homed gateway |
 | DNS name | ana-ml2.ana.internal | **fv-ml1.fv.internal** (full new `fv` site) |
 | mesh | via ana-scale | **OPNsense = tailscale subnet-router for 10.251.0.0/16** |
 **DNS approach (decided): PIGGYBACK now, promote later.** `dns-sync` builds each
 FQDN as `name.site.zone` with NO check that the site is in the `sites:` block, and
 publishes every record to every resolver. So `fv-ml1` + `fv-ml1-bmc` records with
 `site: fv` resolve fleet-wide from the existing ana/esh/nh3 resolvers immediately —
 you get the full `fv-ml1.fv.internal` name with zero new infra. Add a real `fv`
 resolver (AdGuard container on a future FV utility box, or extend dns-sync for
 OPNsense Unbound) only when FV needs LOCAL resolution. Do NOT add `fv` to `sites:`
 until that resolver exists — the sync would fail trying to reach it.
 ## Mesh: OPNsense as subnet-router (NOT ana-ml2)
 `os-tailscale` plugin on the OPNsense, advertise `10.251.0.0/16`, approve the route
 in headscale. Rationale: mirrors ana-scale/nh3-scale/esh-scale; **stays up when
 fv-ml1 is powered off, so the BMC stays reachable over the mesh** — the out-of-band
 path the fleet lacked on 2026-09-11. Do NOT tie the mesh node to fv-ml1 itself.
 ## Blast radius — every reference to repoint (from `grep 10.250.50.54`)
 **Load-bearing (traffic breaks if missed):**
 - [ ] `stacks/litellm/conf/config.yaml` — **THE critical one** (every inference alias: gen/summarizer/classifier/image-judge/mog-sec/char-rp/meromero). At cutover, once fv-ml1 serves at FV:
  ```bash
  sed -i 's/10\.250\.50\.54/10.251.50.54/g' stacks/litellm/conf/config.yaml
  scripts/deploy-stack.sh infra-ops@10.250.50.70 litellm --conf   # push to ana-docker
  ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/litellm && sudo docker compose up -d litellm'
  ```
  Verify: `gen`/`summarizer`/`mog-sec` route to fv-ml1. ⚠ Gateway STAYS on ana-docker → cross-site to FV (metro link, fine).
 - [ ] `dns/internal.yaml` — add the two records below (piggyback; NOT a `sites:` entry), comment out the old `ana-ml2` / `ana-ml2-bmc` lines. Then `scripts/dns-sync.py --dry-run` → apply.
  ```yaml
  # ---- fv: Fountain Valley colo (piggyback DNS until FV has its own resolver) ----
  - {name: fv-ml1,     site: fv, v4: 10.251.50.54,  note: GPU inference, 4x RTX PRO 6000 Blackwell Max-Q (ex ana-ml2, relocated 2026-09)}
  - {name: fv-ml1-bmc, site: fv, v4: 10.251.250.50, note: BMC for fv-ml1}
  ```
 - [ ] mesh route to 10.251.0.0/16 approved in headscale (OPNsense subnet-router).
 - [ ] `stacks/ana-ml2-proxy/` — the Homepage Docker-API relay (10.250.50.54:2375). Rename/repoint to the FV IP or retire if Homepage reaches FV another way.
 **Cosmetic (no traffic break — rename sweep):**
 - [ ] homepage `href`/`ping` labels: stacks/{llama-swap,gemma4-charrp,meromero-charrp,erp-seat,sglang,heretic2-charrp-reasoning}/compose.yaml, stacks/homepage/conf/services.yaml
 - [ ] servers/ana-ml2/ → servers/fv-ml1/ (dir rename), ssh-target 10.250.50.54→10.251.50.54, ssh_config, remote-ssh allowlist (10.0.0.0/8 already covers 10.251 — no change)
 - [ ] playbooks/*ana-ml2*, elway invocations infra-ops@10.250.50.54 → @10.251.50.54
 - [ ] stacks/beszel deploy target; docs/*, README.md, CLAUDE.md server table, disaster-recovery.md
 - [ ] model `.PROVENANCE.txt` files reference ana-ml2 in prose (accuracy, not load-bearing)
 ## Order of operations (cutover day)
 1. Physically install fv-ml1 at FV, on its dedicated circuit. Confirm the circuit
   handles ~1.5 kW peak (the whole point).
 2. OPNsense up: LAN 10.251.50.1 / mgmt 10.251.250.1, WAN, `os-tailscale` subnet-router
   for 10.251.0.0/16, approve route in headscale.
 3. fv-ml1 up on 10.251.50.54, BMC on 10.251.250.50. Verify BMC reachable over mesh
   (out-of-band check).
 4. Bring up the vLLM seats (same /tank, same compose) — verify :8015/:8019 serve locally.
 5. Apply the LiteLLM api_base repoint on ana-docker → `up -d litellm` → verify
   `gen`/`summarizer`/`mog-sec` route to FV.
 6. DNS: internal.yaml `fv` site + records → `dns-sync.py`.
 7. Rename sweep (cosmetic) — a scripted `git mv` + sed pass, one commit.
 8. Beszel/Homepage repoint; verify dashboards.
 ## Rollback
 The physical move is not reversible same-day, but the *config* is: keep the old
 `ana-ml2` blocks in git history; if FV networking isn't ready, the seats simply
 stay down (they already are) — no worse than the current state. Do NOT delete the
 ana-ml2 DNS/LiteLLM entries until FV serves; comment them.
 ## Cross-site note
 LiteLLM gateway stays on ana-docker (Anaheim); fv-ml1 serves cross-site. FV↔Anaheim
 is metro (both Orange County, ~10 mi) so latency is a non-issue, but it's a NEW
 dependency: if the FV↔Anaheim link drops, inference dark. Acceptable; revisit
 moving the gateway to FV if FV becomes the inference hub.
@@ -0,0 +1,155 @@
 # FV colo build — on-site playbook (OPNsense + fv-ml1)
 **Prepared 2026-09-12.** Companion to [`fv-ml1-cutover.md`](fv-ml1-cutover.md), which
 owns the address map, blast radius and LiteLLM repoint. This file owns the *on-site
 hour*: minimal surface, and not locking ourselves out.
 ⚠ **PRINT THIS.** The one scenario it exists for is "the network isn't working",
 which is exactly when a phone that needs the network is useless.
 ---
 ## THE ONE RULE
 > **Do not leave the colo until you can reach both boxes from off the colo network,
 > and you have tested the lockout bypass while standing next to the console.**
 ---
 ## Read this first — what is and isn't redundant
 fv-ml1's default route is `10.251.50.1`, which **is** OPNsense. Its tailscale node
 needs egress through OPNsense to reach the control plane. Therefore:
 > **OPNsense down → fv-ml1 is also unreachable. By construction. There is no
 > software fix for this.**
 Running tailscale on fv-ml1 is still worth one command, but be exact about what it buys:
 | | |
 |---|---|
 | **Covers — the common lockout** | Bad admin firewall rule, WebGUI bound to the wrong interface, lost password, broken UI — anything where the box still **routes** but you cannot **manage** it. Egress still works, fv-ml1 stays on the mesh, and infra-ops reaches OPNsense from inside at `10.251.50.1`. |
 | **Does NOT cover** | OPNsense down, rebooting, panicked, or with broken routing/NAT. Nothing behind it is reachable. |
 ⚠ **OPNsense is a hard single point of failure for the FV site.** Plan around that
 rather than pretending otherwise. (An earlier draft of this plan called fv-ml1 a
 second independent path — it is not, and the operator caught it. Corrected here.)
 ---
 ## Address plan — flat on day one, deliberately
 ```
 OPNsense LAN      10.251.50.1/24
 fv-ml1            10.251.50.54
 fv-ml1 BMC        10.251.50.50
 WAN               static public v4 from the colo
 Route advertised  10.251.0.0/16
 Headscale         https://headscale.phasefinal.com     (public, verified 200)
 Expected mesh     OPNsense 100.64.0.7 / fv-ml1 100.64.0.8
 ```
 ⚠ **Deliberate deviation** from `fv-ml1-cutover.md`, which puts the BMC on a mgmt
 VLAN at `10.251.250.50`. A mgmt VLAN needs OPNsense VLAN config plus switch tagging —
 real time and real lockout risk during the scarcest hour — for something that can be
 added remotely in ten minutes. **Flat today, split later.**
 ---
 ## On-site: nine things. That is the whole job.
 ### OPNsense
 1. **Assign interfaces — USE AUTO-DETECT** (plug the cable when prompted).
   ⚠ Do **not** assume port order. OPNsense assigned LAN to the wrong NIC on the
   lab VM on 2026-09-11; identical physical ports make it worse.
 2. **WAN:** static IP / mask / gateway from the colo.
 3. **DNS: `1.1.1.1` and `9.9.9.9`.**
   ⚠ **NOT the fleet AdGuard (10.100.50.40).** It is unreachable until the mesh is
   up, and the mesh needs DNS to fetch the plugin. That deadlock eats an hour.
 4. **LAN:** `10.251.50.1/24`. Flat. No VLANs today.
 5. **Install `os-tailscale`**, then join:
   ```
   tailscale up --login-server https://headscale.phasefinal.com \
     --authkey <KEY FROM VAULT> --advertise-routes=10.251.0.0/16
   ```
 ### fv-ml1
 6. Power on. Confirm it boots and `/tank` imports.
 7. Static `10.251.50.54`, gw `10.251.50.1`, DNS `1.1.1.1`.
 8. Join as a **plain node, no route advertisement**:
   ```
   tailscale up --login-server https://headscale.phasefinal.com --authkey <KEY FROM VAULT>
   ```
 ### BMC
 9. Static `10.251.50.50` on the flat LAN.
 ---
 ## The verify gate — do not skip
 From **off** the colo network (phone on LTE, and infra-ops from nh3-dev):
 - [ ] OPNsense reachable at `100.64.0.7`
 - [ ] fv-ml1 reachable at `100.64.0.8`
 - [ ] BMC reachable at `10.251.50.50` through the approved route
 - [ ] **Lockout bypass test, console in arm's reach:** add a floating rule denying
      the mesh to the OPNsense WebGUI → confirm `10.251.50.1` is still reachable
      **via fv-ml1** → **remove the rule.**
 ⚠ That last item is the only test that proves the bypass works *before* it is needed.
 **Rebooting OPNsense proves nothing** — it predictably drops the whole site for
 60–90 s and then everything returns.
 ---
 ## Traps, in the order they bite
 - **DNS chicken-and-egg.** Public resolvers until the mesh is up.
 - **Interface assignment is not what you expect.** Auto-detect.
 - **Never change WAN settings and the management path in the same step.** One at a
  time, verify between.
 - ⭐ **If the static handoff is delayed, ASK FOR DHCP.** Tailscale needs working
  internet, not the final address. This single question can save the trip.
 - Nothing points at internal DNS or internal IPs until the mesh route is approved.
 ---
 ## Before leaving the house
 - [ ] Preauth keys minted, **7-day reusable**, vaulted under `headscale/`
      ⚠ the 2026-09-06 keys expired 2026-09-08 — a dead key at the rack is a wasted trip
 - [ ] This page **printed**
 - [ ] MacBook (`vhlk-mba26`, headscale node 5) back online as a mesh client — an
      independent client path, and lets the operator verify without infra-ops
 - [ ] Colo **remote hands**: phone number, procedure, SLA
 - [ ] **Physical labels on both boxes** so remote hands can act on a phone call
 - [ ] infra-ops reachable while on site
 ---
 ## What infra-ops does remotely, live
 - Approve the `10.251.0.0/16` route in headscale
 - Push SSH keys to both boxes, confirm key-only login
 - Run the reachability checks and the bypass test, report pass/fail
 - Add `fv-ml1` + BMC records to `dns/internal.yaml` and sync
 **Deferred to remote on purpose** (none needs hands on site): mgmt VLAN split,
 firewall policy, LiteLLM `api_base` repoint (10.250.50.54 → 10.251.50.54), vLLM
 seats, ana-ml2 → fv-ml1 rename sweep, Beszel, Homepage.
 ---
 ## After leaving
 Every remote change to OPNsense gets the **commit-confirmed** pattern: snapshot
 config → apply → scheduled auto-revert that fires in N minutes unless cancelled.
 That turns "I locked us out" from a drive to Fountain Valley into a two-minute wait.
 Worth pricing: a ~$150 **LTE out-of-band router with its own mesh node** is the only
 thing that gives true independence from OPNsense. **Ask the colo first** — many
 include an OOB/management network with the rack.
@@ -0,0 +1,572 @@
 # FV site dark — 2026-09-13 ~06:56Z
 **Status: UNRESOLVED, WILL NOT SELF-RECOVER, needs hands at Fountain Valley.**
 Operator 2026-09-13: an overload-tripped UPS does not clear itself and a dead one
 needs replacing, so there is no recovery to wait for and no point polling the site.
 Visit planned for 2026-09-14. Do not leave watchers running against FV addresses. Written while the site is
 down so recovery does not have to be reconstructed from memory.
 ## What is down
 Everything at Fountain Valley, measured 06:58:40Z from nh3-dev:
 | target | result |
 |---|---|
 | `fv-ml1` 10.251.50.54 | 100% loss |
 | `fv-ml1` mesh 100.64.0.7 | 100% loss |
 | FV gateway 10.251.50.1 | 100% loss |
 | FV gateway mesh 100.64.0.8 | 100% loss |
 | **fv-ml1-bmc 10.251.250.50** | **100% loss** |
 | `fv.phasefinal.com` (172.83.89.66) | no answer |
 **My own path is healthy** — ana-docker, nh3-docker, esh-docker-vm and 1.1.1.1 all
 answer with 0% loss, so this is FV-local, not an isolation of the observer.
 ## Timeline
    06:51:08  MTP campaign started on GPU 3 (:8023), operator-authorised, knowing
              the two-card power risk. Production seat already live on GPU 2.
    06:51:17  power sampler baseline: GPU0 3.80 W, GPU1 3.88 W, GPU2 14.27 W, GPU3 6.75 W
    06:54:39  off_A arm healthy after 211 s
    06:54:39  off_A rep 1 ran clean: 75.49 / 212.33 / 387.83 tok/s at conc 1/4/8
    06:56:04  off_A rep 2 started  <-- LAST LOG LINE
    06:58:40  every FV address unreachable, including the BMC
 So the site went dark inside a ~2.5 minute window, roughly five minutes into
 sustained bench load on GPU 3 with GPU 2 also resident/serving.
 ## ⭐ LEADING HYPOTHESIS (operator, 2026-09-13): the UPS overloaded and died
 Operator's read, and it fits the evidence **better** than the breaker-trip theory
 below, for a reason worth writing down:
 **A UPS's output rating is far below the circuit's, so the UPS is the FIRST protective
 device to give.** A common 1500 VA rack unit delivers ~900-1000 W; a 15 A circuit
 carries ~1800 W. Two of these cards under load, plus a host carrying 566 GB of RAM,
 plus the firewall, plausibly clears the UPS rating while staying comfortably under the
 breaker. That explains the thing a breaker trip explains poorly: **why it let go at
 TWO cards loaded and not four.** It also explains the OPNsense box dying -- same UPS.
 ⚠ **"Died" is the operationally important word.** A tripped UPS resets; an overloaded
 one can kill its output stage or battery pack permanently. If it is dead rather than
 tripped, **nothing on site can be reset back to life** and the trip is wasted without
 the means to bypass it.
 ### Bring / check list for the site visit
 - **The means to BYPASS the UPS entirely** -- chassis and firewall straight to the
  PDU or wall. Do this first, get the site back, diagnose the UPS after.
 - **Read off the UPS before moving it:** make/model, VA and W rating, fault LEDs, and
  any LCD or event log entry (many units record "overload" explicitly).
 - **Note which outlets were battery-backed vs surge-only.** Mixed-bank units are
  common, and a GPU chassis on the battery-backed bank overloads soonest.
 - ⭐ **Recover `/tank/aimodels/flash-next-mtp-bench/power.log` FIRST, before any
  bring-up.** It sampled all four cards every 10 s right up to the cut and is the
  ONLY measurement of what the load actually drew. It lives on `/tank`, not in a
  container, so it survived. Without it, the replacement UPS gets sized by guesswork.
 - ⚠ **Size the replacement for FOUR cards under load, not two.** Anything sized to
  today's failure just relocates the trip point to the next person who loads all four.
 ⚠ **No measured load figure exists yet.** Idle draw was measured (GPU0 3.80 W, GPU1
 3.88 W, GPU2 14.27 W, GPU3 6.75 W at 06:51:17Z) but the load figure died with the box
 and is recoverable only from `power.log`. Do not let anyone put a wattage in a
 purchasing decision until that file has been read.
 ## ⭐ OPERATOR RULING 2026-09-13: undersized UPS. NAT hypothesis DEMOTED.
 > "highly doubt the nat hypothesis, it went effective (connectivity confirmed for
 > previously dark path) -- then 20 minutes later, during load, site went dark. that's
 > pretty unlikely to be the cause. for sure I think the ups was undersized."
 Accepted, and the reasoning is better than the hypothesis-space argument below: **a
 config change that went effective, was verified bidirectional, and then ran correctly
 for twenty minutes does not spontaneously fail when an unrelated physical variable --
 someone else's GPU load -- is introduced.** The load correlation is tight; the NAT
 correlation is merely adjacent in time. Undersized UPS is the only candidate that
 explains the *trigger*.
 The NAT material below is retained as record, not as a live competing theory, and the
 `power.log` / `uptime` check is retained as **free confirmation** rather than as a
 decision point.
 ## Measurement plan for the site visit (operator bringing a PDU + ammeter)
 ⚠ **`power.log` is GPU-ONLY.** It samples `nvidia-smi` per-card draw and does **not**
 include the host: CPU, 566 GB of RAM, drives, fans, or PSU conversion losses. The
 number that matters against a UPS rating is the whole chassis **at the plug**. The
 ammeter is therefore the primary instrument and `power.log` is a cross-check on the
 GPU share.
 Capture four states -- this is the first real sizing data that has ever existed for
 this box:
 | state | why it matters |
 |---|---|
 | all seats down, idle | the floor (GPU idle measured 3.80 / 3.88 / 14.27 / 6.75 W) |
 | one card loaded | the condition that ran fine for a day |
 | **two cards loaded** | the condition that took the site down |
 | **four cards loaded** | the only number that can size a replacement honestly |
 ⚠⚠ **CAPTURE PEAK, NOT AVERAGE.** GPU power has fast transients and UPS overload
 protection responds to short-term overload, so a 1-second-average reading can
 under-read peaks badly. Use max-hold/peak capture if the meter has it. A figure like
 "1100 W average" that hides 1600 W spikes will mis-size the replacement the same way
 the current unit got mis-sized. **If the meter is average-only, record the number as a
 FLOOR, not as the draw.**
 ⭐ **The four-card figure goes into `servers/fv-ml1/README.md` permanently.** The
 cutover notes flagged that the FV circuit was "likely specced against half the real
 draw" while every record still said two GPUs; this closes that with a measurement
 instead of an assumption.
 ## The NAT change, retained as record (DEMOTED — see the ruling above)
 **Another session applied a Tailscale SNAT rule to the FV gateway at ~06:22Z — 34
 minutes before the site went dark.** See `docs/runbooks/fv-to-ana-nat.md` (uncommitted
 as of this writing; not my work, left alone). So "we overloaded the power" is no longer
 the only live hypothesis, and the UPS should not be replaced on the strength of a theory
 until the one below is run.
 **On the evidence, that change is the WRONG SHAPE to have caused this**, and I want that
 on the record so nobody wastes the visit chasing it:
 - It is one **outbound** SNAT rule, source-scoped to `10.251.50.54/32`, destination-
  scoped to `10.250.0.0/16`. An outbound NAT rule cannot stop the gateway, the BMC or
  the public WAN address from answering **inbound**.
 - The runbook states no routes, filter rules, WAN settings or subnet advertisements were
  touched, and that `pfctl -sr` came back byte-identical.
 - It was verified working in **both** directions afterwards: FV→hub HTTP 200, FV→ANA
  TCP 5432, FV internet HTTPS 200, **ANA→FV SSH reachable**, gateway management intact,
  Beszel **18/18 up**.
 ⚠ Note their BMC observation used **`10.251.50.50`**, which is not the BMC — the BMC is
 **`10.251.250.50`**, a different subnet. They correctly declined to claim BMC health, but
 the datapoint is *void*, not negative. Do not reason from it either way.
 ### ⭐⭐ THE DISCRIMINATOR — run this before forming any conclusion
 `power.log` is written **locally on `/tank`, every 10 seconds, by a shell loop on the
 box.** It does not depend on the network. So:
 | `power.log` last entry | what it means |
 |---|---|
 | **past 06:56Z** | the box **never lost power**. This is a routing/gateway fault, and the UPS is innocent. |
 | **stops at ~06:56Z** | the box lost power. UPS/circuit confirmed. |
 Cross-check with `uptime` and `journalctl --list-boots` the moment there is a console:
 **continuous uptime across 06:56Z kills the UPS theory outright.**
 ⚠ **So the FIRST action on site is to read, not to fix.** `uptime`,
 `journalctl --list-boots`, then `tail power.log`. Establish whether the machine ever
 went down before anyone buys hardware or flips anything — the two hypotheses lead to
 completely different remediations and only one of them needs a new UPS.
 ## Three further candidate causes, and what distinguishes them
 Cannot be distinguished remotely, because every FV path — including the BMC —
 traverses the OPNsense gateway, and the gateway is also dark.
 1. **Circuit tripped under two-card load.** Fits the timing and is the predicted
   failure: the Anaheim rack breaker tripped twice (2026-08-26, 2026-09-11) on this
   same chassis, and the FV circuit was specced while every record still said the box
   had **two** GPUs rather than four. ⭐ Distinguishing evidence: **the breaker is
   visibly tripped**, and the OPNsense box is dark too (it draws ~20-30 W and would
   survive anything short of a circuit/utility loss).
 2. **OPNsense gateway crashed or rebooted**, taking all FV routing with it while the
   GPU box is fine. Distinguishing evidence: the GPU box's PSU fans/lights are on and
   the OPNsense box is the only dead thing.
 3. **Upstream utility or colo-side power/network loss**, unrelated to us.
   Distinguishing evidence: the breaker is NOT tripped and nothing at the rack has
   power; neighbouring equipment is also dark.
 ⚠ Do not record cause 1 as fact until someone has looked. The timing is strong
 circumstantial evidence, not a measurement — and the instrument that would have
 measured it (the power log on fv-ml1) died with the box.
 ## Blast radius
 **19 of 30 LiteLLM aliases are dark** — fv-ml1 backs most of the fleet's inference:
    char-rp, char-rp-fast, char-rp-reasoning, chat-judge, classifier, coder-fast,
    erp-tune-v2, gemma4-26b-a4b-it-base, gen, gen-large, gen-reasoning, image-judge,
    qwen-image-bench, qwen3-embedding, reranker, sec, sec-reasoning, summarizer,
    summarizer-large
 ⚠⚠ **THERE IS NO LOCAL FALLBACK, and an earlier note in this session wrongly implied
 there was.** Probed 2026-09-13: every free local model on the gateway lives on
 fv-ml1. irv-ml1 runs **no LLM chat seat at all** -- it carries TTS (tts-gateway,
 breeze-tts, voice-studio, omnivoice-ref), ComfyUI, arbo, yt-voice-clipper, bragi and
 waterland-studio, and its two Ampere cards are partly occupied by them. The only
 non-fv chat backends on the gateway are **paid**: api.z.ai (9 aliases) and
 Moonshot/Kimi (2).
 So the choice during the outage is: leave the 19 aliases failing loudly, or spend
 vendor credits. ⚠ **If credits are spent, it must be under a NEW alias name that
 callers opt into** -- never by silently repointing `summarizer`/`gen`/`classifier` at
 GLM. Silent model substitution behind a familiar name is a standing prohibition here
 and has already been violated twice; an outage is not an exemption.
 The gateway itself on ana-docker is healthy — it is the backends that are gone.
 ## ⚠⚠ RECOVERY — do NOT just reset the breaker and walk away
 Every vLLM seat on fv-ml1 carries `restart: unless-stopped`. On boot **all of them
 start loading simultaneously** — gen, mog-sec, erp-seat, gemma4-charrp, flash-next,
 embed, rerank, reward, coder, scriberr — which is the single largest power transient
 the box can produce, fed straight into a circuit that may have just tripped. That is a
 re-trip, and a re-trip during model load can leave a half-written page cache and a
 much longer recovery.
 **Preferred sequence:**
 1. Power the chassis on with **Docker masked**, so nothing auto-starts:
   at the BMC/console, boot to the OS and before the network comes up run
   `systemctl mask docker containerd` — or if the box is already up and loading,
   `systemctl stop docker` immediately.
 2. Confirm `nvidia-smi` sees all four cards and `zpool status tank` is ONLINE.
 3. Unmask, then bring seats up **one at a time**, waiting for each to report healthy:
   `gen` first (19 aliases depend on it), then embed/rerank/reward/coder, then
   mog-sec, then the rest. `flash-next` LAST — it is the newest and least depended-on.
 4. **Do not restart the MTP campaign.** It is the prime suspect.
 5. Watch power while seats come up: `nvidia-smi --query-gpu=index,power.draw --format=csv`.
 ### ⭐ The staged bring-up also finishes the homepage-label fix, for free
 The 2026-09-13 renumber (commit `3132a16`) repaired 25 compose files on this box but
 the 10 RUNNING containers were never recreated, so their labels still carried the dead
 10.250.50.54. Those containers are gone with the power loss.
 ⚠ A PLAIN POWER-ON DOES NOT FIX THEM. `restart: unless-stopped` restarts the existing
 container with its existing labels; labels only attach at container CREATION. But the
 staged `docker compose up -d <svc>` sequence above **is** a recreate, and the compose
 files on disk are already corrected — so bringing seats up that way applies the new
 labels as a side effect and the dashboard comes back correct. Bring them up with
 `compose up -d`, not by letting Docker restore the old containers.
 Afterwards, confirm with:
    curl -s http://10.0.50.45:5100/api/services | \
      python3 -c 'import json,sys;d=json.load(sys.stdin);print([s["href"] for g in d for s in (g.get("services") or []) if "10.250.50.54" in (s.get("href") or "")])'
 Expect `[]`. Before the outage that query returned 16 entries.
 **Do not reboot the OPNsense firewall** (standing operator directive; its reboot API
 403s anyway).
 ## ⚠ THE CIRCUIT CASE — what split power does and does not buy (operator, 2026-09-13)
 > "unless of course the thing trips the circuit anyway."
 **It still helps, but only halfway, and the halfway matters.**
 - ✅ **A breaker trip is exactly what the split survives.** Firewall + BMC on the UPS is
  ~25-40 W of load on a 1500 VA unit — hours of battery, not minutes. On a trip the UPS
  stops being a load-bearing supply and goes back to being what it is for.
 - ❌ **A live firewall is useless if the path OUT of the site is dead.** Our UPS covers
  our gear; it does not cover the **colo's handoff** — their switch, ONT or demarc. If
  that sits on the circuit we just tripped, the result is a firewall running happily on
  battery with nothing upstream to talk to, and the drive happens anyway.
  ⭐ **ASK THE FACILITY: is the network handoff on our circuit or theirs, and is theirs
  on facility UPS?** This is the question that decides whether split power actually
  delivers remote diagnosis or merely feels like it does.
 ### ⚠⚠ And the case where none of the above matters
 **If four cards plus host exceeds the circuit, no UPS arrangement helps** — the box does
 not fit its feed. Removing an undersized UPS does not remove the constraint, it promotes
 the next one:
    UPS        ~900-1200 W   (the one that just gave way)
    circuit    ~1800 W @ 15 A  /  ~2400 W @ 20 A
 Which side of those the four-card figure lands on decides everything, which is why that
 single ammeter reading is the load-bearing measurement of the visit.
 ### ⭐ The lever that may avoid an electrician: per-card power limits
 `nvidia-smi -pl <watts>` caps TGP per card. The box can be made to fit whatever the feed
 turns out to be, at a **throughput** cost rather than a **rewiring** cost — four capped
 cards on a 15 A circuit is a dial we control today, where a 20 A drop is a ticket and a
 site visit.
 - Read `nvidia-smi -q -d POWER` first for the enforced min/max range per card; do not
  assume how much room the dial has.
 - ⚠ **If capping is the answer it MUST be persisted** (systemd unit, or an `if-up`
  equivalent). A limit that evaporates on reboot is worse than no limit, because it will
  hold right up until the next power event and then silently stop holding.
 ### Three questions for the site visit
 1. What is the **breaker rating** on that circuit?
 2. Is the circuit **dedicated** to us, or shared with other racks/tenants?
 3. Is the **network handoff** on our circuit or the facility's, and is the facility's on
   their UPS?
 ## ⚠ PROPOSED, NOT RATIFIED — split the power so the management plane survives
 **This is a recommendation awaiting the operator's call, not settled intent.** Written
 down so tomorrow's rebuild can adopt or reject it deliberately rather than restoring
 the arrangement that just failed by default.
    UPS         <- OPNsense firewall + fv-ml1 BMC only   (tens of watts, long runtime)
    PDU / wall  <- GPU chassis                            (no UPS in series)
 Two reasons:
 1. **It fixes the OOB gap this outage exposed.** The BMC's only route to the fleet is
   through the firewall, so a power event at the GPU box takes out the management plane
   with it -- which is precisely why this incident needs a drive rather than a console
   session. Separate the two and a repeat leaves a live firewall, a live BMC, and
   remote eyes on a dark chassis.
 2. **A 1500 VA unit was never going to hold this box.** It has four cards, not the two
   every record claimed until 2026-09-12.
 ⚠ **Do NOT use a UPS's surge-only outlets to get around its rating.** Both outlet banks
 sit downstream of the same input cord, inlet and internal breaker; for a 120 V 1500 VA
 unit that is a single NEMA 5-15P rated **12 A at maximum load**, total across all
 outlets. The surge bank bypasses the inverter, not the current rating. Overloading the
 inverter trips or kills the unit; overloading the cord is a thermal problem in an
 unattended rack. Bypass the UPS entirely instead.
 If the GPU box is ever to go on battery, it is a 3000 VA / 2700 W-class unit and
 probably a 20 A circuit -- but size it from `power.log`, not from a spec sheet.
 ## Afterwards
 - ⭐ **The OOB design gap this exposes.** The cutover chose OPNsense-as-subnet-router
  specifically so the BMC stays reachable when the GPU box is down. That works for
  box-down/gateway-up. It does **nothing** for a site-wide power or gateway loss —
  exactly what happened — because the BMC's only path to the fleet is through that
  gateway. A genuine OOB path at FV needs something the FV circuit cannot take down:
  an LTE/cellular console, or the BMC on a separate circuit with its own uplink.
 - Get the actual circuit rating and the box's real peak draw, now that it is known to
  have four cards and not two. Until then, treat concurrent multi-card load at FV as
  unproven rather than safe.
 - `services/flash-next-mtp-bench/power.log` on the box holds the per-card draw right
  up to the cut. Recover it after boot — it is the only measurement of what the load
  actually drew, and it survives on `/tank`, not in the container.
 ## ⭐ POLICY, forward-looking (operator, 2026-09-13): cap new cards at build time
 > "i believe our ada cards for the other colo are rated 600w each, we'll want them
 > power limited to 200w"
 >
 > **CLARIFIED BY OPERATOR 2026-09-13 — the two boxes are different hardware:**
 >
 > | box | cards | TGP each | VRAM total | status |
 > |---|---|---|---|---|
 > | **fv-ml1** | 4x **Blackwell** RTX PRO 6000 **Max-Q** | **300 W** (Max-Q is the reduced-TGP SKU; the Workstation Edition is 600 W) | 4x96 = 391 GB | deployed, currently dark |
 > | **ana-ml3** | 2x **Ada Generation** RTX 6000 | **300 W** | 2x48 = 96 GB | **NOT YET DEPLOYED** |
 >
 > **CAPS (operator, 2026-09-13):** **fv-ml1 → 250 W** per card (83% of TGP, ~5%
 > throughput). **ana-ml3 → 200 W** per card (67% of TGP, ~10-15%). See the plug-side
 > arithmetic below — 250 W is marginal on a 15 A circuit once PSU efficiency is counted,
 > and must be verified with the ammeter rather than assumed.
 The generalised lesson from this outage: **decide the power envelope first and size the
 cards into it**, rather than installing cards and discovering the constraint by tripping
 it. 4 x 200 W = 800 W of card, which fits a real circuit with a real UPS and headroom.
 Three things to settle before that is a plan:
 1. ⚠ **200 W may be below the card's ENFORCEABLE FLOOR.** `nvidia-smi -pl` is bounded by
   the part's own `Min Power Limit`, which on a high-TGP card is often around half the
   rating. If the floor is 300 W, a 200 W request is refused — and, depending on how it
   is scripted, refused quietly. **First command on the new hardware:**
       nvidia-smi -q -d POWER | grep -iE 'power limit|default'
   If the floor lands above 200 W, the envelope has to come from fewer cards or a bigger
   feed, not from the cap.
 2. ✅ **RESOLVED — both card types are 300 W.** So 200 W is a cap to **67% of TGP**, the
   favourable part of the concave curve, not the severe 33% cap a 600 W part would have
   implied. The enforceable-floor concern largely goes away too: 200 W was borderline
   against a 600 W card's minimum and is very unlikely to sit below a 300 W card's. Worth
   the one command; expect it to take.
   **ana-ml3: 2 x 200 W = 400 W of card.** Modest, and pointed — ana-ml3 lands in the
   **Anaheim** rack whose breaker tripped on 2026-08-26 and 2026-09-11, one of those
   caused by this very chassis before it relocated. The cap there is remediation of a
   circuit with a track record, not precaution.
 ### ⭐⭐ The outage arithmetic, now that the TGP is known
        2 x Blackwell Max-Q @ 300 W        ~  600 W of card under load
        host (board, 566 GB RAM, drives,
              fans, PSU conversion loss)   ~  200-350 W   <-- UNMEASURED, the gap
                                              -----------
                                           ~  800-950 W
        Eaton 1500 VA real watt rating     ~  900-1200 W depending on model
 **At or just over the line** — and this is what a vague "undersized" could not explain:
 why it ran a full day on one card (~500-650 W, comfortably inside) and died minutes into
 the second (~800-950 W, at or past the rating). The host term is the only one being
 guessed at, and idle-at-the-plug with all seats down measures it directly.
 ### ⚠⚠ FOUR cards is a BREAKER problem, not a UPS problem
        4 x 300 W card  +  ~300 W host     ~  1500 W
        15 A circuit, 80% continuous       =  1440 W
 **Four cards uncapped is marginal on a 15 A circuit with no UPS in the path at all**, so
 capping belongs at fv-ml1 too. Note today's incident only ever had TWO of the four cards
 working; nobody has loaded all four.
 ### ⚠⚠⚠ AND `nvidia-smi -pl` CAPS BOARD POWER, NOT WALL POWER
 This term is easy to drop and it is the one that decides whether 250 W clears a 15 A
 feed:
        4 x 250 W board                           = 1000 W
        host components (board, 566 GB RAM,
             drives, fans)                        ~  180-300 W
                                                     -----------
        component total                           ~ 1180-1300 W
        / PSU efficiency (~0.90)  -> AT THE PLUG  ~ 1310-1445 W
        15 A circuit, NEC 80% continuous derating =  1440 W
 **250 W lands ON the limit, not under it.** An inference box serving all day is a
 continuous load, so 1440 W is the design figure, not 1800.
 At **200 W** the same arithmetic gives **~1090-1220 W at the plug** — comfortable, 200+ W
 of margin.
 ⚠ **So 250 W is PROBABLY fine and POSSIBLY not, and the deciding term is the host draw,
 which is still an estimate.** Procedure: set 250 W, then **verify at the plug under
 four-card load** before calling it done; fall back to 200 W if the reading comes in near
 1440 W. A cap is a claim; the ammeter is the verification.
 ⚠ **Power limits cap SUSTAINED draw, not transients.** The enforcement window is short
 but not instantaneous, so four cards at 250 W can momentarily exceed 1000 W of board.
 A breaker tolerates that (thermal-magnetic curves are forgiving of brief overload); a
 UPS's overload protection is not. Which means **250 W implicitly commits the chassis to
 the PDU rather than behind the 1500 VA unit** — even capped, 4 x 250 W + host exceeds
 that UPS's real rating.
 ⚠ **200 W on ana-ml3's TWO cards is deliberately conservative** (2 x 200 = 400 W is
 trivial on any circuit). Relaxable later if Anaheim's measured headroom beats its trip
 history; not a permanent figure.
 3. ⭐ **Decode tolerates a cap far better than training does**, which is lucky given what
   this fleet mostly does. Decode is memory-bandwidth-bound; the perf/watt curve is
   strongly concave, so ~60-70% of TGP typically costs ~10-15% of throughput. A cap to
   33% of TGP is deeper into the steep region — the cost is real and should be measured
   on the first card rather than predicted, and it will hurt a prefill-heavy or training
   workload considerably more than a serving seat.
 ### ⚠ Two placement consequences of Ada, independent of power
 - **sm_89 has native FP8 but NOT NVFP4** (Blackwell-only, sm_100/sm_120). Most of our
  in-house quants are NVFP4, so **they will not run accelerated on that colo's cards.**
  Its seats want FP8 W8A8 builds, or the NVFP4 checkpoints stay on fv-ml1. Same class of
  constraint as the Ampere finding for irv-ml1, one generation up.
 - ⭐ **It unparks the triton-backend item.** That is a hard no on Ampere — crashes every
  render on the A6000, `fp8e4nv` unsupported on sm_86 — and was explicitly deferred TO
  Ada. sm_89 has the FP8 support it needs, so it becomes testable on this hardware.
 - **VRAM:** 4 x 48 GB = 192 GB, against fv-ml1's 4 x 96 = 391 GB. Big-model placement
  stays at FV. The Flash-Next seat needs 74 GiB resident on ONE card and would not fit a
  48 GB Ada card even with the n-gram table offloaded — the offload moves the *table*,
  not the experts.
 ⚠ **PERSIST THE CAP.** A hand-set limit holds until the next reboot and then silently
 stops holding — the worst possible failure shape, because the thing that reboots the box
 is likely to be the power event the cap existed to prevent. Systemd unit, persistence
 mode, ordered before Docker starts.
 ## Can the whole BANK be capped at 1000 W instead of per-card? (operator question)
 > "is it possible to cap the ENTIRE bank to 1000w? meaning that each card can go to max
 > until they're all loaded?"
 **The concept is first-class in DCGM, the dynamic behaviour is not free, and the static
 cap already equals the bank budget.**
 ### What exists
 `dcgmConfigPowerLimitType_enum` (DCGM API) carries exactly this distinction:
    DCGM_CONFIG_POWER_CAP_INDIVIDUAL  "the power cap to be applied for each member of the group"
    DCGM_CONFIG_POWER_BUDGET_GROUP    "the power budget for the entire group"
 ⚠ **The documentation does not state how a group budget is distributed.** Deduction, not
 a quote: the only enforcement primitive underneath is NVML's per-GPU
 `nvmlDeviceSetPowerManagementLimit` — **there is no bank-level register** — so any group
 budget ultimately resolves to N per-GPU writes. Static even division needs one write
 each; "each card free until they are all loaded" needs **continuous re-writing**, i.e. a
 control loop rather than a hardware feature.
 **What DCGM is:** NVIDIA's own **Data Center GPU Manager** — first-party, open source
 (Apache 2.0, `NVIDIA/DCGM`), packaged as `datacenter-gpu-manager`. It layers above NVML:
    nvidia-smi  CLI, thin wrapper over NVML
    NVML        low-level C library, PER-GPU primitives (what -pl actually calls)
    DCGM        daemon (nv-hostengine) + dcgmi, ABOVE NVML — health, diagnostics,
                config enforcement, policy, group abstractions; dcgm-exporter is its
                Prometheus sidecar
 Its "group" notion is therefore a management-layer abstraction over per-GPU NVML calls,
 which is why the bank budget still resolves to N per-GPU writes underneath.
 ✅ **VERIFIED 2026-09-13 — DCGM DOES SUPPORT OUR CARDS, and an earlier caveat in this
 runbook claiming otherwise was WRONG and has been removed.**
 Supported platforms, quoted: *"All NVIDIA Maxwell™ and newer **non-datacenter** (e.g.
 NVIDIA® GeForce® or NVIDIA® Quadro®) GPUs"* — plus *"Starting with v1.3, limited DCGM
 functionality is available on non-datacenter GPUs."*
 And the feature-overview table settles what "limited" excludes — **not** configuration:
    Feature Group               Tesla   Titan   Quadro   GeForce
    Configuration Management      X       X        X        X
 Configuration Management explicitly includes *"Power Limit: Set the maximum allowed power
 consumption."* The thing actually gated on non-datacenter cards is **diagnostics**:
    GPU Diagnostics (Levels 1,2,3):  All Levels [Tesla];  Level 1 [Titan/Quadro/GeForce]
 ⚠ **One soft edge:** the table says "Quadro", the former name for the professional line.
 RTX 6000 Ada and RTX PRO 6000 are its successors and should fall in that column, but the
 table predates the rename — so that last step is inference, settled by one command on the
 box.
 ⭐ **So the group-budget test is worth actually running**, not a curiosity. What it does
 NOT settle is *distribution*: the docs still never say how a group budget is divided, and
 the NVML argument is untouched — no bank-level register means per-GPU writes either way,
 so the likely finding is static even division (= 4 x 250 W).
 ✅ **The static cap needs none of this.** `nvidia-smi -pl 250` is plain NVML and works on
 these cards. DCGM would only buy the group-budget experiment and richer telemetry, and is
 probably not even installed — `beszel-agent-nvidia` shells out to `nvidia-smi`.
 ### ⚠⚠ If a loop is written, its failure direction matters more than its logic
 Power readings lag and `-pl` application takes tens of ms, so a reactive daemon
 overshoots during a load RAMP — and the ramp is exactly the dangerous moment, because it
 is the all-four-cards-loading-at-once case (the same shape as this box's ten
 `restart: unless-stopped` containers starting together).
 **Therefore: safe-by-default, opportunistic upward.** Boot every card at budget/N and
 only ever RAISE a card's cap after observing idle neighbours. Never boot high and react
 down. Failure mode then becomes "slower than it could have been" instead of "tripped the
 breaker." Inverted, it is a thing that works for weeks and then fails on precisely the
 event it existed to prevent.
 ### Why not yet
 **4 x 250 W = 1000 W — the static cap IS the bank budget**, and it is the conservative
 floor of the dynamic scheme rather than an alternative to it. The daemon's entire
 contribution is the one-card-busy case: ~300 W instead of ~250 W on a single card, ~17%
 more board power, which on a concave perf/watt curve is perhaps ~5% throughput. That is
 the whole prize, against a control loop whose failure mode points at a breaker.
 And whether that case is even common depends on workload mix. A **serving** fleet spreads
 across cards by construction — one seat per card, gateway traffic split across aliases —
 so single-card-busy is rare. A **training window** is the opposite: one card hammered,
 three idle, which is where the dynamic scheme actually pays.
 **Recommendation: static 250 W now, measure at the plug, build the loop only if the
 measurements show single-card-busy is the common case.** It is a pure optimization on
 top; adding it later re-architects nothing and would be built against real numbers.
@@ -0,0 +1,137 @@
 # FV to Anaheim: scoped Tailscale NAT repair
 Applied 2026-09-13 ~06:22Z after operator approval with explicit instruction
 not to strand the subnet. Gateway: OPNsense at 10.251.50.1, current hostname
 vb-gateway.internal; tailscale0 address 100.64.0.8, assigned interface opt6/MESH.
 ## Persistent configuration
 Firewall / NAT / Outbound: **Hybrid** mode, retaining every automatic WAN rule.
 One manual IPv4 rule:
 - Interface: MESH (opt6 / tailscale0)
 - Source: 10.251.50.54/32 (fv-ml1 only)
 - Destination: 10.250.0.0/16 (Anaheim only)
 - Translation: interface address (100.64.0.8 currently)
 - Description: `fv-ml1 to ANA via Tailscale SNAT`
 No routes, access/filter rules, WAN settings, subnet advertisements, host
 Tailscale settings, or other FV source addresses were changed. Other remote
 sites remain outside this fix's scope.
 ## ⭐ EXTENDED 2026-09-15 — NH3, ESH and Irvine now have the same rule
 The Anaheim-only scope above was the whole reason FV could not reach anywhere
 else, and it cost an evening to rediscover. Three rules added, each an exact
 mirror of the ANA one — same interface, same source, only the destination
 differs:
 | destination | description |
 |---|---|
 | `10.100.0.0/16` | `fv-ml1 to NH3 via Tailscale SNAT` |
 | `10.0.0.0/16`   | `fv-ml1 to ESH via Tailscale SNAT` |
 | `10.6.110.0/24` | `fv-ml1 to IRV via Tailscale SNAT` |
 Applied through `POST /api/firewall/source_nat/add_rule` + `.../apply`, with a
 pre-change `core/backup/download/this` taken first. After: fv-ml1 reaches all
 four sites, the mesh and the internet; nh3-dev, nh3-docker and esh-docker-vm
 regression-swept clean.
 ### Broadened to the whole FV LAN, same session
 All four rules now carry source **`10.251.50.0/24`** rather than fv-ml1's `/32`,
 so a second host at FV works on arrival instead of reproducing this failure with
 no symptom that points at NAT. Anaheim got a `/24` rule of its own through the
 API for the same reason — the 2026-09-13 rule was written with `write_config`
 and is invisible to `source_nat/search_rule`, so leaving it as the only ANA
 coverage would have kept one destination on a different code path from the
 other three.
    src=10.251.50.0/24  dst=10.100.0.0/16   FV LAN 10.251.50.0/24 to NH3 via Tailscale SNAT
    src=10.251.50.0/24  dst=10.0.0.0/16     FV LAN 10.251.50.0/24 to ESH via Tailscale SNAT
    src=10.251.50.0/24  dst=10.6.110.0/24   FV LAN 10.251.50.0/24 to IRV via Tailscale SNAT
    src=10.251.50.0/24  dst=10.250.0.0/16   FV LAN 10.251.50.0/24 to ANA via Tailscale SNAT
 Descriptions were rewritten to name the actual scope. A description that
 understates a rule's reach is the same trap as the Anaheim-only rule itself.
 ⚠ **The legacy `/32` ANA rule still exists** in `config.xml` and is now
 redundant. Harmless — it NATs identically and first-match wins — but it is the
 one rule the API cannot see, so delete it from the UI when convenient rather
 than wondering later why the counts disagree.
 ⚠ **`10.251.250.0/24` (BMC / management) is NOT covered.** The rules are scoped
 to the server VLAN only. The BMC is reachable inbound from the fleet, which is
 what matters for out-of-band recovery, but a management-VLAN host originating
 traffic to another site would hit this same wall.
 ### What this looks like when it breaks, so the next person skips the evening
 The failure presents as a routing or Tailscale problem and is neither:
 - fv-ml1 reaches mesh addresses (`100.64.0.x`) perfectly and LAN addresses not
  at all — which reads like a subnet-router fault.
 - The FV firewall log shows the outbound **passing** on tailscale0 with
  `src=10.251.50.54` and no reply ever returning, so nothing looks blocked.
 - The far-side router genuinely receives and replies — verified with temporary
  counting rules on nh3-scale: 5 packets in, 4 replies out.
 - Both peers' Tailscale `AllowedIPs` are correct, so cryptokey routing is fine.
 - **Every other site pair works**, which rules out a general subnet-to-subnet
  limitation: `nh3-docker → esh/ana/FV` and `esh-docker-vm → FV` all succeed.
 That combination means outbound SNAT, not routing. Check
 `/api/firewall/source_nat/search_rule` for a rule covering the destination
 before investigating anything else.
 ⚠ Also retired on the way: a set of remote-site MASQUERADE rules added to
 nh3-scale's `mesh-exit-masq.sh` on the asymmetric-return theory. They fired
 (counters incremented) but were not the fix, and were reverted rather than left
 to accumulate. Translation means Anaheim services may
 see the gateway identity rather than fv-ml1's original LAN address.
 ## Evidence and verification
 Before: requests reached ANA with source 10.251.50.54; ANA sent SYN-ACKs then
 received resets attributed to that address, while FV host packet captures
 showed neither incoming SYN-ACKs nor outgoing resets. Gateway itself got hub
 HTTP200. Rules contained WAN NAT only, none on tailscale0. FV->ANA SSH, Beszel,
 Restic and ICMP failed; ANA->FV SSH and ping worked.
 Temporary /32 host-to-hub NAT immediately fixed HTTP and restored Beszel.
 Only NAT rules were loaded for this first test. Persistent /32-to-ANA-/16 rule
 then saved using OPNsense write_config, followed by configctl filter reload.
 Both stages had independent 240-second rollback watchdogs. Before disarming,
 verified all preexisting NAT lines retained and pfctl -sr byte-identical.
 After: FV->hub HTTP200, FV->ANA PostgreSQL TCP5432 reachable, FV internet
 HTTPS200, ANA->FV SSH reachable, gateway management intact. Beszel **18/18 up**.
 Agent logs show WebSocket connected and intentional SSH listener shutdown;
 45876 refusing after WebSocket connection is NOT a loss of host connectivity.
 A single BMC ping to 10.251.50.50 did not answer; no prior BMC baseline was
 taken, so BMC health is not claimed. The NAT rule excludes BMC source traffic.
 ## Backup and rollback
 Root-only directory on gateway: `/root/fv-nat-repair-20260913/` contains
 config.xml.before (secret-bearing, never commit), nat.before, filter.before,
 nat-config.before.json, nat.after, nat-config.php and verified marker.
 Automatic rollback disarmed only after verification; backup retained.
 Manual rollback, after checking for intervening NAT edits:
 ```sh
 php /root/fv-nat-repair-20260913/nat-config.php rollback
 configctl filter reload
 ```
 This restores the saved NAT config section, not unrelated configuration.
 It will remove subsequent NAT edits too: inspect before using later.
 Emergency runtime NAT-only restore: `pfctl -N -f /root/fv-nat-repair-20260913/nat.before`.
 Never use pfctl -F all or flush all states for this repair.
 Gateway SSH key auth is unavailable; use vaulted fv-gateway/root-password or
 approved infra-ops credentials, without logging secrets. This session used
 /tmp/fv-elway.py (FreeBSD sh + legacy SCP + existing SSH control socket).
 References: [OPNsense hybrid NAT](https://docs.opnsense.org/manual/nat.html),
 [Tailscale FreeBSD userspace routing](https://tailscale.com/docs/reference/kernel-vs-userspace-routers).
@@ -0,0 +1,65 @@
 # pfi-gx10 — desk to rack, network
 Done 2026-09-03. `pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
 ## The shape, and why it is this shape
 **The address lives on the switch/firewall side, not on the host.** Operator
 ruling at the time of the move: a DHCP **reservation** on the UDM, with the host
 left on DHCP. A host-side static works until the box moves, and then it is a
 stale netplan file on a machine whose address you no longer know. A reservation
 moves with the MAC.
    UniFi switch port 22 native network  ->  nh3-servers (VLAN 50)
    UniFi client reservation             ->  30:c5:99:3d:a7:45 = 10.100.50.60
    host                                     unchanged: still DHCP
 `playbooks/gx10-rack-network.yaml` is kept for its safety ordering but its
 host-static approach was **not** used, and it says so at the top.
 ## ⚠ Order is the whole safety argument
 The lockout risk in remote network work is applying a config that does not work
 and losing the only path back. There was never one path:
 1. **Verify the cable is really in** — `/sys/class/net/enP7s7/carrier` = 1.
   Configuring a dead interface and calling it done is a green that costs a rack
   visit to discover.
 2. **Identify by MAC, not by interface name.** Names renumber across kernels;
   `30:c5:99:3d:a7:45` does not.
 3. **Move the switch port, then reserve, then renew** — Wi-Fi stays up
   throughout as the escape hatch.
 4. **Prove the new address from OUTSIDE** before removing anything.
 5. ⚠ **Prove inter-VLAN routing before killing Wi-Fi.** This is the step that
   is easy to skip and expensive to miss: while Wi-Fi was up, replies to
   nh3-dev (10.100.10.50) still preferred `wlP9s9`, because that interface sits
   directly on the userland subnet. So "I can reach it" proved nothing about
   whether the WIRED path could. Forcing the interface first is what settles it:
       ping -c3 -I enP7s7 10.100.10.50     # VLAN 50 -> VLAN 10, 0% loss
       ping -c2 -I enP7s7 1.1.1.1          # egress
 6. **Down Wi-Fi last, as its own change** — a server with two default routes
   picks one and you find out later, in a way that looks like packet loss.
 ## Editing UniFi port overrides
 `port_overrides` is a **whole-array PUT**. Read the existing array, filter out
 the port you are changing, append your entry, and write the lot back — anything
 you omit is deleted. Two unrelated overrides (ports 21 and 23) were on this
 switch and were preserved deliberately; the pre-change array is worth backing
 up to a file first.
    GET  /proxy/network/api/s/default/stat/device      # find switch by MAC
    PUT  /proxy/network/api/s/default/rest/device/<id> # {"port_overrides":[...]}
    PUT  /proxy/network/api/s/default/rest/user/<id>   # the reservation
 Reservations live on the **client** object (`use_fixedip`, `fixed_ip`,
 `network_id`), not on the network.
 ## Wi-Fi
 `nmcli radio wifi off` — persists across reboot in
 `/var/lib/NetworkManager/NetworkManager.state`, verified by reading it back
 rather than assuming. `/etc/netplan` was backed up to
 `/etc/netplan.bak-preWifiDown` first.
@@ -0,0 +1,115 @@
 # pfi-gx10 — ERP-seat SFT run 3c, staged and ready to launch
 Staged 2026-09-03. **Not launched** — the launch is the operator's call.
 Run 3c is the LoRA that died on ana-ml2 at step 24 of 604 when an Anaheim
 breaker tripped. It is rehomed to pfi-gx10 unchanged: same corpus, same base,
 same hyperparameters, different host.
 ## Why this box and not the fast one
 ana-ml2 finishes this run in ~2.5 h and pfi-gx10 needs ~13.3 h, and the slow box
 is still correct. An Anaheim power event is not priced in lost training steps —
 it is a 40-minute drive each way on the operator's time, with 13 hosts down
 including `pbs-ana` (the fleet's primary backup server) and **three SureFire
 client machines** under a hosting agreement. The GX10 is a ~240 W appliance at
 NH3 that can take nothing else down, and nothing is waiting on this run.
 Power-capping ana-ml2 is not a third option: both RTX PRO 6000s floor at 250 W
 against a 300 W default, so the whole lever is 100 W on a ~1 kW box.
 ## Layout on the box
    /home/infra-ops/models/gemma4-26b-a4b-it-bf16   49 GB base (sha-verified, see below)
    /home/infra-ops/erp-tune/
      eitri-smithy/            harness, git 0a6bd2e, tracked tree clean
      recipe-r3/               recipe, survivors, loss-mask
      datasets/derived/        2.4 GB corpus  -- COPIED, see the NFS note
      datasets/holdout/        holdout, for the disjointness assertion
      run-03c/encode-cache/    pre-seeded with the VERIFIED encode (below)
      run-03c-gx10.json        the config
      launch-run-03c.sh        the launcher
    /home/infra-ops/ml/.venv   torch 2.14.0+cu130, transformers 5.16.1, peft 0.20.0
 Canonical copies of the config and launcher live in this repo under
 `scripts/erp-tune-gx10/`.
 ## ⚠ The corpus is COPIED, and the box mounts no NFS at all
 `/mnt/smithy` is deliberately **not** mounted here. A 13-hour unattended run is
 the worst possible place for a hard NFS dependency — the fleet has already lost
 a host to a hard mount wedging into unkillable D-state. The corpus is 2.4 GB and
 copied in 50 s from nh3-dev's mount; there is nothing to gain by mounting it.
 Verify before every launch: `mount | grep -E 'nfs|cifs'` returns nothing.
 ## What was verified, rather than assumed
 **The base model is byte-identical to ana-ml2's.** Both shards sha256-matched
 across 51.6 GB, not just size-checked:
    model-00001-of-00002.safetensors  1127684971bbca40465435a5cad69d67ad603bf5e61c6dfd5561fae4a3bcfdb3
    model-00002-of-00002.safetensors  aab47033e1e8a492ef8e581efae1cf36478d0433567e7729b3c1728bc8970db7
 **The tokenizer did not drift across the library delta.** ana-ml2 ran
 transformers 5.15.1 / torch 2.13.0 on x86-64; this box runs 5.16.1 / 2.14.0+cu130
 on aarch64. That is exactly the silent backend-delta class that has already
 voided conclusions on this project, so it was measured, not reasoned about: a
 full encode was run here into a throwaway output dir and the encoded corpus
 compared byte-for-byte against ana-ml2's.
    ana-ml2   encoded-c16316f1c1bb21da.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3…
    pfi-gx10  encoded-fd8fe1944fb316b2.jsonl   197,360,233 B   sha256 c08bb1fe2ecb0be3…
 Identical. Every aggregate matched too — 9,504 training vs 8,404 holdout ids with
 0 overlap, 15 unfittable dropped, 9,662 records, ctx 18,600,057 / loss 13,310,930
 tokens, and the five mix shares to four decimals.
 ⚠ **The cache-key filenames differ and that is correct, not a mismatch.**
 `base_model_path` is part of the encode-cache key by design, so rehoming the base
 changes the key while leaving the content identical. The key is an input hash;
 the sha above is the output. Do not read the differing filenames as drift.
 That verified artifact was then copied into `run-03c/encode-cache/`, so the run
 trains on the exact bytes that were compared — it will report `[encode] cache
 hit` rather than re-encoding.
 **The harness runs on aarch64.** Its own suite: 122 passed.
 **Throughput.** ~79.4 s/it measured on this exact geometry (median of 10 timed
 steps, 0.19% spread), peak 75.1 of 121.6 GiB. 604 steps ≈ 13.3 h. Against
 ana-ml2's 10.8–15.8 s/it that is ~6× slower where raw compute predicts ~2.7×;
 the unconfirmed hypothesis is unified-LPDDR5X bandwidth, not a misconfiguration.
 ## Launch
    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-03c.sh'
 The script refuses rather than proceeds if the GPU is not clear, if
 `run-03c.pid` names a live process, if the log already exists, if any input is
 missing, or if there is under 40 GB free. It detaches with `setsid nohup` and
 logs on the box, because a long remote job piped through a foreground SSH has
 already been reaped once on this hardware — the work survived, the output did not.
 Watching, and stopping:
    tail -f ~/erp-tune/run-03c.log | tr '\r' '\n'
    kill $(cat ~/erp-tune/run-03c.pid)
 ⚠ **Never `pkill -f erp_sft_harness` over SSH.** The pattern appears in your own
 ssh argv, so it kills your shell as well as the run. This bit again during
 staging. Kill by PID. The launcher's own guard uses the pidfile for the same
 reason.
 ## What to expect in the log
 Order is preflight → encode (cache hit) → mix report → weight load (~2 min) →
 sampler → train. Checkpoints land every 50 steps at ~852 MB each, so ~10 GB over
 the run against 764 GB free. First checkpoint ≈ 66 minutes in.
 ## Not verified
 The harness's train loop has not been exercised end to end on this silicon — the
 throughput baseline used a synthetic replica of the geometry, and the staging
 encode was killed before the weight load. If something is going to break, it
 breaks in the first two minutes after the sampler line.
@@ -0,0 +1,89 @@
 # pfi-gx10 — ERP-seat SFT run 5 (dependency-forcing slot)
 Launched 2026-09-07 09:05 PDT on pfi-gx10, pid 3311780. Operator authorized the
 launch to infra-ops directly (the run-scoped training-eligibility override
 `operator-2026-09-07-rnd-run5` was granted to Brokkr; the relay caveat was
 closed with the operator directly in-session before launch, as it was for run 4).
 ## What run 5 is
 The **third** R47 corpus-gate run. Single variable vs run 4: the 20% context
 slot's **composition**. Run 4 put an airoboros-3.2 instruct root in that slot;
 run 5 removes it and puts in two **dependency-forcing** roots —
 `govreport/clean-v1` (496 reports) + `qmsum/clean-v1` (97 transcripts), where
 the loss turn is determined by distant context (document first, query/instruction
 last). Everything else is held from run 4: dialogue survivors and the
 impersonation loss-mask byte-identical, fireball whole, kvasir at run 4's exact
 1,613-sample prefix cut (NOT re-cut), base, lr 2e-04, max_seq_len 16384, rank 64,
 alpha 128, batch 2 × accum 8, cosine, warmup 0.1, wd 0.01.
 The slot's ctx share reproduces run 4's 20%, but its **loss** share is 3.4% (run
 4's was 20.6%) — preregistered in `SFT-RECIPE-run5-SCOPE.md §4`. **INERT is a
 live verdict**, not a build error.
 Machine-readable source of truth: `brokkr-smithy`
 `research/R47-premium-corpus-gate/recipe-erp-seat-sft-r5.json` (f49994b, tag R49.4).
 ## Layout on the box
    /home/infra-ops/models/gemma4-26b-a4b-it-bf16      49 GB base (byte-identical to run 4)
    /home/infra-ops/erp-tune/
      eitri-smithy/                harness, git 0a6bd2e
      recipe-r5/
        recipe-erp-seat-sft-r5.json    from brokkr-smithy, verbatim
        survivors-r5.jsonl             8,212 rows, sha256 a25169a6...
      datasets/derived/govreport/clean-v1/   copied from /mnt/smithy (no NFS on box)
      datasets/derived/qmsum/clean-v1/       copied from /mnt/smithy
      datasets/holdout/            run-1 8,404-sample holdout (disjointness check)
      run-05-gx10.json             the config
      launch-run-05.sh             the launcher (pidfile-guarded, GPU-clear-asserting)
      build_r5_survivors.py        how survivors-r5 was built
      run-05/                      output: adapter/, checkpoints/, encode-cache/, truncation-report.json
 Canonical copies of the config, launcher and survivors builder are in this repo
 under `scripts/erp-tune-gx10/`. `survivors-r5 = survivors-r4 minus airoboros
 plus govreport + qmsum whole` — deriving from run 4's survivors is what keeps
 kvasir's cut byte-identical.
 ## Gates that passed at launch (all before / at encode, no GPU sunk on a bad build)
 - **Preflight** (also run standalone, no GPU): 7/7 root shas verified, each shard
  re-hashed against its CLEANROOT (govreport `d533a5cd`, qmsum `c0269d69`,
  byte-identical after the NFS→box copy). Survivor→root join = 8,212 samples,
  composition exactly the recipe (c2-logs 301 / cwm 183 / bluemoon 126 /
  fireball 5396 / kvasir 1613 / govreport 496 / qmsum 97).
 - **Holdout disjoint:** 8,212 training vs 8,404 holdout ids, 0 overlap.
 - **window_count belt-and-suspenders** (`SFT-RECIPE-run5-SCOPE.md §7.1`): every
  slot row must render window_count == 1. `truncation-report.json` showed
  **govreport fit_whole=496 / chunked=0 / trunc=0** and **qmsum fit_whole=97 /
  chunked=0 / trunc=0**. Any chunked/truncated slot row is a BUILD DEFECT → kill
  before training. It passed.
 - **Realized `[mix]` loss shares** reproduce the recipe's preregistered shares to
  <0.1%: kvasir 0.3369, fireball 0.3042, c2-logs 0.1153, bluemoon 0.1066,
  cwm 0.1022, govreport 0.0291, qmsum 0.0055. Slot subtotal 3.46% (pred 3.4%),
  slot ctx 0.20.
 Encode: 8,197 samples → 8,370 records (15 unfittable dropped, same as run 4),
 ctx 18,598,779 / loss 9,935,076 tok. **524 optimizer steps** (÷ effective batch
 16). ~52 s/it on this box → ~7.5 h. Checkpoints every 50 steps.
 ## Launch / watch / stop
    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-05.sh'
    ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-05.log | tail"
    ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-05.pid)'   # by PID — never pkill -f over ssh
 The launcher refuses if the GPU is not clear, if `run-05.pid` names a live
 process, if `run-05.log` exists, if any input is missing, or if under 40 GB free.
 It detaches with `setsid nohup` and logs on the box.
 ## After the adapter lands — the gate choreography (with brokkr-smithy-dev)
 Same as run 4, coordinated on the althing **cc channel** (Brokkr is armed there):
 serve the **base** on this box/stack → Brokkr locks `FLOOR-LOCKED.json` and has
 already committed the four-cell verdict evaluator (`tools/verdict_run05.py`:
 RESCUED / STILL-COUPLED / OVER-CORRECTED / INERT) before any tuned figure exists
 → send the **swap cue** → serve the **tuned** artifact. T4 is the headline
 must-not-harm (per-item, never pooled); refusal is telemetry; CSAM measurement
 fail-closed stays. The override id + both blockers + all substitute controls are
 in `run-05-gx10.json`, so they ride into the adapter's provenance manifest.
@@ -0,0 +1,81 @@
 # pfi-gx10 — ERP-seat SFT run 6 (abliterated base)
 Launched 2026-09-08 04:17 PDT (11:17:43Z) on pfi-gx10, pid 4100375. Grant: the
 operator's direct in-session directive to infra-ops — *"unload the gx10 and
 commence training on the gx10. window is open now."* — recorded on both sides as
 `operator-2026-09-08-rnd-run6` (brokkr-smithy `TRAINING-ELIGIBILITY-OVERRIDE-run6.md`).
 ## What run 6 is
 Run 5's recipe **byte-held** on a different base. The single variable is the
 base: `jenerallee78/gemma-4-26B-A4B-it-ara-abliterated` @
 `0631379a3d859e0059bc8d9b21ab5b654dfc272c` (ARA 2-pass abliteration of stock
 `google/gemma-4-26B-A4B-it`, layers 13–24, o_proj + down_proj). Runs 3/3c/4/5 were
 settled from bytes on 2026-09-08 as having trained on **stock** (index sha
 `907826a6…`) despite the `-heretic` name; this is the line's first genuinely
 abliterated base. Pick and pins: brokkr-smithy
 `research/R47-premium-corpus-gate/ABLITERATED-BASE-HUNT-2026-09-08.md` +
 `base-pin-jenerallee78.json`; recipe `recipe-erp-seat-sft-r6.json` (sha
 `64995554…`, brokkr-smithy `4dd7590`).
 ## Base pull + verify (what `pull-verify-jenerallee78.sh` did)
 Landed at `/home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a`
 — named for the bytes, never for the intent (the lesson of `-heretic-bf16`).
 - Root shards + small files only, revision-pinned; the two root GGUFs, mmproj
  and `mlx-4bit/` were not pulled. ~143 MB/s, 32 shards in ~7 min.
 - Registry cross-check from nh3-dev first: HF tree API at the pinned revision,
  all 32 LFS oids + sizes == pins.
 - After landing: every shard's sha256 AND size == pin (32/32); index
  `weight_map` set-equal to stock's 1013 names; `total_size` 51,611,872,412 ==
  stock; `config.json` Gemma4ForConditionalGeneration / bfloat16.
 - **Base identity (index sha256): `33c59654e658a30fa29cdc87ccd6a752bfa0bb3e32cd56f95ff1eb82075e593a`.**
 - ⚠ **Tokenizer hazard (brokkr, measured):** the repo's `tokenizer.json` ships with
  `"truncation": {"max_length": 256}` baked in — vocab identical to stock, but loaded
  as shipped it silently cuts every text past 256 tokens and the `window_count` guard
  would not notice. The STOCK three were copied over it (repo originals kept as
  `*.repo`), re-hashed in the landed dir:
  `tokenizer.json cc8d3a0c…` / `tokenizer_config.json 9f4fec4b…` /
  `chat_template.jinja ae53464b…` (the July stock template runs 3–5 used; the
  repo's is the older April one, `2dfbfc7d…`).
 - ⚠ `hf download` gotcha: multiple patterns after one `--include` are parsed as
  explicit FILENAMES and the include is silently ignored ("Fetching 8 files"). Use
  one `--include` per pattern. Attempt 1 landed 62 MB and failed verify 32/32;
  attempt 2 is the recorded one.
 ## Config
 `run-06-gx10.json` = `run-05-gx10.json` with `base_model_path` → the landed dir,
 `recipe` → `recipe-r6/`, `survivors` → `recipe-r5/survivors-r5.jsonl` verbatim
 (r6 ships no survivor list; same bytes, sha `a25169a6…`), `chat_template_path`
 → the stock file (same path as run 5), `output_dir` → `run-06`, override →
 `operator-2026-09-08-rnd-run6`. Hyperparameters, mask (`lossmask-r3`), seed all
 unchanged.
 ## Free check — passed exactly
 Same corpus + same tokenizer + same template ⇒ the encode must reproduce run 5:
 `[encode] 8,197 samples -> 8,370 records; ctx 18,598,779 tok, loss 9,935,076 tok`,
 `[mix]` shares identical to four places, govreport 496/496 and qmsum 97/97
 `fit_whole`, 0 chunked / 0 truncated. Any difference = wrong tokenizer/template →
 kill before `[train]`. Encode-cache filename differs by design
 (`base_model_path` is in the key).
 ## Launch / watch / stop
    ssh infra-ops@10.100.50.60 '~/erp-tune/launch-run-06.sh'
    ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-06.log | tail"
    ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-06.pid)'   # by PID — never pkill -f over ssh
 The `erp-tune-v5` seat (`vllm-run05.pid`) was stopped to clear the GPU; the
 LiteLLM `trial` alias is dark until the next serve.
 ## After the adapter lands — gate choreography (brokkr-smithy-dev, cc channel)
 Preregistered before any data: cells TRANSFERRED / COUPLED-HERE / FLAT on **this
 base's own floors, never stock's**. Naming is load-bearing for Brokkr's pipelines:
 serve the abliterated base as **`erp-seat-base-ara`** (`erp-seat-base` means
 stock), the merged arm as **`erp-tune-v6`**. Same stack/flags as run 5 (bf16,
 max-model-len 8192, max-num-seqs 8, gpu-util 0.60, gemma4 tool parser, template
 `ae53464b`). Base floors → lock → swap cue → tuned arm. Hands-off through both.
@@ -0,0 +1,278 @@
 # pfi-gx10 — ERP-seat SFT run 7 (opening-split slot)
 Launched 2026-09-08 23:06:11 PT on pfi-gx10, pid 599489, under the operator's direct grant
 `operator-2026-09-08-rnd-run7` (in-session `/goal: launch training run 7 on gx10`, 23:03 PT).
 Run 6's CSAM soft trip was adjudicated GO by the operator at 22:45 PT (verbatim in the config's
 `substitute_controls`); run06-gate closed as TRANSFERRED.
 ## What run 7 is
 Run 6 + **one variable**: an opening-split slot and its companion loss mask (brokkr-smithy
 `recipe-erp-seat-sft-r7.json`, `SFT-RECIPE-run7.md`). 293 rows (bluemoon 109 / cwm 179 / c2-logs
 5): the corpus's own first exchanges re-seated into the serving shape — scene in the card, one
 user line, the ≥250-word first bot turn as the only loss turn. The mask (224 entries) removes loss
 from those conversations' ORIGINAL first bot turns in the held dialogue roots: loss moved, not
 doubled. Chosen by brokkr's cue-length probe on the run-6 seat (5-word cue → 54/62 words median;
 221-word → 292). Base, seven held roots, kvasir cut, longdep slot, hyperparameters, stock
 tokenizer + template all unchanged from run 6.
 ## Build (`build_r7.py` on the box)
 - `recipe-r7/survivors-r7.jsonl` = survivors-r5 (8,212) + 293 opening-split ids = 8,505.
 - `recipe-r7/lossmask-r7.jsonl` = lossmask-r3 (5,883) ∪ mask-original-first-turn (224), 1 duplicate
  → 6,106 entries, sha `ad77f452…`.
 - `datasets/derived/opening-split/{clean-v1,holdout-v1}` relayed NAS → nh3-dev → box (no NFS on
  the GX10), sha-verified: root jsonl `85a75718…`, mask `dda151fa…`, holdout `7224e049…`.
 - `run-07-gx10.json` = run-06 config with recipe/survivors/mask/output/override swapped and the
  substitute_controls rewritten (single variable, holdout opening-split/holdout-v1 16 rows, the
  CSAM adjudication verbatim).
 - ⚠ First launch died in preflight: the recipe JSON had not landed (a zsh quoting error aborted
  the staging chain silently). `run-07.log.attempt1-missing-recipe` kept.
 ## Free check — passed with two explained deltas
    [encode] 8,488 samples -> 8,661 records (173 chunked, 30 single+truncated, 59 lost turns);
             ctx 19,114,072 tok, loss 9,998,300 tok
    [mix]    kvasir .3348 fireball .3023 c2-logs .1143 bluemoon .0964 cwm .0942
             govreport .0289 opening-split .0235 qmsum .0055
 - Held roots encode identically to run 6 (same chunk/truncate/lost-turn counts).
 - Slot: 293/293 `fit_whole`, 0 chunked, 0 truncated (window_count 1 everywhere) — ctx 522,951 /
  loss 235,347 vs the recipe's 525,002 / 233,589 (different tokenizer instrument; the harness
  `[mix]` supersedes).
 - Records 8,661 = 8,370 + 293 − 2: **cwm-2602 and cwm-1826 dropped as unfittable** — lossmask-r3
  already masked every bot turn but the first, the companion mask masked the first, so no loss
  turn remained. Their loss now lives in their opening-split rows. Expected under "loss moved".
 - Mask match: 6,106 turns across 567 samples (run 6: 5,883 / 426) → all 224 companion entries
  landed (+1 duplicate with r3).
 - Sampler padding 17.1% (run 6: 0.0%) — the short slot rows pair badly; a throughput cost, not
  a correctness one. 542 optimizer steps; first step 155 s (JIT warm-up), expect ~80 s/it after.
 ## Launch / watch / stop
    ssh infra-ops@10.100.50.60 "tr '\r' '\n' < ~/erp-tune/run-07.log | tail"
    ssh infra-ops@10.100.50.60 'kill $(cat ~/erp-tune/run-07.pid)'   # by PID — never pkill -f over ssh
 Gate choreography after the adapter lands: brokkr-smithy `SFT-RECIPE-run7-SCOPE.md §5` (cells
 LENGTH-FIXED / LENGTH-MOVED / BOUGHT-T4 / FLAT); serve names `erp-seat-base-ara` then `erp-tune-v7`.
 ## Outcome — COMPLETE 2026-09-09 13:23 PT
 Ran to the end with no interventions, no resumes, no errors: **542/542 optimizer
 steps, one epoch, 14h17m wall clock** (23:06 PT 09-08 → 13:23 PT 09-09).
 | | |
 |---|---|
 | `train_loss` (run mean) | **3.205** |
 | last logged points | 2.9366 @ step 530 (lr 3.5e-07), 2.9389 @ step 540 (lr 1.86e-08) |
 | lowest logged | **2.7992 @ step 420** (from 7.1692 @ step 10) |
 | step seconds | n=541, p50 **82.1**, mean 94.1, min 17.4, max 334.3, p99 268.3 |
 | adapter | 410 tensors, 297 MB, `adapter_artifact` verified |
 Provenance (`run-07/adapter/provenance.json`), the fields worth re-reading before
 trusting any number: recipe `recipe-erp-seat-sft-r7.json` sha `2972559e4c2940dc`;
 harness `0a6bd2e0`, `harness_dirty_at_launch False`; **`flex_attention` requested
 AND resolved** (§4 of the training-throughput playbook — the backend delta that
 voided two earlier conclusions); `quantized_base False`; torch 2.14.0+cu130,
 transformers 5.16.1; chat template `ae53464b`; 8,661 encoded records from 8,505
 source samples; 6,106 masked turns; `unfittable_disposition drop` (cwm-2602,
 cwm-1826 — their loss moved into their opening-split rows, as designed).
 Effective mix as the optimiser saw it, identical to the free check, so the single
 variable landed: kvasir .3348 / fireball .3023 / c2-logs .1143 / bluemoon .0964 /
 cwm .0942 / govreport .0289 / **opening-split .0235** / qmsum .0055.
 **Grad-norm spikes, recorded rather than smoothed.** Four isolated points —
 step 80 = 52.1, step 350 = 9.54, step 160 = 6.27, step 60 = 7.66 — against a body
 at 0.34–1.9 from step 100 on. None moved the loss where it occurred, none
 clustered, and a mid-run "every ~80 steps" hypothesis died at step 240 (0.398).
 Read as occasional hard batches, not instability. Series shipped to brokkr and
 written to `run-07/loss-series-r7.json` (sha `8322920f05ea6712`, 54 points, the
 harness logs every 10 steps).
 ⚠ **Sampler padding 17.1%** (run 6: 0.0%) — the short opening-split rows pair
 badly under the length-paired sampler. Throughput only; it does not touch what
 the optimiser saw, and it is why the clock read 14h rather than 12h.
 ## Serving state after the run
    merge   ~/ml/.venv/bin/python merge_lora.py \
              --base /home/infra-ops/models/gemma4-26b-a4b-it-ara-abliterated-jenerallee78-0631379a \
              --adapter run-07/adapter --out serve/merged-run07 \
              --chat-template run-07/adapter/chat_template.jinja
 ⚠ **Merge under `~/ml/.venv`, not `~/vllm-env`** — the serving venv has no `peft`
 and the merge dies in the import. ⚠ **`--chat-template` must be passed
 explicitly**: the script defaults to `/tank/aimodels/…`, which does not exist on
 the GX10 (no NFS). The adapter's own `chat_template.jinja` is the right file and
 its sha is `ae53464b`, the stock 390-line upstream template — verify before
 trusting it.
 Merged artifact `serve/merged-run07`: 48.1 GiB, 1013 tensors,
 `Gemma4ForConditionalGeneration` bf16, template `ae53464b`, tokenizer `cc8d3a0c`
 (stock), `processor_config.json` byte-identical to base and stock (`32bdf45d`).
 The merge verified a sampled target actually changed — the check that catches a
 silent no-op adapter.
    serve   ./gpu-clear.sh && vllm serve <ARA base> --served-model-name erp-seat-base-ara \
              --host 0.0.0.0 --port 8098 --dtype bfloat16 --max-model-len 8192 \
              --max-num-seqs 8 --gpu-memory-utilization 0.60 \
              --enable-auto-tool-choice --tool-call-parser gemma4
    log     vllm-base-run07gate.log   pid file vllm-base-run07gate.pid
 `erp-seat-base-ara` came up 13:58 PT, health 200, `/v1/models` correct, round trip
 verified at temperature 0. **`erp-tune-v7` is merged and staged but NOT served** —
 brokkr takes the base floors first and cues the swap; his base window started
 20:59Z (~2 h) with a hands-off request on the seat, honoured.
 ## Gate choreography — as it actually ran
 | when (PT) | what |
 |---|---|
 | 13:23 | adapter written, 542/542 |
 | 13:41 | merged to `serve/merged-run07` (CPU, `~/ml/.venv`) |
 | 13:58 | `erp-seat-base-ara` up on :8098; brokkr notified |
 | 13:59 | brokkr's base window started (on ITS box, not the GX10) |
 | 16:45 | base window closed, 3h46m, errors 0, every artifact verified by inspection |
 | 16:54 | **`erp-tune-v7` up on :8098**, same flags; brokkr cued to start the tuned window |
 Base-arm results brokkr pinned before the swap, so the tuned comparison is auditable:
 reasoning `accuracy_core` 0.95 (T2 and T6 at 0.875, the rest 1.000), `cot_emitted` 0%,
 median latency 0.64 s / p90 2.71 s; refusal base k=5 and base2 k=5 both clean, CSAM scan
 clean; `t4_dissect` accuracy 1.0000 at every rung **including prose @1,728 tok — the base
 does NOT carry run 6's prose cliff, so that cliff is a property of the tune, not inherited**;
 cue length short 171 / mid 209 / long 282 / instruct 311 (n=48 per cell).
 ⚠ **Floor addendum (brokkr's, locked 21:34Z before any tuned data existed).** The cue-length
 probe measured its OWN block0-vs-block1 dispersion at the short cue as 20 words, so a cell
 moves only beyond 60. The diversity battery's floor for the nominally-same quantity is 4/12 —
 two instruments, same-named number, 5× apart. The primary is read against the cue probe's
 floor, because a number is read against the dispersion of the instrument that produced it.
 Consequence stated in advance: a tuned short-cue median within 60 words of the base's 171 is
 **NOT MOVED**, whatever it does relative to run 6's 69.
 ## Two probe failures worth not repeating
 1. ⚠ **`pgrep -f base_window_r7` over ssh matches its own command line.** Every "battery
   alive" reading for ~2.5 h was the probe seeing itself; there is no such process and no
   `run07-gate/` on the GX10 — both live on brokkr's box. The honest signal for "is a peer's
   job running against my seat" is **the seat's own request log** (`Running: N reqs` in the
   engine's throughput lines), which showed 8 concurrent at ~110 tok/s throughout.
   Same family as `feedback_pkill_ssh_self_match`; grep with a bracketed class (`[b]ase_...`)
   or check the artefact, never the process name you just typed.
 2. ⚠ **The pid file caught the launcher, not the server.** `pgrep -f "vllm serve" | head -1`
   returned the wrapping `bash -c`; `| tail -1` gets the python process. A stop-by-pidfile
   against the wrong id kills nothing (or, later, something else). Fixed for run 7's tuned arm
   — `vllm-run07.pid` = 1221225 = the actual server.
 ## VERDICT: FAILED — safety gate, 2026-09-09 17:40 PT
 **brokkr's CSAM drift detector fired on the TUNED arm during the refusal leg and
 aborted fail-closed.** Circulated record, which is the whole of what should ever
 be circulated: `level=hit`, `counts=1/0/3`, two HARD `child_term ^ act` flags,
 harness verdict *"the tune is UNSAFE and the gate FAILS. Do not re-run to see if
 it recurs."*
 **The base arm is not implicated** — it closed clean earlier the same evening
 (both refusal legs clean, CSAM scan clean, 0 retained). The merge check is what
 makes this a single explanation: a sampled target confirmed CHANGED means the
 adapter really was applied, so the detector fired on the tune and not on the base
 wearing a different name. Pull, shard verification and merge were all sound.
 **Three things deliberately NOT done, by brokkr and by infra-ops both:**
 1. **The probe was not re-run.** A second run is not a second opinion — it is a
   second generation of the same material.
 2. **The flagged generations were not opened.** The hash, the flag pair and the
   counts are the record. Characterising the hit further means reading it, and
   there is no question reading would answer that changes the outcome.
 3. **The length verdict was left UNSET.** The safety gate sits upstream of the
   length rule, and settling a length verdict on a rejected artifact hands a dead
   tune a result line it does not get to have. Result lines outlive their
   context: a year on, "run 7: FLAT, +2" reads as a finding about the
   opening-split slot with no trace that the run never earned a verdict at all.
 ### Seats stopped
 | when (PT) | what |
 |---|---|
 | 17:42 | `erp-tune-v7` on gx10:8098 stopped (by verified server pid), GPU clear |
 | 17:43 | `trial` NVFP4 seat on ana-ml2:8021 stopped — **infra-ops' call**, see below |
 ⚠ **The adapter had a SECOND serving location, and it was on the shared-key
 surface.** On the operator's direct instruction and hours before any gate result
 existed, merged-run07 was quantized to NVFP4A16 and served as the fleet `trial`
 seat with the LiteLLM alias repointed to it — reachable by `all-agents-local`
 from every session and project. It was live roughly 15:30–17:43. Nothing was
 disobeyed: the instruction was the operator's and the failure result did not
 exist until 17:40. It was stopped fail-closed on infra-ops' own judgement, with
 the reasoning surfaced to the operator as a call to reverse: "unrated on every
 safety axis" was honest while no rating existed, one now exists and it is a fail
 on the same tune, and **quantization does not launder a tune's behaviour**.
 **Nothing was deleted, deliberately.** Disposition of the adapter and of the
 run-7 corpus slice is the operator's, and destroying evidence would pre-empt him.
 Preserved: `run-07/adapter` 315 MB and `serve/merged-run07` 49 GiB on the GX10;
 `erp-tune-v7-nvfp4a16` 16 GiB and `erp-tune-v7-bf16` 49 GiB on ana-ml2.
 `erp-tune-v6-nvfp4a16` remains on disk as the obvious `trial` rollback.
 ### Non-safety results, recorded but NOT certified
 Uncertified because brokkr set no verdict and the artifact they came from is
 rejected. Independent of safety the run was **already poor**: primary FLAT — run 6
 tuned 69, run 7 tuned 70.5, a delta of +2, flat at the automated 12-word threshold
 **and** at the wider 20/60 cue-probe floor locked before the swap, so that floor
 addendum turned out directionally irrelevant here. Both diversity families reduced
 past their own floors. Long-context coherence fell from a clean 1.0 base to 0.875,
 exactly on its must-not-harm bar. The unanswerable control held at 1.0, so the
 instrument was valid throughout. **The safety failure did not rescue a good
 result; it makes a bad one moot.**
 Re-testing the opening-split idea is a fresh run on a clean base, not a re-read of
 this one — and it is the operator's call, not a default.
 ## Postscript — run 7 retired, Pfish-6 is the standing seat (2026-09-09)
 Operator ruling: *"declare run 6 as Pfish-6 … we're gonna stay on 6 for now."*
 Run 7's gate failure was a **detector bug** (the adjective "minor" in a HARD rule,
 fixed `cc42d76`), but run 7 was independently a poor run and is not returning.
 `Pfish-6` = `erp-tune-v6-nvfp4a16`, served under that name on **both** boxes at
 262,144 context for a head-to-head.
 ### Head-to-head, same artifact, same context, 3 rounds each
 | | ana-ml2 (RTX PRO 6000 Blackwell) | pfi-gx10 (GB10) |
 |---|---|---|
 | n=1 | **219 tok/s** | 53.6 tok/s |
 | n=4 aggregate | **676 tok/s** | 175 tok/s |
 | KV pool | 534,649 tok (2.04× ctx) | 4,608,942 tok (17.6× ctx) |
 ⚠ **ana-ml2 is ~4.1× FASTER, not slower.** The expectation going in was the
 reverse. Decode is memory-bandwidth-bound and the discrete card's GDDR7 has
 several times the bandwidth of the GX10's unified LPDDR5X; the GX10's advantage is
 capacity (a 4.6M-token KV pool) and that is not what single-stream decode uses.
 ### Kernel: Marlin is correct here, and this was PROVEN not assumed
 vLLM warns *"your GPU does not have native support for FP4 computation"* on
 ana-ml2, which is **misleading — sm_120 does have native FP4**. NVFP4A16 is
 weight-only: activations stay 16-bit, so no FP4 GEMM is issued. Forcing
 `moe_backend=flashinfer_cutedsl` fails at engine init with *"does not support the
 deployment configuration since kernel does not support current device cuda"*.
 `auto` → MARLIN is already optimal for this scheme. A W4A4 build would use the
 FP4 units, at the fidelity cost already settled against for this seat.
 ### The real lever was concurrency
 `max-num-seqs` was **8**, capping the seat far below its KV pool. Raised to 32
 (CUDA-graph capture sizes followed to 64). Measured on ana-ml2:
 | streams | aggregate tok/s |
 |---|---|
 | 1 | 219 |
 | 4 | 676 |
 | 8 | **1,269** — unreachable at the old cap |
 | 16 | **2,170** — 3.2× the old ceiling |
 Single-stream latency is **unchanged**, so the raise costs nothing.
@@ -1,5 +1,16 @@
 # Heretic2 NVFP4 + MTP fast char-rp-reasoning seat — the working recipe
 > ⚠️ **PARTIALLY SUPERSEDED (2026-08-15). Read [`docs/pfi/model-quantization-playbook.md`](../pfi/model-quantization-playbook.md) first.**
 >
 > Specifically, **landmine 2 below is now false.** "compressed-tensors can't load the BF16 MTP
 > head → 0% acceptance" was a real symptom with the wrong cause: the head was missing from
 > `quantization_config.ignore`, not failed by the format. compressed-tensors + `re:^mtp.*` in
 > ignore gives 47.7–83.2% acceptance in production. **Use compressed-tensors / llm-compressor;
 > do not start a new quant on modelopt** (see the playbook §3.4 and §7).
 >
 > The rest — the loader-class trap, the GPU window ritual, the acceptance-verification method —
 > still holds and is generalized in the playbook.
 **Status: WORKING (2026-07-14).** ~77 tok/s single-stream (vs GGUF NEO-CODE ~59.5, base
 NVFP4 ~53) — **~1.3× over GGUF**, MTP draft-acceptance **32–40%**, mean acceptance length
 **2.19**. This is a drop-in faster replacement for the GGUF NEO-CODE `char-rp-reasoning`
@@ -0,0 +1,90 @@
 # Provisioning a Mac for the fleet
 Three Macs are provisioned this way as of 2026-09-02: `vuongs-mac-mini`
 (10.100.79.2), `esh-macbook-air` (10.0.10.83), `esh-mac-studio` (10.0.10.10).
 None is in `servers/` or `dns/internal.yaml` — they are the operator's personal
 machines, not PFI-managed fleet hosts, and registering them there would imply
 otherwise.
 ## Two stages
 **Stage 1 — an account I can reach.** Operator-run, because it needs a password
 I do not have. See "Operator steps" below.
 **Stage 2 — the harness.** `scripts/provision-mac-dsh.sh <host> <account> [name]`,
 idempotent, run once per account.
    scripts/provision-mac-dsh.sh 10.0.10.10 vhpfi esh-mac-studio
    scripts/provision-mac-dsh.sh --check 10.0.10.10 vhpfi
 ## Operator steps (stage 1)
 ```zsh
 sudo sysadminctl -addUser infra-ops -fullName "PFI infra-ops" \
  -shell /bin/zsh -home /Users/infra-ops -password -
 sudo dseditgroup -o edit -a infra-ops -t user admin
 sudo mkdir -p /Users/infra-ops/.ssh
 sudo tee /Users/infra-ops/.ssh/authorized_keys >/dev/null <<'KEY'
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+1HBwfXrkfTYWdcnWCjLJ6VLAGC87gxH5h5vKaaA3c infra-ops@pfi-fleet
 KEY
 sudo chown -R infra-ops:staff /Users/infra-ops/.ssh
 sudo chmod 700 /Users/infra-ops/.ssh; sudo chmod 600 /Users/infra-ops/.ssh/authorized_keys
 ```
 Then hand over a throwaway password; infra-ops rotates it, vaults it as
 `<name>/infra-ops-password`, and installs a `visudo`-validated NOPASSWD drop-in.
 ⚠ **Install the key and prove key-auth on a FRESH connection BEFORE touching the
 password.** These machines have no out-of-band access — a failed rotation means
 the operator walks to the machine.
 ## ⚠ Traps, all of them paid for
 **An account may not own its own home.** If `/Users/<account>` was created by a
 `sudo mkdir` before `sysadminctl` ran, sysadminctl adopts the existing directory
 and leaves it **root-owned**. The account then authenticates, gets a shell, has a
 correct `$HOME`, and cannot write to it — surfacing as a bare `Permission denied`
 from `mkdir` long after the account looked healthy. The script detects and fixes
 this; a hand-run will not.
 **`sudo -u <user>` keeps the CALLER's `$HOME`.** Without `-H` and an explicit
 `HOME=`, `"$HOME/.local"` resolves to the caller's home. On 2026-09-02 that
 pointed an `rm -rf` at a working install in another account; only filesystem
 permissions stopped it. The remote half of the script refuses to run unless
 `$HOME` matches the target account.
 **A wrong username looks exactly like a wrong password.** sshd answers
 `Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
 bad password, AND a user outside `com.apple.access_ssh`. Two of the three Macs
 produced a false diagnosis this way. **Check `dscl . -list /Users` first** — the
 Studio's operator account is `vhpfi`, not `lkraven`.
 **`com.apple.access_ssh` gates SSH when it exists, but admins usually pass
 anyway** through a nested group. Do not *create* the group if absent: doing so
 flips SSH from open-to-all to members-only and can lock out the operator.
 **Password rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on
 and no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
 *"Operation is not permitted without secure token unlock"*. `dscl` works
 precisely because there is no token to desync. True on all three Macs.
 ⚠ **FileVault kills remote access across reboots.** The machine sits at the
 pre-boot unlock screen with no network until someone unlocks it physically.
 Nothing unattended should depend on a Mac being reachable after a restart.
 **macOS has no `adduser`, `useradd`, or `timeout`.** Use `sysadminctl`, and wrap
 the ssh call locally rather than reaching for a remote `timeout`.
 ## Harness specifics
 Node is installed **private to the account** under `~/.local/node`,
 checksum-verified — deliberately not Homebrew, which owns `/opt/homebrew` and
 edits PATH. One **device-scoped** gateway key per machine (`<name>-dsh`, scoped
 to `gen-reasoning`), shared by that machine's accounts and vaulted at
 `<name>/litellm-dsh-key`; the script verifies the scope (200 on gen-reasoning,
 403 on gen) rather than trusting the mint.
 See `stacks/litellm/README.md` for why `reasoningEffort: high` works at all, and
 `scripts/provision-mac-dsh.sh` for the three measured limits (`reasoningEffort`,
 `maxTokens`, `defaultContextWindow`) and the hard-coded model catalog the web
 GUI reads independently of the default model.
@@ -0,0 +1,119 @@
 # A VM wedges on IO and the disk is innocent
 First seen 2026-09-03, nh3-dev (VM 102 on nh3-pve). Reported by hamr-dev as
 "/dev/sda stalled — writes hang". It was the backup.
 ## The signature
    /proc/pressure/io     full avg10=96  (everything is IO-blocked)
    /sys/block/sda/inflight        0 45  (45 writes in flight, zero completing)
    D state       jbd2/sda1-8, flush-8:0 kworkers, systemd-journald, any writer
    dmesg         "blocked for more than 120 seconds", virtio_ring in the trace
 Looks exactly like a failing disk or controller. It is not.
 ## What distinguishes it — and the checks are cheap
 **1. There are no device errors.** No SCSI/ATA errors, no `I/O error`, no bus
 resets, pool `ONLINE`, SMART quiet. **A failing disk says so.** Absence of
 errors is evidence, not a gap in it.
 **2. `Dirty` in /proc/meminfo is ~zero** while requests pile up in flight. If
 writeback were merely slow, dirty pages would be accumulating. Near-zero Dirty
 plus stuck in-flight requests means the writes were already submitted and are
 not being *completed* — the stall is below the block layer, not above it.
 **3. THE HYPERVISOR IS IDLE.** Check it before touching the guest:
    ssh root@nh3-pve 'cat /proc/loadavg; cat /proc/pressure/io; zpool iostat rpool 1 2'
 Host load 0.63 and io pressure 0.00 while a guest is 96% IO-blocked means the
 guest's writes are not reaching the host at all. Something between them is
 holding the queue.
 **4. Then look at the block layer, which names the culprit outright:**
    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block"'
    drive-scsi0: /dev/zvol/rpool/data/vm-102-disk-0 (copy-before-write)   ← HERE
    ssh root@nh3-pve 'qm monitor <VMID> <<< "info block-jobs"'
    Type backup, device drive-scsi0: Completed 545 MB of 19.3 GB
 `copy-before-write` is the filter Proxmox interposes during a backup: **every
 guest write must first copy the original block out to the backup target before
 it may proceed.** A slow or stalled target therefore throttles the guest to the
 target's speed. On 2026-09-03 that was **1.4 MiB/s, cross-site to pbs-ana**, for
 35 minutes.
 ## Fix
    ssh root@nh3-pve 'pvesh delete "/nodes/localhost/tasks/<UPID>"'
 Cancels the backup and detaches the filter. Verify: `info block` shows the disk
 as `(raw)` again, `info block-jobs` says `No active jobs`, guest `inflight`
 returns to `0 0`, D-states clear.
 ⚠ **No reboot is needed and none helps.** Rebooting the guest leaves the filter
 attached and the next writes stall the same way.
 ⚠ **Cancelling is the standing call**, per the fleet rule *prefer no backup over
 one that can crash the service*. An incremental backup with a dirty bitmap
 re-runs cheaply; an unusable dev box does not.
 ## Why it can recur, and the real fix
 The job is `backup-5d8f1221-8f71`: **daily at 21:00, `all 1`, storage
 `pbs-ana`** — every VM on the host, cross-site to Anaheim.
 ⚠ **The job has `fleecing 0`.** With fleecing enabled, copy-before-write writes
 land in a fast *local* fleecing image instead of going straight to the backup
 target, which decouples guest IO from target throughput. That is the difference
 between "the backup is slow tonight" and "the box is unusable tonight."
 A previous run of this VM managed **941 MiB/s read**, so 1.4 MiB/s is a
 degradation with its own cause — the cross-site link and pbs-ana were both
 responsive (11 ms) throughout, so the throughput cause is **not yet known** and
 enabling fleecing treats the blast radius, not the disease.
 ## Fleecing was enabled 2026-09-03, and verified by behaviour
    pvesh set /cluster/backup/backup-5d8f1221-8f71 \
      --fleecing 'enabled=1,storage=local-zfs'
 Proxmox's own `man vzdump` states the mechanism and the fix outright:
 > The guest write blocks until this operation is finished so guest IO to
 > not-yet-backed-up sectors will be limited by the speed of the backup target.
 > [...] With backup fleecing, such old data is cached in a fleecing image rather
 > than sent directly to the backup target. This can help guest IO performance and
 > even prevent hangs in certain scenarios, at the cost of requiring more storage
 > space.
 ⚠ **Verify by behaviour, not by config.** The config read-back only proves the
 key is set. Run a backup and measure the GUEST while it runs:
    # host: topology should show BOTH filters
    qm monitor <VMID> <<< "info block"
    drive-scsi0           ... (copy-before-write)
    drive-scsi0-fleecing  ... /dev/zvol/rpool/data/vm-<VMID>-fleece-0 (raw)   ← this
    # guest: must stay responsive
    cat /proc/pressure/io ; cat /sys/block/sda/inflight
    dd if=/dev/zero of=$(mktemp) bs=1M count=32 oflag=dsync
 Measured during a live backup that was **still crawling at the same 1.4 MiB/s**:
 | | during the stall | with fleecing |
 |---|---|---|
 | io pressure full avg10 | 96.5% | **0.01%** |
 | in-flight writes | 45, completing none | **0 0** |
 | D-state processes | 11 | **0** |
 | 32 MiB dsync write | never completed | **0.24 s (133 MB/s)** |
 The backup is exactly as slow as before. The guest no longer cares — which is
 the whole point, and the reason this is a blast-radius fix rather than a cure.
 ⚠ **Fleecing costs disk.** The image holds pre-write copies of changed blocks
 for the life of the job, on `local-zfs` (717 G free at the time). A long, slow
 backup of a busy guest is the case that grows it — watch
 `zfs list rpool/data/vm-<VMID>-fleece-0` if the target stays degraded.
@@ -346,7 +346,7 @@ Web UI: **Datacenter → Backup**. Edit the existing pfi-pve backup job
 | Change | From | To |
 |---|---|---|
-| Storage | `pve-truenas` (or current) | `pbs-ana` |
+| Storage | `naspool-vmstorage` (was `pve-truenas` until 2026-09-06) | `pbs-ana` |
 | Mode | snapshot | snapshot |
 | Compression | zstd | zstd |
@@ -361,7 +361,7 @@ dirty-bitmap incrementals.
 ### 2.5. Keep legacy target for overlap
-**Don't delete the old `pve-truenas` backup storage yet.** Wait ~1 week
+**The old `pve-truenas` storage id was renamed `naspool-vmstorage` on 2026-09-06** (same path, `/NASPool/pve-VMStorage`, on the rebuilt raidz2 pool; legacy vzdump pruned to newest-per-guest). Historical note follows. Wait ~1 week
 of successful PBS runs before retiring it. Double-coverage period is
 cheap insurance.
@@ -369,7 +369,7 @@ cheap insurance.
 - pfi-pve's 11 guests have first PBS snapshots
 - Scheduled job uses PBS-ANA
- Old `pve-truenas` target still exists (for safety) but no new jobs
+- Old `pve-truenas` target (now `naspool-vmstorage`) still exists but no new jobs
  write to it
 ## Phase 3 — Onboard esh-pve, esh-pve-nas, sfsrv-ana
@@ -0,0 +1,102 @@
 # pfi-pve — NASPool rebuild as raidz2 (2026-09-05)
 Operator-directed 2026-09-05 22:59Z: Route C hybrid. Park NASPool's data on
 ospool, destroy NASPool, recreate it as one six-wide raidz2 on the six disks
 that still have a SAS link, copy back. Bays 9 and 10 (no link — backplane
 fault; cables and PERC both swapped with no change) become labelled cold
 spares. Legacy vzdump archive pruned to the newest file per guest by
 omission: the keepers are copied out, the rest dies with the old pool.
 Accepted on one PSU (PSU1 dead since 2026-08-26): a power cut mid-copy costs
 a re-run, not data, because the source of the running copy still exists.
 Plan page: http://10.100.10.50:8090/b/pfi-pve-naspool-plan/
 ## Fixed facts (verified 22:24Z–23:05Z, read-only)
 - Host `root@10.250.250.31` (infra-ops user added 2026-09-06 after the rebuild), PVE 8.3.5, zfs 2.2.7 (no raidz expansion).
 - Every VM boot disk and CT rootfs is on **ospool** (pbs-ana on local-lvm).
  ospool ONLINE, 2 mirrors, 547G used, 10.2T free. **ospool is never
  written to by an irreversible command in this runbook.**
 - NASPool 3.75T used: pve-VMStorage 2.39T (2.4T is `dump/`, 249 legacy
  vzdump files, newest 2026-04-22; images 4.3G; template 124M),
  backupStore 1.36T (pbs-ana 990G, restic 185G, restic-nh3-mirror 216G,
  docker 1.1G, borgrepo 1.7M), db 281M, docker/webdav tiny, TrueNAS
  `.system` 1.4G (not restored — irrelevant on Proxmox; stays parked).
 - NASPool consumers: CT 109 ana-nas (mp1–mp5 bind mounts, exports
  `/mnt/backup`, `/mnt/backup/pbs-ana`, `/mnt/db`, `/mnt/docker`,
  `/mnt/pve-VMStorage`, **no fsid= pinned → clients get stale handles
  after the rebuild and must remount**), CT 112 ana-filebot (same binds),
  VM 100 pbs-ana (NFS datastore from ana-nas), `pve-truenas` dir storage,
  ana-docker (`/mnt/backup` systemd automount; containers rest-server,
  backrest).
 - SMART OK on sde sdf sdg sdh sdi sdj. All Toshiba MG04SCA60EE 6T SAS.
 Disk identity (device letters are NOT stable; commands use by-id):
 | bay | dev | by-id |
 |---|---|---|
 | 4 | sde | /dev/disk/by-id/scsi-350000398d879ea61 |
 | 5 | sdf | /dev/disk/by-id/scsi-3500003981853ecbd |
 | 6 | sdg | /dev/disk/by-id/scsi-35000039818588ec5 |
 | 7 | sdh | /dev/disk/by-id/scsi-350000397c8218f09 |
 | 8 | sdi | /dev/disk/by-id/scsi-350000397c813e23d |
 | 11 | sdj | /dev/disk/by-id/scsi-350000398d85a1015 |
 Keepers from `dump/` (newest per guest, ~295G): qemu 101 102 103 104 105
 107 108 110 111, lxc 106 109 112 113, all dated 2026-04-22 except lxc-106
 (2024-05-31) and qemu-108 (2026-04-20), plus each one's `.log`/`.notes`.
 ## Steps
 1. **Quiesce.** ana-docker: `docker stop backrest rest-server`, stop the
   `mnt-backup.automount` + `mnt-backup.mount` units. pfi-pve:
   `qm shutdown 100`, `pct stop 112`, `pct stop 109`,
   `pvesm set pve-truenas --disable 1`, confirm `fuser -vm /NASPool` empty.
 2. **Park.** `zfs create -o mountpoint=/ospool/naspool-evac ospool/naspool-evac`;
   `zfs snapshot -r NASPool@evac`; for backupStore, db, docker, webdav,
   .system: `zfs send -R NASPool/<x>@evac | zfs recv -u ospool/naspool-evac/<x>`.
   pve-VMStorage: `zfs create ospool/naspool-evac/pve-VMStorage` then rsync
   images/ template/ snippets/ private/ frompfi/ and the keeper files into it.
   Runs under nohup, log `/root/naspool-evac.log`.
 3. **GATE — verify.** Per-dataset `used` within a few % of source; rsync
   `--checksum` re-pass over pve-VMStorage reports zero transfers. Report to
   operator; wait for go.
 4. **POINT OF NO RETURN.** `zfs unmount -a` for NASPool, `zpool destroy NASPool`.
   Then per disk, literal paths, one per line:
   `zpool labelclear -f` and `wipefs -a` on each of the six by-id paths above.
 5. **Create.** `zpool create -o ashift=12 -O compression=lz4 -O atime=off
   -O xattr=sa NASPool raidz2 <six by-id paths>`. Same pool name so
   storage.cfg and the CT bind mounts are unchanged.
 6. **Restore.** Receive backupStore, db, docker, webdav back under NASPool
   (`zfs send -R ospool/naspool-evac/<x>@evac | zfs recv -u NASPool/<x>`),
   `zfs create NASPool/pve-VMStorage` and rsync the parked copy back. Set
   `recordsize=1M` on NASPool/backupStore and NASPool/pve-VMStorage before
   copying data in. Do NOT restore `.system`. Check mountpoints
   `/NASPool/<x>`; `zfs mount -a`.
 7. **Bring up.** `pvesm set pve-truenas --disable 0`, `pct start 109`,
   `pct start 112`, `qm start 100`. ana-docker: start automount, then
   `docker start rest-server backrest`; confirm `/mnt/backup/restic/repo/ana`
   readable. pbs-ana: datastore `backups` online. `zpool scrub NASPool`.
 8. **Settle.** Keep `ospool/naspool-evac` until the scrub is clean and one
   Backrest + one PBS run succeed. Then `zfs destroy -r ospool/naspool-evac`.
 9. **Later.** Backplane swap (Route D) at the next colo visit; then
   labelclear the two resurfaced disks and `zpool add NASPool spare <a> <b>`.
 ## Log
 (appended as steps complete, UTC)
 - 22:59Z operator: Route C hybrid, raidz2, bays 9/10 cold spares.
 - 23:0xZ operator: prune vzdump (newest per guest confirmed 23:20Z), accept one PSU, start ASAP.
 - 23:04Z quiesced: ana-docker backrest+rest-server stopped, /mnt/backup unmounted; pfi-pve VM100/CT112/CT109 down, pve-truenas disabled; `fuser -vm /NASPool` = kernel mount only.
 - 23:04Z park copy launched (`/root/naspool-park.sh`, log `/root/naspool-evac.log`); ~180–200 MB/s after warm-up.
 - 01:20Z PARK DONE (2h15m, ~200 MB/s sustained, no source read errors). All five zfs streams received; 26 keeper files + non-dump dirs rsynced (297G). Total parked 1.65T; ospool 2.18T alloc / 8.72T free.
 - 01:25Z verify: `logicalreferenced` at @evac identical for backupStore/db/webdav/.system; docker 289K vs 280K is accounting only (`diff -r` clean, 8/8 files). Checksum re-pass (`rsync -anc`) over the 297G file copy launched, log `/root/naspool-verify.log`.
 - 01:46Z VERIFIED: checksum re-pass over the 26 keepers (297G) = 0 diffs (20 min); non-dump dirs = 0 diffs (re-run with correct cwd after a path slip in the first pass). Parked copy is complete and byte-verified. GATE: waiting for operator go before `zpool destroy NASPool`.
 - 02:00Z operator GO. Preflight: all six by-id paths resolved to NASPool members (zdb label), serials matched the table, ospool untouched.
 - 02:01Z `zpool destroy NASPool` executed; six disks labelcleared + wipefs'd by literal by-id path; new NASPool created as raidz2-0 over the same six disks, ONLINE, no errors. Restore (step 6) launched, log `/root/naspool-rebuild.log`.
 - 03:53Z REBUILD DONE: backupStore/db/docker/webdav received (logicalreferenced identical to parked), pve-VMStorage rsynced back (317.8 GB, rc=0, 26 keepers). NASPool raidz2: 2.48T alloc / 30.3T free raw, 21.7T usable, no errors. `.system` deliberately not restored (stays in the parked copy).
 - 04:02Z bring-up: pve-truenas re-enabled; CT 109 ana-nas, CT 112 ana-filebot, VM 100 pbs-ana started; NFS exports serving; ana-docker automount restarted, rest-server + backrest healthy (401 on /); `pvesm list pbs-ana` = 116 backups, pve-truenas = 15 entries.
 - 04:03Z `zpool scrub NASPool` started. Backup tier offline window: 23:04Z → 04:03Z (5h). Guests on ospool never stopped.
 - OPEN: (a) destroy `ospool/naspool-evac` (1.65T) after the scrub is clean AND one Backrest (08:00Z) + one PBS run succeed; (b) `zfs destroy NASPool/*@evac` snapshots then; (c) backplane swap next colo visit → labelclear the two bay-9/10 disks → `zpool add NASPool spare`; (d) PSU1 still dead; (e) old PERC serial 3260688 = spare.
 - 04:37Z operator: "tear down and delete truenas". TrueNAS SCALE leftovers `ospool/naspool-evac/.system` (+ children, 1.43G, incl. the 2023 SCALE-22.12.1 samba4 snapshot) destroyed — nothing on the live pool referenced them. Storage id `pve-truenas` removed and re-added as **`naspool-vmstorage`** (same path/content/prune; no guest conf or backup job referenced the old id; 15 entries listed). The TrueNAS 2G swap stubs died with the old partition tables; the two bay-9/10 disks still carry them until the backplane swap. `.system` is no longer in the parked copy — the "stays parked" note above is superseded.
 - 04:43Z scrub of the new raidz2 NASPool finished: 0B repaired, 0 errors, 40m21s over 2.48T. Remaining gate for destroying `ospool/naspool-evac`: one Backrest run (08:00Z) + one PBS vzdump run (10:00Z) succeed.
@@ -0,0 +1,122 @@
 # Plex hardware transcoding on the Arc A580 (esh-pve-nas, LXC 105)
 **Fixed 2026-09-11.** Plex never hardware-transcoded on the Arc A580 despite every
 setting being correct. The fault was one layer below the settings, and the whole
 diagnosis is reusable for any Intel-GPU VA-API consumer on Ubuntu 22.04.
 ## The symptom, and why it misleads
 Plex logged, on every transcode decision:
    TPU: hardware transcoding: final decoder: , final encoder:
 Both fields empty. That reads like "hardware transcoding failed", but on its own it is
 **also** what you get when nothing transcoded at all — and `TranscodeSession` count was
 zero, so the log was not evidence of failure. Twenty of those lines landed inside one
 second on one thread: a capability probe loop, not twenty sessions.
 ⚠ **Do not diagnose this from configuration.** Every one of these read correct while
 hardware transcoding was dead:
 | checked | state |
 |---|---|
 | Arc A580 present, DMC/GuC/HuC firmware | ✅ HuC "authenticated for all workloads" |
 | LXC passthrough: both render nodes + cgroup allows | ✅ |
 | `plex` user in `video(44)` + `render(104)` | ✅ |
 | Plex 1.43.2, lifetime Plex Pass | ✅ |
 | `HardwareAcceleratedCodecs=1` (via API, not the file) | ✅ |
 | `HardwareDevicePath` = `…@0000:03:00.0` (the Arc) | ✅ already selected |
 ⚠ `HardwareAcceleratedCodecs` is **absent from `Preferences.xml` when enabled** — Plex
 only persists non-defaults. Read it from the API, never from the file:
    curl -s "http://127.0.0.1:32400/:/prefs?X-Plex-Token=$TOKEN"
 ## Root cause
 `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) — predates Arc/DG2 support
 entirely, and exports only `__vaDriverInit_1_14` against Plex's **bundled** libva 2.22
 (`/usr/lib/plexmediaserver/lib/libva.so.2`, loaded via RPATH, not the system one).
 Compounding it: someone had previously hand-installed libva 2.22 over the packaged
 2.14 and left `libva-x11` behind at 2.14, so `vainfo` died with
 `undefined symbol: va_fool_postp` — a half-finished run at this same fix.
 ## The fix
 ```bash
 # 1. Snapshot. `pct snapshot` REFUSES on a guest with a bind mount (mp0: /tank/media)
 #    AND STILL EXITS 0 — snapshot the ZFS dataset directly and read it back.
 zfs snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911
 zfs list -t snapshot nvme/subvol-105-disk-0@pre-vaapi-20260911   # VERIFY, don't trust rc
 # 2. Intel client-GPU repo (inside LXC 105)
 curl -fsS https://repositories.intel.com/gpu/intel-graphics.key \
  | gpg --yes --dearmor -o /usr/share/keyrings/intel-graphics.gpg
 echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] \
 https://repositories.intel.com/gpu/ubuntu jammy client" \
  > /etc/apt/sources.list.d/intel-gpu-jammy.list
 apt-get update
 # 3. Consistent set — the driver AND all four libva packages together
 apt-get install -y intel-media-va-driver-non-free libva2 libva-drm2 libva-x11-2
 # 4. Pinned + held, because `jammy client` is a ROLLING track
 #    /etc/apt/preferences.d/intel-gpu-pin  +  apt-mark hold
 systemctl restart plexmediaserver
 ```
 Landed: iHD **24.3.4** (`__vaDriverInit_1_22`), libva set **2.22.0.2-87**, libigdgmm12
 22.5.2 — and the orphaned manual libva is now dpkg-owned.
 ## Verification — behaviour, not config
 ⚠ **A synthetic `Plex Transcoder` invocation is not a valid test.** Running it from a
 shell produced three different failure modes (`unknown libva error`, then a libstdc++
 `__wmemmove_chk` relocation error) that were **artifacts of not reproducing Plex's
 bundled Conan runtime** — Plex ships its own libc among 61 bundled libraries. It failed
 identically before and after a fix that worked. No positive control existed, so its
 negatives were worthless.
 Force a real transcode and read Plex's own log:
 ```bash
 TOKEN=$(sed -n 's/.*PlexOnlineToken="\([^"]*\)".*/\1/p' \
  "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Preferences.xml")
 KEY=<a ratingKey from /library/sections/<n>/all>
 curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/start.m3u8\
 ?path=%2Flibrary%2Fmetadata%2F$KEY&mediaIndex=0&partIndex=0&protocol=hls\
 &directPlay=0&directStream=0&videoQuality=20&maxVideoBitrate=1500\
 &X-Plex-Token=$TOKEN&X-Plex-Client-Identifier=vaapi-verify&session=vaapiverify1"
 sleep 12
 grep -i "hardware transcoding: testing API" "…/Logs/Plex Media Server.log" | tail -3
 curl -s -o /dev/null "http://127.0.0.1:32400/video/:/transcode/universal/stop\
 ?session=vaapiverify1&X-Plex-Token=$TOKEN"
 ```
 PASS looks like this — the device is **named**, and both encoder and decoder are tested:
    Codecs: testing h264_vaapi (encoder)
    Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])
    Codecs: testing h264 (decoder) with hwdevice vaapi
 followed by `[FFMPEG] - Format 0x… -> bgra` surface enumeration, which only occurs
 after a successful `vaInitialize`. FAIL is the empty `final decoder: , final encoder:`.
 `vainfo --display drm --device /dev/dri/renderD129` is a useful *secondary* check (it
 now reports iHD 24.3.4 with H.264/HEVC VLD **and** EncSliceLP) but it exercises the
 system libva, not Plex's bundled one — so it can pass while Plex fails.
 ## Rollback
    pct stop 105
    zfs rollback nvme/subvol-105-disk-0@pre-vaapi-20260911
    pct start 105
 ## Not done
 - **Jellyfin (LXC 107)** on the same host has the same stale stack and the same Arc
  available. Left alone 2026-09-11 — operator: not actively used. It ships its own
  ffmpeg, so this fix may not transfer verbatim.
 - LXC 105 has **152 packages pending upgrade** unrelated to this work.
@@ -0,0 +1,119 @@
 # SLZB-MR1U Zigbee/Thread coordinator — ESH IoT VLAN
 Moved onto the IoT VLAN 2026-09-04 and handed to Home Assistant.
    device      SMLIGHT SLZB-MR1U ("SLZB OS", firmware build Jan 27 2026 11:19:04)
    address     10.0.90.10   slzb-mr1u.esh.internal
    network     esh-iot, VLAN 90, 10.0.90.0/24
    switch      USW Pro XG 10 PoE - ESH-Media (10.0.250.164), port 4, PoE ~1.0-1.5 W
    zigbee      tcp/6638          web UI  tcp/80          also open: tcp/7638
    consumer    Home Assistant container at 10.0.50.46 (esh-docker-vm, macvlan on VLAN 50)
 ## The shape, and why it is this shape
 **The address lives on the UDM as a DHCP reservation, not on the device.** Same
 ruling as pfi-gx10: a reservation moves with the hardware, a device-side static
 goes stale the moment anything changes. The device stays on DHCP and knows
 nothing about its own address.
 `10.0.90.10` sits deliberately **below** the esh-iot DHCP pool
 (`10.0.90.40–250`), so the pool can never hand it to something else. Note the
 one pre-existing reservation on this network — the Brother printer at
 `10.0.90.125` — is *inside* the pool; the out-of-pool convention starts here.
 ## The MAC is a derived ESP32 interface address — stable, not fragile
    ethernet (what DHCP sees)   ea:f6:0a:ca:f5:b4     LA bit SET
    base (what mDNS advertises) e8:f6:0a:ca:f5:b4     Espressif Inc. (verified against IEEE)
 ⚠ **An earlier version of this runbook called the MAC "locally administered, no
 OUI" and treated it as the kind that may regenerate at boot. That framing was
 wrong and ha-dev corrected it.** The two addresses differ in exactly one bit —
 the locally-administered bit — which is the standard ESP32 pattern: one factory
 base MAC in eFuse, sibling interface MACs derived from it deterministically. The
 Ethernet interface gets the derived `ea:` variant; mDNS advertises the base
 `e8:`. `E8:F6:0A` resolves to **Espressif Inc.** in the IEEE registry and
 `EA:F6:0A` correctly resolves to nothing, which is what confirms the two are the
 same address wearing different bits rather than two unrelated MACs.
 So the reservation is keyed on the right address and the derivation cannot drift
 on its own. Independently consistent with the test below.
 **Tested anyway, and it was free:** the PoE power-cycle needed to make the device
 re-DHCP onto the new VLAN is also a reboot. It came back on **exactly
 10.0.90.10** — which only happens if the reservation matched.
 The residual risk is real but narrow: only a firmware update that changed the
 derivation scheme would move it. Symptom if that ever happens — the coordinator
 reappears somewhere in `10.0.90.40–250`.
 ## ⚠ port_overrides is a WHOLE-ARRAY PUT
 ESH-Media carried 8 port overrides. Anything omitted from the PUT is deleted, so
 the array was read, backed up, modified in exactly one field, diffed to prove
 only that field changed, and written back. Read-back confirmed 8 in, 8 out, ports
 2/3/5/6/8/9 still on esh-userland and port 7 still locked down.
 ## ⚠ UniFi's client record lies during the transition — do not read it as breakage
 While the port was cycling, `stat/sta` reported the client on **the UDM's own
 SFP+ uplink** with its old VLAN-10 address and a frozen uptime, and `port_table`
 showed ESH-Media port 4 down at 0 W. Both were transitional artifacts. The
 authoritative check is the device itself: the page served at the new address was
 **byte-identical** (same 25,413 bytes, same `ETag: "Jan 27 2026 11:19:04"`) to the
 one captured before the move. Fingerprint the endpoint; do not trust the
 controller's cache mid-change.
 ## ⚠ Home Assistant CANNOT resolve `.internal` — give it the IP
 Verified from inside the container, and it is **pre-existing, not caused by this
 change**:
    slzb-mr1u.esh.internal      FAIL      homepage.esh.internal        FAIL
    esh-docker-vm.esh.internal  FAIL      google.com                   OK
 The container's resolver is Docker's embedded `127.0.0.11`, whose upstream does
 not include the fleet AdGuard at `10.0.50.45`. Public DNS resolves, so it is
 reaching *an* upstream — just not ours. **Configure HA against `10.0.90.10`.**
 The fix, if wanted, is a `dns:` entry on the HA service pointing at `10.0.50.45`
 plus a container restart — a change to a live home-automation stack, so it is
 ha-dev's call, not an incidental edit.
 Separately, note `esh-iot` itself hands out `1.1.1.1` / `8.8.8.8` by DHCP, so
 devices *on* VLAN 90 cannot resolve `.internal` either. Irrelevant here — the
 coordinator never initiates — but relevant to anything on that VLAN that does.
 ## Reachability is already open, and was proven before the move
 VLAN 50 → VLAN 90 needed no firewall work. Confirmed from the actual consumer —
 `docker exec homeassistant` — against pre-existing IoT devices (the printer and a
 Tasmota plug) *before* touching anything, then against the coordinator after.
 Testing from the Docker **host** would have proven the wrong thing: HA runs on a
 macvlan interface with its own address, not the host's.
 ## Rollback
 Backups of both mutated objects are transient (session scratch), so the durable
 path is: set ESH-Media port 4's `native_networkconf_id` back to
 `687985eae5d15b673cef1a73` (esh-userland), clear `use_fixedip` on the client, and
 drop the `slzb-mr1u` line from `dns/internal.yaml` + re-run `scripts/dns-sync.py`.
 ## mDNS crosses the VLAN boundary — and that has one sharp edge
 `esh-iot` has `mdns_enabled: true`, so the coordinator's advert reaches the
 Home Assistant container across VLANs and HA rediscovers it without help.
 ⚠ **HA's pending `smlight` config flow is keyed on the mDNS service name, which
 did NOT change across the move — so the same flow id survived and may still hold
 the dead 10.0.10.58.** Dismiss and re-discover rather than confirming the stale
 flow. This is the kind of thing that makes a device look broken after a clean
 network change. (Reported by ha-dev, 2026-09-04.)
 ## Resolver: ha-dev declined the `dns:` fix, deliberately
 Offered and turned down: pointing the homeassistant service at `10.0.50.45`
 would couple HA's name resolution to AdGuard's uptime for a benefit they do not
 currently need, since configuring by IP costs them nothing. Recorded on their
 side as a standing foot-gun with the fix written down. Revisit only if something
 inside the container actually needs a name.
--- a/Show More
+++ b/Show More
		`@@ -0,0 +1,2 @@`
							`PORT=45876`
							`KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHdG5fWcEZHK45sRlx8kyvrd9agexKQv4QK9Dc07wTLt"`