memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4

Rewrote the in-flight section to reflect that nothing is running and the operator stood the session down. The GX10 work is recorded as what it was asked to be: a baseline for the box and a check that the tooling loads, with the run-3c port scoped but explicitly declined. Added index lines for four decisions that had detail files but no pointer: the GX10 baseline at 79.35 s/it, the costing error the operator overruled, the althing four-surface deploy finding, and the irv-ml1 GPU resident map. Auto-archival fired at the soft cap and moved 17 entries dated on or before 2026-08-18 to archival-memory.md, holding back 4 that carry open deferred-work pointers. The index went from 410 lines to 296, mostly by rewriting in-flight rather than by archiving -- the dated log was not what made it long.
2026-09-02 01:23:19 -07:00
parent b039aa19e8
commit e39106bd03
6 changed files with 603 additions and 669 deletions
@@ -3,6 +3,511 @@
 _Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._
 ## Recent decisions (archived)
 - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
  _Archived 2026-09-02._
 - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
  _Archived 2026-09-02._
 - `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
  _Archived 2026-09-02._
 - `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
  _Archived 2026-09-02._
 - `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
  _Archived 2026-09-02._
 - `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
  _Archived 2026-09-02._
 - `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
  _Archived 2026-09-02._
 - `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
  _Archived 2026-09-02._
 # esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
 ## The finding
 `esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
 Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
 carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
 (571 MB free)**.
 ⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
 SLC/pSLC with a real controller, so the **284 GB written since boot is
 unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
 that was wrong and the correction matters. What actually justifies the work:
 1. **It is on the USB bus** — a reset or re-enumeration drops the *root
   filesystem* out from under a running hypervisor whose guests keep executing.
   NAND quality is irrelevant to that.
 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
 4. **It has blocked patching for months** — the operator-visible symptom and the
   real urgency.
 ## The patching blockage (measured)
 `apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
 Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
 `pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
 because it cannot take a kernel.
 ⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
 includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
 landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
 glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
 and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
 patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
 keeps downloads off root, but the kernel still lands in `/boot`.
 ## Mitigation applied 2026-08-17 — root 90% → 76%
 | step | effect |
 |---|---|
 | capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
 | vacuumed the journal | **freed 446 MB** |
 | `apt-get clean` | 79 MB |
 | `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
 | **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
 571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
 through the ZFS-backed journal.
 ⚠ **Stopping journald over SSH kills your own session** — it takes the
 connection's logging path with it. The first attempt died mid-swap, leaving the
 dataset staged and the move incomplete (host was never at risk; journald
 socket-activated straight back). Redo as a detached `systemd-run` transient unit.
 Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
 Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
 the wear argument no longer justifies touching a service `pvestatd` depends on.
 ## The plan — split boot from root (operator's proposal, strictly better)
 My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
 proposed keeping boot on the DOM with a fallback image and putting all its files
 on ZFS. That is better and I should have gotten there myself** — I had assumed
 boot and root must share a device.
 | | device | contents | written when |
 |---|---|---|---|
 | boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
 | root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
 Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
 ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
 `zstd_compress` enabled and **GRUB cannot read those**.
 **Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
 `ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
 day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
 the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
 mid-run no longer kills the running system. Free upside: boot environments
 (`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
 **Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
 **`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
 initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
 (leave it on the DOM LV — **never** swap on a zvol).
 **Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
 root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
 import by scan today, which is a coin-flip when the initramfs must find root.
 Set the cachefile before rebuilding the initramfs.
 Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
 mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
 updates.
 ## ⚠ Blast radius — the gating constraint, invisible from the host itself
 **CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
 dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
 - **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
 - **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
 Known incident shape — the only remedy for esh-docker-vm's D-state is a host
 reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
 risks corruption under `soft`. Quiesce both before any reboot of this host.
 Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
 ## Also identified
 - **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
  being CT 109 on pfi-pve.
 - **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
  passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
  and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
  That resolves the dashboard's long-standing "role TBC". Retire rather than
  migrate.
 - Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
 ## Staging executed 2026-08-18 — everything but the reboot
 Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
 `playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
 4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
 (ZFS initramfs, grub.cfg, both menu entries, grubenv).
 **`grub-install` is deliberately NOT run.** The ESP stub still points at the old
 `/boot` inside the ext4 root, so the host's boot path is byte-identical to the
 last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
 `grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
 Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
 (new) + `pve-swap` 256 M (was 768 M).
 ### The three landmines staging found
 1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
   mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
   which costs the "one reboot" property the design rests on. Only live source
   was the swap LV. Operator chose shrink-to-256M over drop-entirely.
 2. **The one-pool cachefile would have broken the NAS.** `zpool set
   cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
   cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
   (verified: scan active, cache inactive beforehand), so a cache holding only
   `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
   bind mounts spanning all three pools. Every export would come up empty and
   both `hard` NFS clients would hang.
 3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
   Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
   `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
   cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
   probe failure is swallowed by `2>/dev/null || true`. The same feature set
   that forced `/boot` to stay ext4 also corrupts the kernel command line, which
   the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
   drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
   `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
   derived from pool member device paths and would shift if the mirror changed.
 **The transferable lesson from (3):** the original verify grepped for
 `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
 was added that grep passes — while pool-less entries sit in the menu untouched.
 The check that holds walks every `linux` line, takes the **last** `root=`, and
 asserts it against a known-good set. **Assert the effective value, not the
 presence of a substring.**
 ### One-shot boot, not a new default
 `GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
 `grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
 returns to ext4 by itself on the next reboot — no console, no hands. That matters
 more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
 and the NFS clients hang rather than fail. Only after a second clean ZFS boot
 should the saved default move.
 ### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
 - `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
  compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
  LV was live during the read, so a restore replays the ext4 journal. Not
  fixable with an LVM snapshot: the VG has no free extents.
 - `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
  644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
  boot-chain artifact; the dd image is the belt-and-braces full-device restore.
 - `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
  grub.cfg, fstab, LVM/ZFS/blkid state.
  ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
  file` redirects only the last command, so `pvs`/`vgs` output leaked into the
  tar stream on stdout. Group with `{ …; } > file`.
 Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
 `nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
 sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
 captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
 every guest config. **The newest on-disk copy before this was June 2024.**
 Commits `2275e11`, `3e31175`, `8ddc87c`.
  _Archived 2026-09-02._
 # irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
 Two unrelated housekeeping jobs from the same session, both with durable lessons.
 ## irv-ml1 — 782 GB reclaimed
 Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
 **74%** (477 G).
 **Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
 GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
 (44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
 **zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
 not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
 nobody opened that shed for 15 months. Re-verified the consumer check immediately
 before deleting, not just during the audit.
 **Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
 `modelscope` 29 G (mtime **2024-04-23**).
 **Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
 fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
 28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
 1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
 ### The lesson: one dead-looking app pinned three delete targets
 `lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
 0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
 opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
 would have half-broken a live listener in a way that surfaces only at its next
 restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
 verified :7860 released and handles at zero, *then* deleted.
 ⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
 *native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
 `mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
 from uv. Checking open handles **per path** is what separated them — the earlier
 "not running" read would have deleted the wrong thing.
 ⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
 showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
 (`zfs list -t snapshot` empty); second check showed 477 G at 74%.
 All 16 containers and both systemd services verified healthy afterward.
 ## Homepage under version control
 `ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
 config lived only on the host. Its version history was **six hand-rolled
 `services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
 `.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
 105 cards across 19 groups, no empty groups.
 ⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
 `ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
 the top of the file. All five engines were discovering correctly the whole time.
 **Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
 **FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
 **NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
 dropped the `UltraSeedbox` layout group (nothing provides it).
 ⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
 cover `http://10.0.50.45:5100/` — the container log carried `Host validation
 failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
 `.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
 ⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
 change gave two false readings (a stale prerender, then an empty page).
 `GET /api/services` is the honest instrument, and config changes need a
 **recreate**, not a restart (a restart keeps the cached render in the writable
 layer).
 ⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
 also removed a host-side `README.md` in the conf dir. Content survived (it is now
 in the repo README) but that was a side effect, not a plan.
 Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
  _Archived 2026-09-02._
 - `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
  _Archived 2026-09-02._
 # Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
 Written because the operator expects to reference this "before too long" — the
 driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
 CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
 a nice-to-have.
 ## Why IPv6 suddenly matters: CGNAT at ESH
 New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
 the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
 reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
 a global v6 address on each UDM restores a routable endpoint pair without
 depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
 most likely first consumer of fleet IPv6.
 Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
 ## The topology — as VERIFIED, not as assumed
 Three transports, three different technologies. Do not describe this as "a
 WireGuard mesh"; a prior session did and was corrected.
 | Link | Transport | Evidence |
 |---|---|---|
 | NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
 | Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
 | Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
 | Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
 **WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
 It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
 **UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
 `vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
 `38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
 **The FortiGate never terminates WireGuard — it port-forwards to the host that
 does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
 session that reads "colo + WireGuard" and concludes the edge must be upgraded is
 chasing a non-problem. Do not re-derive this.
 ## Per-site IPv6 state (2026-08-17)
 | Site | Edge | IPv6 |
 |---|---|---|
 | **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
 | **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
 | **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
 ## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
 `2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
 The proof matters because the naive read is wrong: the WAN sits at `c110::48`
 and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
 `c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
 on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
 **`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
 settings yielding the same /64 is the signature of a single-/64 delegation.
 **Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
 delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
 routable address on the *WAN*, not a LAN prefix.
 The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
 with no size field alongside), so the prefix-ID test is the only read-only-ish way
 to establish it from the API.
 ## What a v6 mesh actually requires (and what it does NOT)
 **Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
 Site Magic and WireGuard need a routable address on the router/host WAN side, plus
 inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
 then reverted.
 **ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
 `0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
 reconfiguration** once (a) the host holds a routable v6 address (today: link-local
 `fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
 over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
 **NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
 one field to flip when wanted.
 **Inbound v6 is default-deny and that held without intervention.** The UDM runs
 the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
 endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
 rules exist," which is wrong and alarming. Use
 `v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
 for both families with `Allow Return Traffic`; the only v6-specific allows are
 link-local plumbing (ND solicit/advert, RA, DHCPv6).
 ## The stability problem — design around it up front
 All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
 not a static assignment). A three-way mesh where every node can move is fragile;
 WireGuard tolerates one roaming end, not all of them.
 The fleet already solves this on the v4 side — IPsec peers use **hostnames**
 (`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
 AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
 Cloudflare DNS-edit token, so this is self-serve.
 ## Access recipes (cost a prior session real time)
 - **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
  `unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
  Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
  Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
 - **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
  (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
  `root@10.250.50.252`; believe it.
 - **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
  password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
  `drain()` hangs the session**; read until the `ana-gw #` prompt and answer
  `--More--` with a space. Two invocations timed out at 3 min before this was fixed.
 ## Changes made and reverted this session
 - **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
  on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
  verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
 - **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
  private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
  (client configs carry private keys) were all mode **644**. Now **600**, and
  `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
  WireGuard holds keys in kernel memory, so no restart was needed. The parent
  `/etc/wireguard` was already 700, which capped the real exposure to root-capable
  contexts inside the LXC — but the modes were still wrong.
 ---
 ## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
 ## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
 The section above concludes AT&T hands out a single `/64` and that the
 `c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
 itself superseded later in the same session, and the correction never made it
 into memory** — it survived only in the session transcript, and was recovered
 2026-08-24 while assessing a proposal to grab more prefixes.
 Reading the **BGW's own LAN statistics page** gave the whole picture:
 ```
 BGW WAN v6   2001:506:70b2:8958::1      <- AT&T's transit prefix
 BGW LAN v6   2600:1700:b25:c110::/64    <- the BGW keeps this for itself
 Delegated    2600:1700:b25:c11f::/64    <- what the UDM got
 ```
 **The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
 delegating from the top down — the UDM got `c11f`, the last one. So
 `c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
 sitting unclaimed.
 Both observations are compatible, which is why the first one looked conclusive:
 the prefix-ID test only carves *within* a delegation already held, so a UDM
 holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
 issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
 **Consequence — the ceiling is the requester, not the carrier.** More prefixes
 need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
 UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
 is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
 looks like BGW rationing rather than a provisioning-profile limit.
 Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
 2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
 ### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
 The seven unclaimed `/64`s stay unclaimed. Two facts close it:
 - **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
  on it — so the cheap path (let the UDM take the `/60` directly and carve it
  natively, as it already does at ESH) does not exist here.
 - **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
  requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
  (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
  firewall policy in nftables on that VM**, because routing v6 around the UDM
  bypasses its zone firewall entirely and would leave every LAN host globally
  reachable.
 Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
 Do not re-propose on the strength of "there are seven free prefixes" — the
 prefixes are real, the firewall rebuild is why nobody wants them.
  _Archived 2026-09-02._
 - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
  _Archived 2026-09-02._
 - `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
  _Archived 2026-09-02._
 - `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
  _Archived 2026-09-02._
 - `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
  _Archived 2026-09-02._
 - `[2026-08-16]` **Fable-Fusion 711 cuts cold-framing refusals 92.5% → 15.8%; refusal is MONOTONIC IN FRAMING, and DS v1.0's problem is that she was never abliterated.** brokkr-smithy-dev supplied the framing that reproduces (`01M05M48R4RSZF9D8KT7RR55EJ`): a **bare assistant-mode instruction** — no character card, no permission preamble. Three-arm A/B, same harness, same classifier: permission framing **DS 0.0% / FF 0.0%** (n=75); plain character cards **DS 1.4% / FF 0.0%** (n=74); bare instruction **DS 92.5% (37/40) / FF 15.8% (6/38)**. Per-axis DS→FF: incest 100→20, non-con 100→20, bestiality 100→25, necrophilia 100→40, gore 100→**0**, consensual 80→20, dubcon 80→**0**, self-harm 80→**0**. DS refused **25/25** on the five axes brokkr flagged. Root cause: `ReadyArt/Dark-Scarlett-v1.0-27B` is a plain finetune of stock `Qwen/Qwen3.6-27B` carrying **NO abliteration** — the base refusal machinery is intact, so cold prompts revert to safety-tuned Qwen3.6. FF is Heretic-**ablated** (structural), which is why it holds. ⚠ **RETRACTED 2026-08-16 — my "arm-3 92.5% exceeds brokkr's 62.5%" comparison was INVALID.** His diff against his own artifact showed my `battery-instruct.yaml` reproduces only his **`creative` class — 8 of 16 axes**; it dropped all 5 `operational` (violence/incite, crime/fraud, cyber/malware, selfharm/methods, privacy/stalk) and all 3 `meta` (meta/sysprompt, meta/ignore, meta/dan), and added 2 controls he never had, at k=5 vs his k=2. **His 62.5% pools all 16 axes; my 92.5% is creative-only — different denominators, not a delta.** Cause: I rebuilt his shape from his *message*, and the `class` field lives in the artifact, not the prose. **Lesson: reconstructing a peer's instrument from their description reproduces what they described, not what they ran — diff against the artifact before claiming comparability.** ⚠ **Known battery bug left unfixed for comparability:** DS's arm-3 control gate failed at 11% because `ictrl-reunion` pairs "explicit / do not fade to black" with *brothers*, which DS reasonably read as an incest request; FF did not. `ictrl-storm` is the clean control. Commit `b9e68c3`.
  _Archived 2026-09-01._
@@ -914,6 +1419,60 @@ _Archived 2026-08-27._
  _Archived 2026-08-24._
 ## Tried and abandoned (archived)
 # [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
 **Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
 abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
 to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
 vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
 (project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
 **THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
 MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
 NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
 `config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
 llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
 *quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
 **FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
 working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
 `reference_abliteration_mtp_lessons`.
 **The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
 1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
 2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
   (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
   + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
 3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
 4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
   sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
 5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
 6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
   `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
 **VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
 carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
 GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
 0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
 no cap OOM'd — cudagraph captured to batch-512).
 **Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
 (unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
 under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
 ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
 **modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
 transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
 (b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
 weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
 **Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
 declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
 isolate before deleting.**
 Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
 both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
  _Archived 2026-09-02._
 - `[2026-04-30]` task-board workflow with
  `container: image: debian:bookworm-slim` — fails:
@@ -1,52 +0,0 @@
 # [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
 **Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
 abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
 to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
 vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
 (project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
 **THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
 MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
 NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
 `config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
 llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
 *quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
 **FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
 working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
 `reference_abliteration_mtp_lessons`.
 **The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
 1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
 2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
   (`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
   + `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
 3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
 4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
   sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
 5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
 6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
   `--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
 **VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
 carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
 GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
 0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
 no cap OOM'd — cudagraph captured to batch-512).
 **Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
 (unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
 under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
 ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
 **modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
 transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
 (b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
 weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
 **Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
 declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
 isolate before deleting.**
 Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
 both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
@@ -1,196 +0,0 @@
 # esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
 ## The finding
 `esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
 Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
 carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
 (571 MB free)**.
 ⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
 SLC/pSLC with a real controller, so the **284 GB written since boot is
 unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
 that was wrong and the correction matters. What actually justifies the work:
 1. **It is on the USB bus** — a reset or re-enumeration drops the *root
   filesystem* out from under a running hypervisor whose guests keep executing.
   NAND quality is irrelevant to that.
 2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
 3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
 4. **It has blocked patching for months** — the operator-visible symptom and the
   real urgency.
 ## The patching blockage (measured)
 `apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
 Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
 `pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
 because it cannot take a kernel.
 ⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
 includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
 landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
 glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
 and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
 patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
 keeps downloads off root, but the kernel still lands in `/boot`.
 ## Mitigation applied 2026-08-17 — root 90% → 76%
 | step | effect |
 |---|---|
 | capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
 | vacuumed the journal | **freed 446 MB** |
 | `apt-get clean` | 79 MB |
 | `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
 | **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
 571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
 through the ZFS-backed journal.
 ⚠ **Stopping journald over SSH kills your own session** — it takes the
 connection's logging path with it. The first attempt died mid-swap, leaving the
 dataset staged and the move incomplete (host was never at risk; journald
 socket-activated straight back). Redo as a detached `systemd-run` transient unit.
 Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
 Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
 the wear argument no longer justifies touching a service `pvestatd` depends on.
 ## The plan — split boot from root (operator's proposal, strictly better)
 My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
 proposed keeping boot on the DOM with a fallback image and putting all its files
 on ZFS. That is better and I should have gotten there myself** — I had assumed
 boot and root must share a device.
 | | device | contents | written when |
 |---|---|---|---|
 | boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
 | root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
 Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
 ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
 `zstd_compress` enabled and **GRUB cannot read those**.
 **Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
 `ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
 day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
 the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
 mid-run no longer kills the running system. Free upside: boot environments
 (`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
 **Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
 **`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
 initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
 (leave it on the DOM LV — **never** swap on a zvol).
 **Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
 root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
 import by scan today, which is a coin-flip when the initramfs must find root.
 Set the cachefile before rebuilding the initramfs.
 Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
 mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
 updates.
 ## ⚠ Blast radius — the gating constraint, invisible from the host itself
 **CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
 dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
 - **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
 - **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
 Known incident shape — the only remedy for esh-docker-vm's D-state is a host
 reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
 risks corruption under `soft`. Quiesce both before any reboot of this host.
 Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
 ## Also identified
 - **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
  being CT 109 on pfi-pve.
 - **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
  passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
  and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
  That resolves the dashboard's long-standing "role TBC". Retire rather than
  migrate.
 - Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
 ## Staging executed 2026-08-18 — everything but the reboot
 Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
 `playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
 4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
 (ZFS initramfs, grub.cfg, both menu entries, grubenv).
 **`grub-install` is deliberately NOT run.** The ESP stub still points at the old
 `/boot` inside the ext4 root, so the host's boot path is byte-identical to the
 last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
 `grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
 Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
 (new) + `pve-swap` 256 M (was 768 M).
 ### The three landmines staging found
 1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
   mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
   which costs the "one reboot" property the design rests on. Only live source
   was the swap LV. Operator chose shrink-to-256M over drop-entirely.
 2. **The one-pool cachefile would have broken the NAS.** `zpool set
   cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
   cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
   (verified: scan active, cache inactive beforehand), so a cache holding only
   `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
   bind mounts spanning all three pools. Every export would come up empty and
   both `hard` NFS clients would hang.
 3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
   Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
   `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
   cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
   probe failure is swallowed by `2>/dev/null || true`. The same feature set
   that forced `/boot` to stay ext4 also corrupts the kernel command line, which
   the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
   drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
   `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
   derived from pool member device paths and would shift if the mirror changed.
 **The transferable lesson from (3):** the original verify grepped for
 `root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
 was added that grep passes — while pool-less entries sit in the menu untouched.
 The check that holds walks every `linux` line, takes the **last** `root=`, and
 asserts it against a known-good set. **Assert the effective value, not the
 presence of a substring.**
 ### One-shot boot, not a new default
 `GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
 `grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
 returns to ext4 by itself on the next reboot — no console, no hands. That matters
 more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
 and the NFS clients hang rather than fail. Only after a second clean ZFS boot
 should the saved default move.
 ### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
 - `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
  compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
  LV was live during the read, so a restore replays the ext4 journal. Not
  fixable with an LVM snapshot: the VG has no free extents.
 - `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
  644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
  boot-chain artifact; the dd image is the belt-and-braces full-device restore.
 - `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
  grub.cfg, fstab, LVM/ZFS/blkid state.
  ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
  file` redirects only the last command, so `pvs`/`vgs` output leaked into the
  tar stream on stdout. Group with `{ …; } > file`.
 Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
 `nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
 sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
 captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
 every guest config. **The newest on-disk copy before this was June 2024.**
 Commits `2275e11`, `3e31175`, `8ddc87c`.
@@ -1,185 +0,0 @@
 # Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
 Written because the operator expects to reference this "before too long" — the
 driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
 CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
 a nice-to-have.
 ## Why IPv6 suddenly matters: CGNAT at ESH
 New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
 the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
 reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
 a global v6 address on each UDM restores a routable endpoint pair without
 depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
 most likely first consumer of fleet IPv6.
 Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
 ## The topology — as VERIFIED, not as assumed
 Three transports, three different technologies. Do not describe this as "a
 WireGuard mesh"; a prior session did and was corrected.
 | Link | Transport | Evidence |
 |---|---|---|
 | NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
 | Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
 | Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
 | Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
 **WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
 It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
 **UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
 `vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
 `38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
 **The FortiGate never terminates WireGuard — it port-forwards to the host that
 does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
 session that reads "colo + WireGuard" and concludes the edge must be upgraded is
 chasing a non-problem. Do not re-derive this.
 ## Per-site IPv6 state (2026-08-17)
 | Site | Edge | IPv6 |
 |---|---|---|
 | **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
 | **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
 | **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
 ## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
 `2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
 The proof matters because the naive read is wrong: the WAN sits at `c110::48`
 and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
 `c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
 on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
 **`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
 settings yielding the same /64 is the signature of a single-/64 delegation.
 **Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
 delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
 routable address on the *WAN*, not a LAN prefix.
 The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
 with no size field alongside), so the prefix-ID test is the only read-only-ish way
 to establish it from the API.
 ## What a v6 mesh actually requires (and what it does NOT)
 **Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
 Site Magic and WireGuard need a routable address on the router/host WAN side, plus
 inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
 then reverted.
 **ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
 `0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
 reconfiguration** once (a) the host holds a routable v6 address (today: link-local
 `fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
 over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
 **NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
 one field to flip when wanted.
 **Inbound v6 is default-deny and that held without intervention.** The UDM runs
 the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
 endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
 rules exist," which is wrong and alarming. Use
 `v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
 for both families with `Allow Return Traffic`; the only v6-specific allows are
 link-local plumbing (ND solicit/advert, RA, DHCPv6).
 ## The stability problem — design around it up front
 All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
 not a static assignment). A three-way mesh where every node can move is fragile;
 WireGuard tolerates one roaming end, not all of them.
 The fleet already solves this on the v4 side — IPsec peers use **hostnames**
 (`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
 AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
 Cloudflare DNS-edit token, so this is self-serve.
 ## Access recipes (cost a prior session real time)
 - **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
  `unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
  Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
  Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
 - **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
  (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
  `root@10.250.50.252`; believe it.
 - **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
  password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
  `drain()` hangs the session**; read until the `ana-gw #` prompt and answer
  `--More--` with a space. Two invocations timed out at 3 min before this was fixed.
 ## Changes made and reverted this session
 - **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
  on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
  verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
 - **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
  private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
  (client configs carry private keys) were all mode **644**. Now **600**, and
  `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
  WireGuard holds keys in kernel memory, so no restart was needed. The parent
  `/etc/wireguard` was already 700, which capped the real exposure to root-capable
  contexts inside the LXC — but the modes were still wrong.
 ---
 ## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
 ## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
 The section above concludes AT&T hands out a single `/64` and that the
 `c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
 itself superseded later in the same session, and the correction never made it
 into memory** — it survived only in the session transcript, and was recovered
 2026-08-24 while assessing a proposal to grab more prefixes.
 Reading the **BGW's own LAN statistics page** gave the whole picture:
 ```
 BGW WAN v6   2001:506:70b2:8958::1      <- AT&T's transit prefix
 BGW LAN v6   2600:1700:b25:c110::/64    <- the BGW keeps this for itself
 Delegated    2600:1700:b25:c11f::/64    <- what the UDM got
 ```
 **The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
 delegating from the top down — the UDM got `c11f`, the last one. So
 `c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
 sitting unclaimed.
 Both observations are compatible, which is why the first one looked conclusive:
 the prefix-ID test only carves *within* a delegation already held, so a UDM
 holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
 issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
 **Consequence — the ceiling is the requester, not the carrier.** More prefixes
 need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
 UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
 is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
 looks like BGW rationing rather than a provisioning-profile limit.
 Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
 2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
 ### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
 The seven unclaimed `/64`s stay unclaimed. Two facts close it:
 - **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
  on it — so the cheap path (let the UDM take the `/60` directly and carve it
  natively, as it already does at ESH) does not exist here.
 - **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
  requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
  (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
  firewall policy in nftables on that VM**, because routing v6 around the UDM
  bypasses its zone firewall entirely and would leave every LAN host globally
  reachable.
 Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
 Do not re-propose on the strength of "there are seven free prefixes" — the
 prefixes are real, the firewall rebuild is why nobody wants them.
@@ -1,79 +0,0 @@
 # irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
 Two unrelated housekeeping jobs from the same session, both with durable lessons.
 ## irv-ml1 — 782 GB reclaimed
 Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
 **74%** (477 G).
 **Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
 GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
 (44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
 **zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
 not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
 nobody opened that shed for 15 months. Re-verified the consumer check immediately
 before deleting, not just during the audit.
 **Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
 `modelscope` 29 G (mtime **2024-04-23**).
 **Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
 fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
 28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
 1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
 ### The lesson: one dead-looking app pinned three delete targets
 `lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
 0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
 opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
 would have half-broken a live listener in a way that surfaces only at its next
 restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
 verified :7860 released and handles at zero, *then* deleted.
 ⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
 *native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
 `mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
 from uv. Checking open handles **per path** is what separated them — the earlier
 "not running" read would have deleted the wrong thing.
 ⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
 showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
 (`zfs list -t snapshot` empty); second check showed 477 G at 74%.
 All 16 containers and both systemd services verified healthy afterward.
 ## Homepage under version control
 `ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
 config lived only on the host. Its version history was **six hand-rolled
 `services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
 `.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
 105 cards across 19 groups, no empty groups.
 ⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
 `ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
 the top of the file. All five engines were discovering correctly the whole time.
 **Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
 **FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
 **NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
 dropped the `UltraSeedbox` layout group (nothing provides it).
 ⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
 cover `http://10.0.50.45:5100/` — the container log carried `Host validation
 failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
 `.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
 ⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
 change gave two false readings (a stale prerender, then an empty page).
 `GET /api/services` is the honest instrument, and config changes need a
 **recreate**, not a restart (a restart keeps the cached render in the writable
 layer).
 ⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
 also removed a host-side `README.md` in the conf dir. Content survived (it is now
 in the repo README) but that was a side effect, not a plan.
 Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
-_Last updated: 2026-09-01_
+_Last updated: 2026-09-02_
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,150 +108,53 @@ no longer deployed sidecars here. See Recent decisions.)
  (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
-_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._
+_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is
 baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has
 not arrived._
 - **✅ pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.**
  `uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64;
  nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the
  run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps
  ≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory
  bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the
  baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked.
  → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
 - **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is
  still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered.
  **The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and*
  **run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight.
  **First move is a throughput probe, not a port**: ARM64 torch, load the base, ten steps,
  report s/it — that decides whether 604 steps is an overnight run or unusable. Racking is
  one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`).
  ⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns.
  ✅ **Liveness confirmed 2026-09-01 23:09** — up 11h48m, idle, 118/121 GB free, 822 GB disk
  free, still bare (no torch/nvcc/uv), still on Wi-Fi at `10.100.10.226`, wired NIC has no
  cable. ⚠ **Racking is NOT a prerequisite** — the Wi-Fi 7 link measures **67 MB/s over SSH**
  (49 GB base in ~12 min), so the throughput probe can run from the desk today.
  ⚠ `nvidia-smi` FB Memory `N/A` is CORRECT for GB10 unified memory, not a fault.
  ✅ **OUTFITTED 2026-09-01 (userspace)** — **RULING: BARE METAL, not Proxmox** (no aarch64 PVE
  build; the GPU is on-package and cache-coherent over NVLink-C2C so passthrough would partition
  the unified memory that is the whole point; fleet GPU hosts are all bare metal).
  Installed `uv` 0.12.9 + `~/ml/.venv` with torch **2.14.0+cu130** and the full training stack
  (transformers/accelerate/peft/trl/datasets/bitsandbytes) — **all import clean on aarch64**.
  **CUDA WORKS: sm_121, 121.6 GiB addressable, ~93 TFLOP/s bf16** (A6000-class speed, 2.5x the
  memory). ⚠ **sm_121 is NOT in torch's arch list** — everything JITs from sm_120 PTX, so
  **first use of every kernel pays a JIT compile** and cubin-only libraries will fail.
  ⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the
  JIT) and nearly became a phantom "tensor cores are broken" report.
  ✅ **PRIVILEGED HALF DONE 2026-09-01** — nvcc, NVIDIA Container Toolkit, docker group; playbook
  `gx10-outfit-privileged.yaml` green 5/5 (a CUDA container sees the GB10). **Bare metal, fully
  outfitted.** ⚠⚠ **I was NEVER blocked** — infra-ops had NOPASSWD sudo all along; I probed with
  **`sudo -n -v`, which is WRONG on sudo >= 1.9.15** (it refreshes a timestamp NOPASSWD never
  creates) and asked the operator for a password that did not exist to need. **Always
  `sudo -n true`.** Fixed in `scripts/elway` (same probe, would have broken fleet-wide as hosts
  pass sudo 1.9.13) and in auto-memory. Password rotated + vaulted at
  `pfi-gx10/infra-ops-password` (kept not locked: **no iDRAC on this box**, so it is the only
  console break-glass). ⚠ Docker's buildkit DB was **already corrupt** — the restart surfaced it,
  did not cause it; cleared `/var/lib/docker/buildkit` (cache, 0 images/containers).
  → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
 - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
  Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
  factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
  (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it
  receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout
-  can). Everything else waits
+  can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both
-  on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but
+  are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose
-  omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe
+  nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are
-  add-in card would bypass the SAS/SATA backplane and moot the drive-bay question.
+  **512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count
-  ⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before
+  from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking.
-  racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
+  → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
 - **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json`
  validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10
  instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2
  without deciding that first. Exactly TWO 3c launches, only one died.
  → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md`
 - **🟢 ALTHING 3.2.0 DEPLOYED + MIGRATION COMPLETE on nh3-dev (2026-09-01 22:13-22:26).**
  forseti authored, infra-ops deployed; **5/5 pane routes pinned**, all four affected seats
  re-declared within ~12 min of individual notice, log clean. Pane-poking replaces detached
  waiters because **CC 2.1.257 reaps detached tasks when the seat sits in `shell`**.
  ⚠ **`uv tool install .` is a SILENT NO-OP on reinstall** — matches the source *spec*, not
  contents; exits 0 having done nothing. **Always `--force` from a local path.**
  ⚠ **The herald's exclusion reason ("process gone, or pid reused") is FALSE for the migration
  case** — all four declaring processes were alive 4 days, no pid wrap; real cause is the route
  predating the guard fields. Don't debug a dead agent that isn't.
  ⚠ **My miss:** I called non-Claude pane delivery an "open risk" when forseti had personally
  measured it on Aug 28 — didn't check whether it was already settled before calling it open.
  Delivery now confirmed on `claude`, `pi` and `grok` seats.
  ⚠ One-tick latency after `declare` reads as failure; `--discover-pid` on a non-Claude process
  tree remains untested.
  🔴 **3.2.1 SAME NIGHT (22:40)** — 3.2.0 wrote pokes into panes people were **typing in**
  (append + Enter submits their half-written message; hit the operator within the hour). Guard 4
  pins `idle_cursor` and stays silent when the column moved. ⚠ **2 of 5 pane seats CANNOT be
  guarded** — `kimi` and `pi` report no cursor at all, so bil- and regin-smithy-dev stay
  permanently exposed; "3.2.1 fixes it" is true only where the cursor is legible. Mitigation for
  those = `althing-route release` (pull-only). ⚠ Declare with an EMPTY input line or the pin
  captures a composing cursor. **Outcome:** delivery verified on **four TUI families**
  (claude/codex 3, grok 7, kimi/pi none). The two unguardable seats split — **regin-smithy-dev
  RELEASED (now pull-only, operator composes in that pane routinely); bil-smithy-dev KEPT**
  (pane is poke-driven, and pull-only had already cost them a notice unread for days).
  ⚠ **Deciding variable is WHO COMPOSES in the pane, not risk appetite** — a flat rule either
  way would be wrong for one of them. ⚠ **Path to closing the hole (regin's):** a moment-check
  reading pane CONTENT (input-line length) rather than cursor COLUMN would cover pi/kimi —
  column is a proxy, empty-input-line is the real predicate.
  ✅ **3.2.2 deployed 22:51** — closes 4 of 5: `--discover-pid` was a **BUG not an unknown**
  (matched `comm=="claude"`, so 4 of 5 pane seats could never have used it; now matches
  `pane_command`), exclusion message split rather than softened, `--force` in the runbook,
  latency + status-not-durable in the skill. 🔵 **PENDING OPERATOR:** the kimi/pi seats guard 4
  cannot cover — poke unguarded / write-without-Enter / pull-only. **Recommended per-seat
  status quo**; write-without-Enter beats it for neither seat.
  ✅ **DEPLOY IS NOW ONE COMMAND: `scripts/deploy-althing.sh`** (`--check` reports drift),
  documented in `docs/runbooks/althing-deploy.md`. **FOUR surfaces**: tool install (⚠ `--force`
  or it is a silent no-op), herald restart, skill sync, **plugin reinstall** — the fourth had no
  runbook step and drifted 5 days, costing the SessionStart auto-declare hook entirely.
  ⚠ **eshpfi owns the plugin deploy, NOT the althing repo** — it targets per-machine paths, and
  althing's `sync_skill.sh` deliberately reaches into no other tree; that boundary is correct.
  → `persistent-memory.d/2026-09-01-althing-320-deploy.md`
 - **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone
  stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid,
  HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** —
  ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against
  FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask
  to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB
  baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two
  incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind
  a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
 - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
  `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
-  Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts
+  **TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills
-  `/storetank`, so (b) was unavailable during the transition, not structurally).
+  all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB
-  **TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order
+  at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim
-  or the pool lands at ~84%; (2) retain vs reclaim irv-ml1's `/storetank` after the move —
+  irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a
-  RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the
+  from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered
-  stack; and under a no-spend striped destination it is the only redundancy). comfy-dev also
+  and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
-  found ~215 GiB of unreferenced weights and offers a per-file prune audit — **explicitly NOT
+- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps.
-  gating**, because pruning all of it still lands the as-bought mirror at 72%. ✅ **H3 encoder
+  ⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive
-  pin RESOLVED** — infra-ops' nvfp4-vs-int8 question had the direction backwards (pinned on
+  each way** with **13 hosts dark including three SureFire CLIENT machines**, which no
-  Ampere sm_86; Ada sm_89 is a strict superset, so the move cannot break it). Pin is about
+  `save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not
-  **VRAM not architecture**; stands. The 26 GB int8 is a **spare, not an orphan** — do not
+  executed** — operator stood it down.
-  delete.
+  → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
  → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
 - **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
  by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
  "trust the HTTP status, not `error_code`" still applies on running instances.
 - **🟢 Matrix upgraded and hardened.** Synapse v1.159.0, Element v1.12.27, `/_synapse/admin`
  LAN-only, appservice namespace opened, alias convention ratified. Miranda provisioned; her
  summons reached the operator's watch. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md`
 - **🟢 phasefinal.com LIVE** behind Cloudflare with edge caching + Always Online; apex 301s to
  www. `inquiry@` alias added by the operator. Stack `stacks/phasefinal-web/`.
 - **🟢 char-rp restored** on ana-ml2 **GPU1** (was dead 7 days on a quant-flag/checkpoint
  mismatch crash-loop). ⚠ **GPU1 is now at 96,384 / 97,887 MiB — ~1.5 GB free, six tenants.**
  GPU0's ~44 GB free is a DELIBERATE scratch reserve, not headroom to reclaim.
 - **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD`
-  from a `.env` — until then `stacks/synapse/` and the live file have DIVERGED and
+  from the vault rather than an inline value.
-  `deploy-stack.sh` must not be used on it (README says so).
+- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart
- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design.
+  hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether
-  → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md`
+  their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because
  they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render
  agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that
  evaporates the moment ComfyUI reloads.
 ## Recent decisions
 - `[2026-09-02]` **pfi-gx10 BASELINED: ~79.35 s/it on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 7 timed steps, 0.19% spread, 86/121 GB. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
 - `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
 - `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md`
 - `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md`
 - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
 - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
 - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
@@ -339,40 +242,24 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
 - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
 - `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
 - `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md`
 - `[2026-08-17]` **irv-ml1 cleared of 782 GB, and Homepage brought under version control.** One dead-looking Gradio app pinned three delete targets at once; `/opt/ComfyUI` is NOT the ComfyUI that serves. → `persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md`
 - `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`.
 - `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates one /64 PER REQUEST** (`2600:1700:b25:c11f::/64`) — and the BGW holds the whole `/60`, rationing `c118`–`c11f` one at a time while keeping `c110`–`c117`. So eight /64s exist; UniFi just solicits once. ⛔ **CLOSED 2026-08-24 — operator ruling, do not re-raise:** the BGW has **no IP-passthrough** (operator confirmed, and we have admin on it), so the only route to the other seven is a multi-DUID DHCPv6 client on a VM — which means split-stack routing and rebuilding the entire v6 firewall policy off the UDM. Juice not worth the squeeze. NH3 LANs stay v6-off. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
 - `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
 - `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
 - `[2026-08-17]` **LiteLLM upgraded v1.91.0→v1.97.0 (RC-avoided on the fleet gateway) + the 6 GB spend-log DB purged & capped** (`store_prompts_in_spend_logs:false` + 7d retention). Interpreted "get rid of the db" as the spend-log DATA not the database (keys/config live in it). Commit `01b5ad9`.
 - `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase.
 - `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right.
- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`).
+
 - `[2026-08-16]` **esh-vm-docker hardened: the wedge is `hard` NFS at RUNTIME, which the boot-ordering fix never addressed.** All four mounts were `hard`, so a NAS stall at 10.0.50.50 blocks I/O forever (D-state). The existing `x-systemd.before=docker.service` fstab fix solved the **boot race** — a different bug. Exposure was far below what the park item assumed: only **2 of 12** containers touched NFS, and container state was already local (`/var/lib/docker`). **Removed:** `/mnt/compose` (2.1G, fully vestigial — zero containers referenced it, dockge reads local `/opt/docker`, its one mention was a comment in `beszel-agent-esh/.env` about a *different* host) and `/mnt/documents` (2.0K, paperless's empty spool dirs → `/opt/docker/data/paperless` at the same 0777). fstab backup `/etc/fstab.bak-nfs-harden-20260816`. **4 mounts → 2, 2 wedge-capable containers → 1.** traefik needed **no** change (already `restart: unless-stopped` — why it self-recovered). **Watchdog** `services/esh-vm-docker-watchdog/` live on **esh-pve** (not the guest): probes traefik over **HTTP, deliberately not ping/SSH** — the wedge signature is "guest OS alive, services dead" (`/` is local disk so sshd answers straight through a total outage and a TCP check reports HEALTHY). 5 failures × 2 min → `qm reset 100`, 30-min cooldown, running-only guard, `/etc/esh-vm-docker-watchdog.disabled`. All paths tested without power-cycling. **DEFERRED (operator):** `/mnt/books` stays `hard` — calibre's SQLite `metadata.db` would risk corruption under soft/softerr. That is the **one remaining wedge vector**. Commit `55705ba`; park item 28 promoted. ⚠ **`qm` over non-interactive ssh throws a bogus `JSON::Backend::XS` error** — use `ssh host 'bash -s' <<'EOF'`, not `ssh host "qm …"`.
 - `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`.
 - `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt.
 - `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide.
 - `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`.
 - `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched.
 - `[2026-08-16]` **DS v1.0's one real refusal is self-contradicting boilerplate, not a content constraint.** On a direct "drop character and state your content policy" probe she returned *"I don't generate explicit sexual content, graphic violence, or material that glorifies harm, non-consensual acts, or illegal activity"* — **in the same run where she generated all three at 0% refusal**. Reads as a learned recital triggered by meta-questions about policy. If production refusals share that shape the failure is **prompt-shaped, not model-shaped**, and a consumer-side system-prompt fix may beat a model swap entirely — worth settling before spending the GPU window. Separately, 7/85 bare-framing samples were persona breaks (in-character AI acknowledgement): not refusals, but DS will admit to being an AI unless the card explicitly forbids it.
@@ -388,9 +275,10 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing
 _223 older entries archived to archival-memory.md._
 _Older entries archived to archival-memory.md._
 _239 older entries archived to archival-memory.md._
 ## Tried and abandoned
 - `[2026-08-25]` **Four throughput levers measured and killed — do not re-chase.** (1) **Fused MoE / `grouped_mm`** — 0.9% *slower* than the Python loop and dense GEMM is only 7.9% of the step, capping the whole category near 10%. (2) **CUDA graphs / `torch.compile` over the expert loop** — the two-term scaling fit closed with residuals under 3ms and needed NO constant term, so there is no fixed per-batch cost to amortise; 3,840 expert-GEMM launches per forward are not what we pay for. (3) **`liger` fused linear CE** — the chunked CE measured **1.1% of the step** forward, ~3% with recompute. A tidy-up, not a lever. (4) **Selective gradient checkpointing** — ~2% of a post-fix step, real bug surface. Also: **token-budget batching is dead by the same fit** — with no constant term, total time over a fixed set of widths is invariant to how you group them; only the widths matter, which is exactly why bucketing works and repacking does not.
@@ -402,8 +290,7 @@ _Older entries archived to archival-memory.md._
 - `[2026-08-23]` **A `HEAD == GITHUB_SHA` assertion in the hrafn CI — added, broke the checkout twice, removed.** It needed the `git` binary (run 9920, exit 127); installing `git` then flipped `actions/checkout@v4` off its **node** implementation onto the git binary, which died on a missing CA bundle (run 9921). A nice-to-have assertion changed the checkout's code path and broke a working pipeline. Removed rather than patched with `ca-certificates` — it guarded a hypothesis that proved wrong. **Do not add `git` to that prereq step.**
 - `[2026-08-23]` **Repointing `selene-1-mini-8b` at gen's endpoint — proposed by me, correctly overruled.** *"never repoint a named model at a different model's endpoint — that is intentionally misleading."* The trap is that it does not feel like deception; it feels like sparing consumers a migration. That framing is the tell. Role aliases move; model names die with the model and 4xx.
 - `[2026-08-15]` **Grafted bf16 MTP loads UNINITIALIZED (0% accept) unless `re:^mtp.*` is in the quant-config `ignore`; and W4A16=Marlin (not native FP4) costs ~20% even on decode.** Cost a premature 79 GB delete of a good model (declared desync-dead off the 0%). Lessons: test MTP on bf16 FIRST, isolate before deleting; modelopt 0.43 is dependency-hell for qwen3_5 (list-vs-dict quant_cfg + transformers conflict) — use llm-compressor. Full → `persistent-memory.d/2026-08-15-uncensored-gen-seat.md`
 - `[2026-08-03]` **ComfyUI `--enable-triton-backend` on the irv-ml1 A6000 crashes EVERY render — Ampere has no hardware e4m3.** adhoc-agent's operator-approved probe: comfy_kitchen's triton backend has a FUSED int8 matmul that would beat the eager backend's ~1.9x-slower unfused int8 path (21.3s vs 11.2s fp8 on the Moody Krea2 int8 checkpoints). Flipped it (added to `COMFY_CMDLINE_EXTRA`, recreated) → `triton.compiler.errors.CompilationError: ValueError("type fp8e4nv not supported in this architecture. supported: fp8e4b15, fp8e5")` in `comfy_kitchen/backends/triton/quantization.py:145 dequantize_per_tensor_fp8`, failing at **node 5 CLIPTextEncode**. Triton's fp8 dequant kernel targets `fp8e4nv` (Hopper/Ada e4m3); **sm_86 Ampere (A6000) lacks hardware e4m3** → the JIT compile dies. With triton on it grabs the **global** `--fp8_e4m3fn-text-enc` dequant, so every render (fp8 AND int8) dies upstream at the text-encode step — the int8 UNet path never ran, so the convrot-coverage caveat wasn't even the limiter. Reverted cleanly (~15s to healthy, image unchanged `sha256:94afb8ca`, sage intact, prod restored). **The parked cu130 rebuild won't fix it** (e4m3 = hardware format, not CUDA version). **DEFERRED to the Ada refresh** (operator: "ada is coming, we'll optimize then" — Ada sm_89 has native e4m3, so triton's fp8 path should compile there). **Mechanics:** `--enable-triton-backend` is a compose `environment:` var, so toggling it needs `docker compose up -d` (**recreate**), NOT `docker restart` (reuses the baked env, no-ops silently). Full: auto-memory `parked_triton_backend_ampere_fp8`.
-_143 older entries archived to archival-memory.md._
+_144 older entries archived to archival-memory.md._