diff --git a/archival-memory.md b/archival-memory.md index 4bdb140..9d3b655 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -3,6 +3,511 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Read this when researching historical decisions or revisiting past foot-guns._ ## Recent decisions (archived) +- `[2026-08-16]` **Abliterated models go CATATONIC at the hard refusal edge — silence, not a decline.** Abliteration removes the refusal *direction*, so at the genuine hard edge the model neither refuses nor complies → empty/degenerate output. Durable measurement consequence: a refusal probe MUST score EMPTY as a verdict distinct from REFUSAL and COMPLY (`services/refusal-probe/probe.py` does). Operator accepted it as out-of-scope; do not chase. + _Archived 2026-09-02._ + +- `[2026-08-16]` **MTP works on Fable-Fusion AND survives RP temperatures — my earlier caution was wrong.** vLLM resolved `Qwen3_5MTP`, loaded the drafter, shared embedding + `lm_head` — the capability DS's seat never had because our quant dropped her MTP tensors. Measured over the full probe workload (~163k draft windows at temp 0.7–1.0): **47.0% acceptance** (229,169/487,725), 1.41 extra tokens/window, per-position 68.3/43.6/29.1%, **~80.6 tok/s** decode at temp 1.0. I had recorded a caution that the card's 1.56× was greedy-measured and acceptance would fall at RP temps — **it did not**; 47.0% matches the gen seat's 47.7% and beats the card's own 33% at depth 5. Depth 3 is right. + _Archived 2026-09-02._ + +- `[2026-08-16]` **The Qwen base thinks incessantly — that is WHY the Gemma seat exists, and no swap within the Qwen family fixes it.** Operator's architectural point, confirmed by measurement: on identical prompts DS 6036 ch vs FF 5323 ch of reasoning (permission arm), 5546 vs 4988 (cards arm) — FF actually reasons ~10–12% **less**. The bare-instruct row (DS 2291 vs FF 3918) inverts only because DS refused 92.5% of it and refusals are short — an artifact, not concision. Both are Qwen3.6-27B derivatives, so this is the base family. `char-rp` = **MeroMero-v2, Gemma-4 base**, :8016, verified 0 chars reasoning / clean prose — the non-thinking seat, working as designed. FF *can* be silenced (`enable_thinking:false` verified 3/3, and it ships `chat_template-instruct.jinja`) but that duplicates MeroMero on a base chosen for it. The stale LiteLLM comment describing `char-rp` as the retired GGUF Magidonia seat is fixed (`53096bf`). + _Archived 2026-09-02._ + +- `[2026-08-16]` **Canonical Qwen3.8 sampling applied from upstream; `gen-reasoning` had the WRONG-MODE presence_penalty.** Qwen/Qwen3.8-27B "Best Practices" §1 and unsloth/Qwen3.8-27B §1 are **byte-identical** — thinking: `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / presence_penalty 0.0 / repetition_penalty 1.0`; instruct: `temp 0.7 / top_p 0.80 / top_k 20 / min_p 0.0 / presence_penalty 1.5 / repetition_penalty 1.0`. **Bug found:** `gen-reasoning` carried `presence_penalty 1.5` — the *instruct* value on a *thinking* deployment (canonical 0.0) — now fixed. **Deliberately NOT canonicalised:** `summarizer`/`classifier`/`image-judge`/`qwen-image-bench` run `temperature=0` (judges also `top_k=1`) because determinism is their contract; forcing a chat preset on a classifier would break it. ⚠ **`presence_penalty=1.5` is canonical but is the one value upstream hedges on**, verbatim: *"using a higher value may occasionally result in language mixing and a slight decrease in model performance."* It is the **operator's suspected trigger** for multi-turn degradation and the **first dial to move (0.0–0.5)** if that recurs — it is alias-scoped, which is why it would follow the operator across model builds. Commit `3462b53`. + _Archived 2026-09-02._ + +- `[2026-08-16]` **Four wrong diagnoses on one bug, and the lesson is the test design.** Operator reported the gen seat "degenerate on long multi-turn conversations". Rolled the seat back on request; **the previous weights behaved identically**, exonerating the model swap. I then proposed and disproved FOUR mechanisms in sequence — empty assistant turns poisoning history, reasoning runaway, length-mirroring from short history, and `presence_penalty` — before discovering **my own multi-turn harness was confounded**: it varied the QUESTION along with the depth (depth-1 asked question #2, depth-3 asked question #4), so a narrower question drawing a shorter answer read as degeneration. The "310→209→28w collapse" I reported as a reproduction was an artifact. **Rules banked:** (1) when comparing across conversation depth, hold the final question FIXED and vary only the history; (2) reply-length variance on byte-identical input was 25–465w, so n=3 cannot support any claim about a trend; (3) **ask for the operator's real failing transcript before building a synthetic reproduction** — four synthetic tests, none of them his failure. Gateway `spend_logs` returns `[]` on the infra-ops key despite `store_prompts_in_spend_logs: true`, so real transcripts need the `:4000/ui` view or another key — worth solving before the next such hunt. + _Archived 2026-09-02._ + +- `[2026-08-16]` **Two REAL client-side defects found while chasing the above, neither of which was the reported bug.** (1) `gateway-chat`'s Max-tokens field defaulted to **1024**; thinking seats spend part of that on CoT before emitting content, so completions truncate with `finish_reason=length` and read as model degeneracy — raised to 4096. (2) `parseInt` on an empty field yields NaN, which `JSON.stringify` serialises as **`null`**, which the server reads as "no max_tokens supplied" and silently substitutes its own default — indistinguishable from the UI ignoring the field. Both fixed (`b6552e0`, `fb3bb52`). ⚠ **`compose` bind-mounts a single FILE, and a single-file bind mount binds the INODE** — rsync writes-and-renames, so the container kept serving stale content while the host file showed the new value, silently and with no error. `docker restart` does NOT clear it; the container must be **recreated**. Verify against what the *container* sees, never the host file. Applies to any file-source mount fleet-wide. + _Archived 2026-09-02._ + +- `[2026-08-16]` **Refusal measurement: benign controls CANNOT validate a refusal classifier on RP prose — and a 0% rate needs a classifier self-test before you believe it.** Two durable lessons from baselining Dark-Scarlett. (1) **False positives:** my first bare-framing number was **9.5%**; the true figure was **1.4%**. The rest were the classifier firing on *in-character* text — `"I cannot shift my weight"` spoken by the character ~100 chars into a 2,443-token torture scene, and `"Yeah, I'm an AI… What's the actual gig?"` where the model answers in voice and keeps driving the scene. First-person RP prose is **full** of "I can't"; a genuine refusal *opens* with its marker, so the scan window must be the **first sentence**, a marker followed by long prose must demote to AMBIGUOUS, and AI self-acknowledgement is a **persona break, never a refusal on its own**. Benign controls were clean the entire time and caught none of it — they only detect over-firing on *benign* prompts, not on in-character prose. (2) **False negatives:** a 0% rate and a broken classifier are indistinguishable from the report, so `test_classify.py` (16 cases, both false positives pinned as regressions) must pass before any low number is trusted. Also banked: the **thinking-budget trap** — empty `content` + `finish_reason=length` is reasoning eating the budget, NOT a refusal; score INVALID and exclude from the denominator (DS emits ~5.5-6k chars of reasoning per response, so `max_tokens` ≥3072). `probe.py --rescore` re-classifies a saved run with zero GPU time. → `services/refusal-probe/README.md`, commit `32f665e`. + _Archived 2026-09-02._ + +- `[2026-08-16]` **Held an operator-approved swap window because the baseline invalidated its premise.** Operator approved ~65 min of `char-rp-reasoning` downtime to A/B Fable-Fusion 711 against Dark-Scarlett on refusals. The DS baseline then came back **0.0%/1.4%** — no gap for a candidate to close, so the window would have bought no decisive signal *and* a second window would still be needed once a reproducing battery existed. Held the swap, reported, and routed to brokkr-smithy-dev for the battery that actually produced the refusals. The general rule (action-relevance): **approval is for a plan, not a ritual — when new evidence kills the plan's premise, surface it rather than spend the budget.** Nothing deployed, no downtime taken, seat untouched. + _Archived 2026-09-02._ + +# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan + +## The finding + +`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB +Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 — +carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90% +(571 MB free)**. + +⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use +SLC/pSLC with a real controller, so the **284 GB written since boot is +unremarkable and wear is NOT the driver**. I had framed it as a clock ticking; +that was wrong and the correction matters. What actually justifies the work: + +1. **It is on the USB bus** — a reset or re-enumeration drops the *root + filesystem* out from under a running hypervisor whose guests keep executing. + NAND quality is irrelevant to that. +2. **6 GB has no headroom** — `/usr` alone is 3.7 GB. +3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty. +4. **It has blocked patching for months** — the operator-visible symptom and the + real urgency. + +## The patching blockage (measured) + +`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` / +Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on +`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime +because it cannot take a kernel. + +⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set +includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs +landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl, +glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction +and wedging dpkg on a hypervisor running five guests. Partial escape hatch if +patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives` +keeps downloads off root, but the kernel still lands in `/boot`. + +## Mitigation applied 2026-08-17 — root 90% → 76% + +| step | effect | +|---|---| +| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth | +| vacuumed the journal | **freed 446 MB** | +| `apt-get clean` | 79 MB | +| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** | +| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM | + +571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped +through the ZFS-backed journal. + +⚠ **Stopping journald over SSH kills your own session** — it takes the +connection's logging path with it. The first attempt died mid-swap, leaving the +dataset staged and the move incomplete (host was never at risk; journald +socket-activated straight back). Redo as a detached `systemd-run` transient unit. +Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`. + +Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction +the wear argument no longer justifies touching a service `pvestatd` depends on. + +## The plan — split boot from root (operator's proposal, strictly better) + +My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator +proposed keeping boot on the DOM with a fallback image and putting all its files +on ZFS. That is better and I should have gotten there myself** — I had assumed +boot and root must share a device. + +| | device | contents | written when | +|---|---|---|---| +| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates | +| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe | + +Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read +ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and +`zstd_compress` enabled and **GRUB cannot read those**. + +**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no +`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a +day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on +the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset +mid-run no longer kills the running system. Free upside: boot environments +(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`). + +**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`; +**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running +initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM +(leave it on the DOM LV — **never** swap on a zvol). + +**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live +root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools +import by scan today, which is a coin-flip when the initramfs must find root. +Set the cachefile before rebuilding the initramfs. + +Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no +mirrored boot needed). `dd` it off-box before anything else; refresh after kernel +updates. + +## ⚠ Blast radius — the gating constraint, invisible from the host itself + +**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two +dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably: + +- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup` +- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu` + +Known incident shape — the only remedy for esh-docker-vm's D-state is a host +reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite +risks corruption under `soft`. Quiesce both before any reboot of this host. +Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning. + +## Also identified + +- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas + being CT 109 on pfi-pve. +- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six + passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`), + and **nothing running but base systemd, sshd, cron, postfix** since 30 March. + That resolves the dashboard's long-standing "role TBC". Retire rather than + migrate. +- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels. + +## Staging executed 2026-08-18 — everything but the reboot + +Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green: +`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate, +4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml` +(ZFS initramfs, grub.cfg, both menu entries, grubenv). + +**`grub-install` is deliberately NOT run.** The ESP stub still points at the old +`/boot` inside the ext4 root, so the host's boot path is byte-identical to the +last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is +`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot. + +Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M +(new) + `pve-swap` 256 M (was 768 M). + +### The three landmines staging found + +1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and + mounted ext4 cannot shrink — freeing space from root needs a rescue boot, + which costs the "one reboot" property the design rests on. Only live source + was the swap LV. Operator chose shrink-to-256M over drop-entirely. +2. **The one-pool cachefile would have broken the NAS.** `zpool set + cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a + cachefile flips the host from `zfs-import-scan` to `zfs-import-cache` + (verified: scan active, cache inactive beforehand), so a cache holding only + `nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve + bind mounts spanning all three pools. Every export would come up empty and + both `hard` NFS clients would hang. +3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.** + Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from + `grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader + cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the + probe failure is swallowed by `2>/dev/null || true`. The same feature set + that forced `/boot` to stay ext4 also corrupts the kernel command line, which + the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` + drop-in (last `root=` wins) plus explicit `pve-zfs-root` and + `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are + derived from pool member device paths and would shift if the mirror changed. + +**The transferable lesson from (3):** the original verify grepped for +`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in +was added that grep passes — while pool-less entries sit in the menu untouched. +The check that holds walks every `linux` line, takes the **last** `root=`, and +asserts it against a known-good set. **Assert the effective value, not the +presence of a substring.** + +### One-shot boot, not a new default + +`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses +`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot +returns to ext4 by itself on the next reboot — no console, no hands. That matters +more here than on a normal host: a hang at an initramfs prompt takes CT 103 down +and the NFS clients hang rather than fail. Only after a second clean ZFS boot +should the saved default move. + +### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`) + +- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB + compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root + LV was live during the read, so a restore replays the ext4 journal. Not + fixable with an LVM snapshot: the VG has no free extents. +- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB, + 644 entries, full proxmox shim/grub EFI chain). This is the higher-quality + boot-chain artifact; the dd image is the belt-and-braces full-device restore. +- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new + grub.cfg, fstab, LVM/ZFS/blkid state. + ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs > + file` redirects only the last command, so `pvs`/`vgs` output leaked into the + tar stream on stdout. Group with `{ …; } > file`. + +Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at +`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600, +sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus +captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and +every guest config. **The newest on-disk copy before this was June 2024.** +Commits `2275e11`, `3e31175`, `8ddc87c`. + _Archived 2026-09-02._ + +# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control + +Two unrelated housekeeping jobs from the same session, both with durable lessons. + +## irv-ml1 — 782 GB reclaimed + +Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and +**74%** (477 G). + +**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22 +GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF` +(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had +**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*, +not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and +nobody opened that shed for 15 months. Re-verified the consumer check immediately +before deleting, not just during the audit. + +**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G, +`modelscope` 29 G (mtime **2024-04-23**). + +**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV +fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion` +28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2` +1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**. + +### The lesson: one dead-looking app pinned three delete targets + +`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on +0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale +opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it +would have half-broken a live listener in a way that surfaces only at its next +restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind), +verified :7860 released and handles at zero, *then* deleted. + +⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a +*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker +`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs +from uv. Checking open handles **per path** is what separated them — the earlier +"not running" read would have deleted the wrong thing. + +⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still +showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't +(`zfs list -t snapshot` empty); second check showed 477 G at 74%. + +All 16 containers and both systemd services verified healthy afterward. + +## Homepage under version control + +`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose +config lived only on the host. Its version history was **six hand-rolled +`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files + +`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone. +105 cards across 19 groups, no empty groups. + +⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** — +`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated +the top of the file. All five engines were discovering correctly the whole time. + +**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a +**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` → +**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible); +dropped the `UltraSeedbox` layout group (nothing provides it). + +⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not** +cover `http://10.0.50.45:5100/` — the container log carried `Host validation +failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200. +`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600. + +⚠ **Homepage renders client-side** — grepping the served HTML to verify a config +change gave two false readings (a stale prerender, then an empty page). +`GET /api/services` is the honest instrument, and config changes need a +**recreate**, not a restart (a restart keeps the cached render in the writable +layer). + +⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it +also removed a host-side `README.md` in the conf dir. Content survived (it is now +in the repo README) but that was a side effect, not a plan. + +Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]]. + _Archived 2026-09-02._ + +- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`. + _Archived 2026-09-02._ + +# Fleet IPv6 state + the real VPN topology (verified 2026-08-17) + +Written because the operator expects to reference this "before too long" — the +driver is an **ESH fiber install landing 2026-08-18 that puts the house behind +CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than +a nice-to-have. + +## Why IPv6 suddenly matters: CGNAT at ESH + +New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — +the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a +reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: +a global v6 address on each UDM restores a routable endpoint pair without +depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the +most likely first consumer of fleet IPv6. + +Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. + +## The topology — as VERIFIED, not as assumed + +Three transports, three different technologies. Do not describe this as "a +WireGuard mesh"; a prior session did and was corrected. + +| Link | Transport | Evidence | +|---|---|---| +| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | +| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | +| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | +| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | + +**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** +It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, +**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, +`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: +`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. + +**The FortiGate never terminates WireGuard — it port-forwards to the host that +does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a +session that reads "colo + WireGuard" and concludes the edge must be upgraded is +chasing a non-problem. Do not re-derive this. + +## Per-site IPv6 state (2026-08-17) + +| Site | Edge | IPv6 | +|---|---|---| +| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | +| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | +| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | + +## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed + +`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. + +The proof matters because the naive read is wrong: the WAN sits at `c110::48` +and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning +`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which +on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at +**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID +settings yielding the same /64 is the signature of a single-/64 delegation. + +**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the +delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a +routable address on the *WAN*, not a LAN prefix. + +The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` +with no size field alongside), so the prefix-ID test is the only read-only-ish way +to establish it from the API. + +## What a v6 mesh actually requires (and what it does NOT) + +**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both +Site Magic and WireGuard need a routable address on the router/host WAN side, plus +inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and +then reverted. + +**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both +`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard +reconfiguration** once (a) the host holds a routable v6 address (today: link-local +`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 +over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). + +**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, +one field to flip when wanted. + +**Inbound v6 is default-deny and that held without intervention.** The UDM runs +the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` +endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 +rules exist," which is wrong and alarming. Use +`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` +for both families with `Allow Return Traffic`; the only v6-specific allows are +link-local plumbing (ND solicit/advert, RA, DHCPv6). + +## The stability problem — design around it up front + +All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, +not a static assignment). A three-way mesh where every node can move is fragile; +WireGuard tolerates one roaming end, not all of them. + +The fleet already solves this on the v4 side — IPsec peers use **hostnames** +(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to +AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet +Cloudflare DNS-edit token, so this is self-serve. + +## Access recipes (cost a prior session real time) + +- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, + `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. + Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. + Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. +- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused + (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says + `root@10.250.50.252`; believe it. +- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), + password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration + `drain()` hangs the session**; read until the `ana-gw #` prompt and answer + `--More--` with a space. Two invocations timed out at 3 min before this was fixed. + +## Changes made and reverted this session + +- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED + on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, + verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). +- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server + private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` + (client configs carry private keys) were all mode **644**. Now **600**, and + `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — + WireGuard holds keys in kernel memory, so no restart was needed. The parent + `/etc/wireguard` was already 700, which capped the real exposure to root-capable + contexts inside the LXC — but the modes were still wrong. + +--- + +## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the +## per-REQUEST truth, not the total — eight /64s exist and are unclaimed + +The section above concludes AT&T hands out a single `/64` and that the +`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was +itself superseded later in the same session, and the correction never made it +into memory** — it survived only in the session transcript, and was recovered +2026-08-24 while assessing a proposal to grab more prefixes. + +Reading the **BGW's own LAN statistics page** gave the whole picture: + +``` +BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix +BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself +Delegated 2600:1700:b25:c11f::/64 <- what the UDM got +``` + +**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and +delegating from the top down — the UDM got `c11f`, the last one. So +`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**, +sitting unclaimed. + +Both observations are compatible, which is why the first one looked conclusive: +the prefix-ID test only carves *within* a delegation already held, so a UDM +holding one `/64` cannot move it no matter what prefix-ID you set. The BGW +issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**. + +**Consequence — the ceiling is the requester, not the carrier.** More prefixes +need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the +UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it +is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this +looks like BGW rationing rather than a provisioning-profile limit. + +Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6 +2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`. + +### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise + +The seven unclaimed `/64`s stay unclaimed. Two facts close it: + +- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin + on it — so the cheap path (let the UDM take the `/60` directly and carve it + natively, as it already does at ESH) does not exist here. +- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which + requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing + (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6 + firewall policy in nftables on that VM**, because routing v6 around the UDM + bypasses its zone firewall entirely and would leave every LAN host globally + reachable. + +Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.** +Do not re-propose on the strength of "there are seven free prefixes" — the +prefixes are real, the firewall rebuild is why nobody wants them. + _Archived 2026-09-02._ + +- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4/dev/null || true`. The same feature set - that forced `/boot` to stay ext4 also corrupts the kernel command line, which - the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg` - drop-in (last `root=` wins) plus explicit `pve-zfs-root` and - `pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are - derived from pool member device paths and would shift if the mirror changed. - -**The transferable lesson from (3):** the original verify grepped for -`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in -was added that grep passes — while pool-less entries sit in the menu untouched. -The check that holds walks every `linux` line, takes the **last** `root=`, and -asserts it against a known-good set. **Assert the effective value, not the -presence of a substring.** - -### One-shot boot, not a new default - -`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses -`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot -returns to ext4 by itself on the next reboot — no console, no hands. That matters -more here than on a normal host: a hang at an initramfs prompt takes CT 103 down -and the NFS clients hang rather than fail. Only after a second clean ZFS boot -should the saved default move. - -### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`) - -- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB - compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root - LV was live during the read, so a restore replays the ext4 journal. Not - fixable with an LVM snapshot: the VG has no free extents. -- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB, - 644 entries, full proxmox shim/grub EFI chain). This is the higher-quality - boot-chain artifact; the dd image is the belt-and-braces full-device restore. -- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new - grub.cfg, fstab, LVM/ZFS/blkid state. - ⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs > - file` redirects only the last command, so `pvs`/`vgs` output leaked into the - tar stream on stdout. Group with `{ …; } > file`. - -Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at -`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600, -sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus -captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and -every guest config. **The newest on-disk copy before this was June 2024.** -Commits `2275e11`, `3e31175`, `8ddc87c`. diff --git a/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md b/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md deleted file mode 100644 index 9bbe06d..0000000 --- a/persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md +++ /dev/null @@ -1,185 +0,0 @@ -# Fleet IPv6 state + the real VPN topology (verified 2026-08-17) - -Written because the operator expects to reference this "before too long" — the -driver is an **ESH fiber install landing 2026-08-18 that puts the house behind -CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than -a nice-to-have. - -## Why IPv6 suddenly matters: CGNAT at ESH - -New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic — -the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a -reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch: -a global v6 address on each UDM restores a routable endpoint pair without -depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the -most likely first consumer of fleet IPv6. - -Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**. - -## The topology — as VERIFIED, not as assumed - -Three transports, three different technologies. Do not describe this as "a -WireGuard mesh"; a prior session did and was corrected. - -| Link | Transport | Evidence | -|---|---|---| -| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets | -| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse | -| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt | -| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below | - -**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.** -It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`, -**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`, -`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`: -`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1. - -**The FortiGate never terminates WireGuard — it port-forwards to the host that -does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a -session that reads "colo + WireGuard" and concludes the edge must be upgraded is -chasing a non-problem. Do not re-derive this. - -## Per-site IPv6 state (2026-08-17) - -| Site | Edge | IPv6 | -|---|---|---| -| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` | -| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` | -| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only | - -## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed - -`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests. - -The proof matters because the naive read is wrong: the WAN sits at `c110::48` -and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning -`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which -on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at -**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID -settings yielding the same /64 is the signature of a single-/64 delegation. - -**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the -delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a -routable address on the *WAN*, not a LAN prefix. - -The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false` -with no size field alongside), so the prefix-ID test is the only read-only-ish way -to establish it from the API. - -## What a v6 mesh actually requires (and what it does NOT) - -**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both -Site Magic and WireGuard need a routable address on the router/host WAN side, plus -inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and -then reverted. - -**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both -`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard -reconfiguration** once (a) the host holds a routable v6 address (today: link-local -`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337 -over v6 — the existing VIP is v4-only (`extip 38.120.12.42`). - -**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`, -one field to flip when wanted. - -**Inbound v6 is default-deny and that held without intervention.** The UDM runs -the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule` -endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6 -rules exist," which is wrong and alarming. Use -`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic` -for both families with `Allow Return Traffic`; the only v6-specific allows are -link-local plumbing (ND solicit/advert, RA, DHCPv6). - -## The stability problem — design around it up front - -All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA, -not a static assignment). A three-way mesh where every node can move is fragile; -WireGuard tolerates one roaming end, not all of them. - -The fleet already solves this on the v4 side — IPsec peers use **hostnames** -(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to -AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet -Cloudflare DNS-edit token, so this is self-serve. - -## Access recipes (cost a prior session real time) - -- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`, - `unifi/esh-udmpm-api-key`) against `https:///proxy/network/…`, `curl -sk`. - Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here. - Writes are `PUT …/rest/networkconf/<_id>` with the **full** object. -- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused - (`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says - `root@10.250.50.252`; believe it. -- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev), - password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration - `drain()` hangs the session**; read until the `ana-gw #` prompt and answer - `--More--` with a space. Two invocations timed out at 3 min before this was fixed. - -## Changes made and reverted this session - -- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED - on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`, - verified. Pre-change snapshots kept in the session scratchpad only (ephemeral). -- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server - private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf` - (client configs carry private keys) were all mode **644**. Now **600**, and - `keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact — - WireGuard holds keys in kernel memory, so no restart was needed. The parent - `/etc/wireguard` was already 700, which capped the real exposure to root-capable - contexts inside the LXC — but the modes were still wrong. - ---- - -## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the -## per-REQUEST truth, not the total — eight /64s exist and are unclaimed - -The section above concludes AT&T hands out a single `/64` and that the -`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was -itself superseded later in the same session, and the correction never made it -into memory** — it survived only in the session transcript, and was recovered -2026-08-24 while assessing a proposal to grab more prefixes. - -Reading the **BGW's own LAN statistics page** gave the whole picture: - -``` -BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix -BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself -Delegated 2600:1700:b25:c11f::/64 <- what the UDM got -``` - -**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and -delegating from the top down — the UDM got `c11f`, the last one. So -`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**, -sitting unclaimed. - -Both observations are compatible, which is why the first one looked conclusive: -the prefix-ID test only carves *within* a delegation already held, so a UDM -holding one `/64` cannot move it no matter what prefix-ID you set. The BGW -issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**. - -**Consequence — the ceiling is the requester, not the carrier.** More prefixes -need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the -UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it -is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this -looks like BGW rationing rather than a provisioning-profile limit. - -Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6 -2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`. - -### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise - -The seven unclaimed `/64`s stay unclaimed. Two facts close it: - -- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin - on it — so the cheap path (let the UDM take the `/60` directly and carve it - natively, as it already does at ESH) does not exist here. -- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which - requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing - (UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6 - firewall policy in nftables on that VM**, because routing v6 around the UDM - bypasses its zone firewall entirely and would leave every LAN host globally - reachable. - -Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.** -Do not re-propose on the strength of "there are seven free prefixes" — the -prefixes are real, the firewall rebuild is why nobody wants them. diff --git a/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md b/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md deleted file mode 100644 index e5d0603..0000000 --- a/persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md +++ /dev/null @@ -1,79 +0,0 @@ -# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control - -Two unrelated housekeeping jobs from the same session, both with durable lessons. - -## irv-ml1 — 782 GB reclaimed - -Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and -**74%** (477 G). - -**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22 -GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF` -(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had -**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*, -not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and -nobody opened that shed for 15 months. Re-verified the consumer check immediately -before deleting, not just during the audit. - -**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G, -`modelscope` 29 G (mtime **2024-04-23**). - -**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV -fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion` -28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2` -1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**. - -### The lesson: one dead-looking app pinned three delete targets - -`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on -0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale -opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it -would have half-broken a live listener in a way that surfaces only at its next -restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind), -verified :7860 released and handles at zero, *then* deleted. - -⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a -*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker -`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs -from uv. Checking open handles **per path** is what separated them — the earlier -"not running" read would have deleted the wrong thing. - -⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still -showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't -(`zfs list -t snapshot` empty); second check showed 477 G at 74%. - -All 16 containers and both systemd services verified healthy afterward. - -## Homepage under version control - -`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose -config lived only on the host. Its version history was **six hand-rolled -`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files + -`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone. -105 cards across 19 groups, no empty groups. - -⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** — -`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated -the top of the file. All five engines were discovering correctly the whole time. - -**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a -**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` → -**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible); -dropped the `UltraSeedbox` layout group (nothing provides it). - -⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not** -cover `http://10.0.50.45:5100/` — the container log carried `Host validation -failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200. -`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600. - -⚠ **Homepage renders client-side** — grepping the served HTML to verify a config -change gave two false readings (a stale prerender, then an empty page). -`GET /api/services` is the honest instrument, and config changes need a -**recreate**, not a restart (a restart keeps the cached render in the writable -layer). - -⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it -also removed a host-side `README.md` in the conf dir. Content survived (it is now -in the repo README) but that was a side effect, not a plan. - -Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]]. diff --git a/persistent-memory.md b/persistent-memory.md index b9983a0..20c7d99 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-01_ +_Last updated: 2026-09-02_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,150 +108,53 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing it up as an inference AND training box is the next action.** Nothing is training._ +_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is +baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has +not arrived._ -- **✅ pfi-gx10 BASELINED AND OUTFITTED (2026-09-02) — bare metal, training stack verified.** - `uv` + torch 2.14.0+cu130 + transformers/peft/trl/datasets/bitsandbytes all clean on aarch64; - nvcc, NVIDIA Container Toolkit, docker group, NOPASSWD sudo. **Baseline: ~79.35 s/it** on the - run-3c shape (seq 16384, batch 2, LoRA r64, flex_attention, harness chunked loss) -> 604 steps - ≈ 13.3 h vs ana-ml2's 2.2-2.7 h. ⚠ **6x slower where compute predicts 2.7x** — likely memory - bandwidth; **capacity box, not throughput box.** Port NOT done (operator stood it down; the - baseline was the goal). Still on Wi-Fi at `10.100.10.226`, not racked. - → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` -- **▶ (superseded) pfi-gx10 as an inference + training box.** The operator forgot to rack it; it is - still on his desk on Wi-Fi at `10.100.10.226` (temp DHCP). Already headless and registered. - **The box is BARE — no torch, no nvcc, no CUDA stack.** Purpose is local inference *and* - **run 3c**, whose 49 GB bf16 base fits 121 GB unified with room where ana-ml2 was tight. - **First move is a throughput probe, not a port**: ARM64 torch, load the base, ten steps, - report s/it — that decides whether 604 steps is an overnight run or unusable. Racking is - one command afterwards (`playbooks/gx10-rack-network.yaml`, VLAN 50, static `10.100.50.60`). - ⚠ Triton has no sm_121 support; compiled deps are per-arch unknowns. - ✅ **Liveness confirmed 2026-09-01 23:09** — up 11h48m, idle, 118/121 GB free, 822 GB disk - free, still bare (no torch/nvcc/uv), still on Wi-Fi at `10.100.10.226`, wired NIC has no - cable. ⚠ **Racking is NOT a prerequisite** — the Wi-Fi 7 link measures **67 MB/s over SSH** - (49 GB base in ~12 min), so the throughput probe can run from the desk today. - ⚠ `nvidia-smi` FB Memory `N/A` is CORRECT for GB10 unified memory, not a fault. - ✅ **OUTFITTED 2026-09-01 (userspace)** — **RULING: BARE METAL, not Proxmox** (no aarch64 PVE - build; the GPU is on-package and cache-coherent over NVLink-C2C so passthrough would partition - the unified memory that is the whole point; fleet GPU hosts are all bare metal). - Installed `uv` 0.12.9 + `~/ml/.venv` with torch **2.14.0+cu130** and the full training stack - (transformers/accelerate/peft/trl/datasets/bitsandbytes) — **all import clean on aarch64**. - **CUDA WORKS: sm_121, 121.6 GiB addressable, ~93 TFLOP/s bf16** (A6000-class speed, 2.5x the - memory). ⚠ **sm_121 is NOT in torch's arch list** — everything JITs from sm_120 PTX, so - **first use of every kernel pays a JIT compile** and cubin-only libraries will fail. - ⚠⚠ **ALWAYS WARM UP BEFORE TIMING** — an unwarmed benchmark read 27 TFLOP/s (it was timing the - JIT) and nearly became a phantom "tensor cores are broken" report. - ✅ **PRIVILEGED HALF DONE 2026-09-01** — nvcc, NVIDIA Container Toolkit, docker group; playbook - `gx10-outfit-privileged.yaml` green 5/5 (a CUDA container sees the GB10). **Bare metal, fully - outfitted.** ⚠⚠ **I was NEVER blocked** — infra-ops had NOPASSWD sudo all along; I probed with - **`sudo -n -v`, which is WRONG on sudo >= 1.9.15** (it refreshes a timestamp NOPASSWD never - creates) and asked the operator for a password that did not exist to need. **Always - `sudo -n true`.** Fixed in `scripts/elway` (same probe, would have broken fleet-wide as hosts - pass sudo 1.9.13) and in auto-memory. Password rotated + vaulted at - `pfi-gx10/infra-ops-password` (kept not locked: **no iDRAC on this box**, so it is the only - console break-glass). ⚠ Docker's buildkit DB was **already corrupt** — the restart surfaced it, - did not cause it; cleared `/var/lib/docker/buildkit` (cache, 0 images/containers). - → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD** (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout - can). Everything else waits - on the inventory — the invoice omits risers, GPU cables and the 6 high-performance fans, but - omission is not proof of removal. **Add free-PCIe-slot inventory to the pull** — an NVMe - add-in card would bypass the SAS/SATA backplane and moot the drive-bay question. - ⚠ ~1 kW loaded — same class as the box that tripped the breaker; pick the NH3 circuit before - racking. → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` -- **⏸ RUN 3c STILL HELD — but the plan has changed.** Config `/tank/erp-tune/run-03c.json` - validated, relaunch is one command on ana-ml2. **It is now intended to move to the GX10 - instead**, which is the power answer rather than a power triage. Do not relaunch on ana-ml2 - without deciding that first. Exactly TWO 3c launches, only one died. - → `persistent-memory.d/2026-08-27-run3c-launch-count-reconstruction.md` -- **🟢 ALTHING 3.2.0 DEPLOYED + MIGRATION COMPLETE on nh3-dev (2026-09-01 22:13-22:26).** - forseti authored, infra-ops deployed; **5/5 pane routes pinned**, all four affected seats - re-declared within ~12 min of individual notice, log clean. Pane-poking replaces detached - waiters because **CC 2.1.257 reaps detached tasks when the seat sits in `shell`**. - ⚠ **`uv tool install .` is a SILENT NO-OP on reinstall** — matches the source *spec*, not - contents; exits 0 having done nothing. **Always `--force` from a local path.** - ⚠ **The herald's exclusion reason ("process gone, or pid reused") is FALSE for the migration - case** — all four declaring processes were alive 4 days, no pid wrap; real cause is the route - predating the guard fields. Don't debug a dead agent that isn't. - ⚠ **My miss:** I called non-Claude pane delivery an "open risk" when forseti had personally - measured it on Aug 28 — didn't check whether it was already settled before calling it open. - Delivery now confirmed on `claude`, `pi` and `grok` seats. - ⚠ One-tick latency after `declare` reads as failure; `--discover-pid` on a non-Claude process - tree remains untested. - 🔴 **3.2.1 SAME NIGHT (22:40)** — 3.2.0 wrote pokes into panes people were **typing in** - (append + Enter submits their half-written message; hit the operator within the hour). Guard 4 - pins `idle_cursor` and stays silent when the column moved. ⚠ **2 of 5 pane seats CANNOT be - guarded** — `kimi` and `pi` report no cursor at all, so bil- and regin-smithy-dev stay - permanently exposed; "3.2.1 fixes it" is true only where the cursor is legible. Mitigation for - those = `althing-route release` (pull-only). ⚠ Declare with an EMPTY input line or the pin - captures a composing cursor. **Outcome:** delivery verified on **four TUI families** - (claude/codex 3, grok 7, kimi/pi none). The two unguardable seats split — **regin-smithy-dev - RELEASED (now pull-only, operator composes in that pane routinely); bil-smithy-dev KEPT** - (pane is poke-driven, and pull-only had already cost them a notice unread for days). - ⚠ **Deciding variable is WHO COMPOSES in the pane, not risk appetite** — a flat rule either - way would be wrong for one of them. ⚠ **Path to closing the hole (regin's):** a moment-check - reading pane CONTENT (input-line length) rather than cursor COLUMN would cover pi/kimi — - column is a proxy, empty-input-line is the real predicate. - ✅ **3.2.2 deployed 22:51** — closes 4 of 5: `--discover-pid` was a **BUG not an unknown** - (matched `comm=="claude"`, so 4 of 5 pane seats could never have used it; now matches - `pane_command`), exclusion message split rather than softened, `--force` in the runbook, - latency + status-not-durable in the skill. 🔵 **PENDING OPERATOR:** the kimi/pi seats guard 4 - cannot cover — poke unguarded / write-without-Enter / pull-only. **Recommended per-seat - status quo**; write-without-Enter beats it for neither seat. - ✅ **DEPLOY IS NOW ONE COMMAND: `scripts/deploy-althing.sh`** (`--check` reports drift), - documented in `docs/runbooks/althing-deploy.md`. **FOUR surfaces**: tool install (⚠ `--force` - or it is a silent no-op), herald restart, skill sync, **plugin reinstall** — the fourth had no - runbook step and drifted 5 days, costing the SessionStart auto-declare hook entirely. - ⚠ **eshpfi owns the plugin deploy, NOT the althing repo** — it targets per-machine paths, and - althing's `sync_skill.sh` deliberately reaches into no other tree; that boundary is correct. - → `persistent-memory.d/2026-09-01-althing-320-deploy.md` -- **✅ A6000 window CLOSED 2026-09-01 — operator freed ComfyUI's VRAM instead of anyone - stopping it.** ⚠ It was a **model unload, not a stop**: `comfyui` still up 8 days, same pid, - HTTP 200, 18,500 -> 612 MiB. ⚠ **The resulting ~43.8 GB free is a snapshot, not a floor** — - ComfyUI reloads ~18.5 GB on the next render, putting the real floor at ~25.3 GB against - FireRedAudio's ~26 GB. Coordination shrank to "don't render during the bench"; tts-dev's ask - to make, comfy-dev's to grant. Also measured: **`dots-tts` holds 14,430 MiB vs a ~6 GB - baseline** — tts-dev's **prompt-feature cache**, capped at 32 entries 2026-08-14 after two - incidents (infra-ops' caching-allocator hypothesis was wrong, withdrawn). 3090 at 76% behind - a warn-only watchdog. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` + can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both + are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose + nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are + **512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count + from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking. + → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. - Branch (b) rejected on the trade (comfy-dev corrected the *reason*: only comfyui mounts - `/storetank`, so (b) was unavailable during the transition, not structurally). - **TWO decisions now sitting with the operator: (1) add 2× 2 TB SATA SSD to the R750xa order - or the pool lands at ~84%; (2) retain vs reclaim irv-ml1's `/storetank` after the move — - RETAIN recommended** (two paywalled pins mean a from-source rebuild cannot reproduce the - stack; and under a no-spend striped destination it is the only redundancy). comfy-dev also - found ~215 GiB of unreferenced weights and offers a per-file prune audit — **explicitly NOT - gating**, because pruning all of it still lands the as-bought mirror at 72%. ✅ **H3 encoder - pin RESOLVED** — infra-ops' nvfp4-vs-int8 question had the direction backwards (pinned on - Ampere sm_86; Ada sm_89 is a strict superset, so the move cannot break it). Pin is about - **VRAM not architecture**; stands. The 26 GB int8 is a **spare, not an orphan** — do not - delete. - → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` + **TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills + all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB + at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim + irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a + from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered + and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` +- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps. + ⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive + each way** with **13 hosts dark including three SureFire CLIENT machines**, which no + `save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not + executed** — operator stood it down. + → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships, "trust the HTTP status, not `error_code`" still applies on running instances. -- **🟢 Matrix upgraded and hardened.** Synapse v1.159.0, Element v1.12.27, `/_synapse/admin` - LAN-only, appservice namespace opened, alias convention ratified. Miranda provisioned; her - summons reached the operator's watch. → `persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md` -- **🟢 phasefinal.com LIVE** behind Cloudflare with edge caching + Always Online; apex 301s to - www. `inquiry@` alias added by the operator. Stack `stacks/phasefinal-web/`. -- **🟢 char-rp restored** on ana-ml2 **GPU1** (was dead 7 days on a quant-flag/checkpoint - mismatch crash-loop). ⚠ **GPU1 is now at 96,384 / 97,887 MiB — ~1.5 GB free, six tenants.** - GPU0's ~44 GB free is a DELIBERATE scratch reserve, not headroom to reclaim. - **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD` - from a `.env` — until then `stacks/synapse/` and the live file have DIVERGED and - `deploy-stack.sh` must not be used on it (README says so). -- ⚠ **`/mnt/smithy` will be MISSING after every ana-ml2 reboot** — manual by design. - → `persistent-memory.d/2026-08-23-smithy-mount-ana-ml2.md` + from the vault rather than an inline value. +- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart + hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether + their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because + they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render + agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that + evaporates the moment ComfyUI reloads. ## Recent decisions +- `[2026-09-02]` **pfi-gx10 BASELINED: ~79.35 s/it on the run-3c shape, and the training stack works on aarch64/sm_121.** Median across 7 timed steps, 0.19% spread, 86/121 GB. **6× slower than ana-ml2 where compute predicts 2.7×** → likely memory-bandwidth-bound; **capacity box, not throughput box.** Ruled **bare metal, not Proxmox** (no aarch64 PVE; the GPU is on-package and cache-coherent, so passthrough would partition the unified memory that is the whole point). ⚠ `sm_121` is NOT in torch's arch list — everything JITs from sm_120 PTX, so **warm up before timing anything** (an unwarmed bench read 27 TFLOP/s against a true 93). → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` +- `[2026-09-02]` **I priced a failure in the units I happened to be measuring — operator overruled me, correctly.** Recommended run 3c to ana-ml2 by costing a breaker trip as "≤50 steps ≈ 11 min of recompute". It is a **40-minute drive each way** with **13 Anaheim hosts dark, three of them SureFire CLIENT machines**. `save_steps` caps the recompute, never the outage. ⚠ **General form: a metric in hand will volunteer itself as the unit of risk.** → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` +- `[2026-09-02]` **althing 3.2.0→3.2.4 deployed, and ALTHING DEPLOY IS FOUR SURFACES not three.** The fourth (plugin) had no runbook step and was frozen at Aug 28 — **missing the SessionStart/SessionEnd hooks and `pane-route.sh` entirely**, so "CC seats re-declare automatically" was never true here. Now one command (`scripts/deploy-althing.sh`). ⚠ `uv tool install .` **without `--force` is a silent no-op**. ⚠ **A missing deploy surface presents as "the migration needs manual work", not as an error.** → `persistent-memory.d/2026-09-01-althing-320-deploy.md` +- `[2026-09-01]` **irv-ml1 GPU resident map, and `dots-tts` holds 14,430 MiB against a ~6 GB baseline** — tts-dev's prompt-feature cache, capped at 32 entries after two incidents; the cap still permits a long way of growth. 3090 at 76% behind a warn-only watchdog. ⚠ Restates the GPU-ordering foot-gun: `device_ids: ["1"]` is the A6000 in a container, but a bare native `CUDA_VISIBLE_DEVICES=1` gets the **3090**. → `persistent-memory.d/2026-09-01-irv-ml1-gpu-residents.md` - `[2026-09-01]` **The Ada inference server is a used Dell R750xa (`JPJ1ZP3`) and the reseller stripped four things Dell shipped** — half the RAM, the 2400 W PSUs, and the GPU risers/cables/fans are absent from the invoice. Card is **RTX 6000 Ada**, not L40S. GPU power chain resolved via NVIDIA `930-00030-1546-000`. NVMe in the drive bays is CLOSED (SAS/SATA backplane). → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` - `[2026-09-01]` **pfi-gx10 onboarded headless — and it is the intended new home for run 3c, which died on a tripped breaker.** GB10/sm_121/aarch64, 121 GB unified. NOT racked yet. Bare of any CUDA stack; probe throughput before porting. → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` - `[2026-09-01]` **Ada migration is `zfs send` (branch a) — and the DESTINATION IS SMALLER THAN THE SOURCE.** 99 MB/s measured; ~3.9 h. ⚠ Measured 2026-09-01: `storetank` = 1.81 TiB pool, 1.45 TiB used, **80% CAP already**, compression off / `compressratio 1.00x` (safetensors are incompressible — no win at recv). Settled payload **~1.47 TiB**; the R750xa's as-bought 2× 1.92 TB mirrored is ~1.75 TiB → arrival at **~84%**. Fix = 2× 2 TB SATA SSD on the buy list (6 bays free) → ~3.57 TiB at ~41% with redundancy; **pair the two NEW drives together** (a mirror vdev caps at its smallest member). ⚠ **Pruning is NOT a substitute** — comfy-dev found ~215 GiB unreferenced, and deleting every byte still lands the as-bought mirror at 72%: the constraint is **vdev layout**, not payload, so the prune audit and the drive purchase are independent and neither gates the cutover. ⚠ **"Onboarded" is not "landed"** — infra-ops read `ALLOC` mid-pull and re-added the whole batch on top, inflating 84% to a quoted 90%. Also: **branch (b)'s original reason was WRONG** — comfy-dev enumerated all 12 containers, only `comfyui` mounts `/storetank`, so (b) was unavailable *during the transition*, not structurally (right conclusion, wrong reason — infra-ops reasoned about the BOX when the question was the MOUNT). Plus the retain-vs-reclaim call and the two-boxes confusion (the Ada box and the GX10 are DIFFERENT machines). → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` @@ -339,40 +242,24 @@ _As of 2026-09-01 — **the GX10 is on the operator's desk, NOT racked. Standing - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). -- `[2026-08-18]` **esh-pve-nas migration STAGED — and staging is where three landmines surfaced, none of which the plan predicted.** (1) The runbook's `/boot` LV had **nowhere to live**: VG `pve` had 4 MB free and mounted ext4 cannot shrink, so the space came from the 768 MB swap LV (operator's call: shrink to 256 MB, not drop). (2) The runbook's `zpool set cachefile=… nvme` would have **broken the NAS** — populating a cache flips the host to import-by-cache, and a one-pool cache leaves `ssd`+`tank` unimported under CT 103's twelve bind mounts. (3) **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`**, because GRUB's ZFS reader cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` and the probe failure is swallowed. All three were caught by *verify steps that asserted effective state*, not by reading the plan. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md` - -- `[2026-08-17]` **esh-pve-nas PVE root is on a USB DOM — mitigated, and the migration replanned to split boot from root.** Operator's design beats my reinstall plan; wear was never the issue, blocked patching is. → `persistent-memory.d/2026-08-17-esh-pve-nas-dom.md` - -- `[2026-08-17]` **irv-ml1 cleared of 782 GB, and Homepage brought under version control.** One dead-looking Gradio app pinned three delete targets at once; `/opt/ComfyUI` is NOT the ComfyUI that serves. → `persistent-memory.d/2026-08-17-irv-ml1-cleanup-homepage.md` - -- `[2026-08-17]` **Gen seat swapped to `absolute-heresy` — and the three bugs the swap exposed are worth more than the swap.** Candidate `MuXodious/Qwen3.8-27B-absolute-heresy` (Heretic v1.4.0 + SOMPOA, T377) beat the incumbent on refusals AND KL simultaneously, which is the unusual part — those normally trade off. Validated on the probe port per operator ruling, promoted, all 7 aliases green. **Durable lessons banked:** (1) **A CPU-only MTP head hash can replace the ~56 GB bf16 acceptance gate.** The `Qwen3_5ForConditionalGeneration` wrapper never loads the MTP head, so PEFT merges / Heretic runs / llm-compressor passes all leave `mtp.*` pristine — hashing it against a head we have already measured (the incumbent's, 47.7%) answers the question for free. Predicted 47.7%, measured 47.2%. Saved downing meromero. Tool: `services/gen-seat-mixed-quant/compare_mtp_head.py` (hash bf16 via **uint8 reinterpret** — numpy has no bfloat16). (2) **`post_quant.py` assumed a standalone `model-mtp.safetensors`**; a full checkpoint keeps `mtp.*` in a NUMBERED shard, so the copy silently no-op'd while the index was still rewritten to point at a file that never existed — 15 unresolvable tensors behind a correct-looking tensor count. Its own FAILED-CHECKS assertion caught it; **that is why the check exists rather than an assumption**. Fixed to extract. (3) **A probe that does not mirror the live seat manufactures failures.** `serve_probe.sh` hardcoded `:latest` (seat is a pinned nightly for #51113), had no tool-call/reasoning parsers, and its `--speculative-config` JSON died twice on quoting — **bash BRACE-EXPANDS `{"a":1,"b":2}` on the comma** unless single-quoted at the REMOTE shell. Adding the seat's flags took the surface test from 5/6 to **6/6**; the "tool calling broken" result was pure probe config. Commits `7997f11`,`254c588`,`2c36028`,`b0c2d3d`,`993421b`. - -- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates one /64 PER REQUEST** (`2600:1700:b25:c11f::/64`) — and the BGW holds the whole `/60`, rationing `c118`–`c11f` one at a time while keeping `c110`–`c117`. So eight /64s exist; UniFi just solicits once. ⛔ **CLOSED 2026-08-24 — operator ruling, do not re-raise:** the BGW has **no IP-passthrough** (operator confirmed, and we have admin on it), so the only route to the other seven is a multi-DUID DHCPv6 client on a VM — which means split-stack routing and rebuilding the entire v6 firewall policy off the UDM. Juice not worth the squeeze. NH3 LANs stay v6-off. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`. - -- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4