memory: snapshot — session stand-down; GX10 baselined, althing at 3.2.4
Rewrote the in-flight section to reflect that nothing is running and the operator stood the session down. The GX10 work is recorded as what it was asked to be: a baseline for the box and a check that the tooling loads, with the run-3c port scoped but explicitly declined. Added index lines for four decisions that had detail files but no pointer: the GX10 baseline at 79.35 s/it, the costing error the operator overruled, the althing four-surface deploy finding, and the irv-ml1 GPU resident map. Auto-archival fired at the soft cap and moved 17 entries dated on or before 2026-08-18 to archival-memory.md, holding back 4 that carry open deferred-work pointers. The index went from 410 lines to 296, mostly by rewriting in-flight rather than by archiving -- the dated log was not what made it long.
This commit is contained in:
@@ -1,52 +0,0 @@
|
||||
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
|
||||
|
||||
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
|
||||
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
|
||||
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
|
||||
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
|
||||
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
|
||||
|
||||
**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
|
||||
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
|
||||
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
|
||||
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
|
||||
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
|
||||
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
|
||||
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
|
||||
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
|
||||
`reference_abliteration_mtp_lessons`.
|
||||
|
||||
**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
|
||||
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
|
||||
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
|
||||
(`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
|
||||
+ `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
|
||||
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
|
||||
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
|
||||
sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
|
||||
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
|
||||
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
|
||||
`--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
|
||||
|
||||
**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
|
||||
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
|
||||
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
|
||||
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
|
||||
no cap OOM'd — cudagraph captured to batch-512).
|
||||
|
||||
**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
|
||||
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
|
||||
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
|
||||
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
|
||||
|
||||
**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
|
||||
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
|
||||
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
|
||||
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
|
||||
|
||||
**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
|
||||
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
|
||||
isolate before deleting.**
|
||||
|
||||
Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
|
||||
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
|
||||
@@ -1,196 +0,0 @@
|
||||
# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
|
||||
|
||||
## The finding
|
||||
|
||||
`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
|
||||
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
|
||||
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
|
||||
(571 MB free)**.
|
||||
|
||||
⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
|
||||
SLC/pSLC with a real controller, so the **284 GB written since boot is
|
||||
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
|
||||
that was wrong and the correction matters. What actually justifies the work:
|
||||
|
||||
1. **It is on the USB bus** — a reset or re-enumeration drops the *root
|
||||
filesystem* out from under a running hypervisor whose guests keep executing.
|
||||
NAND quality is irrelevant to that.
|
||||
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
|
||||
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
|
||||
4. **It has blocked patching for months** — the operator-visible symptom and the
|
||||
real urgency.
|
||||
|
||||
## The patching blockage (measured)
|
||||
|
||||
`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
|
||||
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
|
||||
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
|
||||
because it cannot take a kernel.
|
||||
|
||||
⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
|
||||
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
|
||||
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
|
||||
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
|
||||
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
|
||||
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
|
||||
keeps downloads off root, but the kernel still lands in `/boot`.
|
||||
|
||||
## Mitigation applied 2026-08-17 — root 90% → 76%
|
||||
|
||||
| step | effect |
|
||||
|---|---|
|
||||
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
|
||||
| vacuumed the journal | **freed 446 MB** |
|
||||
| `apt-get clean` | 79 MB |
|
||||
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
|
||||
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
|
||||
|
||||
571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
|
||||
through the ZFS-backed journal.
|
||||
|
||||
⚠ **Stopping journald over SSH kills your own session** — it takes the
|
||||
connection's logging path with it. The first attempt died mid-swap, leaving the
|
||||
dataset staged and the move incomplete (host was never at risk; journald
|
||||
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
|
||||
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
|
||||
|
||||
Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
|
||||
the wear argument no longer justifies touching a service `pvestatd` depends on.
|
||||
|
||||
## The plan — split boot from root (operator's proposal, strictly better)
|
||||
|
||||
My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
|
||||
proposed keeping boot on the DOM with a fallback image and putting all its files
|
||||
on ZFS. That is better and I should have gotten there myself** — I had assumed
|
||||
boot and root must share a device.
|
||||
|
||||
| | device | contents | written when |
|
||||
|---|---|---|---|
|
||||
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
|
||||
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
|
||||
|
||||
Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
|
||||
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
|
||||
`zstd_compress` enabled and **GRUB cannot read those**.
|
||||
|
||||
**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
|
||||
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
|
||||
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
|
||||
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
|
||||
mid-run no longer kills the running system. Free upside: boot environments
|
||||
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
|
||||
|
||||
**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
|
||||
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
|
||||
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
|
||||
(leave it on the DOM LV — **never** swap on a zvol).
|
||||
|
||||
**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
|
||||
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
|
||||
import by scan today, which is a coin-flip when the initramfs must find root.
|
||||
Set the cachefile before rebuilding the initramfs.
|
||||
|
||||
Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
|
||||
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
|
||||
updates.
|
||||
|
||||
## ⚠ Blast radius — the gating constraint, invisible from the host itself
|
||||
|
||||
**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
|
||||
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
|
||||
|
||||
- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
|
||||
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
|
||||
|
||||
Known incident shape — the only remedy for esh-docker-vm's D-state is a host
|
||||
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
|
||||
risks corruption under `soft`. Quiesce both before any reboot of this host.
|
||||
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
|
||||
|
||||
## Also identified
|
||||
|
||||
- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
|
||||
being CT 109 on pfi-pve.
|
||||
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
|
||||
passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
|
||||
and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
|
||||
That resolves the dashboard's long-standing "role TBC". Retire rather than
|
||||
migrate.
|
||||
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
|
||||
|
||||
## Staging executed 2026-08-18 — everything but the reboot
|
||||
|
||||
Two rerunnable elway playbooks, 0 failed steps, 17/17 verify green:
|
||||
`playbooks/esh-pve-nas-stage-zfs-root.yaml` (LV surgery, `/boot` populate,
|
||||
4.3 GB root rsync in 228 s, fstab) and `playbooks/esh-pve-nas-stage-bootloader.yaml`
|
||||
(ZFS initramfs, grub.cfg, both menu entries, grubenv).
|
||||
|
||||
**`grub-install` is deliberately NOT run.** The ESP stub still points at the old
|
||||
`/boot` inside the ext4 root, so the host's boot path is byte-identical to the
|
||||
last 140 days and an unplanned reboot mid-staging is a non-event. Cutover is
|
||||
`grub-install` + `grub-reboot pve-zfs-root` + `zfs set mountpoint=/` + reboot.
|
||||
|
||||
Final DOM layout: `pve-root` 6.04 G (untouched, the rollback) + `pve-boot` 512 M
|
||||
(new) + `pve-swap` 256 M (was 768 M).
|
||||
|
||||
### The three landmines staging found
|
||||
|
||||
1. **The `/boot` LV had nowhere to live.** VG `pve` had **4 MB free**, and
|
||||
mounted ext4 cannot shrink — freeing space from root needs a rescue boot,
|
||||
which costs the "one reboot" property the design rests on. Only live source
|
||||
was the swap LV. Operator chose shrink-to-256M over drop-entirely.
|
||||
2. **The one-pool cachefile would have broken the NAS.** `zpool set
|
||||
cachefile=… nvme` looks scoped and safe; it is the opposite. Populating a
|
||||
cachefile flips the host from `zfs-import-scan` to `zfs-import-cache`
|
||||
(verified: scan active, cache inactive beforehand), so a cache holding only
|
||||
`nvme` leaves `ssd` and `tank` unimported at boot — and CT 103 has twelve
|
||||
bind mounts spanning all three pools. Every export would come up empty and
|
||||
both `hard` NFS clients would hang.
|
||||
3. **`update-grub` silently emitted a pool-less `root=ZFS=/ROOT/pve-1`.**
|
||||
Debian's `10_linux` builds `${rpool}${bootfs}`; `rpool` comes from
|
||||
`grub-probe --target=fs_label`, which returns empty because GRUB's ZFS reader
|
||||
cannot open a pool with `encryption`/`large_dnode`/`zstd_compress` — and the
|
||||
probe failure is swallowed by `2>/dev/null || true`. The same feature set
|
||||
that forced `/boot` to stay ext4 also corrupts the kernel command line, which
|
||||
the design did not anticipate. Fixed with a `/etc/default/grub.d/zfs-root.cfg`
|
||||
drop-in (last `root=` wins) plus explicit `pve-zfs-root` and
|
||||
`pve-ext4-rollback` entries carrying stable ids — the auto-generated ids are
|
||||
derived from pool member device paths and would shift if the mirror changed.
|
||||
|
||||
**The transferable lesson from (3):** the original verify grepped for
|
||||
`root=ZFS=nvme/ROOT/pve-1` *appearing somewhere* in grub.cfg. Once the drop-in
|
||||
was added that grep passes — while pool-less entries sit in the menu untouched.
|
||||
The check that holds walks every `linux` line, takes the **last** `root=`, and
|
||||
asserts it against a known-good set. **Assert the effective value, not the
|
||||
presence of a substring.**
|
||||
|
||||
### One-shot boot, not a new default
|
||||
|
||||
`GRUB_DEFAULT=saved` with grubenv pinned to `pve-ext4-rollback`, and cutover uses
|
||||
`grub-reboot pve-zfs-root` so ZFS is tried **exactly once**. A failed ZFS boot
|
||||
returns to ext4 by itself on the next reboot — no console, no hands. That matters
|
||||
more here than on a normal host: a hang at an initramfs prompt takes CT 103 down
|
||||
and the NFS clients hang rather than fail. Only after a second clean ZFS boot
|
||||
should the saved default move.
|
||||
|
||||
### Off-box artifacts (`nh3-dev:~/backups/esh-pve-nas/`)
|
||||
|
||||
- `dom-sdq-20260818.img.zst` — full DOM image, 7,837,450,240 B raw / 2.38 GiB
|
||||
compressed, zstd XXH64 verified. ⚠ **Crash-consistent, not clean** — the root
|
||||
LV was live during the read, so a restore replays the ext4 journal. Not
|
||||
fixable with an LVM snapshot: the VG has no free extents.
|
||||
- `bootchain-20260818.tar.gz` — clean, consistent tar of `/boot` + ESP (88 MB,
|
||||
644 entries, full proxmox shim/grub EFI chain). This is the higher-quality
|
||||
boot-chain artifact; the dd image is the belt-and-braces full-device restore.
|
||||
- `pve-config-snapshot-20260818T051*.tar.gz` — 147 entries incl. the new
|
||||
grub.cfg, fstab, LVM/ZFS/blkid state.
|
||||
⚠ Building this the first time produced a **corrupt archive**: `pvs; vgs; lvs >
|
||||
file` redirects only the last command, so `pvs`/`vgs` output leaked into the
|
||||
tar stream on stdout. Group with `{ …; } > file`.
|
||||
|
||||
Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Earlier config snapshot at
|
||||
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
|
||||
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
|
||||
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
|
||||
every guest config. **The newest on-disk copy before this was June 2024.**
|
||||
Commits `2275e11`, `3e31175`, `8ddc87c`.
|
||||
@@ -1,185 +0,0 @@
|
||||
# Fleet IPv6 state + the real VPN topology (verified 2026-08-17)
|
||||
|
||||
Written because the operator expects to reference this "before too long" — the
|
||||
driver is an **ESH fiber install landing 2026-08-18 that puts the house behind
|
||||
CGNAT**, which breaks Site Magic on IPv4 and makes IPv6 load-bearing rather than
|
||||
a nice-to-have.
|
||||
|
||||
## Why IPv6 suddenly matters: CGNAT at ESH
|
||||
|
||||
New ESH fiber (installing 2026-08-18) hands out a **CGNAT IPv4**. Site Magic —
|
||||
the UniFi-to-UniFi SD-WAN mesh tunnel that currently links NH3 ↔ ESH — needs a
|
||||
reachable endpoint, and a CGNAT address is not one. **IPv6 is the escape hatch:
|
||||
a global v6 address on each UDM restores a routable endpoint pair without
|
||||
depending on the ISP's v4 at all.** That, not the WireGuard RA mesh, is the
|
||||
most likely first consumer of fleet IPv6.
|
||||
|
||||
Operator expects addresses at **Anaheim shortly** and **ESH 2026-08-18**.
|
||||
|
||||
## The topology — as VERIFIED, not as assumed
|
||||
|
||||
Three transports, three different technologies. Do not describe this as "a
|
||||
WireGuard mesh"; a prior session did and was corrected.
|
||||
|
||||
| Link | Transport | Evidence |
|
||||
|---|---|---|
|
||||
| NH3 UDM ↔ ESH UDM | **Site Magic** (`vpn_type: sdwan-mesh-tunnel`) | UDM `networkconf`, carries all 7 ESH subnets |
|
||||
| Colo FortiGate ↔ NH3 UDM | **IPsec IKEv2** | FG `pfi-ana-nh3` → 70.230.226.88, **158M pkt rx / 165M tx** — the fleet workhorse |
|
||||
| Colo FortiGate ↔ ESH UDM | **IPsec IKEv2** | FG `ana-to-eshudm` → 70.181.90.232, 53K/56K pkt |
|
||||
| Remote-access VPN | **WireGuard, host-based on `ana-wg`** | see below |
|
||||
|
||||
**WireGuard is an RA (remote-access) convention only — it is NOT the site mesh.**
|
||||
It runs on `ana-wg` (LXC 113, Debian 12, 10.250.50.252), interface `wg0`,
|
||||
**UDP 31337**, tunnel subnet `10.30.10.0/24`, 3 peers (`tc2-mac`, `vh-iphone`,
|
||||
`vh-mba26`). Reached from outside via a FortiGate VIP `wg-to-ana-wg`:
|
||||
`38.120.12.42:31337/udp → 10.250.50.252:31337` on wan1.
|
||||
|
||||
**The FortiGate never terminates WireGuard — it port-forwards to the host that
|
||||
does.** FortiOS 7.2.10 has no native WireGuard (Fortinet added it in 7.4), so a
|
||||
session that reads "colo + WireGuard" and concludes the edge must be upgraded is
|
||||
chasing a non-problem. Do not re-derive this.
|
||||
|
||||
## Per-site IPv6 state (2026-08-17)
|
||||
|
||||
| Site | Edge | IPv6 |
|
||||
|---|---|---|
|
||||
| **NH3** | UDM SE | **WAN live** — `2600:1700:b25:c110::48` via DHCPv6 on ATTFiber. All 5 LANs `ipv6_interface_type=none` |
|
||||
| **Anaheim colo** | FortiGate-80F, FortiOS 7.2.10 | **None.** `diagnose ipv6 address list` → only loopback `::1`; every physical iface `ipv6: ::/0` |
|
||||
| **ESH home** | UDM Pro Max | **None.** Both WANs `wan_type_v6=disabled`; link-local only |
|
||||
|
||||
## AT&T delegates exactly ONE /64 at NH3 — proven, not assumed
|
||||
|
||||
`2600:1700:b25:c11f::/64`. **One.** Not the /60 the addressing pattern suggests.
|
||||
|
||||
The proof matters because the naive read is wrong: the WAN sits at `c110::48`
|
||||
and the LAN got `c11f::1/64`, which looks exactly like slot 15 of a /60 spanning
|
||||
`c110`–`c11f`. It isn't. Forcing the prefix ID from auto to a manual `0` — which
|
||||
on a real /60 would relocate the LAN to `c110::1/64` — left the subnet at
|
||||
**`c11f::1/64`, stable across a 4-minute settle**. Two different prefix-ID
|
||||
settings yielding the same /64 is the signature of a single-/64 delegation.
|
||||
|
||||
**Consequence: exactly one VLAN can have IPv6 at NH3**, unless AT&T enlarges the
|
||||
delegation. If Site Magic-over-v6 is the goal that is fine — Site Magic needs a
|
||||
routable address on the *WAN*, not a LAN prefix.
|
||||
|
||||
The controller never exposes the PD size directly (`wan_dhcpv6_pd_size_auto:false`
|
||||
with no size field alongside), so the prefix-ID test is the only read-only-ish way
|
||||
to establish it from the API.
|
||||
|
||||
## What a v6 mesh actually requires (and what it does NOT)
|
||||
|
||||
**Does NOT require prefix delegation.** PD hands addresses to LAN *clients*. Both
|
||||
Site Magic and WireGuard need a routable address on the router/host WAN side, plus
|
||||
inbound reachability. Enabling PD on a LAN is orthogonal — this was tested and
|
||||
then reverted.
|
||||
|
||||
**ana-wg's WireGuard socket is ALREADY dual-stack** — `ss` shows both
|
||||
`0.0.0.0:31337` and `[::]:31337`. It will accept IPv6 peers with **no WireGuard
|
||||
reconfiguration** once (a) the host holds a routable v6 address (today: link-local
|
||||
`fe80::be24:11ff:fed7:e4b7` only) and (b) the FortiGate passes inbound UDP 31337
|
||||
over v6 — the existing VIP is v4-only (`extip 38.120.12.42`).
|
||||
|
||||
**NH3 UDM's own WG server is v4-pinned** — `wireguard_interface_binding_mode_ip_version: 'v4'`,
|
||||
one field to flip when wanted.
|
||||
|
||||
**Inbound v6 is default-deny and that held without intervention.** The UDM runs
|
||||
the **zone-based** firewall (66 policies). ⚠ The legacy `rest/firewallrule`
|
||||
endpoint returns **0 rules** on this box — a quick check there reads as "no IPv6
|
||||
rules exist," which is wrong and alarming. Use
|
||||
`v2/api/site/default/firewall-policies`. WAN→LAN default is `Block All Traffic`
|
||||
for both families with `Allow Return Traffic`; the only v6-specific allows are
|
||||
link-local plumbing (ND solicit/advert, RA, DHCPv6).
|
||||
|
||||
## The stability problem — design around it up front
|
||||
|
||||
All three endpoints will hold **dynamic** addresses (NH3's came via DHCPv6 IA_NA,
|
||||
not a static assignment). A three-way mesh where every node can move is fragile;
|
||||
WireGuard tolerates one roaming end, not all of them.
|
||||
|
||||
The fleet already solves this on the v4 side — IPsec peers use **hostnames**
|
||||
(`ana-fw.phasefinal.com`, `nh3.phasefinal.com`), not raw IPs. **Extend that to
|
||||
AAAA records** and dynamic prefixes stop mattering. infra-ops holds the fleet
|
||||
Cloudflare DNS-edit token, so this is self-serve.
|
||||
|
||||
## Access recipes (cost a prior session real time)
|
||||
|
||||
- **UniFi UDMs** — `X-API-KEY` from the vault (`secret get unifi/pfi-udmse-api-key`,
|
||||
`unifi/esh-udmpm-api-key`) against `https://<ip>/proxy/network/…`, `curl -sk`.
|
||||
Classic `api/s/default/rest/networkconf` + `stat/device` carry everything here.
|
||||
Writes are `PUT …/rest/networkconf/<_id>` with the **full** object.
|
||||
- **`ana-wg` is `root@`, NOT `infra-ops@`** — the shared infra-ops key is refused
|
||||
(`Permission denied (publickey,password)`). `servers/ana-wg/ssh-target` says
|
||||
`root@10.250.50.252`; believe it.
|
||||
- **FortiGate** — paramiko via `uv run --with paramiko` (no sshpass on nh3-dev),
|
||||
password `secret get fortigate/ana-gw-infra-ops-password`. ⚠ **A fixed-duration
|
||||
`drain()` hangs the session**; read until the `ana-gw #` prompt and answer
|
||||
`--More--` with a space. Two invocations timed out at 3 min before this was fixed.
|
||||
|
||||
## Changes made and reverted this session
|
||||
|
||||
- **Enabled PD on `nh3-iot` (VLAN 90)** to measure the delegation, then **REVERTED
|
||||
on operator instruction** — all 5 NH3 LANs are back to `ipv6_interface_type=none`,
|
||||
verified. Pre-change snapshots kept in the session scratchpad only (ephemeral).
|
||||
- **`ana-wg` WireGuard key material was world-readable** — `wg0.conf` (server
|
||||
private key + 2 peer PSKs), `keys/*_priv`, `keys/*_psk`, and `configs/*.conf`
|
||||
(client configs carry private keys) were all mode **644**. Now **600**, and
|
||||
`keys/` + `configs/` dirs **700**. `wg-quick@wg0` stayed active, 3 peers intact —
|
||||
WireGuard holds keys in kernel memory, so no restart was needed. The parent
|
||||
`/etc/wireguard` was already 700, which capped the real exposure to root-capable
|
||||
contexts inside the LXC — but the modes were still wrong.
|
||||
|
||||
---
|
||||
|
||||
## CORRECTION (recorded 2026-08-24): "AT&T delegates exactly ONE /64" is the
|
||||
## per-REQUEST truth, not the total — eight /64s exist and are unclaimed
|
||||
|
||||
The section above concludes AT&T hands out a single `/64` and that the
|
||||
`c110`/`c11f` pattern reading as a `/60` was a misread. **That conclusion was
|
||||
itself superseded later in the same session, and the correction never made it
|
||||
into memory** — it survived only in the session transcript, and was recovered
|
||||
2026-08-24 while assessing a proposal to grab more prefixes.
|
||||
|
||||
Reading the **BGW's own LAN statistics page** gave the whole picture:
|
||||
|
||||
```
|
||||
BGW WAN v6 2001:506:70b2:8958::1 <- AT&T's transit prefix
|
||||
BGW LAN v6 2600:1700:b25:c110::/64 <- the BGW keeps this for itself
|
||||
Delegated 2600:1700:b25:c11f::/64 <- what the UDM got
|
||||
```
|
||||
|
||||
**The BGW holds the `/60` and rations it**, keeping `c110`–`c117` for itself and
|
||||
delegating from the top down — the UDM got `c11f`, the last one. So
|
||||
`c118`–`c11f` are **eight delegatable /64s that genuinely exist and are yours**,
|
||||
sitting unclaimed.
|
||||
|
||||
Both observations are compatible, which is why the first one looked conclusive:
|
||||
the prefix-ID test only carves *within* a delegation already held, so a UDM
|
||||
holding one `/64` cannot move it no matter what prefix-ID you set. The BGW
|
||||
issues **one `/64` per IA_PD request**, and **UniFi solicits exactly once**.
|
||||
|
||||
**Consequence — the ceiling is the requester, not the carrier.** More prefixes
|
||||
need more IA_PD requests (multiple IAIDs, or multiple client DUIDs), which the
|
||||
UDM will not do. That is what makes a separate DHCPv6-PD client viable, and it
|
||||
is why "ask AT&T for a bigger delegation" may be aimed at the wrong party: this
|
||||
looks like BGW rationing rather than a provisioning-profile limit.
|
||||
|
||||
Live state at correction time: `wan_dhcpv6_pd_size: 64`, `wan1 v6
|
||||
2600:1700:b25:c110::48`, all 5 NH3 LANs still `ipv6_interface_type: none`.
|
||||
|
||||
### ⛔ CLOSED 2026-08-24 — operator ruling, do not re-raise
|
||||
|
||||
The seven unclaimed `/64`s stay unclaimed. Two facts close it:
|
||||
|
||||
- **The BGW has no IP-passthrough mode.** Operator confirmed, and we hold admin
|
||||
on it — so the cheap path (let the UDM take the `/60` directly and carve it
|
||||
natively, as it already does at ESH) does not exist here.
|
||||
- **The only remaining route is a multi-DUID DHCPv6 client on a VM**, which
|
||||
requires re-cabling to reach the BGW's DHCPv6 server, split-stack routing
|
||||
(UDM for v4, VM for v6), and — the actual cost — **rebuilding the whole IPv6
|
||||
firewall policy in nftables on that VM**, because routing v6 around the UDM
|
||||
bypasses its zone firewall entirely and would leave every LAN host globally
|
||||
reachable.
|
||||
|
||||
Operator's call: not worth it. **NH3 LANs stay `ipv6_interface_type: none`.**
|
||||
Do not re-propose on the strength of "there are seven free prefixes" — the
|
||||
prefixes are real, the firewall rebuild is why nobody wants them.
|
||||
@@ -1,79 +0,0 @@
|
||||
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
|
||||
|
||||
Two unrelated housekeeping jobs from the same session, both with durable lessons.
|
||||
|
||||
## irv-ml1 — 782 GB reclaimed
|
||||
|
||||
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
|
||||
**74%** (477 G).
|
||||
|
||||
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
|
||||
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
|
||||
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
|
||||
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
|
||||
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
|
||||
nobody opened that shed for 15 months. Re-verified the consumer check immediately
|
||||
before deleting, not just during the audit.
|
||||
|
||||
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
|
||||
`modelscope` 29 G (mtime **2024-04-23**).
|
||||
|
||||
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
|
||||
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
|
||||
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
|
||||
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
|
||||
|
||||
### The lesson: one dead-looking app pinned three delete targets
|
||||
|
||||
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
|
||||
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
|
||||
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
|
||||
would have half-broken a live listener in a way that surfaces only at its next
|
||||
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
|
||||
verified :7860 released and handles at zero, *then* deleted.
|
||||
|
||||
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
|
||||
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
|
||||
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
|
||||
from uv. Checking open handles **per path** is what separated them — the earlier
|
||||
"not running" read would have deleted the wrong thing.
|
||||
|
||||
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
|
||||
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
|
||||
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
|
||||
|
||||
All 16 containers and both systemd services verified healthy afterward.
|
||||
|
||||
## Homepage under version control
|
||||
|
||||
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
|
||||
config lived only on the host. Its version history was **six hand-rolled
|
||||
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
|
||||
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
|
||||
105 cards across 19 groups, no empty groups.
|
||||
|
||||
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
|
||||
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
|
||||
the top of the file. All five engines were discovering correctly the whole time.
|
||||
|
||||
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
|
||||
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
|
||||
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
|
||||
dropped the `UltraSeedbox` layout group (nothing provides it).
|
||||
|
||||
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
|
||||
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
|
||||
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
|
||||
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
|
||||
|
||||
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
|
||||
change gave two false readings (a stale prerender, then an empty page).
|
||||
`GET /api/services` is the honest instrument, and config changes need a
|
||||
**recreate**, not a restart (a restart keeps the cached render in the writable
|
||||
layer).
|
||||
|
||||
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
|
||||
also removed a host-side `README.md` in the conf dir. Content survived (it is now
|
||||
in the repo README) but that was a side effect, not a plan.
|
||||
|
||||
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
Reference in New Issue
Block a user