memory: snapshot — heresy gen seat live, irv-ml1 cleared, homepage repo'd, esh-pve-nas DOM planned
This commit is contained in:
@@ -0,0 +1,126 @@
|
||||
# esh-pve-nas — PVE root on a USB DOM: diagnosis, mitigation, migration plan
|
||||
|
||||
## The finding
|
||||
|
||||
`esh-pve-nas` (`esh-nas-pve.esteban.net`, 10.0.50.55) runs PVE root off a **USB
|
||||
Disk-on-Module** — `sdq`, 7.3 GB, `ID_BUS=usb`, `ID_VENDOR=NORELSYS`, model 1081 —
|
||||
carved into a 512 MB ESP + 768 MB swap + a **6 GB ext4 root** that was at **90%
|
||||
(571 MB free)**.
|
||||
|
||||
⚠ **Operator corrected my first read: it is a DOM, not a thumb drive.** DOMs use
|
||||
SLC/pSLC with a real controller, so the **284 GB written since boot is
|
||||
unremarkable and wear is NOT the driver**. I had framed it as a clock ticking;
|
||||
that was wrong and the correction matters. What actually justifies the work:
|
||||
|
||||
1. **It is on the USB bus** — a reset or re-enumeration drops the *root
|
||||
filesystem* out from under a running hypervisor whose guests keep executing.
|
||||
NAND quality is irrelevant to that.
|
||||
2. **6 GB has no headroom** — `/usr` alone is 3.7 GB.
|
||||
3. **Unmirrored**, while 928 GB of mirrored NVMe sits 96% empty.
|
||||
4. **It has blocked patching for months** — the operator-visible symptom and the
|
||||
real urgency.
|
||||
|
||||
## The patching blockage (measured)
|
||||
|
||||
`apt-get -s dist-upgrade`: **225 packages pending, 161 carrying `deb12uN` /
|
||||
Debian-Security bumps** including `ssh 1:9.2p1-2+deb12u10`. Host sits on
|
||||
`pve-manager/8.4.11` vs sibling esh-pve's **8.4.14**, with 20 weeks uptime
|
||||
because it cannot take a kernel.
|
||||
|
||||
⚠ **Ordering is load-bearing: migrate FIRST, patch after.** The pending set
|
||||
includes `proxmox-kernel-6.8.12-42-pve-signed` — ~250 MB of kernel + initramfs
|
||||
landing in `/boot`, **which is on root**. Unpacking 225 packages (dpkg, perl,
|
||||
glibc-adjacent) into 1.3 GB of headroom risks filling the disk mid-transaction
|
||||
and wedging dpkg on a hypervisor running five guests. Partial escape hatch if
|
||||
patching truly cannot wait: `apt-get -o Dir::Cache::Archives=/nvme/tmp/apt-archives`
|
||||
keeps downloads off root, but the kernel still lands in `/boot`.
|
||||
|
||||
## Mitigation applied 2026-08-17 — root 90% → 76%
|
||||
|
||||
| step | effect |
|
||||
|---|---|
|
||||
| capped journald (`SystemMaxUse=64M`; was **fully default/uncapped**) | stops unbounded growth |
|
||||
| vacuumed the journal | **freed 446 MB** |
|
||||
| `apt-get clean` | 79 MB |
|
||||
| `/root/neo` (2024 Intel NEO OpenCL debs) → `/nvme/tmp/root-neo-20260817/` | 259 MB — **moved, not deleted** |
|
||||
| **`/var/log/journal` relocated onto ZFS** (`nvme/varlog`) | dominant writer off the DOM |
|
||||
|
||||
571 MB → **1.4 GB free**. All five guests stayed up; a fresh `logger` round-tripped
|
||||
through the ZFS-backed journal.
|
||||
|
||||
⚠ **Stopping journald over SSH kills your own session** — it takes the
|
||||
connection's logging path with it. The first attempt died mid-swap, leaving the
|
||||
dataset staged and the move incomplete (host was never at risk; journald
|
||||
socket-activated straight back). Redo as a detached `systemd-run` transient unit.
|
||||
Script + reason live at `root@10.0.50.55:/root/move-journal-to-zfs.sh`.
|
||||
|
||||
Deliberately **not** done: moving `/var/lib/rrdcached`. With the DOM correction
|
||||
the wear argument no longer justifies touching a service `pvestatd` depends on.
|
||||
|
||||
## The plan — split boot from root (operator's proposal, strictly better)
|
||||
|
||||
My first plan was a full reinstall to a mirrored-NVMe ZFS root. **The operator
|
||||
proposed keeping boot on the DOM with a fallback image and putting all its files
|
||||
on ZFS. That is better and I should have gotten there myself** — I had assumed
|
||||
boot and root must share a device.
|
||||
|
||||
| | device | contents | written when |
|
||||
|---|---|---|---|
|
||||
| boot | DOM `sdq` | ESP + `/boot` (ext4) | only on kernel/GRUB updates |
|
||||
| root | `nvme` pool | `nvme/ROOT/pve-1` | constantly, on mirrored NVMe |
|
||||
|
||||
Keeping `/boot` on **ext4** is the point, not a compromise: GRUB never has to read
|
||||
ZFS, which matters because the `nvme` pool has `encryption`, `large_dnode` and
|
||||
`zstd_compress` enabled and **GRUB cannot read those**.
|
||||
|
||||
**Why it beats the reinstall:** the `nvme` pool survives (no guest migration, no
|
||||
`ssd`/`tank` export-import, no reinstall); downtime is **one reboot** not half a
|
||||
day; **rollback is a GRUB menu entry** because the ext4 root stays untouched on
|
||||
the DOM; and it retires the actual top risk — with root on NVMe a USB bus reset
|
||||
mid-run no longer kills the running system. Free upside: boot environments
|
||||
(`zfs snapshot nvme/ROOT/pve-1@pre-upgrade`).
|
||||
|
||||
**Preconditions verified already met:** UEFI + `grub-efi-amd64 2.06-13+pmx7`;
|
||||
**`zfs-initramfs 2.2.8-pve1` already installed with 76 ZFS files in the running
|
||||
initrd**; root only 4.3 GB to copy; swap 767 MB / 123 MB used against 125 GB RAM
|
||||
(leave it on the DOM LV — **never** swap on a zvol).
|
||||
|
||||
**Two traps:** `canmount=noauto` on the root dataset or ZFS mounts over the live
|
||||
root; and `cachefile` is `none` with a **0-byte `/etc/zfs/zpool.cache`** — pools
|
||||
import by scan today, which is a coin-flip when the initramfs must find root.
|
||||
Set the cachefile before rebuilding the initramfs.
|
||||
|
||||
Operator ruled a **cloned DOM image is sufficient** boot-path insurance (no
|
||||
mirrored boot needed). `dd` it off-box before anything else; refresh after kernel
|
||||
updates.
|
||||
|
||||
## ⚠ Blast radius — the gating constraint, invisible from the host itself
|
||||
|
||||
**CT 103 `esh-nas` (10.0.50.50) IS the NAS, and it runs on this host.** Two
|
||||
dependents mount it over **`hard`** NFS — they do not fail, they hang unkillably:
|
||||
|
||||
- **esh-docker-vm** (10.0.50.45): `/mnt/books`, `/mnt/backup`
|
||||
- **esh-pve** (10.0.250.35): `/mnt/pve/esh-nas`, `/mnt/pve/tank-vmbu`
|
||||
|
||||
Known incident shape — the only remedy for esh-docker-vm's D-state is a host
|
||||
reboot, and `/mnt/books` was *deliberately* left `hard` because calibre's SQLite
|
||||
risks corruption under `soft`. Quiesce both before any reboot of this host.
|
||||
Recorded in `servers/esh-pve-nas/README.md` as a never-reboot-casually warning.
|
||||
|
||||
## Also identified
|
||||
|
||||
- **`esh-nas` is CT 103** on esh-pve-nas — structurally the same shape as ana-nas
|
||||
being CT 109 on pfi-pve.
|
||||
- **`ESH-FileBot` (CT 106, 10.0.50.70) is an empty shell** — 80 GB rootfs, six
|
||||
passthrough mounts (`books`/`documents`/`music`/`share`/`pvestore`/`ssd-pvestore`),
|
||||
and **nothing running but base systemd, sshd, cron, postfix** since 30 March.
|
||||
That resolves the dashboard's long-standing "role TBC". Retire rather than
|
||||
migrate.
|
||||
- Both ESH hypervisors have **20 weeks uptime** and differing PVE patch levels.
|
||||
|
||||
Runbook: `docs/runbooks/esh-pve-nas-boot-migration.md`. Config snapshot off-box at
|
||||
`nh3-dev:~/backups/esh-pve-nas/pve-config-snapshot-20260818T043027Z.tar.gz` (0600,
|
||||
sha256 `dc312793d027dc43…`) — `/etc/pve`, network, fstab, apt, authorized_keys plus
|
||||
captured `zpool`/`zfs`/`disk-by-id`/`lsblk`-with-serials/`pvesm`/`dpkg` state and
|
||||
every guest config. **The newest on-disk copy before this was June 2024.**
|
||||
Commits `2275e11`, `3e31175`, `8ddc87c`.
|
||||
@@ -0,0 +1,79 @@
|
||||
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
|
||||
|
||||
Two unrelated housekeeping jobs from the same session, both with durable lessons.
|
||||
|
||||
## irv-ml1 — 782 GB reclaimed
|
||||
|
||||
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
|
||||
**74%** (477 G).
|
||||
|
||||
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
|
||||
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
|
||||
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
|
||||
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
|
||||
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
|
||||
nobody opened that shed for 15 months. Re-verified the consumer check immediately
|
||||
before deleting, not just during the audit.
|
||||
|
||||
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
|
||||
`modelscope` 29 G (mtime **2024-04-23**).
|
||||
|
||||
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
|
||||
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
|
||||
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
|
||||
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
|
||||
|
||||
### The lesson: one dead-looking app pinned three delete targets
|
||||
|
||||
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
|
||||
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
|
||||
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
|
||||
would have half-broken a live listener in a way that surfaces only at its next
|
||||
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
|
||||
verified :7860 released and handles at zero, *then* deleted.
|
||||
|
||||
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
|
||||
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
|
||||
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
|
||||
from uv. Checking open handles **per path** is what separated them — the earlier
|
||||
"not running" read would have deleted the wrong thing.
|
||||
|
||||
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
|
||||
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
|
||||
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
|
||||
|
||||
All 16 containers and both systemd services verified healthy afterward.
|
||||
|
||||
## Homepage under version control
|
||||
|
||||
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
|
||||
config lived only on the host. Its version history was **six hand-rolled
|
||||
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
|
||||
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
|
||||
105 cards across 19 groups, no empty groups.
|
||||
|
||||
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
|
||||
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
|
||||
the top of the file. All five engines were discovering correctly the whole time.
|
||||
|
||||
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
|
||||
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
|
||||
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
|
||||
dropped the `UltraSeedbox` layout group (nothing provides it).
|
||||
|
||||
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
|
||||
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
|
||||
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
|
||||
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
|
||||
|
||||
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
|
||||
change gave two false readings (a stale prerender, then an empty page).
|
||||
`GET /api/services` is the honest instrument, and config changes need a
|
||||
**recreate**, not a restart (a restart keeps the cached render in the writable
|
||||
layer).
|
||||
|
||||
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
|
||||
also removed a host-side `README.md` in the conf dir. Content survived (it is now
|
||||
in the repo README) but that was a side effect, not a plan.
|
||||
|
||||
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
|
||||
Reference in New Issue
Block a user