memory: snapshot — heresy gen seat live, irv-ml1 cleared, homepage repo'd, esh-pve-nas DOM planned

2026-08-17 21:44:29 -07:00
parent 8ddc87c852
commit 356752d99c
3 changed files with 213 additions and 2 deletions
@@ -0,0 +1,79 @@
+# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
+
+Two unrelated housekeeping jobs from the same session, both with durable lessons.
+
+## irv-ml1 — 782 GB reclaimed
+
+Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
+**74%** (477 G).
+
+**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
+GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
+(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
+**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
+not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
+nobody opened that shed for 15 months. Re-verified the consumer check immediately
+before deleting, not just during the audit.
+
+**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
+`modelscope` 29 G (mtime **2024-04-23**).
+
+**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
+fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
+28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
+1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
+
+### The lesson: one dead-looking app pinned three delete targets
+
+`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
+0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
+opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
+would have half-broken a live listener in a way that surfaces only at its next
+restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
+verified :7860 released and handles at zero, *then* deleted.
+
+⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
+*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
+`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
+from uv. Checking open handles **per path** is what separated them — the earlier
+"not running" read would have deleted the wrong thing.
+
+⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
+showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
+(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
+
+All 16 containers and both systemd services verified healthy afterward.
+
+## Homepage under version control
+
+`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
+config lived only on the host. Its version history was **six hand-rolled
+`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
+`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
+105 cards across 19 groups, no empty groups.
+
+⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
+`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
+the top of the file. All five engines were discovering correctly the whole time.
+
+**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
+**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
+**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
+dropped the `UltraSeedbox` layout group (nothing provides it).
+
+⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
+cover `http://10.0.50.45:5100/` — the container log carried `Host validation
+failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
+`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
+
+⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
+change gave two false readings (a stale prerender, then an empty page).
+`GET /api/services` is the honest instrument, and config changes need a
+**recreate**, not a restart (a restart keeps the cached render in the writable
+layer).
+
+⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
+also removed a host-side `README.md` in the conf dir. Content survived (it is now
+in the repo README) but that was a side effect, not a plan.
+
+Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].