80 lines
4.2 KiB
Markdown
80 lines
4.2 KiB
Markdown
# irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
|
||
|
||
Two unrelated housekeeping jobs from the same session, both with durable lessons.
|
||
|
||
## irv-ml1 — 782 GB reclaimed
|
||
|
||
Root was at **92%** (148 G free), storetank **86%**. Now **64%** (635 G free) and
|
||
**74%** (477 G).
|
||
|
||
**Tier 1 — dead weights, 286 GB.** `/storetank/llm-models/Storage` (**217 G**, 22
|
||
GGUF repos, atimes Jan–May **2025**) plus `models--MaziyarPanahi--WizardLM-2-8x22B-GGUF`
|
||
(44 G) and `models--h2oai--h2ogpt-4096-llama2-13b-chat` (25 G). The 217 G pile had
|
||
**zero consumers** — no llama-swap, no llama.cpp, no textgen running *or installed*,
|
||
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
|
||
nobody opened that shed for 15 months. Re-verified the consumer check immediately
|
||
before deleting, not just during the audit.
|
||
|
||
**Tier 2 — regenerable caches, 194 GB.** `uv` 65 G + 60 G, `pip` 31 G + 8.7 G,
|
||
`modelscope` 29 G (mtime **2024-04-23**).
|
||
|
||
**Tier 3 — retired stacks, 302 GB** (operator: "those were old days… we're a UV
|
||
fleet now"): `/opt/fluxgym` 64 G, `/opt/ComfyUI` **native** 41 G, `/opt/stablediffusion`
|
||
28 G, `/opt/alltalk` 19 G, `/opt/o-textgen` 12 G, `/opt/sdnext` 3 G, `/opt/xttsv2`
|
||
1.8 G, `tabbyAPI` 3.1 G, **`miniconda3` 130 G**.
|
||
|
||
### The lesson: one dead-looking app pinned three delete targets
|
||
|
||
`lsof +D` per path found **PID 281192 — fluxgym, up 42 days, listening on
|
||
0.0.0.0:7860** — holding 15 open handles into `miniconda3/envs/vllm` (stale
|
||
opencv wheels) **and 41 into `/opt/ComfyUI`**. Deleting miniconda underneath it
|
||
would have half-broken a live listener in a way that surfaces only at its next
|
||
restart. Stopped it by **explicit PID** (never `pkill -f` — handle-blind),
|
||
verified :7860 released and handles at zero, *then* deleted.
|
||
|
||
⚠ **Name collision that nearly cost a production service:** `/opt/ComfyUI` is a
|
||
*native* install; the ComfyUI that actually serves (:8188, 200 OK) is the **Docker
|
||
`mmartial` container** reading `/worktank/comfyui`, and arbo's `comfy_engine` runs
|
||
from uv. Checking open handles **per path** is what separated them — the earlier
|
||
"not running" read would have deleted the wrong thing.
|
||
|
||
⚠ **`df` lags an async ZFS free.** Right after the 217 G delete, storetank still
|
||
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
|
||
(`zfs list -t snapshot` empty); second check showed 477 G at 74%.
|
||
|
||
All 16 containers and both systemd services verified healthy afterward.
|
||
|
||
## Homepage under version control
|
||
|
||
`ghcr.io/gethomepage/homepage` on **esh-docker-vm:5100** was the one stack whose
|
||
config lived only on the host. Its version history was **six hand-rolled
|
||
`services.yaml.bak-*` files**. Now `stacks/homepage/` (compose + 9 config files +
|
||
`.env.example` + README), deployed via `deploy-stack.sh`; `.bak` files gone.
|
||
105 cards across 19 groups, no empty groups.
|
||
|
||
⚠ **I claimed ana-docker wasn't wired into `docker.yaml`. It already was** —
|
||
`ana-pfi-docker: 10.250.50.70` — and I built a theory on a `tail` that truncated
|
||
the top of the file. All five engines were discovering correctly the whole time.
|
||
|
||
**Corrections landed:** `ANA-Firewall` said "Fortigate 81F" → it is a
|
||
**FortiGate-80F, FortiOS 7.2.10** (verified against the device); `NH3-Ansible` →
|
||
**NH3-ExtDev** (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
|
||
dropped the `UltraSeedbox` layout group (nothing provides it).
|
||
|
||
⚠ **`HOMEPAGE_ALLOWED_HOSTS` matches host AND port.** `10.0.50.45` did **not**
|
||
cover `http://10.0.50.45:5100/` — the container log carried `Host validation
|
||
failed` while the Traefik hostnames worked. Fixed; direct IP:port now 200.
|
||
`.env` was **mode 644** holding Plex + Jellyfin API keys → now 600.
|
||
|
||
⚠ **Homepage renders client-side** — grepping the served HTML to verify a config
|
||
change gave two false readings (a stale prerender, then an empty page).
|
||
`GET /api/services` is the honest instrument, and config changes need a
|
||
**recreate**, not a restart (a restart keeps the cached render in the writable
|
||
layer).
|
||
|
||
⚠ `deploy-stack.sh` runs rsync with `--delete` — alongside the six `.bak` files it
|
||
also removed a host-side `README.md` in the conf dir. Content survived (it is now
|
||
in the repo README) but that was a side effect, not a plan.
|
||
|
||
Commits `c5beeac`, `d1f4f1c`. See also [[2026-08-17-fleet-ipv6-mesh]].
|