4.2 KiB
irv-ml1 weight cleanup (782 GB) + Homepage brought under version control
Two unrelated housekeeping jobs from the same session, both with durable lessons.
irv-ml1 — 782 GB reclaimed
Root was at 92% (148 G free), storetank 86%. Now 64% (635 G free) and 74% (477 G).
Tier 1 — dead weights, 286 GB. /storetank/llm-models/Storage (217 G, 22
GGUF repos, atimes Jan–May 2025) plus models--MaziyarPanahi--WizardLM-2-8x22B-GGUF
(44 G) and models--h2oai--h2ogpt-4096-llama2-13b-chat (25 G). The 217 G pile had
zero consumers — no llama-swap, no llama.cpp, no textgen running or installed,
not even a stopped container. The fleet moved to vLLM/NVFP4 seats on ana-ml2 and
nobody opened that shed for 15 months. Re-verified the consumer check immediately
before deleting, not just during the audit.
Tier 2 — regenerable caches, 194 GB. uv 65 G + 60 G, pip 31 G + 8.7 G,
modelscope 29 G (mtime 2024-04-23).
Tier 3 — retired stacks, 302 GB (operator: "those were old days… we're a UV
fleet now"): /opt/fluxgym 64 G, /opt/ComfyUI native 41 G, /opt/stablediffusion
28 G, /opt/alltalk 19 G, /opt/o-textgen 12 G, /opt/sdnext 3 G, /opt/xttsv2
1.8 G, tabbyAPI 3.1 G, miniconda3 130 G.
The lesson: one dead-looking app pinned three delete targets
lsof +D per path found PID 281192 — fluxgym, up 42 days, listening on
0.0.0.0:7860 — holding 15 open handles into miniconda3/envs/vllm (stale
opencv wheels) and 41 into /opt/ComfyUI. Deleting miniconda underneath it
would have half-broken a live listener in a way that surfaces only at its next
restart. Stopped it by explicit PID (never pkill -f — handle-blind),
verified :7860 released and handles at zero, then deleted.
⚠ Name collision that nearly cost a production service: /opt/ComfyUI is a
native install; the ComfyUI that actually serves (:8188, 200 OK) is the Docker
mmartial container reading /worktank/comfyui, and arbo's comfy_engine runs
from uv. Checking open handles per path is what separated them — the earlier
"not running" read would have deleted the wrong thing.
⚠ df lags an async ZFS free. Right after the 217 G delete, storetank still
showed 86%/261 G — the exact shape of a snapshot-retention problem. It wasn't
(zfs list -t snapshot empty); second check showed 477 G at 74%.
All 16 containers and both systemd services verified healthy afterward.
Homepage under version control
ghcr.io/gethomepage/homepage on esh-docker-vm:5100 was the one stack whose
config lived only on the host. Its version history was six hand-rolled
services.yaml.bak-* files. Now stacks/homepage/ (compose + 9 config files +
.env.example + README), deployed via deploy-stack.sh; .bak files gone.
105 cards across 19 groups, no empty groups.
⚠ I claimed ana-docker wasn't wired into docker.yaml. It already was —
ana-pfi-docker: 10.250.50.70 — and I built a theory on a tail that truncated
the top of the file. All five engines were discovering correctly the whole time.
Corrections landed: ANA-Firewall said "Fortigate 81F" → it is a
FortiGate-80F, FortiOS 7.2.10 (verified against the device); NH3-Ansible →
NH3-ExtDev (10.100.50.42 is nh3-extdev, successor to the retired nh3-ansible);
dropped the UltraSeedbox layout group (nothing provides it).
⚠ HOMEPAGE_ALLOWED_HOSTS matches host AND port. 10.0.50.45 did not
cover http://10.0.50.45:5100/ — the container log carried Host validation failed while the Traefik hostnames worked. Fixed; direct IP:port now 200.
.env was mode 644 holding Plex + Jellyfin API keys → now 600.
⚠ Homepage renders client-side — grepping the served HTML to verify a config
change gave two false readings (a stale prerender, then an empty page).
GET /api/services is the honest instrument, and config changes need a
recreate, not a restart (a restart keeps the cached render in the writable
layer).
⚠ deploy-stack.sh runs rsync with --delete — alongside the six .bak files it
also removed a host-side README.md in the conf dir. Content survived (it is now
in the repo README) but that was a side effect, not a plan.
Commits c5beeac, d1f4f1c. See also 2026-08-17-fleet-ipv6-mesh.