docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m

servers/irv-ml1/README.md

@@ -66,10 +66,19 @@ current behaviour.
   - GPU 1: **NVIDIA RTX A6000** (48 GB VRAM)
   - Total VRAM: 72 GB across both
 - **OS:** Debian 12 (bookworm), kernel 6.1.0-37
-- **Storage:**
-  - `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free)
-  - `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows)
-  - `/storetank` — 1.8 TB (64% used, ~660 GB free)
+- **Storage** (measured 2026-10-05 0115, after that night's cleanup):
+  - `/` on `/dev/nvme0n1p2`: Samsung 970 EVO Plus 2 TB, **single disk, no mirror**. 54% used, 802 GB free. Holds
+    `/var/lib/docker` and `/home`.
+  - `worktank`: ZFS mirror of 2× Samsung PM981 512 GB NVMe, the fastest storage on the box. 86% used, ~52 GB free.
+    ComfyUI run dirs, TTS seat caches, and arbo's text encoders (next bullet).
+  - `storetank`: ZFS mirror of 2× Crucial MX500 2 TB SATA SSD. 79% used, 377 GB free. arbo's models.
+  - ⚠ **`/storetank/arbo/models/text_encoders` is NOT on storetank.** It is ZFS dataset `worktank/arbo-text-encoders`
+    (recordsize 1M, atime off) mounted at that path since 2026-10-05, so every path (host, the comfyui container's
+    `/basedir/models/text_encoders`, arbo's catalog) stayed the same. It moved for NVMe load speed and to relieve storetank
+    (87% → 79%). Consequences: `zfs list` shows it under worktank, and `du /storetank` does not count it. A container that
+    bind-mounts `/storetank/arbo/models` sees it only if it was started AFTER the mount (default rprivate propagation);
+    zfs-mount runs before local-fs.target, so at boot it is in place before docker. If worktank fails to import, the
+    path shows an empty directory and ComfyUI reports no text encoders.
 
 ## What it runs
 
@@ -123,7 +132,20 @@ esh-docker-vm, which auto-discovers containers on this host).
 
 ## Storage watch
 
-Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
+**2026-10-05 cleanup (Prime):** Docker went from 968 GB to ~527 GB. 318 old pullable tags were removed (arbo except
+v1.17.0 + v1.16.2, vllm, old comfyui cuda12.6/12.8 tags, CUDA base images, python:3.11, tts-generation-webui,
+llama.cpp), plus dangling images and build cache unused for more than 30 days: +441 GB. Home entries untouched for more
+than 2 months were deleted: +82 GB (aimodels, granite-4.1-8b-bf16, r16-vmoan-harness, r17smoke/separator, morpheus,
+neovim, panns_data, small scripts/logs, old `.cache` dirs, 13 old HF cache entries). Kept although old, because they
+are live: `act-runner` (act_runner.service), `arbo-webhook.py`, `ytvc-webhook.py`, `arbo-deploy.sh`,
+`deploy-engine.sh`, `dotfiles` (stow target for `.zshrc` and friends) and `fzf-git.sh` (sourced by `.zshrc`).
+⚠ An old mtime does not make a file unused: daemons started in May never update atime. Check `/proc/*/{cwd,cmdline,fd,maps}`,
+units and symlinks before deleting.
+Not done: ~81 GB of locally built TTS seat images with no container (qwen3-tts, ace-step, vibevoice, zonos, parakeet,
+kyutai-tts, old dots-tts, stable-audio-open). They cannot be re-pulled, and the asset-engine still lists several of
+those seats, so the TTS owner decides.
+
+Before that: nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
 on the native-toolchain side; keep an eye on it since model weights
 and training outputs accumulate steadily (misbehavior starts around
 ~95% on either ext4 or ZFS).