diff --git a/persistent-memory.md b/persistent-memory.md index 81a53ba..492dcc0 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -300,6 +300,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca ## Recent decisions +- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md` +- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md` - `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml` - `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md` - `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md` diff --git a/servers/ana-docker/README.md b/servers/ana-docker/README.md index 53b9424..797fa00 100644 --- a/servers/ana-docker/README.md +++ b/servers/ana-docker/README.md @@ -8,6 +8,12 @@ General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0 - **FQDN:** `ana-docker.phasefinal.com` - **SSH:** standard port 22 - **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt) + - ⚠ **`readTimeout: 10m` on both entrypoints since 2026-10-05.** Traefik v3 defaults it to 60s, which cut every + request body slower than 60 s. Measured: skaldsong's 2.5 GB registry blob PUTs died at exactly 60.0 s (504, then 499), + and 16 of 16 deploys failed from 2026-05-28. The runner here reaches gitea through the FortiGate hairpin + (ClientHost 10.250.50.1, ~40 MB/s). Static config is at `/opt/docker/conf/traefik-ana/config/traefik.yml` on the host; + it is NOT in `stacks/` (pre-edit copy: `traefik.yml.pre-readtimeout-20261005`). A static change needs a + `docker restart traefik` (~20 s of public HTTPS down). ## Hardware diff --git a/servers/irv-ml1/README.md b/servers/irv-ml1/README.md index c1624cb..7d4c96f 100644 --- a/servers/irv-ml1/README.md +++ b/servers/irv-ml1/README.md @@ -66,10 +66,19 @@ current behaviour. - GPU 1: **NVIDIA RTX A6000** (48 GB VRAM) - Total VRAM: 72 GB across both - **OS:** Debian 12 (bookworm), kernel 6.1.0-37 -- **Storage:** - - `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free) - - `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows) - - `/storetank` — 1.8 TB (64% used, ~660 GB free) +- **Storage** (measured 2026-10-05 0115, after that night's cleanup): + - `/` on `/dev/nvme0n1p2`: Samsung 970 EVO Plus 2 TB, **single disk, no mirror**. 54% used, 802 GB free. Holds + `/var/lib/docker` and `/home`. + - `worktank`: ZFS mirror of 2× Samsung PM981 512 GB NVMe, the fastest storage on the box. 86% used, ~52 GB free. + ComfyUI run dirs, TTS seat caches, and arbo's text encoders (next bullet). + - `storetank`: ZFS mirror of 2× Crucial MX500 2 TB SATA SSD. 79% used, 377 GB free. arbo's models. + - ⚠ **`/storetank/arbo/models/text_encoders` is NOT on storetank.** It is ZFS dataset `worktank/arbo-text-encoders` + (recordsize 1M, atime off) mounted at that path since 2026-10-05, so every path (host, the comfyui container's + `/basedir/models/text_encoders`, arbo's catalog) stayed the same. It moved for NVMe load speed and to relieve storetank + (87% → 79%). Consequences: `zfs list` shows it under worktank, and `du /storetank` does not count it. A container that + bind-mounts `/storetank/arbo/models` sees it only if it was started AFTER the mount (default rprivate propagation); + zfs-mount runs before local-fs.target, so at boot it is in place before docker. If worktank fails to import, the + path shows an empty directory and ComfyUI reports no text encoders. ## What it runs @@ -123,7 +132,20 @@ esh-docker-vm, which auto-discovers containers on this host). ## Storage watch -Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass +**2026-10-05 cleanup (Prime):** Docker went from 968 GB to ~527 GB. 318 old pullable tags were removed (arbo except +v1.17.0 + v1.16.2, vllm, old comfyui cuda12.6/12.8 tags, CUDA base images, python:3.11, tts-generation-webui, +llama.cpp), plus dangling images and build cache unused for more than 30 days: +441 GB. Home entries untouched for more +than 2 months were deleted: +82 GB (aimodels, granite-4.1-8b-bf16, r16-vmoan-harness, r17smoke/separator, morpheus, +neovim, panns_data, small scripts/logs, old `.cache` dirs, 13 old HF cache entries). Kept although old, because they +are live: `act-runner` (act_runner.service), `arbo-webhook.py`, `ytvc-webhook.py`, `arbo-deploy.sh`, +`deploy-engine.sh`, `dotfiles` (stow target for `.zshrc` and friends) and `fzf-git.sh` (sourced by `.zshrc`). +⚠ An old mtime does not make a file unused: daemons started in May never update atime. Check `/proc/*/{cwd,cmdline,fd,maps}`, +units and symlinks before deleting. +Not done: ~81 GB of locally built TTS seat images with no container (qwen3-tts, ace-step, vibevoice, zonos, parakeet, +kyutai-tts, old dots-tts, stable-audio-open). They cannot be re-pulled, and the asset-engine still lists several of +those seats, so the TTS owner decides. + +Before that: nothing acute. `/storetank` dropped from 92% → 64% after a prune pass on the native-toolchain side; keep an eye on it since model weights and training outputs accumulate steadily (misbehavior starts around ~95% on either ext4 or ZFS).