docs(irv-ml1, ana-docker): storage cleanup, text encoders on worktank, Traefik readTimeout 10m
This commit is contained in:
@@ -300,6 +300,8 @@ _As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-05]` **irv-ml1 storage cleanup + text encoders on worktank (Prime)** — Docker prune +441 G (dangling, old pullable tags, cache >30d); home entries untouched >2 months −82 G (live-but-old kept: act-runner, webhooks, dotfiles, fzf-git); `/storetank/arbo/models/text_encoders` is now ZFS `worktank/arbo-text-encoders` mounted at the same path (sha-verified, ComfyUI sees all 23). Root 84%→54%, storetank 87%→79%, worktank 62%→86%. → `servers/irv-ml1/README.md`
|
||||
- `[2026-10-05]` **ana-docker Traefik `readTimeout` 60s → 10m on both entrypoints** — the Traefik v3 default cut skaldsong's 2.5 G registry blob PUTs at exactly 60.0 s (16/16 deploys failed since 05-28). Static file on host, not in `stacks/`. → `servers/ana-docker/README.md`
|
||||
- `[2026-10-05]` **Embedder routing split (Prime 0030, via albok-dev, confirmed directly): `qwen3-embedding` = esh-ml1 primary / nh3-ml1 failover; new `qwen3-embedding-bulk` = nh3-ml1 primary / esh-ml1 failover, for albok writes** — LiteLLM `order` (read at source, v1.97.0); failover proved by stopping nh3-ml1 tei-embed (bulk 6/6 → esh-ml1, 200s). Gateway restart = 43 s outage. albok-service key granted the bulk alias; Nemi key not. TEI weights revision unpinned on both (model_sha null). → `stacks/litellm/conf/config.yaml`
|
||||
- `[2026-10-04]` **albok-service moved to nh3-albok, a VM that runs only albok (albok-dev relaying Prime)** — nh3-docker's socket containers and plaintext tcp/2375 were root over the restricted wings; nh3-pve-2 vmbr0 VLAN-aware by reboot; PBS ns `nh3-pve-2`; copy verified by digest (dir st_size differs on ext4, not data). → `persistent-memory.d/2026-10-04-albok-service-moved-to-nh3-albok.md`
|
||||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
|
||||
@@ -8,6 +8,12 @@ General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0
|
||||
- **FQDN:** `ana-docker.phasefinal.com`
|
||||
- **SSH:** standard port 22
|
||||
- **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt)
|
||||
- ⚠ **`readTimeout: 10m` on both entrypoints since 2026-10-05.** Traefik v3 defaults it to 60s, which cut every
|
||||
request body slower than 60 s. Measured: skaldsong's 2.5 GB registry blob PUTs died at exactly 60.0 s (504, then 499),
|
||||
and 16 of 16 deploys failed from 2026-05-28. The runner here reaches gitea through the FortiGate hairpin
|
||||
(ClientHost 10.250.50.1, ~40 MB/s). Static config is at `/opt/docker/conf/traefik-ana/config/traefik.yml` on the host;
|
||||
it is NOT in `stacks/` (pre-edit copy: `traefik.yml.pre-readtimeout-20261005`). A static change needs a
|
||||
`docker restart traefik` (~20 s of public HTTPS down).
|
||||
|
||||
## Hardware
|
||||
|
||||
|
||||
@@ -66,10 +66,19 @@ current behaviour.
|
||||
- GPU 1: **NVIDIA RTX A6000** (48 GB VRAM)
|
||||
- Total VRAM: 72 GB across both
|
||||
- **OS:** Debian 12 (bookworm), kernel 6.1.0-37
|
||||
- **Storage:**
|
||||
- `/` on `/dev/nvme0n1p2` — 1.8 TB (78% used, ~393 GB free)
|
||||
- `/worktank` — 462 GB (1% used — dedicated to Docker stacks' user state, e.g. ComfyUI models + workflows)
|
||||
- `/storetank` — 1.8 TB (64% used, ~660 GB free)
|
||||
- **Storage** (measured 2026-10-05 0115, after that night's cleanup):
|
||||
- `/` on `/dev/nvme0n1p2`: Samsung 970 EVO Plus 2 TB, **single disk, no mirror**. 54% used, 802 GB free. Holds
|
||||
`/var/lib/docker` and `/home`.
|
||||
- `worktank`: ZFS mirror of 2× Samsung PM981 512 GB NVMe, the fastest storage on the box. 86% used, ~52 GB free.
|
||||
ComfyUI run dirs, TTS seat caches, and arbo's text encoders (next bullet).
|
||||
- `storetank`: ZFS mirror of 2× Crucial MX500 2 TB SATA SSD. 79% used, 377 GB free. arbo's models.
|
||||
- ⚠ **`/storetank/arbo/models/text_encoders` is NOT on storetank.** It is ZFS dataset `worktank/arbo-text-encoders`
|
||||
(recordsize 1M, atime off) mounted at that path since 2026-10-05, so every path (host, the comfyui container's
|
||||
`/basedir/models/text_encoders`, arbo's catalog) stayed the same. It moved for NVMe load speed and to relieve storetank
|
||||
(87% → 79%). Consequences: `zfs list` shows it under worktank, and `du /storetank` does not count it. A container that
|
||||
bind-mounts `/storetank/arbo/models` sees it only if it was started AFTER the mount (default rprivate propagation);
|
||||
zfs-mount runs before local-fs.target, so at boot it is in place before docker. If worktank fails to import, the
|
||||
path shows an empty directory and ComfyUI reports no text encoders.
|
||||
|
||||
## What it runs
|
||||
|
||||
@@ -123,7 +132,20 @@ esh-docker-vm, which auto-discovers containers on this host).
|
||||
|
||||
## Storage watch
|
||||
|
||||
Nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
|
||||
**2026-10-05 cleanup (Prime):** Docker went from 968 GB to ~527 GB. 318 old pullable tags were removed (arbo except
|
||||
v1.17.0 + v1.16.2, vllm, old comfyui cuda12.6/12.8 tags, CUDA base images, python:3.11, tts-generation-webui,
|
||||
llama.cpp), plus dangling images and build cache unused for more than 30 days: +441 GB. Home entries untouched for more
|
||||
than 2 months were deleted: +82 GB (aimodels, granite-4.1-8b-bf16, r16-vmoan-harness, r17smoke/separator, morpheus,
|
||||
neovim, panns_data, small scripts/logs, old `.cache` dirs, 13 old HF cache entries). Kept although old, because they
|
||||
are live: `act-runner` (act_runner.service), `arbo-webhook.py`, `ytvc-webhook.py`, `arbo-deploy.sh`,
|
||||
`deploy-engine.sh`, `dotfiles` (stow target for `.zshrc` and friends) and `fzf-git.sh` (sourced by `.zshrc`).
|
||||
⚠ An old mtime does not make a file unused: daemons started in May never update atime. Check `/proc/*/{cwd,cmdline,fd,maps}`,
|
||||
units and symlinks before deleting.
|
||||
Not done: ~81 GB of locally built TTS seat images with no container (qwen3-tts, ace-step, vibevoice, zonos, parakeet,
|
||||
kyutai-tts, old dots-tts, stable-audio-open). They cannot be re-pulled, and the asset-engine still lists several of
|
||||
those seats, so the TTS owner decides.
|
||||
|
||||
Before that: nothing acute. `/storetank` dropped from 92% → 64% after a prune pass
|
||||
on the native-toolchain side; keep an eye on it since model weights
|
||||
and training outputs accumulate steadily (misbehavior starts around
|
||||
~95% on either ext4 or ZFS).
|
||||
|
||||
Reference in New Issue
Block a user