vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(sweep): retire the dead 10.100.79.3 address across the fleet

Browse Source 
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is
on no interface anywhere, so anything pointing at it gets no route at all. Homepage
went from 9 dead cards to 0 of 112.

The load-bearing part is that there is no single right target: it depends on who
resolves it. The operator's browser and the Homepage and open-webui containers on
esh-docker-vm all resolve nh3.internal, so those get the name and survive the next
renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those
get the IP.

litellm on ana-docker looked like a counterexample and is not: it resolves the name
only through its own extra_hosts entry, while asset-engine on the same host fails on
it. Test from the container you are about to change, never from a neighbour. Before
committing to the name I confirmed the Homepage container actually fetches ytvc's
healthz through it in production rather than assuming resolution implies reach.

On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history.
Seven running containers recreated so the labels took. Seven dormant ones carried
stale labels because editing a compose file does not touch an existing container
object - fixed with compose create --force-recreate, which rebuilds the container
without starting it, the right tool for a deliberately dormant stack.

The sweep's real find was off irv-ml1 entirely: four live values on two other hosts,
silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS,
asset-engine's inference host, and two skaldsong TTS URLs. Both running services were
recreated and verified reaching their targets afterwards rather than merely carrying
the new string.

One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic
label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I
caught it only because I had taken a baseline before touching it. A label-only edit
still costs a full model reload on a GPU container.
This commit is contained in:
vh
2026-09-15 08:41:44 -07:00
parent 8bc46e5132
commit 66c860d6c1
4 changed files with 84 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.d/2026-09-15-irv-ml1-address-sweep-done.md
+81
View File
@@ -0,0 +1,81 @@
# irv-ml1 dead-address sweep — EXECUTED 2026-09-15 (operator-directed)
Closes `2026-09-15-irv-ml1-dead-wg0-address.md`. `10.100.79.3` (the wg0 lifeline
retired at the 2026-09-06 headscale cutover) is on **no interface anywhere**; a
request to it gets *no route*, not a refusal.
**Result: 0 of 112 Homepage cards point at it. Was 9.**
## Which address, and why it differs per host — this is the load-bearing part
⚠ **There is no single right answer.** The correct target depends on **who resolves
it**, and getting this wrong swaps a dead address for an unresolvable one — which is
what tts-dev nearly did on voice-studio.
| resolver | can it resolve `*.nh3.internal`? | use |
|---|---|---|
| operator's browser (`homepage.href`) | yes | **name** |
| Homepage container @ esh-docker-vm (`siteMonitor`) | **yes** (measured) | **name** |
| open-webui container @ esh-docker-vm | **yes** (measured) | **name** |
| containers on **irv-ml1** | **NO** | IP |
| containers on **ana-docker** | **NO** | IP |
⚠⚠ **litellm on ana-docker resolves the name ONLY because of its own `extra_hosts`
entry.** Testing from litellm and generalising would have been wrong: `asset-engine`
on the same host, with no `extra_hosts`, gets `Name or service not known`. **Test from
the container you are about to change, not a neighbour.**
Positive control used before committing to the name: the Homepage container actually
*fetched* ytvc's `/healthz` through `irv-ml1.nh3.internal` and got `{"ok":true,…}` —
the pattern already worked in production.
## What was changed
**irv-ml1 — 24 files swept**, 14 comment-only hits deliberately left as
port-allocation history.
- **7 running** containers recreated so labels took: `arbo`, `breeze-tts`, `comfyui`,
`dockge`, `kokoro`, `tts-gateway`, `waterland-studio` (+ its `siteMonitor`).
`dockge`'s href is built from `DOCKGE_HOST_IP` in `.env`, so one var fixed it.
- **7 dormant** containers in `Created`/`Exited` state carried stale labels because a
compose edit alone does not touch an existing container object. Fixed with
**`docker compose create --force-recreate`** — rebuilds the container with new
labels **without starting it**, which is exactly right for a deliberately dormant
stack: `chatterbox-fast`, `dots-tts`, `speaches`, `cosyvoice`, `fish-s2`,
`index-tts`, `omnivoice`. `omnivoice-ref` (a different stack, Up 4 weeks) untouched.
- 17 more dormant compose files swept; inert until someone starts them.
## ⚠⚠ FOUR MORE LIVE BREAKAGES on OTHER hosts — the sweep's real find
The renumber left a trail off irv-ml1 that nobody grepped for. All silently dead nine
days, none alerting:
| host | service | was | now |
|---|---|---|---|
| esh-docker-vm | **open-webui** read-aloud TTS | `AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1` | `irv-ml1.nh3.internal` |
| ana-docker | **asset-engine** (Up, healthy) | `INFERENCE_HOST=10.100.79.3` | `10.6.110.50` |
| ana-docker | skaldsong TTS (Exited 2 mo) | `SKALDSONG_TTS_BASE_URL=…8193` | `10.6.110.50` |
| ana-docker | skaldsong vibevoice | `…8194` | `10.6.110.50` |
Both running services recreated and **verified reaching their targets after the
change**, not just carrying the new string. Repo `.env.example` templates updated to
match per-host (a template hands the next person a dead address otherwise).
## ⚠ My own error, worth not repeating
I recreated **`breeze-tts` for a cosmetic label change** and took the fleet TTS path
down for its ~90 s CUDA-graph warm-up. `ext-tts` returned **500** until it finished.
Caught it only because I had taken a **baseline before touching it** (200 / 35,180
bytes) — without that, a 500 would have looked like a pre-existing fault.
⭐ **A label-only edit still costs a full model reload on a GPU container.** Batch
those into a window or accept a stale label; do not bounce a warming model for a link.
## Still open
- `speaches`'s label says `:8204`, which is **breeze-tts's live port**. Latent conflict
if anyone starts it. Not fixed — it is a port-register question, not an address one.
- ⭐ The durable fix for ana-docker/irv-ml1 containers is **`extra_hosts` + the name**
(what `litellm` and `talk` already do), so the IP lives in one place a single `.env`
line moves. Left as an improvement rather than done, because `asset-engine` is
CI-deployed and its compose is not mine to edit from here.
stacks/asset-engine/.env.example
+1 -1
View File
@@ -31,7 +31,7 @@ ASSET_ENGINE_SSH_DIR=/opt/docker/conf/asset-engine/ssh
# Inference target. Default is irv-ml1 over WG. Override if the fleet's
# inference host moves.
INFERENCE_HOST=10.100.79.3
INFERENCE_HOST=10.6.110.50
# OIDC seam — empty in v1 (auth is no-op). Populate when v2 forward-auth
# lands. Pre-allocated here so the surface is visible in the config file
stacks/open-webui/.env.example
+1 -1
View File
@@ -40,7 +40,7 @@ RAG_OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1
# --- TTS (G4 — tts-dev tunes model/voice). Direct at the dots gateway, no LiteLLM. ---
AUDIO_TTS_ENGINE=openai
AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1
AUDIO_TTS_OPENAI_API_BASE_URL=http://irv-ml1.nh3.internal:8198/v1
AUDIO_TTS_OPENAI_API_KEY=sk-no-auth-lan-seat
AUDIO_TTS_MODEL=tts-1
AUDIO_TTS_VOICE=nova
stacks/skaldsong/.env.example
+1 -1
View File
@@ -61,4 +61,4 @@ SKALDSONG_HOST_CORS_ORIGINS=["http://10.250.50.70:8300"]
# TTS — Kokoro on irv-ml1, reached over WireGuard via ana-wg. Same URL
# works fleet-wide; the WG tunnel handles the cross-site routing.
SKALDSONG_TTS_ENGINE=kokoro
SKALDSONG_TTS_BASE_URL=http://10.100.79.3:8193
SKALDSONG_TTS_BASE_URL=http://10.6.110.50:8193