fix(sweep): retire the dead 10.100.79.3 address across the fleet

Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is
on no interface anywhere, so anything pointing at it gets no route at all. Homepage
went from 9 dead cards to 0 of 112.

The load-bearing part is that there is no single right target: it depends on who
resolves it. The operator's browser and the Homepage and open-webui containers on
esh-docker-vm all resolve nh3.internal, so those get the name and survive the next
renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those
get the IP.

litellm on ana-docker looked like a counterexample and is not: it resolves the name
only through its own extra_hosts entry, while asset-engine on the same host fails on
it. Test from the container you are about to change, never from a neighbour. Before
committing to the name I confirmed the Homepage container actually fetches ytvc's
healthz through it in production rather than assuming resolution implies reach.

On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history.
Seven running containers recreated so the labels took. Seven dormant ones carried
stale labels because editing a compose file does not touch an existing container
object - fixed with compose create --force-recreate, which rebuilds the container
without starting it, the right tool for a deliberately dormant stack.

The sweep's real find was off irv-ml1 entirely: four live values on two other hosts,
silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS,
asset-engine's inference host, and two skaldsong TTS URLs. Both running services were
recreated and verified reaching their targets afterwards rather than merely carrying
the new string.

One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic
label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I
caught it only because I had taken a baseline before touching it. A label-only edit
still costs a full model reload on a GPU container.

This commit is contained in:

2026-09-15 08:41:44 -07:00

parent 8bc46e5132

commit 66c860d6c1

4 changed files with 84 additions and 3 deletions

									
										stacks/open-webui/.env.example
									
		+1
		-1
	
												View File
												
				@@ -40,7 +40,7 @@ RAG_OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1

				# --- TTS (G4 — tts-dev tunes model/voice). Direct at the dots gateway, no LiteLLM. ---

				AUDIO_TTS_ENGINE=openai

				AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1

				AUDIO_TTS_OPENAI_API_BASE_URL=http://irv-ml1.nh3.internal:8198/v1

				AUDIO_TTS_OPENAI_API_KEY=sk-no-auth-lan-seat

				AUDIO_TTS_MODEL=tts-1

				AUDIO_TTS_VOICE=nova