Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is
on no interface anywhere, so anything pointing at it gets no route at all. Homepage
went from 9 dead cards to 0 of 112.
The load-bearing part is that there is no single right target: it depends on who
resolves it. The operator's browser and the Homepage and open-webui containers on
esh-docker-vm all resolve nh3.internal, so those get the name and survive the next
renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those
get the IP.
litellm on ana-docker looked like a counterexample and is not: it resolves the name
only through its own extra_hosts entry, while asset-engine on the same host fails on
it. Test from the container you are about to change, never from a neighbour. Before
committing to the name I confirmed the Homepage container actually fetches ytvc's
healthz through it in production rather than assuming resolution implies reach.
On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history.
Seven running containers recreated so the labels took. Seven dormant ones carried
stale labels because editing a compose file does not touch an existing container
object - fixed with compose create --force-recreate, which rebuilds the container
without starting it, the right tool for a deliberately dormant stack.
The sweep's real find was off irv-ml1 entirely: four live values on two other hosts,
silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS,
asset-engine's inference host, and two skaldsong TTS URLs. Both running services were
recreated and verified reaching their targets afterwards rather than merely carrying
the new string.
One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic
label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I
caught it only because I had taken a baseline before touching it. A label-only edit
still costs a full model reload on a GPU container.
Two corrections surfaced by the first end-to-end deploy that didn't
land in the pre-flight align:
- SPA static assets are at /app/spa, not /app/web/dist (Dockerfile
COPYs the SvelteKit build output flat into /app/spa, not into
/app/spa/dist). Mismatch caused /health to 500 with
"RuntimeError: File at path /app/web/dist/index.html does not
exist."
- SKALDSONG_HOST_CORS_ORIGINS must be a JSON array literal in .env.
Pydantic-settings parses complex-typed env vars via json.loads();
bare URL string fails first-boot with SettingsError.
Container now reports Up (healthy) on ana-docker; /health 200.
skaldsong-dev surfaced three contract corrections before the first
deploy:
- WORLDTREE_TOKEN (outbound HTTP Bearer) was missing — separate code
path from SKALDSONG_BIFROST_JWT_KEY (inbound HS256 verify) but
same secret value.
- WORLDTREE_BASE_URL replaces SKALDSONG_WORLDTREE_API_URL (the
former is what the app actually reads).
- SKALDSONG_HOST_WIZARD_AGENT_ID was missing entirely — must pin to
skaldsong:wizard-v2 to inherit the existing Worldtree agent slot;
blank would burn another slot of the 50-per-key Heimdall quota.
Registry-pull pattern matching Worldtree: CI on vh/skaldsong builds and
pushes gitea.phasefinal.com/vh/skaldsong:<sha>, this playbook pulls +
recreates. SHA-pin only per current preference; no :latest moving-tag
advance yet (revisit once /health exercises Worldtree + Kokoro reach).
Host port 8300 (host) → 8000 (container). Persistent state under
/opt/docker/conf/skaldsong/{db,runs}.
Bifrost endpoint URL 10.250.50.70:8300 will need a paired
BIFROST_CLIENT_ALLOWED_HOSTS update on corviduo-dev Worldtree at first
deploy.
vh