vh
66c860d6c1
Operator-directed. The wg0 lifeline retired at the 2026-09-06 headscale cutover is on no interface anywhere, so anything pointing at it gets no route at all. Homepage went from 9 dead cards to 0 of 112. The load-bearing part is that there is no single right target: it depends on who resolves it. The operator's browser and the Homepage and open-webui containers on esh-docker-vm all resolve nh3.internal, so those get the name and survive the next renumber. Containers on irv-ml1 and ana-docker cannot resolve it at all, so those get the IP. litellm on ana-docker looked like a counterexample and is not: it resolves the name only through its own extra_hosts entry, while asset-engine on the same host fails on it. Test from the container you are about to change, never from a neighbour. Before committing to the name I confirmed the Homepage container actually fetches ytvc's healthz through it in production rather than assuming resolution implies reach. On irv-ml1, 24 files swept and 14 comment-only hits left as port-allocation history. Seven running containers recreated so the labels took. Seven dormant ones carried stale labels because editing a compose file does not touch an existing container object - fixed with compose create --force-recreate, which rebuilds the container without starting it, the right tool for a deliberately dormant stack. The sweep's real find was off irv-ml1 entirely: four live values on two other hosts, silently dead for nine days and alerting nobody. Open WebUI's read-aloud TTS, asset-engine's inference host, and two skaldsong TTS URLs. Both running services were recreated and verified reaching their targets afterwards rather than merely carrying the new string. One self-inflicted outage worth recording: I recreated breeze-tts for a cosmetic label change and took ext-tts down for its ~90s CUDA-graph warm-up, returning 500. I caught it only because I had taken a baseline before touching it. A label-only edit still costs a full model reload on a GPU container.
open-webui
Env-declarative chat frontend over the LiteLLM gateway, on esh-docker-vm
(10.0.50.45:3211). Stood up 2026-08-21 as a parallel bake-off against
lobe-chat (:3210) — operator-approved candidate replacement. Nothing about
lobe-chat changes unless/until the operator retires it.
- URL: http://10.0.50.45:3211
- Image:
ghcr.io/open-webui/open-webui:main(deployed v0.11.0) - Compose (server):
/opt/docker/compose/open-webui/ - Admin:
lkraven/vh@phasefinal.com(first account = admin). Signup is disabled in steady state; add users via the Admin UI.
Why this exists
Lobe's config does not survive a fleet that changes daily: its model list is a static env string that never auto-discovers (rotted both directions), and half its config (TTS/model/voice) lives in each browser's local store, so "configured" is a property of whoever's laptop. Open WebUI is env-declarative (the deploy is the single source of config truth) and auto-tracks the gateway roster with no pins.
The deal-sealer — ENABLE_PERSISTENT_CONFIG=False
Default is True, under which env vars are read once on first boot, written to
the DB, then ignored forever (edit-env-and-bounce silently does nothing). We set
it False: every knob is re-read from env on each boot, and Admin-UI changes
do not survive a restart. Verified both directions at deploy (see gates).
⚠️ A var only reaches the container if
compose.yamlreferences it in itsenvironment:block. Adding a line to.envalone does nothing — Compose uses.envfor${VAR}substitution, not as anenv_file. This bit us once:ENABLE_API_KEYS(see below) was set in.envbut not wired in compose, so API-key minting stayed 403 until the compose var was added.
Acceptance gate results (verified on the box, 2026-08-21)
| Gate | Result | Notes |
|---|---|---|
| G1 declarative config works | ✅ PASS | env change takes on bounce (signup 200→403); UI/runtime change reverts on restart (first-admin runtime signup-disable + a runtime ENABLE_API_KEYS=true both reverted to env on restart). No persistent-config bug bit this deploy (no Redis; flags honored once named right). |
| G2 model list auto-tracks LiteLLM, no pins | ✅ PASS | Picker = the 31 live gateway models 1:1, nothing hand-listed; a retired gateway model disappears with no OWUI edit. arena-model (+1) is OWUI's own eval-arena feature, not a pin. Tradeoff: the picker also shows non-chat seats (embeddings/rerankers/tts) — the flip side of "no hand-listing"; can't be pruned to chat-only without reintroducing exactly the pinning G2 forbids. |
| G3 reconciliation API is real | ✅ PASS | GET /api/v1/models/export round-trips JSON; POST /api/v1/models/sync genuinely reconciles — create and delete to match desired state (verified: syncing set B removed set A). Sync requires full records (user_id/created_at/updated_at) — it round-trips the export shape. API keys enabled via ENABLE_API_KEYS=True (plural — see warning above). |
| G5 task model pinned | ✅ PASS | TASK_MODEL_EXTERNAL=summarizer (gen seat @ temp 0). |
| G4 TTS by env, second-browser zero-setup | ⏳ tts-dev | Env knobs set (direct at http://10.100.79.3:8198/v1, no LiteLLM); tts-dev runs the gate. |
End-to-end sanity: a chat completion to summarizer through OWUI → LiteLLM → seat returns correctly.
Credential posture (G6)
- Fresh, capped LiteLLM key
key_alias: open-webui-esh—all-proxy-models(reaches paid passthroughs per the operator's standing posture) but capped:max_budget: $50,budget_duration: 1mo(unlikelobe-chat-esh, which is uncapped — deliberately not inherited). A paid call spends real vendor credits until the cap trips. - Signup off after admin creation (no first-registrant LAN race).
- Secrets in the vault, never in git:
secret get esh-docker-vm/open-webui-litellm-key # -> OPENAI_API_KEY secret get esh-docker-vm/open-webui-secret-key # -> WEBUI_SECRET_KEY secret get esh-docker-vm/open-webui-admin # -> lkraven admin cred state
TTS (G4 — tts-dev owns)
Routed direct at the fleet dots gateway http://10.100.79.3:8198/v1, not
through LiteLLM (:8198 is itself the swap-proof switchboard; LiteLLM would add a
hop + a wrong audio/mpeg stamp; direct needs no gateway key). Five knobs:
AUDIO_TTS_ENGINE=openai, AUDIO_TTS_OPENAI_API_BASE_URL,
AUDIO_TTS_OPENAI_API_KEY (placeholder — seat is LAN/WG-only, no auth),
AUDIO_TTS_MODEL, AUDIO_TTS_VOICE (model/voice are tts-dev's to tune).
Reversible to ext-tts via the base URL if audio spend/log visibility is wanted.
Deploy / operate
# Canonical lives here; push to the host:
scripts/deploy-stack.sh esh-docker-vm open-webui --compose
# Real .env lives on the host (secrets from the vault), never committed.
# On the host:
cd /opt/docker/compose/open-webui
$EDITOR .env # change any knob HERE (env is the source of truth)
docker compose up -d # bounce to apply — UI changes do NOT persist
# RAG embeddings route through the gateway (qwen3-embedding) to avoid a
# boot-time HuggingFace download and stay fleet-consistent.