vh
e0d1c44137
The 2026-09-06 headscale cutover retired irv-ml1's wg0 tunnel IP 10.100.79.3 (now 10.6.110.50). Repointed all LIVE canonical refs to the DNS NAME so the next move can't re-break them: homepage.href/siteMonitor labels across 25 stack composes, load-bearing env defaults (asset-engine INFERENCE_HOST, open-webui AUDIO_TTS_OPENAI_API_BASE_URL, skaldsong SKALDSONG_TTS_BASE_URL, zonos-gateway ZONOS_URL, dia), homepage services.yaml manual cards (Voice Design Studio, IRV-ML1), and servers/irv-ml1/ssh-target. Updated the stale 'WG tunnel' comment to the mesh reality. Left as-is: README curl-examples and .env.example comments (docs), and historical mentions in CLAUDE.md/persistent-memory. NOTE: applying the label repoints to the RUNNING irv-ml1 containers needs a recreate per service (labels read at creation); deployed .env values are separate from these canonical defaults.
open-webui
Env-declarative chat frontend over the LiteLLM gateway, on esh-docker-vm
(10.0.50.45:3211). Stood up 2026-08-21 as a parallel bake-off against
lobe-chat (:3210) — operator-approved candidate replacement. Nothing about
lobe-chat changes unless/until the operator retires it.
- URL: http://10.0.50.45:3211
- Image:
ghcr.io/open-webui/open-webui:main(deployed v0.11.0) - Compose (server):
/opt/docker/compose/open-webui/ - Admin:
lkraven/vh@phasefinal.com(first account = admin). Signup is disabled in steady state; add users via the Admin UI.
Why this exists
Lobe's config does not survive a fleet that changes daily: its model list is a static env string that never auto-discovers (rotted both directions), and half its config (TTS/model/voice) lives in each browser's local store, so "configured" is a property of whoever's laptop. Open WebUI is env-declarative (the deploy is the single source of config truth) and auto-tracks the gateway roster with no pins.
The deal-sealer — ENABLE_PERSISTENT_CONFIG=False
Default is True, under which env vars are read once on first boot, written to
the DB, then ignored forever (edit-env-and-bounce silently does nothing). We set
it False: every knob is re-read from env on each boot, and Admin-UI changes
do not survive a restart. Verified both directions at deploy (see gates).
⚠️ A var only reaches the container if
compose.yamlreferences it in itsenvironment:block. Adding a line to.envalone does nothing — Compose uses.envfor${VAR}substitution, not as anenv_file. This bit us once:ENABLE_API_KEYS(see below) was set in.envbut not wired in compose, so API-key minting stayed 403 until the compose var was added.
Acceptance gate results (verified on the box, 2026-08-21)
| Gate | Result | Notes |
|---|---|---|
| G1 declarative config works | ✅ PASS | env change takes on bounce (signup 200→403); UI/runtime change reverts on restart (first-admin runtime signup-disable + a runtime ENABLE_API_KEYS=true both reverted to env on restart). No persistent-config bug bit this deploy (no Redis; flags honored once named right). |
| G2 model list auto-tracks LiteLLM, no pins | ✅ PASS | Picker = the 31 live gateway models 1:1, nothing hand-listed; a retired gateway model disappears with no OWUI edit. arena-model (+1) is OWUI's own eval-arena feature, not a pin. Tradeoff: the picker also shows non-chat seats (embeddings/rerankers/tts) — the flip side of "no hand-listing"; can't be pruned to chat-only without reintroducing exactly the pinning G2 forbids. |
| G3 reconciliation API is real | ✅ PASS | GET /api/v1/models/export round-trips JSON; POST /api/v1/models/sync genuinely reconciles — create and delete to match desired state (verified: syncing set B removed set A). Sync requires full records (user_id/created_at/updated_at) — it round-trips the export shape. API keys enabled via ENABLE_API_KEYS=True (plural — see warning above). |
| G5 task model pinned | ✅ PASS | TASK_MODEL_EXTERNAL=summarizer (gen seat @ temp 0). |
| G4 TTS by env, second-browser zero-setup | ⏳ tts-dev | Env knobs set (direct at http://10.100.79.3:8198/v1, no LiteLLM); tts-dev runs the gate. |
End-to-end sanity: a chat completion to summarizer through OWUI → LiteLLM → seat returns correctly.
Credential posture (G6)
- Fresh, capped LiteLLM key
key_alias: open-webui-esh—all-proxy-models(reaches paid passthroughs per the operator's standing posture) but capped:max_budget: $50,budget_duration: 1mo(unlikelobe-chat-esh, which is uncapped — deliberately not inherited). A paid call spends real vendor credits until the cap trips. - Signup off after admin creation (no first-registrant LAN race).
- Secrets in the vault, never in git:
secret get esh-docker-vm/open-webui-litellm-key # -> OPENAI_API_KEY secret get esh-docker-vm/open-webui-secret-key # -> WEBUI_SECRET_KEY secret get esh-docker-vm/open-webui-admin # -> lkraven admin cred state
TTS (G4 — tts-dev owns)
Routed direct at the fleet dots gateway http://10.100.79.3:8198/v1, not
through LiteLLM (:8198 is itself the swap-proof switchboard; LiteLLM would add a
hop + a wrong audio/mpeg stamp; direct needs no gateway key). Five knobs:
AUDIO_TTS_ENGINE=openai, AUDIO_TTS_OPENAI_API_BASE_URL,
AUDIO_TTS_OPENAI_API_KEY (placeholder — seat is LAN/WG-only, no auth),
AUDIO_TTS_MODEL, AUDIO_TTS_VOICE (model/voice are tts-dev's to tune).
Reversible to ext-tts via the base URL if audio spend/log visibility is wanted.
Deploy / operate
# Canonical lives here; push to the host:
scripts/deploy-stack.sh esh-docker-vm open-webui --compose
# Real .env lives on the host (secrets from the vault), never committed.
# On the host:
cd /opt/docker/compose/open-webui
$EDITOR .env # change any knob HERE (env is the source of truth)
docker compose up -d # bounce to apply — UI changes do NOT persist
# RAG embeddings route through the gateway (qwen3-embedding) to avoid a
# boot-time HuggingFace download and stay fleet-consistent.