vh
39da1d4a97
The board mixed tools with endpoints. A vLLM seat whose href is a /docs page
sat in the same band as ComfyUI; the MQTT broker and the RustDesk relay, which
have no page at all, sat in Apps; and `Service Networking` was thirteen members
spanning three AdGuards, five Dockges, two Traefiks and four headless agents.
Every group is now one of two kinds and they never mix. TOOLS are expanded and
sit at the top of their tab. ENDPOINTS — an API, a broker, a background agent,
an href that is /docs or /ping or nothing — carry `initiallyCollapsed: true`
and sit at the bottom. Collapsed is not hidden: the eyebrow and its rule still
render, so the tab still says the thing exists and one click expands it.
A second rule fell out of the same pass and now shapes the group boundaries: a
group's members should all carry a widget or none should. A stat strip makes a
card ~50px taller, so one widget card in a row of plain ones opens a void under
the plain ones. That is why AdGuard and Traefik get their own groups rather
than sharing one with Dockge, and it is most of why the old Service Networking
band looked broken. AdGuard (ANA) was the last short card in its row and now
carries the same query/blocked/latency strip as its two siblings — one
infra-ops AdGuard login authenticates against all three instances, verified
against each; it lives in that stack's .env on the host and is vaulted.
The sixteen GPU-backed model seats were deliberately NOT relabelled.
`homepage.group` is read at container creation, so clearer names for
`AI - Inference` and friends would have cost a recreate on six vLLM seats, four
eval seats and four TTS engines — multi-minute model reloads on endpoints peers
reach through the gateway. Order plus `initiallyCollapsed` buys the same
separation for nothing, so those names stay as they are on purpose.
28 containers that ARE cheap to bounce were relabelled, across five hosts, via
rerunnable elway playbooks. Their label steps are gated on the old value still
being present, so a second run reports skipped rather than churning. Two verify
steps were wrong on first contact and are fixed with the reason recorded: the
traefik check raced its own recreate, and asserting a model seat is "running"
cannot answer "did I bounce it" when a seat may be legitimately stopped —
container age can, and now does.
The canonical stacks/ tree was synced to the deployed labels afterwards, so
intent and reality agree again on all fourteen tracked stacks.
Also documents the real nature of the post-recreate blank dashboard, which cost
~25 minutes here and an hour on 2026-08-19. `initialSettings":{}` in the served
HTML is the catch branch of the page's data loader, not a warm-up and not a
cache — and the error can vanish entirely, because the logger is assigned inside
the same try and the catch only logs if the logger exists. Ruled out by
measurement this time: all four API routes return 200 with correct content while
the page serves {}, and the previous known-good settings.yaml reproduces it
identically. The README now carries the one-command test and the next lead.
Before/after, all four tabs: http://10.100.10.50:8090/b/homepage-relayout/
open-webui
Env-declarative chat frontend over the LiteLLM gateway, on esh-docker-vm
(10.0.50.45:3211). Stood up 2026-08-21 as a parallel bake-off against
lobe-chat (:3210) — operator-approved candidate replacement. Nothing about
lobe-chat changes unless/until the operator retires it.
- URL: http://10.0.50.45:3211
- Image:
ghcr.io/open-webui/open-webui:main(deployed v0.11.0) - Compose (server):
/opt/docker/compose/open-webui/ - Admin:
lkraven/vh@phasefinal.com(first account = admin). Signup is disabled in steady state; add users via the Admin UI.
Why this exists
Lobe's config does not survive a fleet that changes daily: its model list is a static env string that never auto-discovers (rotted both directions), and half its config (TTS/model/voice) lives in each browser's local store, so "configured" is a property of whoever's laptop. Open WebUI is env-declarative (the deploy is the single source of config truth) and auto-tracks the gateway roster with no pins.
The deal-sealer — ENABLE_PERSISTENT_CONFIG=False
Default is True, under which env vars are read once on first boot, written to
the DB, then ignored forever (edit-env-and-bounce silently does nothing). We set
it False: every knob is re-read from env on each boot, and Admin-UI changes
do not survive a restart. Verified both directions at deploy (see gates).
⚠️ A var only reaches the container if
compose.yamlreferences it in itsenvironment:block. Adding a line to.envalone does nothing — Compose uses.envfor${VAR}substitution, not as anenv_file. This bit us once:ENABLE_API_KEYS(see below) was set in.envbut not wired in compose, so API-key minting stayed 403 until the compose var was added.
Acceptance gate results (verified on the box, 2026-08-21)
| Gate | Result | Notes |
|---|---|---|
| G1 declarative config works | ✅ PASS | env change takes on bounce (signup 200→403); UI/runtime change reverts on restart (first-admin runtime signup-disable + a runtime ENABLE_API_KEYS=true both reverted to env on restart). No persistent-config bug bit this deploy (no Redis; flags honored once named right). |
| G2 model list auto-tracks LiteLLM, no pins | ✅ PASS | Picker = the 31 live gateway models 1:1, nothing hand-listed; a retired gateway model disappears with no OWUI edit. arena-model (+1) is OWUI's own eval-arena feature, not a pin. Tradeoff: the picker also shows non-chat seats (embeddings/rerankers/tts) — the flip side of "no hand-listing"; can't be pruned to chat-only without reintroducing exactly the pinning G2 forbids. |
| G3 reconciliation API is real | ✅ PASS | GET /api/v1/models/export round-trips JSON; POST /api/v1/models/sync genuinely reconciles — create and delete to match desired state (verified: syncing set B removed set A). Sync requires full records (user_id/created_at/updated_at) — it round-trips the export shape. API keys enabled via ENABLE_API_KEYS=True (plural — see warning above). |
| G5 task model pinned | ✅ PASS | TASK_MODEL_EXTERNAL=summarizer (gen seat @ temp 0). |
| G4 TTS by env, second-browser zero-setup | ⏳ tts-dev | Env knobs set (direct at http://10.100.79.3:8198/v1, no LiteLLM); tts-dev runs the gate. |
End-to-end sanity: a chat completion to summarizer through OWUI → LiteLLM → seat returns correctly.
Credential posture (G6)
- Fresh, capped LiteLLM key
key_alias: open-webui-esh—all-proxy-models(reaches paid passthroughs per the operator's standing posture) but capped:max_budget: $50,budget_duration: 1mo(unlikelobe-chat-esh, which is uncapped — deliberately not inherited). A paid call spends real vendor credits until the cap trips. - Signup off after admin creation (no first-registrant LAN race).
- Secrets in the vault, never in git:
secret get esh-docker-vm/open-webui-litellm-key # -> OPENAI_API_KEY secret get esh-docker-vm/open-webui-secret-key # -> WEBUI_SECRET_KEY secret get esh-docker-vm/open-webui-admin # -> lkraven admin cred state
TTS (G4 — tts-dev owns)
Routed direct at the fleet dots gateway http://10.100.79.3:8198/v1, not
through LiteLLM (:8198 is itself the swap-proof switchboard; LiteLLM would add a
hop + a wrong audio/mpeg stamp; direct needs no gateway key). Five knobs:
AUDIO_TTS_ENGINE=openai, AUDIO_TTS_OPENAI_API_BASE_URL,
AUDIO_TTS_OPENAI_API_KEY (placeholder — seat is LAN/WG-only, no auth),
AUDIO_TTS_MODEL, AUDIO_TTS_VOICE (model/voice are tts-dev's to tune).
Reversible to ext-tts via the base URL if audio spend/log visibility is wanted.
Deploy / operate
# Canonical lives here; push to the host:
scripts/deploy-stack.sh esh-docker-vm open-webui --compose
# Real .env lives on the host (secrets from the vault), never committed.
# On the host:
cd /opt/docker/compose/open-webui
$EDITOR .env # change any knob HERE (env is the source of truth)
docker compose up -d # bounce to apply — UI changes do NOT persist
# RAG embeddings route through the gateway (qwen3-embedding) to avoid a
# boot-time HuggingFace download and stay fleet-consistent.