feat(open-webui): deploy env-declarative chat bake-off on esh-docker-vm:3211

Stand up Open WebUI v0.11.0 parallel to lobe-chat (:3210) as an
operator-approved candidate replacement. Env-declarative config
(ENABLE_PERSISTENT_CONFIG=False = the deploy is the config source of
truth), model picker auto-tracks the LiteLLM gateway with no pins,
background task model pinned to summarizer, TTS routed direct at the
dots gateway (:8198).

Gates verified on the box: G1 (declarative config, both directions),
G2 (picker == live gateway roster, no hand-listing), G3 (models/sync
genuinely reconciles create+delete), G5 (task model pinned). G4 (TTS)
handed to tts-dev.

Credential posture (G6): fresh capped LiteLLM key open-webui-esh
(all-proxy-models, $50/1mo budget) rather than inheriting the uncapped
lobe-chat-esh key; signup locked off after admin creation. Secrets
vaulted under esh-docker-vm/open-webui-*.

Note: the API-key toggle env var is ENABLE_API_KEYS (plural); a var
only reaches the container if compose names it in environment:.
lobe-chat is untouched.

persistent-memory.md

@@ -122,6 +122,8 @@ _As of 2026-08-21 00:35 — **the Heretic-300 session, and its reversal** (see t
 
 - **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
 
+- **🟢 OPEN WEBUI — deployed as a Lobe bake-off, esh-docker-vm:3211 (2026-08-21).** Operator-approved candidate replacement for `lobe-chat` (:3210), stood up **parallel** — Lobe untouched. `stacks/open-webui/` (v0.11.0, `ENABLE_PERSISTENT_CONFIG=False` = deploy is the config source of truth). Gates (verified on the box): **G1** declarative-config PASS both directions (env change takes on bounce, UI change reverts on restart — no persistent-config bug bit it); **G2** picker auto-tracks the 31 live gateway models 1:1, no pins (also shows non-chat seats — the flip side of no-hand-listing); **G3** `POST /api/v1/models/sync` genuinely reconciles (create+delete), `export` round-trips; **G5** task model pinned `summarizer`; **G4** (TTS, direct at `:8198`) handed to tts-dev. Admin = **lkraven** (temp pw, signup then locked off). Fresh **capped** key `open-webui-esh` (`all-proxy-models` + **$50/1mo** cap — NOT inherited from uncapped `lobe-chat-esh`). Secrets vaulted `esh-docker-vm/open-webui-{litellm-key,secret-key,admin}`. Folded in a `docker image prune -af` → **73.6 GB reclaimed**. ⚠ **LESSON:** in Open WebUI a `.env` var only reaches the container if `compose.yaml` names it in `environment:` (Compose uses `.env` for `${VAR}` substitution, not as an `env_file`); and the API-key toggle env var is **`ENABLE_API_KEYS`** (plural) — singular is inert. Detail lives in `stacks/open-webui/README.md`. **Operator's open call:** whether Lobe retires once G4 passes.
+
 - **⛔ RESULT 2026-08-21 — the gemma4 CoT test on a STABLE (v0.27.1) is BLOCKED by a config incompatibility, NOT the parser.** Tried serving the MeroMero NVFP4A16 quant on `vllm/vllm-openai:v0.27.1`. Two-stage failure: (1) v0.27.1's stricter transformers raised `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is per-layer` on the Gemma-4 config; setting `allow_global_per_layer_attribute_access:true` on `text_config` downgraded it to a warning BUT (2) then `gemma4.py load_weights` crashed with **`AssertionError: load weight (512) into parameter (256)`** — **Gemma-4-31B is genuinely HETEROGENEOUS (some layers head_dim 512, not a uniform 256)**, so forcing the global value built wrong-shaped params. The transformers guard was CORRECT; there is no safe override. **The MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot load on v0.27.x without a config migration (proper per_layer_config) or a re-quant against the newer transformers.** ⚠ **This also means the eventual gen-seat move to v0.27.2 stable must re-verify any Gemma-4 seat's config-compat** — the transformers heterogeneity change affects all Gemma-4 quants of this vintage. **FULLY REVERTED:** config.json restored (flags removed), compose + image back to `latest` (v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified on v0.24.0. Net: char-rp stays pinned to v0.24.0; MeroMero CoT remains undelivered. **The per-request-kwargs hypothesis was never even reachable** — couldn't load the model to test it. **For RP-with-CoT: gen-reasoning (works now) or a re-quant of MeroMero against v0.27.x transformers (real work, unproven payoff).**
 
 - **⚠️ CORRECTED 2026-08-21 — MeroMero-v2 CoT: NOT a hard wall, and NOT MeroMero-specific. My first conclusion ("gemma4 parser is process-wide") was WRONG.** Read the actual code, not the stale compose comment. **The real mechanism (gemma4-GENERAL, applies to any gemma4 finetune on this template family):** thinking is a **per-request** template toggle — `chat_template.jinja:347-352` emits the generation prompt `<|turn>model\n`, and **only when `enable_thinking` is false** does it prefill an empty `<|channel>thought\n<channel|>` to SUPPRESS thinking; `enable_thinking:true` omits the prefill so the model is free to open a real `<|channel>thought…<channel|>` block. The vLLM parser (`vllm/reasoning/gemma4_utils.py:parse_thinking_output`) **splits on `<|channel>`/`<channel|>` tag PRESENCE — "works with or without enable_thinking," NOT a process-wide flag.** The stale compose comment I trusted cited an OLD parser API (`vllm/parser/gemma4.py:439`) that this container does not run. **So there is no architectural blocker; the two-served-name gen pattern SHOULD work.** **What actually failed my test:** meromero runs `vllm/vllm-openai:latest` (v0.24.0); per-request `chat_template_kwargs.enable_thinking:true` produced no thinking on it, whereas the **gen seat's pinned nightly demonstrably applies per-request `chat_template_kwargs`** (gen-reasoning works). So the practical block is a **vLLM-version / per-request-plumbing issue on v0.24.0**, not the model and not the architecture — and it would hit ANY gemma4 finetune served on that image the same way. **UNVERIFIED FIX (needs a GPU window): re-serve meromero on the nightly image + no process default + per-request enable_thinking; likely yields clean split CoT.** Currently REVERTED to known-good (char-rp prose, process default false, single served-name). ⚠ Kept `MEROMERO_GPU_MEM_UTIL` 0.52→0.51 (0.52 no longer boots next to the bigger orcarouter gen; free 49.02 < 49.38 GiB; 0.51 = KV 2.00× @ 262K).

stacks/open-webui/.env.example

@@ -0,0 +1,51 @@
+# open-webui on esh-docker-vm. Real .env lives on the host and is NEVER committed.
+# Populate secrets from the vault:
+#   secret get esh-docker-vm/open-webui-litellm-key   -> OPENAI_API_KEY
+#   secret get esh-docker-vm/open-webui-secret-key     -> WEBUI_SECRET_KEY
+#
+# ⚠️ ENABLE_PERSISTENT_CONFIG=False is load-bearing (G1): env is re-read every
+#    boot, UI changes do NOT persist across restart. Change config HERE + bounce.
+
+OPENWEBUI_PORT=3211
+OPENWEBUI_TAG=main
+OPENWEBUI_ENV=prod
+
+# --- The deal-sealer ---
+ENABLE_PERSISTENT_CONFIG=False
+
+# --- Auth / signup (G6). Signup is OFF in steady state; the first/admin account
+#     (lkraven / vh@phasefinal.com) was created during deploy with signup briefly
+#     enabled, then locked. To add a user: create via Admin UI (signup stays off). ---
+WEBUI_AUTH=True
+WEBUI_SECRET_KEY=
+ENABLE_SIGNUP=False
+ENABLE_LOGIN_FORM=True
+DEFAULT_USER_ROLE=pending
+# Open WebUI v0.11.0 env var is ENABLE_API_KEYS (plural); singular is inert.
+ENABLE_API_KEYS=True
+
+# --- Model source: LiteLLM gateway, NO pins (G2 auto-tracks the live roster). ---
+ENABLE_OPENAI_API=True
+OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1
+OPENAI_API_KEY=
+ENABLE_OLLAMA_API=False
+
+# --- Background task model (G5): summarizer (gen seat @ temp 0). ---
+TASK_MODEL_EXTERNAL=summarizer
+
+# --- RAG embeddings via the gateway (avoids boot-time HF download). ---
+RAG_EMBEDDING_ENGINE=openai
+RAG_EMBEDDING_MODEL=qwen3-embedding
+RAG_OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1
+
+# --- TTS (G4 — tts-dev tunes model/voice). Direct at the dots gateway, no LiteLLM. ---
+AUDIO_TTS_ENGINE=openai
+AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1
+AUDIO_TTS_OPENAI_API_KEY=sk-no-auth-lan-seat
+AUDIO_TTS_MODEL=tts-1
+AUDIO_TTS_VOICE=nova
+
+# --- Telemetry off ---
+ANONYMIZED_TELEMETRY=False
+DO_NOT_TRACK=True
+SCARF_NO_ANALYTICS=True

stacks/open-webui/README.md

@@ -0,0 +1,87 @@
+# open-webui
+
+Env-declarative chat frontend over the LiteLLM gateway, on **esh-docker-vm**
+(`10.0.50.45:3211`). Stood up 2026-08-21 as a **parallel bake-off** against
+`lobe-chat` (`:3210`) — operator-approved candidate replacement. Nothing about
+`lobe-chat` changes unless/until the operator retires it.
+
+- **URL:** http://10.0.50.45:3211
+- **Image:** `ghcr.io/open-webui/open-webui:main` (deployed v0.11.0)
+- **Compose (server):** `/opt/docker/compose/open-webui/`
+- **Admin:** `lkraven` / `vh@phasefinal.com` (first account = admin). Signup is
+  **disabled** in steady state; add users via the Admin UI.
+
+## Why this exists
+
+Lobe's config does not survive a fleet that changes daily: its model list is a
+static env string that never auto-discovers (rotted both directions), and half
+its config (TTS/model/voice) lives in each browser's local store, so
+"configured" is a property of whoever's laptop. Open WebUI is **env-declarative**
+(the deploy is the single source of config truth) and **auto-tracks** the gateway
+roster with no pins.
+
+## The deal-sealer — `ENABLE_PERSISTENT_CONFIG=False`
+
+Default is `True`, under which env vars are read once on first boot, written to
+the DB, then ignored forever (edit-env-and-bounce silently does nothing). We set
+it **False**: every knob is re-read from env on each boot, and Admin-UI changes
+do **not** survive a restart. Verified both directions at deploy (see gates).
+
+> ⚠️ **A var only reaches the container if `compose.yaml` references it** in its
+> `environment:` block. Adding a line to `.env` alone does nothing — Compose uses
+> `.env` for `${VAR}` substitution, not as an `env_file`. This bit us once:
+> `ENABLE_API_KEYS` (see below) was set in `.env` but not wired in compose, so
+> API-key minting stayed 403 until the compose var was added.
+
+## Acceptance gate results (verified on the box, 2026-08-21)
+
+| Gate | Result | Notes |
+|---|---|---|
+| **G1** declarative config works | ✅ PASS | env change takes on bounce (signup 200→403); UI/runtime change reverts on restart (first-admin runtime signup-disable + a runtime `ENABLE_API_KEYS=true` both reverted to env on restart). No persistent-config bug bit this deploy (no Redis; flags honored once named right). |
+| **G2** model list auto-tracks LiteLLM, no pins | ✅ PASS | Picker = the 31 live gateway models 1:1, nothing hand-listed; a retired gateway model disappears with no OWUI edit. `arena-model` (+1) is OWUI's own eval-arena feature, not a pin. **Tradeoff:** the picker also shows non-chat seats (embeddings/rerankers/tts) — the flip side of "no hand-listing"; can't be pruned to chat-only without reintroducing exactly the pinning G2 forbids. |
+| **G3** reconciliation API is real | ✅ PASS | `GET /api/v1/models/export` round-trips JSON; `POST /api/v1/models/sync` genuinely **reconciles** — create **and delete** to match desired state (verified: syncing set B removed set A). Sync requires full records (`user_id`/`created_at`/`updated_at`) — it round-trips the `export` shape. API keys enabled via `ENABLE_API_KEYS=True` (plural — see warning above). |
+| **G5** task model pinned | ✅ PASS | `TASK_MODEL_EXTERNAL=summarizer` (gen seat @ temp 0). |
+| **G4** TTS by env, second-browser zero-setup | ⏳ tts-dev | Env knobs set (direct at `http://10.100.79.3:8198/v1`, no LiteLLM); tts-dev runs the gate. |
+
+End-to-end sanity: a chat completion to `summarizer` through OWUI → LiteLLM → seat returns correctly.
+
+## Credential posture (G6)
+
+- **Fresh, capped LiteLLM key** `key_alias: open-webui-esh` — `all-proxy-models`
+  (reaches paid passthroughs per the operator's standing posture) **but capped:
+  `max_budget: $50`, `budget_duration: 1mo`** (unlike `lobe-chat-esh`, which is
+  uncapped — deliberately not inherited). A paid call spends real vendor credits
+  until the cap trips.
+- **Signup off** after admin creation (no first-registrant LAN race).
+- Secrets in the vault, never in git:
+  ```
+  secret get esh-docker-vm/open-webui-litellm-key   # -> OPENAI_API_KEY
+  secret get esh-docker-vm/open-webui-secret-key      # -> WEBUI_SECRET_KEY
+  secret get esh-docker-vm/open-webui-admin           # -> lkraven admin cred state
+  ```
+
+## TTS (G4 — tts-dev owns)
+
+Routed **direct at the fleet dots gateway** `http://10.100.79.3:8198/v1`, not
+through LiteLLM (:8198 is itself the swap-proof switchboard; LiteLLM would add a
+hop + a wrong `audio/mpeg` stamp; direct needs no gateway key). Five knobs:
+`AUDIO_TTS_ENGINE=openai`, `AUDIO_TTS_OPENAI_API_BASE_URL`,
+`AUDIO_TTS_OPENAI_API_KEY` (placeholder — seat is LAN/WG-only, no auth),
+`AUDIO_TTS_MODEL`, `AUDIO_TTS_VOICE` (model/voice are tts-dev's to tune).
+Reversible to `ext-tts` via the base URL if audio spend/log visibility is wanted.
+
+## Deploy / operate
+
+```bash
+# Canonical lives here; push to the host:
+scripts/deploy-stack.sh esh-docker-vm open-webui --compose
+# Real .env lives on the host (secrets from the vault), never committed.
+
+# On the host:
+cd /opt/docker/compose/open-webui
+$EDITOR .env          # change any knob HERE (env is the source of truth)
+docker compose up -d  # bounce to apply — UI changes do NOT persist
+
+# RAG embeddings route through the gateway (qwen3-embedding) to avoid a
+# boot-time HuggingFace download and stay fleet-consistent.
+```

stacks/open-webui/compose.yaml

@@ -0,0 +1,122 @@
+# open-webui — evaluation chat frontend over the LiteLLM gateway, on esh-docker-vm.
+#
+# WHY THIS EXISTS (bake-off against lobe-chat, 2026-08-21)
+#   Lobe's config model does not survive a fleet that changes daily: its model
+#   list is a static env string that never auto-discovers (it rotted in BOTH
+#   directions — missed `sec`/`sec-reasoning` going in, kept retired `char-rp`
+#   names that 400 on click), and half its config (TTS service/model/voice)
+#   lives in each browser's local store, so "configured" is a property of
+#   whoever's laptop rather than of the deploy. Open WebUI is env-declarative and
+#   auto-tracks the gateway roster. This stack stands up PARALLEL to lobe-chat
+#   (Lobe on :3210, this on :3211); nothing about Lobe changes until/unless the
+#   operator retires it. The June-2026 weight objection (1,825 MB vs 143 MB) is
+#   moot here: esh-docker-vm has ~138 GB free.
+#
+# THE DEAL-SEALER — ENABLE_PERSISTENT_CONFIG=False (see .env)
+#   Default is True, and under True env vars are read ONCE on first boot,
+#   written to the DB, then ignored forever — so edit-env-and-bounce silently
+#   does nothing. We set it False so the deploy is the single source of config
+#   truth: every knob below is re-read from env on each boot, and a change made
+#   in the Admin UI does NOT survive a restart. Verified both directions at
+#   deploy (G1). ⚠️ Open upstream bugs touch this flag (open-webui #20830 Redis
+#   cache — N/A, we use no Redis; #15505; #21839) — re-verify after image bumps.
+#
+# CREDENTIAL POSTURE (G6 — chosen, not inherited)
+#   Purpose-minted LiteLLM virtual key `key_alias: open-webui-esh`, scoped to the
+#   `all-proxy-models` access group (reaches EVERYTHING incl. paid passthroughs —
+#   GLM/Kimi/gen-frontier) per the operator's standing posture, BUT capped with a
+#   `max_budget` ceiling (unlike lobe-chat-esh which is uncapped). A paid call
+#   from this UI spends real vendor credits until the cap trips. Signup is turned
+#   OFF after the first/admin account is created, so the surface is not open-
+#   registration on the LAN. Secrets live in the vault, never in git:
+#     secret get esh-docker-vm/open-webui-litellm-key   -> OPENAI_API_KEY
+#     secret get esh-docker-vm/open-webui-secret-key     -> WEBUI_SECRET_KEY
+#     secret get esh-docker-vm/open-webui-admin          -> lkraven admin cred state
+#
+# TTS (G4 — tts-dev owns verification; env knobs set here)
+#   Routed DIRECT at the fleet dots gateway http://10.100.79.3:8198/v1, NOT
+#   through LiteLLM. Rationale (agreed with tts-dev): :8198 is itself the swap-
+#   proof switchboard — engines swap behind it — so LiteLLM would add only a hop
+#   and a wrong `audio/mpeg` content-type stamp; direct needs no gateway key on
+#   the audio path. Reversible: flip AUDIO_TTS_OPENAI_API_BASE_URL to the gateway
+#   + `ext-tts` if audio spend/log visibility is ever wanted. The five AUDIO_TTS_*
+#   vars are tts-dev's to tune (model/voice); base URL + engine are settled.
+
+name: open-webui
+
+services:
+  open-webui:
+    image: ghcr.io/open-webui/open-webui:${OPENWEBUI_TAG:-main}
+    container_name: ${OPENWEBUI_CONTAINER:-open-webui}
+    restart: unless-stopped
+    ports:
+      - "${OPENWEBUI_PORT:-3211}:8080"
+    environment:
+      # --- The deal-sealer: deploy is the single source of config truth. ---
+      - ENABLE_PERSISTENT_CONFIG=${ENABLE_PERSISTENT_CONFIG:-False}
+      - ENV=${OPENWEBUI_ENV:-prod}
+      - WEBUI_NAME=${WEBUI_NAME:-PFI Open WebUI}
+      - WEBUI_URL=${WEBUI_URL:-http://10.0.50.45:3211}
+      # --- Auth / signup posture (G6). Signup is flipped False post-admin. ---
+      - WEBUI_AUTH=${WEBUI_AUTH:-True}
+      - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:?set in .env, from the vault}
+      - ENABLE_SIGNUP=${ENABLE_SIGNUP:-False}
+      - ENABLE_LOGIN_FORM=${ENABLE_LOGIN_FORM:-True}
+      - DEFAULT_USER_ROLE=${DEFAULT_USER_ROLE:-pending}
+      # NOTE: the env var is ENABLE_API_KEYS (plural) in Open WebUI v0.11.0
+      # (config.py: os.getenv('ENABLE_API_KEYS','False')). The singular form is
+      # inert — a deploy that sets ENABLE_API_KEY silently leaves key-mint 403.
+      - ENABLE_API_KEYS=${ENABLE_API_KEYS:-True}
+      # --- Model source: the LiteLLM gateway, no pins (G2 auto-tracks). ---
+      - ENABLE_OPENAI_API=${ENABLE_OPENAI_API:-True}
+      - OPENAI_API_BASE_URL=${OPENAI_API_BASE_URL:-http://10.250.50.70:4000/v1}
+      - OPENAI_API_KEY=${OPENAI_API_KEY:?set in .env, from the vault}
+      # Ollama is banned fleet-wide; keep the client from probing localhost:11434.
+      - ENABLE_OLLAMA_API=${ENABLE_OLLAMA_API:-False}
+      # --- Background task model (G5): pin it or an unpinned default hammers a
+      #     name the gateway does not serve and fills logs with 4xx. External
+      #     (OpenAI-type) models use TASK_MODEL_EXTERNAL. summarizer = gen seat
+      #     at temp 0, the right fit for auto-title/tag/query-rewrite. ---
+      - TASK_MODEL_EXTERNAL=${TASK_MODEL_EXTERNAL:-summarizer}
+      # --- RAG embeddings through the gateway (fleet-consistent; avoids a boot-
+      #     time HuggingFace download of the default local sentence-transformer). ---
+      - RAG_EMBEDDING_ENGINE=${RAG_EMBEDDING_ENGINE:-openai}
+      - RAG_EMBEDDING_MODEL=${RAG_EMBEDDING_MODEL:-qwen3-embedding}
+      - RAG_OPENAI_API_BASE_URL=${RAG_OPENAI_API_BASE_URL:-http://10.250.50.70:4000/v1}
+      - RAG_OPENAI_API_KEY=${OPENAI_API_KEY:?set in .env, from the vault}
+      # --- TTS (G4): direct at the dots gateway, engine + base URL settled;
+      #     model/voice are tts-dev's to tune. Seat is LAN/WG-only, no auth. ---
+      - AUDIO_TTS_ENGINE=${AUDIO_TTS_ENGINE:-openai}
+      - AUDIO_TTS_OPENAI_API_BASE_URL=${AUDIO_TTS_OPENAI_API_BASE_URL:-http://10.100.79.3:8198/v1}
+      - AUDIO_TTS_OPENAI_API_KEY=${AUDIO_TTS_OPENAI_API_KEY:-sk-no-auth-lan-seat}
+      - AUDIO_TTS_MODEL=${AUDIO_TTS_MODEL:-tts-1}
+      - AUDIO_TTS_VOICE=${AUDIO_TTS_VOICE:-nova}
+      # --- Telemetry off. ---
+      - ANONYMIZED_TELEMETRY=${ANONYMIZED_TELEMETRY:-False}
+      - DO_NOT_TRACK=${DO_NOT_TRACK:-True}
+      - SCARF_NO_ANALYTICS=${SCARF_NO_ANALYTICS:-True}
+    volumes:
+      - open-webui_data:/app/backend/data
+    healthcheck:
+      test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/health').status==200 else 1)\" || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 5
+      start_period: 120s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=Apps
+      - homepage.name=Open WebUI
+      - homepage.icon=mdi-chat-question
+      - homepage.description=Env-declarative chat frontend over the LiteLLM gateway (eval vs Lobe)
+      - homepage.href=http://10.0.50.45:${OPENWEBUI_PORT:-3211}
+
+volumes:
+  open-webui_data:
+    name: open-webui_data
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true