feat(lobe-chat): stand up Lobe Chat on esh-docker-vm over the LiteLLM gateway

Replacement candidate for the hand-rolled gateway-chat HTML surface, which
the operator does not want to keep improving -- it has already produced two
defects tonight.

Chosen over Open WebUI on weight, measured from the registries rather than
recalled: Lobe 143 MB compressed / 1 layer vs Open WebUI 1,825 MB / 19
layers, a 12.8x difference. Open WebUI was declined in June 2026 on weight
and that still holds; its secondary recorded objection (empty-tools 400
against vLLM) is now moot since strip_empty_tools covers the normal API
path and only missed LiteLLM's built-in playground.

CREDENTIAL POSTURE: deliberately NOT the shared all-agents key, which
reaches the paid GLM/Kimi passthroughs -- a LAN-exposed chat UI holding it
would let anyone reaching the port spend vendor credits from a pool shared
across every project. Minted a scoped LiteLLM virtual key
(key_alias lobe-chat-esh) limited to the 20 free local models, and verified
the scoping BOTH ways: gen answers, glm-5.2 / kimi-k3 / gen-frontier all
return 'key not allowed to access model'. Secrets vaulted, host .env 0600.

Verified from INSIDE the container, not just from the host: /v1/models
returns the fleet seats and a gen round-trip returns 'ok', so the app's own
network path and key both work. Container healthy, / -> 307 -> /chat -> 200.

Documents the open question this deploy exists to answer: whether Lobe's
TTS is ENV-configurable or UI-only. That is the operator's deciding
criterion and is NOT yet established -- Open WebUI has dedicated AUDIO_TTS_*
vars, Lobe documents a shared OPENAI_PROXY_URL which should carry TTS since
LiteLLM serves audio/speech on the same base, but that is inference.

Also records the ext-tts voice foot-gun: unknown voices 404 and can trip
the router cooldown, so the voice must be pinned rather than left at a UI
default.

.gitignore

@@ -39,3 +39,4 @@ graphify-out/*
 # Python bytecode (e.g. from local py_compile of stack wrappers)
 __pycache__/
 *.pyc
+stacks/lobe-chat/.env

stacks/lobe-chat/.env.example

@@ -0,0 +1,10 @@
+# lobe-chat on esh-docker-vm. Real .env lives on the host and is NEVER committed.
+# Populate from the vault:
+#   secret get esh-docker-vm/lobe-chat-litellm-key        -> OPENAI_API_KEY
+#   secret get esh-docker-vm/lobe-chat-access-code        -> ACCESS_CODE
+#   secret get esh-docker-vm/lobe-chat-key-vaults-secret  -> KEY_VAULTS_SECRET
+LOBE_PORT=3210
+OPENAI_PROXY_URL=http://10.250.50.70:4000/v1
+OPENAI_API_KEY=
+ACCESS_CODE=
+KEY_VAULTS_SECRET=

stacks/lobe-chat/README.md

@@ -0,0 +1,84 @@
+# lobe-chat — chat frontend over the LiteLLM gateway (esh-docker-vm)
+
+Evaluation replacement for the hand-rolled `gateway-chat` single-file HTML
+surface, which the operator does not want to keep improving — it has already
+produced two defects (a 1024 `max_tokens` default that read as model degeneracy,
+and a `NaN`→`null` `max_tokens` bug).
+
+- **Host:** esh-docker-vm (10.0.50.45) · **Port:** 3210 · **URL:** http://10.0.50.45:3210
+- **Backend:** LiteLLM gateway at `10.250.50.70:4000/v1` (reachable from ESH, ~30 ms)
+
+## Why Lobe over Open WebUI
+
+Weight, measured from the registries rather than from marketing:
+
+| | compressed | layers |
+|---|---|---|
+| Lobe Chat | **143 MB** | 1 |
+| Open WebUI | 1,825 MB | 19 |
+
+12.8×. Open WebUI was declined by the operator in June 2026 on weight grounds and
+that objection still holds. (Its recorded *secondary* objection — the empty-`tools`
+400 against vLLM — is now moot: the `strip_empty_tools` callback covers the normal
+API path and only failed to protect LiteLLM's own built-in playground.)
+
+## ⚠️ The open question this deploy exists to answer
+
+**Is Lobe's TTS configurable by ENV, or only through the settings UI?** That is the
+operator's deciding criterion — manageable/scriptable by an agent — and it is
+unresolved. Open WebUI has dedicated `AUDIO_TTS_ENGINE` / `AUDIO_TTS_OPENAI_API_BASE_URL`
+/ `AUDIO_TTS_MODEL` / `AUDIO_TTS_VOICE`. Lobe documents a **shared** `OPENAI_PROXY_URL`,
+which *should* carry TTS because LiteLLM serves `/v1/chat/completions` and
+`/v1/audio/speech` on the same base — but that is inference, not verification.
+
+If it turns out UI-only, the real choice is: 1.8 GB with genuine scriptability, or
+143 MB with click-ops. That is an operator call, not an agent one.
+
+## ⚠️ ext-tts voice foot-gun
+
+`ext-tts` accepts `donut` / `emmie` / `glados` / `miranda` (+ emotion variants) and
+the OpenAI aliases `nova` / `alloy`. **Any other OpenAI voice name (`echo`, `fable`,
+`onyx`, `shimmer`) 404s and can trip the LiteLLM router cooldown.** Pin the voice
+explicitly rather than accepting whatever the UI defaults to.
+
+## Credential posture
+
+Deliberately **not** the shared all-agents key — that reaches the paid passthroughs
+(GLM, Kimi), and a LAN-exposed chat UI holding it would let anyone who can reach the
+port spend vendor credits from a pool shared across every project.
+
+This stack uses a purpose-minted LiteLLM virtual key, `key_alias: lobe-chat-esh`,
+scoped to the 20 free **local** models. Scoping was verified at mint time, both
+directions:
+
+- `gen` → answers
+- `glm-5.2`, `kimi-k3`, `gen-frontier` → `key not allowed to access model`
+
+Secrets are in the vault, never in git. `.env` on the host is `0600`:
+
+```bash
+secret get esh-docker-vm/lobe-chat-litellm-key        # -> OPENAI_API_KEY
+secret get esh-docker-vm/lobe-chat-access-code        # -> ACCESS_CODE (UI gate)
+secret get esh-docker-vm/lobe-chat-key-vaults-secret  # -> KEY_VAULTS_SECRET
+```
+
+`ACCESS_CODE` matters: this is a home-lab LAN segment with nothing in front of it.
+
+## Verified on deploy (2026-08-16)
+
+- container healthy; `http://10.0.50.45:3210/` → 307 → `/chat` → 200
+- from **inside** the container: `GET /v1/models` returns the fleet seats, and a
+  `gen` chat round-trip returns `"ok"` — so the app's own network path and key work,
+  not merely the host's
+- image on disk 617 MB (143 MB compressed)
+
+## Deploy
+
+```bash
+scripts/deploy-stack.sh esh-docker-vm lobe-chat --compose
+# on host: populate .env from the vault (see above), chmod 600, then
+ssh lkraven@10.0.50.45 'cd /opt/docker/compose/lobe-chat && docker compose up -d'
+```
+
+`lkraven` owns `/opt/docker` and is in the `docker` group on this host, so no sudo
+is needed. Note ESH is outside the infra-ops NOPASSWD grant.

stacks/lobe-chat/compose.yaml

@@ -0,0 +1,75 @@
+# lobe-chat — evaluation chat frontend over the LiteLLM gateway, on esh-docker-vm.
+#
+# WHY THIS EXISTS
+#   `gateway-chat` is a hand-rolled single-file HTML surface the operator does not
+#   want to keep improving; it has already cost two defects (a 1024 max_tokens
+#   default that read as model degeneracy, and a NaN->null max_tokens bug).
+#   Lobe was chosen over Open WebUI on weight, measured from the registries
+#   rather than from marketing: 143 MB compressed / 1 layer, against Open WebUI's
+#   1,825 MB / 19 layers. 12.8x.
+#
+# OPEN QUESTION THIS DEPLOY EXISTS TO ANSWER
+#   Whether Lobe's TTS base URL / model / voice are settable by ENV or only via
+#   the settings UI. Open WebUI has dedicated AUDIO_TTS_* vars; Lobe documents a
+#   SHARED OPENAI_PROXY_URL, which should work because LiteLLM serves both
+#   /v1/chat/completions and /v1/audio/speech on one base -- but "should" is not
+#   "does". If it turns out UI-only it fails the operator's manageable-by-agent
+#   criterion even while winning on weight.
+#
+# ⚠️ CREDENTIAL POSTURE — deliberately NOT the shared all-agents key.
+#   That key reaches the PAID passthroughs (GLM, Kimi) and a LAN-exposed chat UI
+#   holding it would let anyone who can reach the port spend vendor credits on a
+#   pool shared across every project. This stack uses a purpose-minted LiteLLM
+#   virtual key (`key_alias: lobe-chat-esh`) scoped to the 20 free LOCAL models.
+#   Verified at mint time: `gen` answers; `glm-5.2`, `kimi-k3` and `gen-frontier`
+#   all return "key not allowed to access model".
+#   Secrets live in the vault, never in git:
+#     secret get esh-docker-vm/lobe-chat-litellm-key
+#     secret get esh-docker-vm/lobe-chat-access-code
+#     secret get esh-docker-vm/lobe-chat-key-vaults-secret
+#
+# ⚠️ ext-tts FOOT-GUN: an unknown voice 404s and can trip the LiteLLM router
+#   cooldown. `ext-tts` accepts donut/emmie/glados/miranda (+ emotion variants)
+#   and the OpenAI aliases nova/alloy. If Lobe sends any other OpenAI voice name
+#   (echo, fable, onyx, shimmer) it will 404 -- pin the voice rather than leaving
+#   it at whatever the UI defaults to.
+
+name: lobe-chat
+
+services:
+  lobe-chat:
+    image: lobehub/lobe-chat:${LOBE_TAG:-latest}
+    container_name: ${LOBE_CONTAINER:-lobe-chat}
+    restart: unless-stopped
+    ports:
+      - "${LOBE_PORT:-3210}:3210"
+    environment:
+      # Chat + TTS both resolve through the one LiteLLM base.
+      - OPENAI_API_KEY=${OPENAI_API_KEY:?set in .env, from the vault}
+      - OPENAI_PROXY_URL=${OPENAI_PROXY_URL:-http://10.250.50.70:4000/v1}
+      # Pin the visible model list so the picker shows fleet seats rather than
+      # Lobe's built-in OpenAI catalogue (which would 404 against the gateway).
+      - OPENAI_MODEL_LIST=${OPENAI_MODEL_LIST:--all,+gen,+gen-reasoning,+char-rp,+char-rp-reasoning,+char-rp-fable,+summarizer,+summarizer-large,+classifier,+image-judge,+coder-fast,+chat-judge}
+      # LAN-exposed on a home-lab segment with no auth in front: gate it.
+      - ACCESS_CODE=${ACCESS_CODE:?set in .env, from the vault}
+      # Encrypts any provider keys a user stores in the UI.
+      - KEY_VAULTS_SECRET=${KEY_VAULTS_SECRET:?set in .env, from the vault}
+    healthcheck:
+      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3210/ >/dev/null 2>&1 || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 60s
+    networks:
+      - tnet
+    labels:
+      - homepage.group=Apps
+      - homepage.name=Lobe Chat
+      - homepage.icon=mdi-chat-processing
+      - homepage.description=Chat frontend over the LiteLLM gateway (eval)
+      - homepage.href=http://10.0.50.45:${LOBE_PORT:-3210}
+
+networks:
+  tnet:
+    name: traefik-net
+    external: true