From 76834777a4409bf4ab334c218be5d320a98167e7 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Fri, 21 Aug 2026 09:52:07 -0700 Subject: [PATCH] feat(open-webui): deploy env-declarative chat bake-off on esh-docker-vm:3211 Stand up Open WebUI v0.11.0 parallel to lobe-chat (:3210) as an operator-approved candidate replacement. Env-declarative config (ENABLE_PERSISTENT_CONFIG=False = the deploy is the config source of truth), model picker auto-tracks the LiteLLM gateway with no pins, background task model pinned to summarizer, TTS routed direct at the dots gateway (:8198). Gates verified on the box: G1 (declarative config, both directions), G2 (picker == live gateway roster, no hand-listing), G3 (models/sync genuinely reconciles create+delete), G5 (task model pinned). G4 (TTS) handed to tts-dev. Credential posture (G6): fresh capped LiteLLM key open-webui-esh (all-proxy-models, $50/1mo budget) rather than inheriting the uncapped lobe-chat-esh key; signup locked off after admin creation. Secrets vaulted under esh-docker-vm/open-webui-*. Note: the API-key toggle env var is ENABLE_API_KEYS (plural); a var only reaches the container if compose names it in environment:. lobe-chat is untouched. --- persistent-memory.md | 2 + stacks/open-webui/.env.example | 51 ++++++++++++++ stacks/open-webui/README.md | 87 +++++++++++++++++++++++ stacks/open-webui/compose.yaml | 122 +++++++++++++++++++++++++++++++++ 4 files changed, 262 insertions(+) create mode 100644 stacks/open-webui/.env.example create mode 100644 stacks/open-webui/README.md create mode 100644 stacks/open-webui/compose.yaml diff --git a/persistent-memory.md b/persistent-memory.md index fea11c1..eea3bcc 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,6 +122,8 @@ _As of 2026-08-21 00:35 β€” **the Heretic-300 session, and its reversal** (see t - **🟒 HOMEPAGE β€” cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own β€” do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. β†’ `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` +- **🟒 OPEN WEBUI β€” deployed as a Lobe bake-off, esh-docker-vm:3211 (2026-08-21).** Operator-approved candidate replacement for `lobe-chat` (:3210), stood up **parallel** β€” Lobe untouched. `stacks/open-webui/` (v0.11.0, `ENABLE_PERSISTENT_CONFIG=False` = deploy is the config source of truth). Gates (verified on the box): **G1** declarative-config PASS both directions (env change takes on bounce, UI change reverts on restart β€” no persistent-config bug bit it); **G2** picker auto-tracks the 31 live gateway models 1:1, no pins (also shows non-chat seats β€” the flip side of no-hand-listing); **G3** `POST /api/v1/models/sync` genuinely reconciles (create+delete), `export` round-trips; **G5** task model pinned `summarizer`; **G4** (TTS, direct at `:8198`) handed to tts-dev. Admin = **lkraven** (temp pw, signup then locked off). Fresh **capped** key `open-webui-esh` (`all-proxy-models` + **$50/1mo** cap β€” NOT inherited from uncapped `lobe-chat-esh`). Secrets vaulted `esh-docker-vm/open-webui-{litellm-key,secret-key,admin}`. Folded in a `docker image prune -af` β†’ **73.6 GB reclaimed**. ⚠ **LESSON:** in Open WebUI a `.env` var only reaches the container if `compose.yaml` names it in `environment:` (Compose uses `.env` for `${VAR}` substitution, not as an `env_file`); and the API-key toggle env var is **`ENABLE_API_KEYS`** (plural) β€” singular is inert. Detail lives in `stacks/open-webui/README.md`. **Operator's open call:** whether Lobe retires once G4 passes. + - **β›” RESULT 2026-08-21 β€” the gemma4 CoT test on a STABLE (v0.27.1) is BLOCKED by a config incompatibility, NOT the parser.** Tried serving the MeroMero NVFP4A16 quant on `vllm/vllm-openai:v0.27.1`. Two-stage failure: (1) v0.27.1's stricter transformers raised `AmbiguousGlobalPerLayerAttributeError: 'head_dim' is per-layer` on the Gemma-4 config; setting `allow_global_per_layer_attribute_access:true` on `text_config` downgraded it to a warning BUT (2) then `gemma4.py load_weights` crashed with **`AssertionError: load weight (512) into parameter (256)`** β€” **Gemma-4-31B is genuinely HETEROGENEOUS (some layers head_dim 512, not a uniform 256)**, so forcing the global value built wrong-shaped params. The transformers guard was CORRECT; there is no safe override. **The MeroMero quant's config was authored for v0.24.0's Gemma4 loader and cannot load on v0.27.x without a config migration (proper per_layer_config) or a re-quant against the newer transformers.** ⚠ **This also means the eventual gen-seat move to v0.27.2 stable must re-verify any Gemma-4 seat's config-compat** β€” the transformers heterogeneity change affects all Gemma-4 quants of this vintage. **FULLY REVERTED:** config.json restored (flags removed), compose + image back to `latest` (v0.24.0), gateway char-rp-reasoning removed, char-rp prose verified on v0.24.0. Net: char-rp stays pinned to v0.24.0; MeroMero CoT remains undelivered. **The per-request-kwargs hypothesis was never even reachable** β€” couldn't load the model to test it. **For RP-with-CoT: gen-reasoning (works now) or a re-quant of MeroMero against v0.27.x transformers (real work, unproven payoff).** - **⚠️ CORRECTED 2026-08-21 β€” MeroMero-v2 CoT: NOT a hard wall, and NOT MeroMero-specific. My first conclusion ("gemma4 parser is process-wide") was WRONG.** Read the actual code, not the stale compose comment. **The real mechanism (gemma4-GENERAL, applies to any gemma4 finetune on this template family):** thinking is a **per-request** template toggle β€” `chat_template.jinja:347-352` emits the generation prompt `<|turn>model\n`, and **only when `enable_thinking` is false** does it prefill an empty `<|channel>thought\n` to SUPPRESS thinking; `enable_thinking:true` omits the prefill so the model is free to open a real `<|channel>thought…` block. The vLLM parser (`vllm/reasoning/gemma4_utils.py:parse_thinking_output`) **splits on `<|channel>`/`` tag PRESENCE β€” "works with or without enable_thinking," NOT a process-wide flag.** The stale compose comment I trusted cited an OLD parser API (`vllm/parser/gemma4.py:439`) that this container does not run. **So there is no architectural blocker; the two-served-name gen pattern SHOULD work.** **What actually failed my test:** meromero runs `vllm/vllm-openai:latest` (v0.24.0); per-request `chat_template_kwargs.enable_thinking:true` produced no thinking on it, whereas the **gen seat's pinned nightly demonstrably applies per-request `chat_template_kwargs`** (gen-reasoning works). So the practical block is a **vLLM-version / per-request-plumbing issue on v0.24.0**, not the model and not the architecture β€” and it would hit ANY gemma4 finetune served on that image the same way. **UNVERIFIED FIX (needs a GPU window): re-serve meromero on the nightly image + no process default + per-request enable_thinking; likely yields clean split CoT.** Currently REVERTED to known-good (char-rp prose, process default false, single served-name). ⚠ Kept `MEROMERO_GPU_MEM_UTIL` 0.52β†’0.51 (0.52 no longer boots next to the bigger orcarouter gen; free 49.02 < 49.38 GiB; 0.51 = KV 2.00Γ— @ 262K). diff --git a/stacks/open-webui/.env.example b/stacks/open-webui/.env.example new file mode 100644 index 0000000..8e14747 --- /dev/null +++ b/stacks/open-webui/.env.example @@ -0,0 +1,51 @@ +# open-webui on esh-docker-vm. Real .env lives on the host and is NEVER committed. +# Populate secrets from the vault: +# secret get esh-docker-vm/open-webui-litellm-key -> OPENAI_API_KEY +# secret get esh-docker-vm/open-webui-secret-key -> WEBUI_SECRET_KEY +# +# ⚠️ ENABLE_PERSISTENT_CONFIG=False is load-bearing (G1): env is re-read every +# boot, UI changes do NOT persist across restart. Change config HERE + bounce. + +OPENWEBUI_PORT=3211 +OPENWEBUI_TAG=main +OPENWEBUI_ENV=prod + +# --- The deal-sealer --- +ENABLE_PERSISTENT_CONFIG=False + +# --- Auth / signup (G6). Signup is OFF in steady state; the first/admin account +# (lkraven / vh@phasefinal.com) was created during deploy with signup briefly +# enabled, then locked. To add a user: create via Admin UI (signup stays off). --- +WEBUI_AUTH=True +WEBUI_SECRET_KEY= +ENABLE_SIGNUP=False +ENABLE_LOGIN_FORM=True +DEFAULT_USER_ROLE=pending +# Open WebUI v0.11.0 env var is ENABLE_API_KEYS (plural); singular is inert. +ENABLE_API_KEYS=True + +# --- Model source: LiteLLM gateway, NO pins (G2 auto-tracks the live roster). --- +ENABLE_OPENAI_API=True +OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1 +OPENAI_API_KEY= +ENABLE_OLLAMA_API=False + +# --- Background task model (G5): summarizer (gen seat @ temp 0). --- +TASK_MODEL_EXTERNAL=summarizer + +# --- RAG embeddings via the gateway (avoids boot-time HF download). --- +RAG_EMBEDDING_ENGINE=openai +RAG_EMBEDDING_MODEL=qwen3-embedding +RAG_OPENAI_API_BASE_URL=http://10.250.50.70:4000/v1 + +# --- TTS (G4 β€” tts-dev tunes model/voice). Direct at the dots gateway, no LiteLLM. --- +AUDIO_TTS_ENGINE=openai +AUDIO_TTS_OPENAI_API_BASE_URL=http://10.100.79.3:8198/v1 +AUDIO_TTS_OPENAI_API_KEY=sk-no-auth-lan-seat +AUDIO_TTS_MODEL=tts-1 +AUDIO_TTS_VOICE=nova + +# --- Telemetry off --- +ANONYMIZED_TELEMETRY=False +DO_NOT_TRACK=True +SCARF_NO_ANALYTICS=True diff --git a/stacks/open-webui/README.md b/stacks/open-webui/README.md new file mode 100644 index 0000000..e81e2c4 --- /dev/null +++ b/stacks/open-webui/README.md @@ -0,0 +1,87 @@ +# open-webui + +Env-declarative chat frontend over the LiteLLM gateway, on **esh-docker-vm** +(`10.0.50.45:3211`). Stood up 2026-08-21 as a **parallel bake-off** against +`lobe-chat` (`:3210`) β€” operator-approved candidate replacement. Nothing about +`lobe-chat` changes unless/until the operator retires it. + +- **URL:** http://10.0.50.45:3211 +- **Image:** `ghcr.io/open-webui/open-webui:main` (deployed v0.11.0) +- **Compose (server):** `/opt/docker/compose/open-webui/` +- **Admin:** `lkraven` / `vh@phasefinal.com` (first account = admin). Signup is + **disabled** in steady state; add users via the Admin UI. + +## Why this exists + +Lobe's config does not survive a fleet that changes daily: its model list is a +static env string that never auto-discovers (rotted both directions), and half +its config (TTS/model/voice) lives in each browser's local store, so +"configured" is a property of whoever's laptop. Open WebUI is **env-declarative** +(the deploy is the single source of config truth) and **auto-tracks** the gateway +roster with no pins. + +## The deal-sealer β€” `ENABLE_PERSISTENT_CONFIG=False` + +Default is `True`, under which env vars are read once on first boot, written to +the DB, then ignored forever (edit-env-and-bounce silently does nothing). We set +it **False**: every knob is re-read from env on each boot, and Admin-UI changes +do **not** survive a restart. Verified both directions at deploy (see gates). + +> ⚠️ **A var only reaches the container if `compose.yaml` references it** in its +> `environment:` block. Adding a line to `.env` alone does nothing β€” Compose uses +> `.env` for `${VAR}` substitution, not as an `env_file`. This bit us once: +> `ENABLE_API_KEYS` (see below) was set in `.env` but not wired in compose, so +> API-key minting stayed 403 until the compose var was added. + +## Acceptance gate results (verified on the box, 2026-08-21) + +| Gate | Result | Notes | +|---|---|---| +| **G1** declarative config works | βœ… PASS | env change takes on bounce (signup 200β†’403); UI/runtime change reverts on restart (first-admin runtime signup-disable + a runtime `ENABLE_API_KEYS=true` both reverted to env on restart). No persistent-config bug bit this deploy (no Redis; flags honored once named right). | +| **G2** model list auto-tracks LiteLLM, no pins | βœ… PASS | Picker = the 31 live gateway models 1:1, nothing hand-listed; a retired gateway model disappears with no OWUI edit. `arena-model` (+1) is OWUI's own eval-arena feature, not a pin. **Tradeoff:** the picker also shows non-chat seats (embeddings/rerankers/tts) β€” the flip side of "no hand-listing"; can't be pruned to chat-only without reintroducing exactly the pinning G2 forbids. | +| **G3** reconciliation API is real | βœ… PASS | `GET /api/v1/models/export` round-trips JSON; `POST /api/v1/models/sync` genuinely **reconciles** β€” create **and delete** to match desired state (verified: syncing set B removed set A). Sync requires full records (`user_id`/`created_at`/`updated_at`) β€” it round-trips the `export` shape. API keys enabled via `ENABLE_API_KEYS=True` (plural β€” see warning above). | +| **G5** task model pinned | βœ… PASS | `TASK_MODEL_EXTERNAL=summarizer` (gen seat @ temp 0). | +| **G4** TTS by env, second-browser zero-setup | ⏳ tts-dev | Env knobs set (direct at `http://10.100.79.3:8198/v1`, no LiteLLM); tts-dev runs the gate. | + +End-to-end sanity: a chat completion to `summarizer` through OWUI β†’ LiteLLM β†’ seat returns correctly. + +## Credential posture (G6) + +- **Fresh, capped LiteLLM key** `key_alias: open-webui-esh` β€” `all-proxy-models` + (reaches paid passthroughs per the operator's standing posture) **but capped: + `max_budget: $50`, `budget_duration: 1mo`** (unlike `lobe-chat-esh`, which is + uncapped β€” deliberately not inherited). A paid call spends real vendor credits + until the cap trips. +- **Signup off** after admin creation (no first-registrant LAN race). +- Secrets in the vault, never in git: + ``` + secret get esh-docker-vm/open-webui-litellm-key # -> OPENAI_API_KEY + secret get esh-docker-vm/open-webui-secret-key # -> WEBUI_SECRET_KEY + secret get esh-docker-vm/open-webui-admin # -> lkraven admin cred state + ``` + +## TTS (G4 β€” tts-dev owns) + +Routed **direct at the fleet dots gateway** `http://10.100.79.3:8198/v1`, not +through LiteLLM (:8198 is itself the swap-proof switchboard; LiteLLM would add a +hop + a wrong `audio/mpeg` stamp; direct needs no gateway key). Five knobs: +`AUDIO_TTS_ENGINE=openai`, `AUDIO_TTS_OPENAI_API_BASE_URL`, +`AUDIO_TTS_OPENAI_API_KEY` (placeholder β€” seat is LAN/WG-only, no auth), +`AUDIO_TTS_MODEL`, `AUDIO_TTS_VOICE` (model/voice are tts-dev's to tune). +Reversible to `ext-tts` via the base URL if audio spend/log visibility is wanted. + +## Deploy / operate + +```bash +# Canonical lives here; push to the host: +scripts/deploy-stack.sh esh-docker-vm open-webui --compose +# Real .env lives on the host (secrets from the vault), never committed. + +# On the host: +cd /opt/docker/compose/open-webui +$EDITOR .env # change any knob HERE (env is the source of truth) +docker compose up -d # bounce to apply β€” UI changes do NOT persist + +# RAG embeddings route through the gateway (qwen3-embedding) to avoid a +# boot-time HuggingFace download and stay fleet-consistent. +``` diff --git a/stacks/open-webui/compose.yaml b/stacks/open-webui/compose.yaml new file mode 100644 index 0000000..9447316 --- /dev/null +++ b/stacks/open-webui/compose.yaml @@ -0,0 +1,122 @@ +# open-webui β€” evaluation chat frontend over the LiteLLM gateway, on esh-docker-vm. +# +# WHY THIS EXISTS (bake-off against lobe-chat, 2026-08-21) +# Lobe's config model does not survive a fleet that changes daily: its model +# list is a static env string that never auto-discovers (it rotted in BOTH +# directions β€” missed `sec`/`sec-reasoning` going in, kept retired `char-rp` +# names that 400 on click), and half its config (TTS service/model/voice) +# lives in each browser's local store, so "configured" is a property of +# whoever's laptop rather than of the deploy. Open WebUI is env-declarative and +# auto-tracks the gateway roster. This stack stands up PARALLEL to lobe-chat +# (Lobe on :3210, this on :3211); nothing about Lobe changes until/unless the +# operator retires it. The June-2026 weight objection (1,825 MB vs 143 MB) is +# moot here: esh-docker-vm has ~138 GB free. +# +# THE DEAL-SEALER β€” ENABLE_PERSISTENT_CONFIG=False (see .env) +# Default is True, and under True env vars are read ONCE on first boot, +# written to the DB, then ignored forever β€” so edit-env-and-bounce silently +# does nothing. We set it False so the deploy is the single source of config +# truth: every knob below is re-read from env on each boot, and a change made +# in the Admin UI does NOT survive a restart. Verified both directions at +# deploy (G1). ⚠️ Open upstream bugs touch this flag (open-webui #20830 Redis +# cache β€” N/A, we use no Redis; #15505; #21839) β€” re-verify after image bumps. +# +# CREDENTIAL POSTURE (G6 β€” chosen, not inherited) +# Purpose-minted LiteLLM virtual key `key_alias: open-webui-esh`, scoped to the +# `all-proxy-models` access group (reaches EVERYTHING incl. paid passthroughs β€” +# GLM/Kimi/gen-frontier) per the operator's standing posture, BUT capped with a +# `max_budget` ceiling (unlike lobe-chat-esh which is uncapped). A paid call +# from this UI spends real vendor credits until the cap trips. Signup is turned +# OFF after the first/admin account is created, so the surface is not open- +# registration on the LAN. Secrets live in the vault, never in git: +# secret get esh-docker-vm/open-webui-litellm-key -> OPENAI_API_KEY +# secret get esh-docker-vm/open-webui-secret-key -> WEBUI_SECRET_KEY +# secret get esh-docker-vm/open-webui-admin -> lkraven admin cred state +# +# TTS (G4 β€” tts-dev owns verification; env knobs set here) +# Routed DIRECT at the fleet dots gateway http://10.100.79.3:8198/v1, NOT +# through LiteLLM. Rationale (agreed with tts-dev): :8198 is itself the swap- +# proof switchboard β€” engines swap behind it β€” so LiteLLM would add only a hop +# and a wrong `audio/mpeg` content-type stamp; direct needs no gateway key on +# the audio path. Reversible: flip AUDIO_TTS_OPENAI_API_BASE_URL to the gateway +# + `ext-tts` if audio spend/log visibility is ever wanted. The five AUDIO_TTS_* +# vars are tts-dev's to tune (model/voice); base URL + engine are settled. + +name: open-webui + +services: + open-webui: + image: ghcr.io/open-webui/open-webui:${OPENWEBUI_TAG:-main} + container_name: ${OPENWEBUI_CONTAINER:-open-webui} + restart: unless-stopped + ports: + - "${OPENWEBUI_PORT:-3211}:8080" + environment: + # --- The deal-sealer: deploy is the single source of config truth. --- + - ENABLE_PERSISTENT_CONFIG=${ENABLE_PERSISTENT_CONFIG:-False} + - ENV=${OPENWEBUI_ENV:-prod} + - WEBUI_NAME=${WEBUI_NAME:-PFI Open WebUI} + - WEBUI_URL=${WEBUI_URL:-http://10.0.50.45:3211} + # --- Auth / signup posture (G6). Signup is flipped False post-admin. --- + - WEBUI_AUTH=${WEBUI_AUTH:-True} + - WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY:?set in .env, from the vault} + - ENABLE_SIGNUP=${ENABLE_SIGNUP:-False} + - ENABLE_LOGIN_FORM=${ENABLE_LOGIN_FORM:-True} + - DEFAULT_USER_ROLE=${DEFAULT_USER_ROLE:-pending} + # NOTE: the env var is ENABLE_API_KEYS (plural) in Open WebUI v0.11.0 + # (config.py: os.getenv('ENABLE_API_KEYS','False')). The singular form is + # inert β€” a deploy that sets ENABLE_API_KEY silently leaves key-mint 403. + - ENABLE_API_KEYS=${ENABLE_API_KEYS:-True} + # --- Model source: the LiteLLM gateway, no pins (G2 auto-tracks). --- + - ENABLE_OPENAI_API=${ENABLE_OPENAI_API:-True} + - OPENAI_API_BASE_URL=${OPENAI_API_BASE_URL:-http://10.250.50.70:4000/v1} + - OPENAI_API_KEY=${OPENAI_API_KEY:?set in .env, from the vault} + # Ollama is banned fleet-wide; keep the client from probing localhost:11434. + - ENABLE_OLLAMA_API=${ENABLE_OLLAMA_API:-False} + # --- Background task model (G5): pin it or an unpinned default hammers a + # name the gateway does not serve and fills logs with 4xx. External + # (OpenAI-type) models use TASK_MODEL_EXTERNAL. summarizer = gen seat + # at temp 0, the right fit for auto-title/tag/query-rewrite. --- + - TASK_MODEL_EXTERNAL=${TASK_MODEL_EXTERNAL:-summarizer} + # --- RAG embeddings through the gateway (fleet-consistent; avoids a boot- + # time HuggingFace download of the default local sentence-transformer). --- + - RAG_EMBEDDING_ENGINE=${RAG_EMBEDDING_ENGINE:-openai} + - RAG_EMBEDDING_MODEL=${RAG_EMBEDDING_MODEL:-qwen3-embedding} + - RAG_OPENAI_API_BASE_URL=${RAG_OPENAI_API_BASE_URL:-http://10.250.50.70:4000/v1} + - RAG_OPENAI_API_KEY=${OPENAI_API_KEY:?set in .env, from the vault} + # --- TTS (G4): direct at the dots gateway, engine + base URL settled; + # model/voice are tts-dev's to tune. Seat is LAN/WG-only, no auth. --- + - AUDIO_TTS_ENGINE=${AUDIO_TTS_ENGINE:-openai} + - AUDIO_TTS_OPENAI_API_BASE_URL=${AUDIO_TTS_OPENAI_API_BASE_URL:-http://10.100.79.3:8198/v1} + - AUDIO_TTS_OPENAI_API_KEY=${AUDIO_TTS_OPENAI_API_KEY:-sk-no-auth-lan-seat} + - AUDIO_TTS_MODEL=${AUDIO_TTS_MODEL:-tts-1} + - AUDIO_TTS_VOICE=${AUDIO_TTS_VOICE:-nova} + # --- Telemetry off. --- + - ANONYMIZED_TELEMETRY=${ANONYMIZED_TELEMETRY:-False} + - DO_NOT_TRACK=${DO_NOT_TRACK:-True} + - SCARF_NO_ANALYTICS=${SCARF_NO_ANALYTICS:-True} + volumes: + - open-webui_data:/app/backend/data + healthcheck: + test: ["CMD-SHELL", "python3 -c \"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/health').status==200 else 1)\" || exit 1"] + interval: 30s + timeout: 10s + retries: 5 + start_period: 120s + networks: + - tnet + labels: + - homepage.group=Apps + - homepage.name=Open WebUI + - homepage.icon=mdi-chat-question + - homepage.description=Env-declarative chat frontend over the LiteLLM gateway (eval vs Lobe) + - homepage.href=http://10.0.50.45:${OPENWEBUI_PORT:-3211} + +volumes: + open-webui_data: + name: open-webui_data + +networks: + tnet: + name: traefik-net + external: true