vh
b6552e0546
Operator reported the gen seat 'cutting off'. It is not the seat. The chat UI's max_tokens field defaults to 1024, and every thinking seat spends part of that budget on CoT before emitting content, so the completion truncates mid-sentence with finish_reason=length and reads as model degeneracy. Measured through the gateway: gen 1024 -> finish=stop, 716w (survives, but marginally) gen-reasoning 1024 -> finish=length, cut mid-word <-- the symptom gen-reasoning 4096 -> finish=stop, 839w Seat itself is clean: direct long-form generations return finish=stop with complete sentences and a max repeated 6-gram of 1 (no degeneration), and enable_thinking:false still holds on every non-thinking alias, so the AEON swap did not cause this. Also documents a trap that made the fix look like it had not applied: compose bind-mounts a single FILE, and a single-file bind mount binds the INODE. rsync writes-and-renames, producing a new inode, so the container kept serving the old content while the host file showed the new value -- silently, with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount; directory mounts are unaffected.
81 lines
3.2 KiB
Markdown
81 lines
3.2 KiB
Markdown
# gateway-chat
|
|
|
|
Persistent static-serve of **`tools/gateway-chat.html`** — the zero-dependency web chat
|
|
for **smoking models on the LiteLLM gateway** (`10.250.50.70:4000`). It auto-discovers
|
|
every gateway model via `/v1/models` (the ↻ control — new models just appear), takes
|
|
system prompts, streams responses (renders `reasoning_content`), and supports image
|
|
upload for vision models (Qwopus, image-judge). It deliberately never sends a `tools`
|
|
field, sidestepping the vLLM empty-`tools` 400.
|
|
|
|
- **Host:** ana-docker (non-GPU)
|
|
- **URL:** http://10.250.50.70:8091
|
|
- **Image:** `nginx:alpine` (tiny static server — no GPU, no DB)
|
|
- **Served file:** `conf/index.html` → mounted read-only at
|
|
`/usr/share/nginx/html/index.html`
|
|
|
|
## The served file mirrors `tools/gateway-chat.html`
|
|
|
|
The canonical/editable source is the repo's **`tools/gateway-chat.html`** (also openable
|
|
`file://` or via `python3 -m http.server -d tools`). `conf/index.html` here is the
|
|
deployed copy. After editing the tool, re-sync + redeploy:
|
|
|
|
```bash
|
|
cp tools/gateway-chat.html stacks/gateway-chat/conf/index.html
|
|
scripts/deploy-stack.sh ana-docker gateway-chat --conf
|
|
```
|
|
|
|
No restart needed — the file is bind-mounted, so nginx serves the new content on the next
|
|
request. (Restart only if you want a forced reload.)
|
|
|
|
## Deploy
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh ana-docker gateway-chat # compose + conf
|
|
ssh ana-docker 'cd /opt/docker/compose/gateway-chat && docker compose up -d'
|
|
```
|
|
|
|
Set the gateway base URL + an API key in the page's sidebar (persists in `localStorage`),
|
|
then hit ↻ to load the model list.
|
|
|
|
## ⚠️ Single-file bind mount — a `--conf` deploy is NOT enough
|
|
|
|
`compose.yaml` bind-mounts **one file**:
|
|
|
|
```
|
|
/opt/docker/conf/gateway-chat/index.html -> /usr/share/nginx/html/index.html
|
|
```
|
|
|
|
A single-file bind mount binds the **inode**, not the path. `deploy-stack.sh`
|
|
uses rsync, which writes a new file and renames it over the old one — a *new*
|
|
inode — so the container keeps serving the **old** content indefinitely. The host
|
|
file and the container's view silently disagree, and nothing errors.
|
|
|
|
Observed 2026-08-16: host showed `value="4096"`, container and HTTP still served
|
|
`value="1024"`.
|
|
|
|
**Always follow a conf deploy of this stack with a recreate:**
|
|
|
|
```bash
|
|
scripts/deploy-stack.sh ana-docker gateway-chat --conf --yes
|
|
ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/gateway-chat && sudo docker compose up -d --force-recreate'
|
|
```
|
|
|
|
`docker restart` does **not** fix it — the stale inode is already bound. Verify
|
|
against what the container actually sees, never the host file:
|
|
|
|
```bash
|
|
docker exec gateway-chat grep -oE 'id="max"[^>]*value="[0-9]+"' /usr/share/nginx/html/index.html
|
|
```
|
|
|
|
Applies to any stack whose mount source is a FILE rather than a directory.
|
|
Directory mounts do not have this problem.
|
|
|
|
## Max tokens default
|
|
|
|
Raised 1024 → 4096 (2026-08-16). Every thinking seat on this gateway
|
|
(`gen-reasoning`, `char-rp-reasoning`) spends part of the completion budget on
|
|
CoT before emitting content, so a 1024 cap truncates mid-sentence with
|
|
`finish_reason=length` — which reads as the *model* being degenerate when it is
|
|
purely a client-side cap. Measured: `gen-reasoning` at 1024 → `finish=length`,
|
|
cut mid-word; at 4096 → clean stop at 839 words.
|