vh
b6552e0546
Operator reported the gen seat 'cutting off'. It is not the seat. The chat UI's max_tokens field defaults to 1024, and every thinking seat spends part of that budget on CoT before emitting content, so the completion truncates mid-sentence with finish_reason=length and reads as model degeneracy. Measured through the gateway: gen 1024 -> finish=stop, 716w (survives, but marginally) gen-reasoning 1024 -> finish=length, cut mid-word <-- the symptom gen-reasoning 4096 -> finish=stop, 839w Seat itself is clean: direct long-form generations return finish=stop with complete sentences and a max repeated 6-gram of 1 (no degeneration), and enable_thinking:false still holds on every non-thinking alias, so the AEON swap did not cause this. Also documents a trap that made the fix look like it had not applied: compose bind-mounts a single FILE, and a single-file bind mount binds the INODE. rsync writes-and-renames, producing a new inode, so the container kept serving the old content while the host file showed the new value -- silently, with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount; directory mounts are unaffected.
3.2 KiB
gateway-chat
Persistent static-serve of tools/gateway-chat.html — the zero-dependency web chat
for smoking models on the LiteLLM gateway (10.250.50.70:4000). It auto-discovers
every gateway model via /v1/models (the ↻ control — new models just appear), takes
system prompts, streams responses (renders reasoning_content), and supports image
upload for vision models (Qwopus, image-judge). It deliberately never sends a tools
field, sidestepping the vLLM empty-tools 400.
- Host: ana-docker (non-GPU)
- URL: http://10.250.50.70:8091
- Image:
nginx:alpine(tiny static server — no GPU, no DB) - Served file:
conf/index.html→ mounted read-only at/usr/share/nginx/html/index.html
The served file mirrors tools/gateway-chat.html
The canonical/editable source is the repo's tools/gateway-chat.html (also openable
file:// or via python3 -m http.server -d tools). conf/index.html here is the
deployed copy. After editing the tool, re-sync + redeploy:
cp tools/gateway-chat.html stacks/gateway-chat/conf/index.html
scripts/deploy-stack.sh ana-docker gateway-chat --conf
No restart needed — the file is bind-mounted, so nginx serves the new content on the next request. (Restart only if you want a forced reload.)
Deploy
scripts/deploy-stack.sh ana-docker gateway-chat # compose + conf
ssh ana-docker 'cd /opt/docker/compose/gateway-chat && docker compose up -d'
Set the gateway base URL + an API key in the page's sidebar (persists in localStorage),
then hit ↻ to load the model list.
⚠️ Single-file bind mount — a --conf deploy is NOT enough
compose.yaml bind-mounts one file:
/opt/docker/conf/gateway-chat/index.html -> /usr/share/nginx/html/index.html
A single-file bind mount binds the inode, not the path. deploy-stack.sh
uses rsync, which writes a new file and renames it over the old one — a new
inode — so the container keeps serving the old content indefinitely. The host
file and the container's view silently disagree, and nothing errors.
Observed 2026-08-16: host showed value="4096", container and HTTP still served
value="1024".
Always follow a conf deploy of this stack with a recreate:
scripts/deploy-stack.sh ana-docker gateway-chat --conf --yes
ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/gateway-chat && sudo docker compose up -d --force-recreate'
docker restart does not fix it — the stale inode is already bound. Verify
against what the container actually sees, never the host file:
docker exec gateway-chat grep -oE 'id="max"[^>]*value="[0-9]+"' /usr/share/nginx/html/index.html
Applies to any stack whose mount source is a FILE rather than a directory. Directory mounts do not have this problem.
Max tokens default
Raised 1024 → 4096 (2026-08-16). Every thinking seat on this gateway
(gen-reasoning, char-rp-reasoning) spends part of the completion budget on
CoT before emitting content, so a 1024 cap truncates mid-sentence with
finish_reason=length — which reads as the model being degenerate when it is
purely a client-side cap. Measured: gen-reasoning at 1024 → finish=length,
cut mid-word; at 4096 → clean stop at 839 words.