vh
fb3bb521fe
An empty or non-numeric Max tokens field makes parseInt return NaN, and
JSON.stringify serialises NaN as null. The server reads null as 'no
max_tokens supplied' and substitutes its own default -- which is
indistinguishable from the UI ignoring the field, and is the most likely
explanation for a typed value appearing to have no effect. Falls back to
the same 4096 the input defaults to.
Ruled out on the way to this, all measured rather than assumed:
- LiteLLM caps nothing: max_tokens=None on both aliases, no max-token
keys in litellm_settings or general_settings.
- The gateway honours large values end-to-end: 5,346 completion tokens
returned at max_tokens=8192, finish=stop.
- The UI has ONE chat send path, no duplicate element ids, a standard
getElementById helper, and the request body is never mutated after
construction -- so the field is read live at send time.
Remaining client-side cause if it recurs is a stale cached page: nginx
serves this file with only Last-Modified/ETag and no Cache-Control, so an
already-open tab will not re-fetch. ETag changes on each deploy, so a
reload picks it up.
gateway-chat
Persistent static-serve of tools/gateway-chat.html — the zero-dependency web chat
for smoking models on the LiteLLM gateway (10.250.50.70:4000). It auto-discovers
every gateway model via /v1/models (the ↻ control — new models just appear), takes
system prompts, streams responses (renders reasoning_content), and supports image
upload for vision models (Qwopus, image-judge). It deliberately never sends a tools
field, sidestepping the vLLM empty-tools 400.
- Host: ana-docker (non-GPU)
- URL: http://10.250.50.70:8091
- Image:
nginx:alpine(tiny static server — no GPU, no DB) - Served file:
conf/index.html→ mounted read-only at/usr/share/nginx/html/index.html
The served file mirrors tools/gateway-chat.html
The canonical/editable source is the repo's tools/gateway-chat.html (also openable
file:// or via python3 -m http.server -d tools). conf/index.html here is the
deployed copy. After editing the tool, re-sync + redeploy:
cp tools/gateway-chat.html stacks/gateway-chat/conf/index.html
scripts/deploy-stack.sh ana-docker gateway-chat --conf
No restart needed — the file is bind-mounted, so nginx serves the new content on the next request. (Restart only if you want a forced reload.)
Deploy
scripts/deploy-stack.sh ana-docker gateway-chat # compose + conf
ssh ana-docker 'cd /opt/docker/compose/gateway-chat && docker compose up -d'
Set the gateway base URL + an API key in the page's sidebar (persists in localStorage),
then hit ↻ to load the model list.
⚠️ Single-file bind mount — a --conf deploy is NOT enough
compose.yaml bind-mounts one file:
/opt/docker/conf/gateway-chat/index.html -> /usr/share/nginx/html/index.html
A single-file bind mount binds the inode, not the path. deploy-stack.sh
uses rsync, which writes a new file and renames it over the old one — a new
inode — so the container keeps serving the old content indefinitely. The host
file and the container's view silently disagree, and nothing errors.
Observed 2026-08-16: host showed value="4096", container and HTTP still served
value="1024".
Always follow a conf deploy of this stack with a recreate:
scripts/deploy-stack.sh ana-docker gateway-chat --conf --yes
ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/gateway-chat && sudo docker compose up -d --force-recreate'
docker restart does not fix it — the stale inode is already bound. Verify
against what the container actually sees, never the host file:
docker exec gateway-chat grep -oE 'id="max"[^>]*value="[0-9]+"' /usr/share/nginx/html/index.html
Applies to any stack whose mount source is a FILE rather than a directory. Directory mounts do not have this problem.
Max tokens default
Raised 1024 → 4096 (2026-08-16). Every thinking seat on this gateway
(gen-reasoning, char-rp-reasoning) spends part of the completion budget on
CoT before emitting content, so a 1024 cap truncates mid-sentence with
finish_reason=length — which reads as the model being degenerate when it is
purely a client-side cap. Measured: gen-reasoning at 1024 → finish=length,
cut mid-word; at 4096 → clean stop at 839 words.