From b6552e0546df50dcb4bbbf100199fa66f0719f2b Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sun, 16 Aug 2026 15:10:27 -0700 Subject: [PATCH] fix(gateway-chat): raise max_tokens default 1024 -> 4096; document single-file bind-mount trap Operator reported the gen seat 'cutting off'. It is not the seat. The chat UI's max_tokens field defaults to 1024, and every thinking seat spends part of that budget on CoT before emitting content, so the completion truncates mid-sentence with finish_reason=length and reads as model degeneracy. Measured through the gateway: gen 1024 -> finish=stop, 716w (survives, but marginally) gen-reasoning 1024 -> finish=length, cut mid-word <-- the symptom gen-reasoning 4096 -> finish=stop, 839w Seat itself is clean: direct long-form generations return finish=stop with complete sentences and a max repeated 6-gram of 1 (no degeneration), and enable_thinking:false still holds on every non-thinking alias, so the AEON swap did not cause this. Also documents a trap that made the fix look like it had not applied: compose bind-mounts a single FILE, and a single-file bind mount binds the INODE. rsync writes-and-renames, producing a new inode, so the container kept serving the old content while the host file showed the new value -- silently, with no error. docker restart does NOT clear it; the container must be recreated. Verify against what the container sees, never the host file. Applies to any file-source mount; directory mounts are unaffected. --- stacks/gateway-chat/README.md | 42 +++++++++++++++++++++++++++++ stacks/gateway-chat/conf/index.html | 8 +++++- 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/stacks/gateway-chat/README.md b/stacks/gateway-chat/README.md index dcf0484..d2c631c 100644 --- a/stacks/gateway-chat/README.md +++ b/stacks/gateway-chat/README.md @@ -36,3 +36,45 @@ ssh ana-docker 'cd /opt/docker/compose/gateway-chat && docker compose up -d' Set the gateway base URL + an API key in the page's sidebar (persists in `localStorage`), then hit ↻ to load the model list. + +## ⚠️ Single-file bind mount — a `--conf` deploy is NOT enough + +`compose.yaml` bind-mounts **one file**: + +``` +/opt/docker/conf/gateway-chat/index.html -> /usr/share/nginx/html/index.html +``` + +A single-file bind mount binds the **inode**, not the path. `deploy-stack.sh` +uses rsync, which writes a new file and renames it over the old one — a *new* +inode — so the container keeps serving the **old** content indefinitely. The host +file and the container's view silently disagree, and nothing errors. + +Observed 2026-08-16: host showed `value="4096"`, container and HTTP still served +`value="1024"`. + +**Always follow a conf deploy of this stack with a recreate:** + +```bash +scripts/deploy-stack.sh ana-docker gateway-chat --conf --yes +ssh infra-ops@10.250.50.70 'cd /opt/docker/compose/gateway-chat && sudo docker compose up -d --force-recreate' +``` + +`docker restart` does **not** fix it — the stale inode is already bound. Verify +against what the container actually sees, never the host file: + +```bash +docker exec gateway-chat grep -oE 'id="max"[^>]*value="[0-9]+"' /usr/share/nginx/html/index.html +``` + +Applies to any stack whose mount source is a FILE rather than a directory. +Directory mounts do not have this problem. + +## Max tokens default + +Raised 1024 → 4096 (2026-08-16). Every thinking seat on this gateway +(`gen-reasoning`, `char-rp-reasoning`) spends part of the completion budget on +CoT before emitting content, so a 1024 cap truncates mid-sentence with +`finish_reason=length` — which reads as the *model* being degenerate when it is +purely a client-side cap. Measured: `gen-reasoning` at 1024 → `finish=length`, +cut mid-word; at 4096 → clean stop at 839 words. diff --git a/stacks/gateway-chat/conf/index.html b/stacks/gateway-chat/conf/index.html index e575e44..df7ae53 100644 --- a/stacks/gateway-chat/conf/index.html +++ b/stacks/gateway-chat/conf/index.html @@ -65,7 +65,13 @@
-
+ +