fix(gen-seat): diagnose the unterminated-<think> leak — model defect, temp-triggered
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# think-leak — reproducers for the h300 unterminated-`<think>` defect
|
||||
|
||||
`[2026-08-21]` The Cold-Fusion Heretic-300 gen seat emits an **unterminated
|
||||
`<think>` block into `content`** on any alias sampling at temperature > 0.
|
||||
Operator-reported through Lobe as "sends CoT but never completes the turn".
|
||||
|
||||
## What actually happens
|
||||
|
||||
With `enable_thinking: false`, the Qwen3.8 chat template appends a *pre-closed*
|
||||
`<think>\n\n</think>\n\n` to the prompt (`chat_template.jinja` L165-166). The
|
||||
model opens a **fresh** `<think>` anyway and never closes it. Because the prompt
|
||||
already closed the block, vLLM's `qwen3` reasoning parser is not in reasoning
|
||||
state, so the tag is passed through as ordinary text: `reasoning_content` is
|
||||
empty, `reasoning_tokens` is 0, and the whole reasoning-plus-answer blob lands in
|
||||
`content`. Any client that renders `<think>…</think>` as a collapsible trace then
|
||||
shows an endless thought bubble and no answer. The client is behaving correctly.
|
||||
|
||||
## The trigger is TEMPERATURE, not presence_penalty
|
||||
|
||||
`arms_de.py`, n=12 per arm, same reproducer prompt:
|
||||
|
||||
| arm | leaks |
|
||||
|---|---|
|
||||
| temp 0.7, presence_penalty 1.5 (current `gen`) | 4/12 |
|
||||
| temp 0.7, presence_penalty 0.0 | 4/12 |
|
||||
| temp 0.7, presence_penalty 0.5 | 3/12 |
|
||||
| **temp 0**, presence_penalty 1.5 | **0/12** |
|
||||
|
||||
This **falsifies** the standing hypothesis (in the litellm config comment, and
|
||||
the operator's own 2026-08-16 note) that `presence_penalty: 1.5` is the first
|
||||
dial to move. It is not this bug's cause.
|
||||
|
||||
It also explains the blast radius exactly — only the temp-0.7 aliases leak:
|
||||
|
||||
| alias | temp | leak (n=12) |
|
||||
|---|---|---|
|
||||
| `gen` | 0.7 | 2 |
|
||||
| `summarizer-large` | 0.7 | 2 |
|
||||
| `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` | 0 | 0 |
|
||||
|
||||
## The fix
|
||||
|
||||
`final_validate.py`, n=30 across 4 prompt types plus a 3-turn conversation:
|
||||
|
||||
| config | leaks | empty content |
|
||||
|---|---|---|
|
||||
| `{enable_thinking: false}` (current) | **8/30** | 0 |
|
||||
| `{enable_thinking: true, reasoning_effort: low}` | **0/30** | 1 |
|
||||
|
||||
Give the model a legitimately open `<think>` and it closes it properly, the
|
||||
parser does its job, and `content` comes out clean. Costs ~+27% completion
|
||||
tokens and leaves a ~3% empty-content residual.
|
||||
|
||||
## Scripts
|
||||
|
||||
| script | what it does |
|
||||
|---|---|
|
||||
| `probe_gen.py <model> <max_tokens> <n>` | leak/finish/empty tally through the LiteLLM gateway |
|
||||
| `blast.py` | leak rate across all seven seat aliases |
|
||||
| `arms_de.py` | isolates temperature vs presence_penalty |
|
||||
| `validate_fix.py` | thinking off vs low vs medium, n=12 |
|
||||
| `final_validate.py` | the n=30 multi-prompt + multi-turn confirmation |
|
||||
| `test_fix.py` | direct-to-vLLM A/B, bypasses the gateway |
|
||||
|
||||
## Using these on any future seat
|
||||
|
||||
**A "7/7 aliases return 200" smoke test cannot catch this.** Trivial prompts
|
||||
never invite reasoning, so they never sample the leaking token. Probe with a
|
||||
reasoning-inviting prompt at n>=12 and grep raw `content` for `<think>`.
|
||||
Reference in New Issue
Block a user