vh
91f4cf22e1
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
think-leak — reproducers for the h300 unterminated-<think> defect
[2026-08-21] The Cold-Fusion Heretic-300 gen seat emits an unterminated
<think> block into content on any alias sampling at temperature > 0.
Operator-reported through Lobe as "sends CoT but never completes the turn".
What actually happens
With enable_thinking: false, the Qwen3.8 chat template appends a pre-closed
<think>\n\n</think>\n\n to the prompt (chat_template.jinja L165-166). The
model opens a fresh <think> anyway and never closes it. Because the prompt
already closed the block, vLLM's qwen3 reasoning parser is not in reasoning
state, so the tag is passed through as ordinary text: reasoning_content is
empty, reasoning_tokens is 0, and the whole reasoning-plus-answer blob lands in
content. Any client that renders <think>…</think> as a collapsible trace then
shows an endless thought bubble and no answer. The client is behaving correctly.
The trigger is TEMPERATURE, not presence_penalty
arms_de.py, n=12 per arm, same reproducer prompt:
| arm | leaks |
|---|---|
temp 0.7, presence_penalty 1.5 (current gen) |
4/12 |
| temp 0.7, presence_penalty 0.0 | 4/12 |
| temp 0.7, presence_penalty 0.5 | 3/12 |
| temp 0, presence_penalty 1.5 | 0/12 |
This falsifies the standing hypothesis (in the litellm config comment, and
the operator's own 2026-08-16 note) that presence_penalty: 1.5 is the first
dial to move. It is not this bug's cause.
It also explains the blast radius exactly — only the temp-0.7 aliases leak:
| alias | temp | leak (n=12) |
|---|---|---|
gen |
0.7 | 2 |
summarizer-large |
0.7 | 2 |
summarizer, classifier, image-judge, qwen-image-bench |
0 | 0 |
The fix
final_validate.py, n=30 across 4 prompt types plus a 3-turn conversation:
| config | leaks | empty content |
|---|---|---|
{enable_thinking: false} (current) |
8/30 | 0 |
{enable_thinking: true, reasoning_effort: low} |
0/30 | 1 |
Give the model a legitimately open <think> and it closes it properly, the
parser does its job, and content comes out clean. Costs ~+27% completion
tokens and leaves a ~3% empty-content residual.
Scripts
| script | what it does |
|---|---|
probe_gen.py <model> <max_tokens> <n> |
leak/finish/empty tally through the LiteLLM gateway |
blast.py |
leak rate across all seven seat aliases |
arms_de.py |
isolates temperature vs presence_penalty |
validate_fix.py |
thinking off vs low vs medium, n=12 |
final_validate.py |
the n=30 multi-prompt + multi-turn confirmation |
test_fix.py |
direct-to-vLLM A/B, bypasses the gateway |
Using these on any future seat
A "7/7 aliases return 200" smoke test cannot catch this. Trivial prompts
never invite reasoning, so they never sample the leaking token. Probe with a
reasoning-inviting prompt at n>=12 and grep raw content for <think>.