fix(gen-seat): diagnose the unterminated-<think> leak — model defect, temp-triggered
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# think-leak — reproducers for the h300 unterminated-`<think>` defect
|
||||
|
||||
`[2026-08-21]` The Cold-Fusion Heretic-300 gen seat emits an **unterminated
|
||||
`<think>` block into `content`** on any alias sampling at temperature > 0.
|
||||
Operator-reported through Lobe as "sends CoT but never completes the turn".
|
||||
|
||||
## What actually happens
|
||||
|
||||
With `enable_thinking: false`, the Qwen3.8 chat template appends a *pre-closed*
|
||||
`<think>\n\n</think>\n\n` to the prompt (`chat_template.jinja` L165-166). The
|
||||
model opens a **fresh** `<think>` anyway and never closes it. Because the prompt
|
||||
already closed the block, vLLM's `qwen3` reasoning parser is not in reasoning
|
||||
state, so the tag is passed through as ordinary text: `reasoning_content` is
|
||||
empty, `reasoning_tokens` is 0, and the whole reasoning-plus-answer blob lands in
|
||||
`content`. Any client that renders `<think>…</think>` as a collapsible trace then
|
||||
shows an endless thought bubble and no answer. The client is behaving correctly.
|
||||
|
||||
## The trigger is TEMPERATURE, not presence_penalty
|
||||
|
||||
`arms_de.py`, n=12 per arm, same reproducer prompt:
|
||||
|
||||
| arm | leaks |
|
||||
|---|---|
|
||||
| temp 0.7, presence_penalty 1.5 (current `gen`) | 4/12 |
|
||||
| temp 0.7, presence_penalty 0.0 | 4/12 |
|
||||
| temp 0.7, presence_penalty 0.5 | 3/12 |
|
||||
| **temp 0**, presence_penalty 1.5 | **0/12** |
|
||||
|
||||
This **falsifies** the standing hypothesis (in the litellm config comment, and
|
||||
the operator's own 2026-08-16 note) that `presence_penalty: 1.5` is the first
|
||||
dial to move. It is not this bug's cause.
|
||||
|
||||
It also explains the blast radius exactly — only the temp-0.7 aliases leak:
|
||||
|
||||
| alias | temp | leak (n=12) |
|
||||
|---|---|---|
|
||||
| `gen` | 0.7 | 2 |
|
||||
| `summarizer-large` | 0.7 | 2 |
|
||||
| `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` | 0 | 0 |
|
||||
|
||||
## The fix
|
||||
|
||||
`final_validate.py`, n=30 across 4 prompt types plus a 3-turn conversation:
|
||||
|
||||
| config | leaks | empty content |
|
||||
|---|---|---|
|
||||
| `{enable_thinking: false}` (current) | **8/30** | 0 |
|
||||
| `{enable_thinking: true, reasoning_effort: low}` | **0/30** | 1 |
|
||||
|
||||
Give the model a legitimately open `<think>` and it closes it properly, the
|
||||
parser does its job, and `content` comes out clean. Costs ~+27% completion
|
||||
tokens and leaves a ~3% empty-content residual.
|
||||
|
||||
## Scripts
|
||||
|
||||
| script | what it does |
|
||||
|---|---|
|
||||
| `probe_gen.py <model> <max_tokens> <n>` | leak/finish/empty tally through the LiteLLM gateway |
|
||||
| `blast.py` | leak rate across all seven seat aliases |
|
||||
| `arms_de.py` | isolates temperature vs presence_penalty |
|
||||
| `validate_fix.py` | thinking off vs low vs medium, n=12 |
|
||||
| `final_validate.py` | the n=30 multi-prompt + multi-turn confirmation |
|
||||
| `test_fix.py` | direct-to-vLLM A/B, bypasses the gateway |
|
||||
|
||||
## Using these on any future seat
|
||||
|
||||
**A "7/7 aliases return 200" smoke test cannot catch this.** Trivial prompts
|
||||
never invite reasoning, so they never sample the leaking token. Probe with a
|
||||
reasoning-inviting prompt at n>=12 and grep raw `content` for `<think>`.
|
||||
@@ -0,0 +1,21 @@
|
||||
import json, urllib.request, collections
|
||||
P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."
|
||||
def run(label, params, n=12):
|
||||
t=collections.Counter()
|
||||
for i in range(n):
|
||||
body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}],
|
||||
"max_tokens":2048,"chat_template_kwargs":{"enable_thinking":False},
|
||||
"top_k":20,"min_p":0.0,"repetition_penalty":1.0}
|
||||
body.update(params)
|
||||
req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; cont=c["message"].get("content") or ""
|
||||
t["n"]+=1
|
||||
if "<think>" in cont: t["LEAK"]+=1
|
||||
t["ctok"]+=d.get("usage",{}).get("completion_tokens") or 0
|
||||
print(f" {label:<52} n={t['n']} LEAK={t.get('LEAK',0)} avg_ctok={t['ctok']//t['n']}")
|
||||
run("A: temp0.7 top_p0.8 pp1.5 (CURRENT gen)", {"temperature":0.7,"top_p":0.8,"presence_penalty":1.5})
|
||||
run("D: temp0.7 top_p0.8 pp0.0 (drop presence_penalty)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.0})
|
||||
run("E: temp0.7 top_p0.8 pp0.5 (upstream low end)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.5})
|
||||
run("F: temp0 top_p0.8 pp1.5 (temp-0, like summarizer)", {"temperature":0,"top_p":0.8,"presence_penalty":1.5})
|
||||
@@ -0,0 +1,34 @@
|
||||
import json, urllib.request, collections
|
||||
KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
|
||||
URL="http://10.250.50.70:4000/v1/chat/completions"
|
||||
PROMPTS=[
|
||||
("reasoning","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."),
|
||||
("summarize","Summarise these in one sentence each:\n1. Fed holds rates, signals two cuts\n2. Port strike enters third week\n3. Study links microplastics to soil carbon loss"),
|
||||
("classify","Classify the sentiment of each line as POSITIVE, NEGATIVE or NEUTRAL:\n1. The deploy finally went clean.\n2. Third outage this week.\n3. Meeting moved to Thursday."),
|
||||
]
|
||||
def probe(model,n=4,maxtok=2048):
|
||||
t=collections.Counter(); ex=None
|
||||
for label,p in PROMPTS:
|
||||
for i in range(n):
|
||||
body={"model":model,"messages":[{"role":"user","content":p}],"max_tokens":maxtok}
|
||||
req=urllib.request.Request(URL,data=json.dumps(body).encode(),
|
||||
headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
|
||||
try:
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
except Exception as e:
|
||||
t["error"]+=1; continue
|
||||
c=d["choices"][0]; cont=c["message"].get("content") or ""
|
||||
t["n"]+=1
|
||||
if "<think>" in cont:
|
||||
t["LEAK"]+=1; t[f"leak_{label}"]+=1
|
||||
if ex is None: ex=(label,cont[:120])
|
||||
if not cont.strip(): t["empty"]+=1
|
||||
if c.get("finish_reason")!="stop": t[f"finish_{c.get('finish_reason')}"]+=1
|
||||
return t,ex
|
||||
|
||||
for m in ["gen","summarizer","summarizer-large","classifier","image-judge","qwen-image-bench","gen-reasoning"]:
|
||||
t,ex=probe(m)
|
||||
leak=t.get("LEAK",0); n=t.get("n",0)
|
||||
pct=(100.0*leak/n) if n else 0
|
||||
print(f" {m:<20} n={n:<3} LEAK={leak:<3} ({pct:4.1f}%) empty={t.get('empty',0)} " + " ".join(f"{k}={v}" for k,v in t.items() if k.startswith(('leak_','finish_'))))
|
||||
if ex: print(f" e.g. [{ex[0]}] {ex[1]!r}")
|
||||
@@ -0,0 +1,40 @@
|
||||
import json, urllib.request, collections
|
||||
BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}
|
||||
PROMPTS=[
|
||||
("math","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."),
|
||||
("tech","Compare NVMe raidz2 vs mirrored vdevs for a write-heavy Postgres workload. Which would you pick and why?"),
|
||||
("prose","Write a short scene: two engineers argue about shipping a known-flaky feature."),
|
||||
("openq","Why did the Bronze Age collapse happen? Give your best reasoning."),
|
||||
]
|
||||
CONVO=[{"role":"user","content":"Explain what a heat pump does."},
|
||||
{"role":"assistant","content":"A heat pump moves heat rather than generating it, using a refrigerant cycle to pull warmth from outside air into a house."},
|
||||
{"role":"user","content":"Why does its efficiency drop when it gets very cold out? Think it through."}]
|
||||
def run(label, ctk, n=6):
|
||||
t=collections.Counter()
|
||||
for lbl,p in PROMPTS:
|
||||
for i in range(n):
|
||||
body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":p}],
|
||||
"max_tokens":3072,"chat_template_kwargs":ctk}; body.update(BASE)
|
||||
req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; cont=c["message"].get("content") or ""
|
||||
t["n"]+=1
|
||||
if "<think>" in cont: t["LEAK"]+=1; t["leak_"+lbl]+=1
|
||||
if not cont.strip(): t["EMPTY"]+=1
|
||||
if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1
|
||||
# multi-turn
|
||||
for i in range(n):
|
||||
body={"model":"qwen3.8-27b-uncensored","messages":CONVO,"max_tokens":3072,"chat_template_kwargs":ctk}
|
||||
body.update(BASE)
|
||||
req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; cont=c["message"].get("content") or ""
|
||||
t["n"]+=1; t["mt"]+=1
|
||||
if "<think>" in cont: t["LEAK"]+=1; t["leak_multiturn"]+=1
|
||||
if not cont.strip(): t["EMPTY"]+=1
|
||||
extra=" ".join(f"{k}={v}" for k,v in sorted(t.items()) if k.startswith(("leak_","finish_")))
|
||||
print(f" {label:<40} n={t['n']:<3} LEAK={t.get('LEAK',0):<3} EMPTY={t.get('EMPTY',0)} {extra}")
|
||||
run("CURRENT: enable_thinking=false", {"enable_thinking":False})
|
||||
run("FIX B: thinking=true, effort=low", {"enable_thinking":True,"reasoning_effort":"low"})
|
||||
@@ -0,0 +1,39 @@
|
||||
import json, sys, urllib.request, collections
|
||||
KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
|
||||
URL="http://10.250.50.70:4000/v1/chat/completions"
|
||||
|
||||
PROMPTS = [
|
||||
"A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning.",
|
||||
"Compare the trade-offs of NVMe RAIDZ2 versus mirrored vdevs for a write-heavy database workload.",
|
||||
"Write a short scene: two engineers argue about whether to ship a known-flaky feature.",
|
||||
]
|
||||
|
||||
def call(model, prompt, max_tokens):
|
||||
body={"model":model,"messages":[{"role":"user","content":prompt}],"max_tokens":max_tokens}
|
||||
req=urllib.request.Request(URL,data=json.dumps(body).encode(),
|
||||
headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; m=c["message"]
|
||||
return {"finish":c.get("finish_reason"),
|
||||
"content":m.get("content") or "",
|
||||
"reasoning":m.get("reasoning_content") or "",
|
||||
"ctok":d.get("usage",{}).get("completion_tokens"),
|
||||
"rtok":(d.get("usage",{}).get("completion_tokens_details") or {}).get("reasoning_tokens")}
|
||||
|
||||
model=sys.argv[1]; maxtok=int(sys.argv[2]); n=int(sys.argv[3])
|
||||
print(f"=== {model} | max_tokens={maxtok} | n={n} per prompt ===")
|
||||
tally=collections.Counter()
|
||||
for pi,p in enumerate(PROMPTS):
|
||||
for i in range(n):
|
||||
try:
|
||||
r=call(model,p,maxtok)
|
||||
except Exception as e:
|
||||
print(f" p{pi} #{i}: ERROR {e}"); tally["error"]+=1; continue
|
||||
empty = len(r["content"].strip())==0
|
||||
has_think = "<think>" in r["content"] or "<think>" in r["reasoning"]
|
||||
flag = " <<< EMPTY CONTENT" if empty else ""
|
||||
tally[r["finish"]]+=1
|
||||
if empty: tally["empty_content"]+=1
|
||||
if r["reasoning"]: tally["had_reasoning"]+=1
|
||||
print(f" p{pi} #{i}: finish={r['finish']:<8} ctok={r['ctok']:<5} content={len(r['content']):<5} reasoning={len(r['reasoning']):<6} rtok={r['rtok']} think_tag={has_think}{flag}")
|
||||
print(" TALLY:", dict(tally))
|
||||
@@ -0,0 +1,34 @@
|
||||
import json, urllib.request, collections, sys
|
||||
KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
|
||||
P="A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning."
|
||||
|
||||
def raw(model, extra, n=6, maxtok=2048):
|
||||
tally=collections.Counter()
|
||||
for i in range(n):
|
||||
body={"model":model,"messages":[{"role":"user","content":P}],"max_tokens":maxtok}
|
||||
body.update(extra)
|
||||
req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; m=c["message"]
|
||||
cont=m.get("content") or ""; reas=m.get("reasoning_content") or ""
|
||||
leak = "<think>" in cont
|
||||
tally["n"]+=1
|
||||
if leak: tally["LEAK_think_in_content"]+=1
|
||||
if not cont.strip(): tally["empty_content"]+=1
|
||||
if reas: tally["reasoning_captured"]+=1
|
||||
tally[f"finish_{c.get('finish_reason')}"]+=1
|
||||
print(f" #{i}: finish={c.get('finish_reason'):<7} content={len(cont):<5} reasoning={len(reas):<6} leak={leak}")
|
||||
return tally
|
||||
|
||||
print("=== ARM A: current gen config (enable_thinking=false), direct to vLLM ===")
|
||||
a=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":False},
|
||||
"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,
|
||||
"top_k":20,"min_p":0.0,"repetition_penalty":1.0})
|
||||
print(" TALLY:", dict(a)); print()
|
||||
|
||||
print("=== ARM B: enable_thinking=TRUE + reasoning_effort=low (proposed fix) ===")
|
||||
b=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":True,"reasoning_effort":"low"},
|
||||
"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,
|
||||
"top_k":20,"min_p":0.0,"repetition_penalty":1.0})
|
||||
print(" TALLY:", dict(b))
|
||||
@@ -0,0 +1,23 @@
|
||||
import json, urllib.request, collections
|
||||
P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."
|
||||
BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}
|
||||
def run(label, ctk, n=12):
|
||||
t=collections.Counter()
|
||||
for i in range(n):
|
||||
body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}],
|
||||
"max_tokens":2048,"chat_template_kwargs":ctk}; body.update(BASE)
|
||||
req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
|
||||
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
||||
c=d["choices"][0]; cont=c["message"].get("content") or ""; reas=c["message"].get("reasoning_content") or ""
|
||||
t["n"]+=1
|
||||
if "<think>" in cont: t["LEAK"]+=1
|
||||
if not cont.strip(): t["EMPTY"]+=1
|
||||
if reas: t["reasoning_captured"]+=1
|
||||
t["ctok_total"]+=d.get("usage",{}).get("completion_tokens") or 0
|
||||
if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1
|
||||
n_=t["n"]
|
||||
print(f" {label:<44} n={n_} LEAK={t.get('LEAK',0)} EMPTY={t.get('EMPTY',0)} reas_captured={t.get('reasoning_captured',0)} avg_ctok={t['ctok_total']//n_}")
|
||||
run("A: enable_thinking=false (CURRENT gen)", {"enable_thinking":False})
|
||||
run("B: enable_thinking=true + effort=low", {"enable_thinking":True,"reasoning_effort":"low"})
|
||||
run("C: enable_thinking=true + effort=medium", {"enable_thinking":True,"reasoning_effort":"medium"})
|
||||
Reference in New Issue
Block a user