vh
91f4cf22e1
Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).
The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.
The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):
temp 0.7, pp 1.5 (current gen) 4/12
temp 0.7, pp 0.0 4/12
temp 0.7, pp 0.5 3/12
temp 0, pp 1.5 0/12
That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.
It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.
Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.
The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.
Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.
40 lines
2.1 KiB
Python
40 lines
2.1 KiB
Python
import json, sys, urllib.request, collections
|
|
KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
|
|
URL="http://10.250.50.70:4000/v1/chat/completions"
|
|
|
|
PROMPTS = [
|
|
"A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning.",
|
|
"Compare the trade-offs of NVMe RAIDZ2 versus mirrored vdevs for a write-heavy database workload.",
|
|
"Write a short scene: two engineers argue about whether to ship a known-flaky feature.",
|
|
]
|
|
|
|
def call(model, prompt, max_tokens):
|
|
body={"model":model,"messages":[{"role":"user","content":prompt}],"max_tokens":max_tokens}
|
|
req=urllib.request.Request(URL,data=json.dumps(body).encode(),
|
|
headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
|
|
d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
|
|
c=d["choices"][0]; m=c["message"]
|
|
return {"finish":c.get("finish_reason"),
|
|
"content":m.get("content") or "",
|
|
"reasoning":m.get("reasoning_content") or "",
|
|
"ctok":d.get("usage",{}).get("completion_tokens"),
|
|
"rtok":(d.get("usage",{}).get("completion_tokens_details") or {}).get("reasoning_tokens")}
|
|
|
|
model=sys.argv[1]; maxtok=int(sys.argv[2]); n=int(sys.argv[3])
|
|
print(f"=== {model} | max_tokens={maxtok} | n={n} per prompt ===")
|
|
tally=collections.Counter()
|
|
for pi,p in enumerate(PROMPTS):
|
|
for i in range(n):
|
|
try:
|
|
r=call(model,p,maxtok)
|
|
except Exception as e:
|
|
print(f" p{pi} #{i}: ERROR {e}"); tally["error"]+=1; continue
|
|
empty = len(r["content"].strip())==0
|
|
has_think = "<think>" in r["content"] or "<think>" in r["reasoning"]
|
|
flag = " <<< EMPTY CONTENT" if empty else ""
|
|
tally[r["finish"]]+=1
|
|
if empty: tally["empty_content"]+=1
|
|
if r["reasoning"]: tally["had_reasoning"]+=1
|
|
print(f" p{pi} #{i}: finish={r['finish']:<8} ctok={r['ctok']:<5} content={len(r['content']):<5} reasoning={len(r['reasoning']):<6} rtok={r['rtok']} think_tag={has_think}{flag}")
|
|
print(" TALLY:", dict(tally))
|