fix(gen-seat): diagnose the unterminated-<think> leak — model defect, temp-triggered

Operator reported the new Heretic-300 gen seat "sends CoT but never completes
the turn" through Lobe. Diagnosed; not yet fixed (the fix changes gen's
semantics, so it is the operator's call).

The Qwen3.8 chat template appends a pre-closed <think>\n\n</think>\n\n when
enable_thinking is false. The h300 model opens a fresh <think> anyway and never
closes it. Because the prompt already closed the block, vLLM's qwen3 reasoning
parser is not in reasoning state, so the tag passes through as ordinary text --
reasoning_content empty, reasoning_tokens 0, and the whole reasoning-plus-answer
blob lands in content. Lobe then correctly treats the unterminated tag as
still-thinking and renders no answer. The client and the serving stack are both
behaving correctly; the model is not.

The trigger is TEMPERATURE, not presence_penalty (n=12 per arm):

  temp 0.7, pp 1.5  (current gen)   4/12
  temp 0.7, pp 0.0                  4/12
  temp 0.7, pp 0.5                  3/12
  temp 0,   pp 1.5                  0/12

That falsifies the standing hypothesis, recorded in the litellm config comment
and in the operator's own 2026-08-16 note, that presence_penalty 1.5 is the
first dial to move. It is not this bug's cause.

It also explains the blast radius: only the two temp-0.7 aliases leak, `gen`
and `summarizer-large`. summarizer, classifier, image-judge and qwen-image-bench
all run at temp 0 and are clean, so nevermore's summarizer path is unaffected.

Candidate fix, validated n=30 over 4 prompt types plus a 3-turn conversation:
chat_template_kwargs {enable_thinking: true, reasoning_effort: low} takes 8/30
leaks to 0/30, at ~+27% completion tokens and a ~3% empty-content residual.

The tell appears in eval_coldfusion_h300.json and in none of the aeon, heresy,
mixed or w4a16 evals, so it is new with this build -- but L35 was never evaled,
so this does not separate a Cold-Fusion base trait from a Heretic-300
abliteration artifact.

Reproducers and the full method land in bench/think-leak/. Note in particular
that the 7/7 alias smoke test run at cutover structurally could not catch this:
trivial prompts never invite reasoning, so they never sample the leaking token.

persistent-memory.md

@@ -122,6 +122,14 @@ _As of 2026-08-20 23:30 — **the Heretic-300 session** (see the 🔴 entry abov
 
 - **🟢 HOMEPAGE — cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). ⚠ **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own — do not chase it.** ⚠ CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. → `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md`
 
+- **🔴 GEN SEAT DEFECT 2026-08-21 — the h300 build emits an UNTERMINATED `<think>` into `content`, ~27% of the time, on any temp>0 alias. Operator-reported via Lobe ("sends CoT, never completes the turn").**
+  - **Mechanism.** With `enable_thinking:false` the chat template appends a **pre-closed** `<think>\n\n</think>\n\n` to the prompt (jinja L165-166). The h300 model **opens a fresh `<think>` anyway and never closes it** — verified raw: `has </think>: False`, `finish_reason: stop`, reasoning *and* answer in one `content` blob starting `<think>Ok, let's figure this out:`. vLLM's `qwen3` reasoning parser can't catch it: the prompt already closed the block, so the parser isn't in reasoning state and the tag is just text (`reasoning_content` empty, `reasoning_tokens: 0`). **The client is blameless** — Lobe correctly treats an unterminated `<think>` as still-thinking, so it renders an endless thought bubble and never shows the answer.
+  - **★ It is a SAMPLING event, and the trigger is TEMPERATURE — not presence_penalty.** n=12 per arm on the reproducer: `pp 1.5` → 4 leaks, `pp 0.0` → 4, `pp 0.5` → 3 (all the same), **`temperature 0` → 0**. ⚠️ **This FALSIFIES the standing "presence_penalty 1.5 is the first dial to move" hypothesis** recorded in the litellm config comment and by the operator 2026-08-16 — it is not this bug's cause. Leave that dial alone for this symptom.
+  - **Blast radius = exactly the two temp-0.7 aliases.** `gen` and `summarizer-large` leak (~17-27%); `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` are all **temp=0 and clean at 0/12** — so **nevermore's `summarizer` path is NOT affected**. `gen-reasoning` doesn't leak (its think block is legitimately open) but shows the *other* symptom, empty `content`, at ~1/12.
+  - **New with this build, probably.** The `<think>`/"figure this out" tell appears in `bench/eval_coldfusion_h300.json` (x2) and in **none** of `eval_aeon`, `eval_heresy`, `eval_mixed`, `eval_w4a16`. ⚠ **L35 was never evaled**, so this does NOT separate "a Cold-Fusion base trait" from "a Heretic-300 abliteration artifact" — both are Cold-Fusion. Abliteration optimizes refusal count under a KL cap and has **no notion of format compliance**, so degrading the pre-closed-think-block convention is a plausible cost of the 8/100 result.
+  - **FIX, validated n=30 over 4 prompt types + a 3-turn conversation:** `chat_template_kwargs: {enable_thinking: true, reasoning_effort: low}` on `gen` → **0/30 leaks** (current config: **8/30**, worst on prose 4/6). Give the model a legitimately open `<think>` and it closes it properly, the parser does its job, `content` comes out clean. Cost ~+27% completion tokens (257 vs 202 avg) and a residual **1/30 empty-content**. Semantic change: `gen` stops being a non-thinking deployment — **operator's call, not applied.**
+  - **★ PROCESS LESSON: the 7/7 alias smoke test structurally CANNOT catch this.** Trivial prompts ("Reply with exactly: OK-gen") never invite reasoning, so they never sample the leaking token. Same shape as the compose file's own warning that single-turn probes missed the xhigh budget bug. **Probe with a reasoning-inviting prompt at n≥12, and grep the raw `content` for `<think>` — never just check HTTP 200.** The tell was sitting in my own `eval_coldfusion_h300.json` output the night of the cutover and I read past it.
+
 - **🟢 GEN SEAT — LIVE = COLD-FUSION HERETIC-300 (cut over 2026-08-20 23:05).** `GEN_MODEL=/tank/aimodels/qwen38-27b-coldfusion-h300-nvfp4-mixed` on `gen-seat`/`vllm-gen`, ana-ml2 GPU0 `:8015`, served-name unchanged (`qwen3.8-27b-uncensored` / `-thinking`) so all 7 LiteLLM aliases route without a gateway edit. **Verified end to end:** healthy in 5.5 min; KV **401,550 tok / 1.53×** (baseline 403k/1.54× — within noise); **7/7 aliases green** through LiteLLM; **VISION INTACT** (correctly enumerated colour/form/position of 3 shapes — the surface that had never been exercised after abliteration → MTP-dropping export → graft → quant); **MTP acceptance 59.7% median @ 118.37 tok/s** (`bench/mtp_coldfusion_h300.json`) — statistically identical to L35's **59.1% @ 118.71** on the same instrument, so **the roadmap's "~47% for a pristine graft" prediction was WRONG — a pristine graft accepts as well as an in-band one.** ⚠ A single long-prose sample read 47.5%; the 8-run spread is 49.0–65.4%, so **one sample cannot characterize acceptance** — always use `quickbench.py`. Deterministic quality gens all correct (heat-pump, primes=77, 14:20→17:05, prose); abliteration survival **4/4 compliance**. **⏳ PPL NOT MEASURED** — `eval_quality.py` aborts with "prompt_logprobs look uniform" because the seat runs `--speculative-config`; the documented workaround is a spec-decode-free probe seat (`bench/serve_probe.sh`, :8017), and **there is no VRAM for one** (GPU0 5.2 GB free, GPU1 2.4 GB free). Comparison target = heresy's **6.910 mean / 5.625 median**. **ROLLBACK (one line):** `sudo cp /opt/docker/compose/gen-seat/.env.bak-pre-h300-20260820 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen` → back to `qwen38-27b-coldfusion-L35-nvfp4-mixed`. **Do NOT delete** `-L35-nvfp4-mixed` or `qwen38-27b-heresy-nvfp4-mixed`. ⚠ **The self-harm guardrail is GONE on this build** (operator's own next work item). Also normalized the quant dir from root:0600 to `llmuser:llmuser` 0664 to match every other model dir. **Confirmed the right weights are mounted on TWO discriminating views** — mtime and a 64 MB head-hash both match h300 and differ from L35; `config.json` sha256 is **identical** across both builds and therefore useless as a discriminator (it carries no weight-specific content — don't reach for it again).
 
 - **⚪ PRIOR GEN SEAT — `absolute-heresy` 2026-08-17 (validated, promoted; superseded by L35 then h300 on 2026-08-20).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` — **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 — 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** ⚠ speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) — read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). ⚠ **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover — the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` — **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`.

services/gen-seat-mixed-quant/bench/think-leak/README.md

@@ -0,0 +1,69 @@
+# think-leak — reproducers for the h300 unterminated-`<think>` defect
+
+`[2026-08-21]` The Cold-Fusion Heretic-300 gen seat emits an **unterminated
+`<think>` block into `content`** on any alias sampling at temperature > 0.
+Operator-reported through Lobe as "sends CoT but never completes the turn".
+
+## What actually happens
+
+With `enable_thinking: false`, the Qwen3.8 chat template appends a *pre-closed*
+`<think>\n\n</think>\n\n` to the prompt (`chat_template.jinja` L165-166). The
+model opens a **fresh** `<think>` anyway and never closes it. Because the prompt
+already closed the block, vLLM's `qwen3` reasoning parser is not in reasoning
+state, so the tag is passed through as ordinary text: `reasoning_content` is
+empty, `reasoning_tokens` is 0, and the whole reasoning-plus-answer blob lands in
+`content`. Any client that renders `<think>…</think>` as a collapsible trace then
+shows an endless thought bubble and no answer. The client is behaving correctly.
+
+## The trigger is TEMPERATURE, not presence_penalty
+
+`arms_de.py`, n=12 per arm, same reproducer prompt:
+
+| arm | leaks |
+|---|---|
+| temp 0.7, presence_penalty 1.5 (current `gen`) | 4/12 |
+| temp 0.7, presence_penalty 0.0 | 4/12 |
+| temp 0.7, presence_penalty 0.5 | 3/12 |
+| **temp 0**, presence_penalty 1.5 | **0/12** |
+
+This **falsifies** the standing hypothesis (in the litellm config comment, and
+the operator's own 2026-08-16 note) that `presence_penalty: 1.5` is the first
+dial to move. It is not this bug's cause.
+
+It also explains the blast radius exactly — only the temp-0.7 aliases leak:
+
+| alias | temp | leak (n=12) |
+|---|---|---|
+| `gen` | 0.7 | 2 |
+| `summarizer-large` | 0.7 | 2 |
+| `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` | 0 | 0 |
+
+## The fix
+
+`final_validate.py`, n=30 across 4 prompt types plus a 3-turn conversation:
+
+| config | leaks | empty content |
+|---|---|---|
+| `{enable_thinking: false}` (current) | **8/30** | 0 |
+| `{enable_thinking: true, reasoning_effort: low}` | **0/30** | 1 |
+
+Give the model a legitimately open `<think>` and it closes it properly, the
+parser does its job, and `content` comes out clean. Costs ~+27% completion
+tokens and leaves a ~3% empty-content residual.
+
+## Scripts
+
+| script | what it does |
+|---|---|
+| `probe_gen.py <model> <max_tokens> <n>` | leak/finish/empty tally through the LiteLLM gateway |
+| `blast.py` | leak rate across all seven seat aliases |
+| `arms_de.py` | isolates temperature vs presence_penalty |
+| `validate_fix.py` | thinking off vs low vs medium, n=12 |
+| `final_validate.py` | the n=30 multi-prompt + multi-turn confirmation |
+| `test_fix.py` | direct-to-vLLM A/B, bypasses the gateway |
+
+## Using these on any future seat
+
+**A "7/7 aliases return 200" smoke test cannot catch this.** Trivial prompts
+never invite reasoning, so they never sample the leaking token. Probe with a
+reasoning-inviting prompt at n>=12 and grep raw `content` for `<think>`.

services/gen-seat-mixed-quant/bench/think-leak/arms_de.py

@@ -0,0 +1,21 @@
+import json, urllib.request, collections
+P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."
+def run(label, params, n=12):
+    t=collections.Counter()
+    for i in range(n):
+        body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}],
+              "max_tokens":2048,"chat_template_kwargs":{"enable_thinking":False},
+              "top_k":20,"min_p":0.0,"repetition_penalty":1.0}
+        body.update(params)
+        req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
+            data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+        d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+        c=d["choices"][0]; cont=c["message"].get("content") or ""
+        t["n"]+=1
+        if "<think>" in cont: t["LEAK"]+=1
+        t["ctok"]+=d.get("usage",{}).get("completion_tokens") or 0
+    print(f"  {label:<52} n={t['n']} LEAK={t.get('LEAK',0)} avg_ctok={t['ctok']//t['n']}")
+run("A: temp0.7 top_p0.8 pp1.5   (CURRENT gen)", {"temperature":0.7,"top_p":0.8,"presence_penalty":1.5})
+run("D: temp0.7 top_p0.8 pp0.0   (drop presence_penalty)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.0})
+run("E: temp0.7 top_p0.8 pp0.5   (upstream low end)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.5})
+run("F: temp0   top_p0.8 pp1.5   (temp-0, like summarizer)", {"temperature":0,"top_p":0.8,"presence_penalty":1.5})

services/gen-seat-mixed-quant/bench/think-leak/blast.py

@@ -0,0 +1,34 @@
+import json, urllib.request, collections
+KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
+URL="http://10.250.50.70:4000/v1/chat/completions"
+PROMPTS=[
+ ("reasoning","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."),
+ ("summarize","Summarise these in one sentence each:\n1. Fed holds rates, signals two cuts\n2. Port strike enters third week\n3. Study links microplastics to soil carbon loss"),
+ ("classify","Classify the sentiment of each line as POSITIVE, NEGATIVE or NEUTRAL:\n1. The deploy finally went clean.\n2. Third outage this week.\n3. Meeting moved to Thursday."),
+]
+def probe(model,n=4,maxtok=2048):
+    t=collections.Counter(); ex=None
+    for label,p in PROMPTS:
+        for i in range(n):
+            body={"model":model,"messages":[{"role":"user","content":p}],"max_tokens":maxtok}
+            req=urllib.request.Request(URL,data=json.dumps(body).encode(),
+                headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
+            try:
+                d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+            except Exception as e:
+                t["error"]+=1; continue
+            c=d["choices"][0]; cont=c["message"].get("content") or ""
+            t["n"]+=1
+            if "<think>" in cont:
+                t["LEAK"]+=1; t[f"leak_{label}"]+=1
+                if ex is None: ex=(label,cont[:120])
+            if not cont.strip(): t["empty"]+=1
+            if c.get("finish_reason")!="stop": t[f"finish_{c.get('finish_reason')}"]+=1
+    return t,ex
+
+for m in ["gen","summarizer","summarizer-large","classifier","image-judge","qwen-image-bench","gen-reasoning"]:
+    t,ex=probe(m)
+    leak=t.get("LEAK",0); n=t.get("n",0)
+    pct=(100.0*leak/n) if n else 0
+    print(f"  {m:<20} n={n:<3} LEAK={leak:<3} ({pct:4.1f}%)  empty={t.get('empty',0)}  " + " ".join(f"{k}={v}" for k,v in t.items() if k.startswith(('leak_','finish_'))))
+    if ex: print(f"      e.g. [{ex[0]}] {ex[1]!r}")

services/gen-seat-mixed-quant/bench/think-leak/final_validate.py

@@ -0,0 +1,40 @@
+import json, urllib.request, collections
+BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}
+PROMPTS=[
+ ("math","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."),
+ ("tech","Compare NVMe raidz2 vs mirrored vdevs for a write-heavy Postgres workload. Which would you pick and why?"),
+ ("prose","Write a short scene: two engineers argue about shipping a known-flaky feature."),
+ ("openq","Why did the Bronze Age collapse happen? Give your best reasoning."),
+]
+CONVO=[{"role":"user","content":"Explain what a heat pump does."},
+       {"role":"assistant","content":"A heat pump moves heat rather than generating it, using a refrigerant cycle to pull warmth from outside air into a house."},
+       {"role":"user","content":"Why does its efficiency drop when it gets very cold out? Think it through."}]
+def run(label, ctk, n=6):
+    t=collections.Counter()
+    for lbl,p in PROMPTS:
+        for i in range(n):
+            body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":p}],
+                  "max_tokens":3072,"chat_template_kwargs":ctk}; body.update(BASE)
+            req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
+                data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+            d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+            c=d["choices"][0]; cont=c["message"].get("content") or ""
+            t["n"]+=1
+            if "<think>" in cont: t["LEAK"]+=1; t["leak_"+lbl]+=1
+            if not cont.strip(): t["EMPTY"]+=1
+            if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1
+    # multi-turn
+    for i in range(n):
+        body={"model":"qwen3.8-27b-uncensored","messages":CONVO,"max_tokens":3072,"chat_template_kwargs":ctk}
+        body.update(BASE)
+        req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
+            data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+        d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+        c=d["choices"][0]; cont=c["message"].get("content") or ""
+        t["n"]+=1; t["mt"]+=1
+        if "<think>" in cont: t["LEAK"]+=1; t["leak_multiturn"]+=1
+        if not cont.strip(): t["EMPTY"]+=1
+    extra=" ".join(f"{k}={v}" for k,v in sorted(t.items()) if k.startswith(("leak_","finish_")))
+    print(f"  {label:<40} n={t['n']:<3} LEAK={t.get('LEAK',0):<3} EMPTY={t.get('EMPTY',0)}  {extra}")
+run("CURRENT: enable_thinking=false", {"enable_thinking":False})
+run("FIX B:   thinking=true, effort=low", {"enable_thinking":True,"reasoning_effort":"low"})

services/gen-seat-mixed-quant/bench/think-leak/probe_gen.py

@@ -0,0 +1,39 @@
+import json, sys, urllib.request, collections
+KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
+URL="http://10.250.50.70:4000/v1/chat/completions"
+
+PROMPTS = [
+ "A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning.",
+ "Compare the trade-offs of NVMe RAIDZ2 versus mirrored vdevs for a write-heavy database workload.",
+ "Write a short scene: two engineers argue about whether to ship a known-flaky feature.",
+]
+
+def call(model, prompt, max_tokens):
+    body={"model":model,"messages":[{"role":"user","content":prompt}],"max_tokens":max_tokens}
+    req=urllib.request.Request(URL,data=json.dumps(body).encode(),
+        headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"})
+    d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+    c=d["choices"][0]; m=c["message"]
+    return {"finish":c.get("finish_reason"),
+            "content":m.get("content") or "",
+            "reasoning":m.get("reasoning_content") or "",
+            "ctok":d.get("usage",{}).get("completion_tokens"),
+            "rtok":(d.get("usage",{}).get("completion_tokens_details") or {}).get("reasoning_tokens")}
+
+model=sys.argv[1]; maxtok=int(sys.argv[2]); n=int(sys.argv[3])
+print(f"=== {model} | max_tokens={maxtok} | n={n} per prompt ===")
+tally=collections.Counter()
+for pi,p in enumerate(PROMPTS):
+    for i in range(n):
+        try:
+            r=call(model,p,maxtok)
+        except Exception as e:
+            print(f"  p{pi} #{i}: ERROR {e}"); tally["error"]+=1; continue
+        empty = len(r["content"].strip())==0
+        has_think = "<think>" in r["content"] or "<think>" in r["reasoning"]
+        flag = "  <<< EMPTY CONTENT" if empty else ""
+        tally[r["finish"]]+=1
+        if empty: tally["empty_content"]+=1
+        if r["reasoning"]: tally["had_reasoning"]+=1
+        print(f"  p{pi} #{i}: finish={r['finish']:<8} ctok={r['ctok']:<5} content={len(r['content']):<5} reasoning={len(r['reasoning']):<6} rtok={r['rtok']} think_tag={has_think}{flag}")
+print("  TALLY:", dict(tally))

services/gen-seat-mixed-quant/bench/think-leak/test_fix.py

@@ -0,0 +1,34 @@
+import json, urllib.request, collections, sys
+KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip()
+P="A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning."
+
+def raw(model, extra, n=6, maxtok=2048):
+    tally=collections.Counter()
+    for i in range(n):
+        body={"model":model,"messages":[{"role":"user","content":P}],"max_tokens":maxtok}
+        body.update(extra)
+        req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
+            data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+        d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+        c=d["choices"][0]; m=c["message"]
+        cont=m.get("content") or ""; reas=m.get("reasoning_content") or ""
+        leak = "<think>" in cont
+        tally["n"]+=1
+        if leak: tally["LEAK_think_in_content"]+=1
+        if not cont.strip(): tally["empty_content"]+=1
+        if reas: tally["reasoning_captured"]+=1
+        tally[f"finish_{c.get('finish_reason')}"]+=1
+        print(f"    #{i}: finish={c.get('finish_reason'):<7} content={len(cont):<5} reasoning={len(reas):<6} leak={leak}")
+    return tally
+
+print("=== ARM A: current gen config (enable_thinking=false), direct to vLLM ===")
+a=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":False},
+      "temperature":0.7,"top_p":0.8,"presence_penalty":1.5,
+      "top_k":20,"min_p":0.0,"repetition_penalty":1.0})
+print("  TALLY:", dict(a)); print()
+
+print("=== ARM B: enable_thinking=TRUE + reasoning_effort=low (proposed fix) ===")
+b=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":True,"reasoning_effort":"low"},
+      "temperature":0.7,"top_p":0.8,"presence_penalty":1.5,
+      "top_k":20,"min_p":0.0,"repetition_penalty":1.0})
+print("  TALLY:", dict(b))

services/gen-seat-mixed-quant/bench/think-leak/validate_fix.py

@@ -0,0 +1,23 @@
+import json, urllib.request, collections
+P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."
+BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}
+def run(label, ctk, n=12):
+    t=collections.Counter()
+    for i in range(n):
+        body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}],
+              "max_tokens":2048,"chat_template_kwargs":ctk}; body.update(BASE)
+        req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions",
+            data=json.dumps(body).encode(),headers={"Content-Type":"application/json"})
+        d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False)
+        c=d["choices"][0]; cont=c["message"].get("content") or ""; reas=c["message"].get("reasoning_content") or ""
+        t["n"]+=1
+        if "<think>" in cont: t["LEAK"]+=1
+        if not cont.strip(): t["EMPTY"]+=1
+        if reas: t["reasoning_captured"]+=1
+        t["ctok_total"]+=d.get("usage",{}).get("completion_tokens") or 0
+        if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1
+    n_=t["n"]
+    print(f"  {label:<44} n={n_} LEAK={t.get('LEAK',0)} EMPTY={t.get('EMPTY',0)} reas_captured={t.get('reasoning_captured',0)} avg_ctok={t['ctok_total']//n_}")
+run("A: enable_thinking=false  (CURRENT gen)", {"enable_thinking":False})
+run("B: enable_thinking=true + effort=low", {"enable_thinking":True,"reasoning_effort":"low"})
+run("C: enable_thinking=true + effort=medium", {"enable_thinking":True,"reasoning_effort":"medium"})