diff --git a/persistent-memory.md b/persistent-memory.md index a6ecfe9..b4dcb43 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -122,6 +122,14 @@ _As of 2026-08-20 23:30 ā€” **the Heretic-300 session** (see the šŸ”“ entry abov - **šŸŸ¢ HOMEPAGE ā€” cleaned + themed (Australis Skyfall).** Three real defects fixed (UltraSeedbox on all tabs, Uptime Kuma double-rendered, fiction column counts), AI tab reordered by clickability, then themed from the operator's Skyfall handoff bundle with a background generated by **Arbo** (`t2i-ui-background`, job `13f0891f4e42`). āš  **After any recreate the tab bar/wallpaper/i18n vanish for up to ~an hour and then return on their own ā€” do not chase it.** āš  CSS is served per-request: a theme change needs a **reload**, not a recreate, and candidate CSS can be injected live via Playwright for seconds-long iteration. ā†’ `persistent-memory.d/2026-08-19-homepage-skyfall-theme.md` +- **šŸ”“ GEN SEAT DEFECT 2026-08-21 ā€” the h300 build emits an UNTERMINATED `` into `content`, ~27% of the time, on any temp>0 alias. Operator-reported via Lobe ("sends CoT, never completes the turn").** + - **Mechanism.** With `enable_thinking:false` the chat template appends a **pre-closed** `\n\n\n\n` to the prompt (jinja L165-166). The h300 model **opens a fresh `` anyway and never closes it** ā€” verified raw: `has : False`, `finish_reason: stop`, reasoning *and* answer in one `content` blob starting `Ok, let's figure this out:`. vLLM's `qwen3` reasoning parser can't catch it: the prompt already closed the block, so the parser isn't in reasoning state and the tag is just text (`reasoning_content` empty, `reasoning_tokens: 0`). **The client is blameless** ā€” Lobe correctly treats an unterminated `` as still-thinking, so it renders an endless thought bubble and never shows the answer. + - **ā˜… It is a SAMPLING event, and the trigger is TEMPERATURE ā€” not presence_penalty.** n=12 per arm on the reproducer: `pp 1.5` ā†’ 4 leaks, `pp 0.0` ā†’ 4, `pp 0.5` ā†’ 3 (all the same), **`temperature 0` ā†’ 0**. āš ļø **This FALSIFIES the standing "presence_penalty 1.5 is the first dial to move" hypothesis** recorded in the litellm config comment and by the operator 2026-08-16 ā€” it is not this bug's cause. Leave that dial alone for this symptom. + - **Blast radius = exactly the two temp-0.7 aliases.** `gen` and `summarizer-large` leak (~17-27%); `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` are all **temp=0 and clean at 0/12** ā€” so **nevermore's `summarizer` path is NOT affected**. `gen-reasoning` doesn't leak (its think block is legitimately open) but shows the *other* symptom, empty `content`, at ~1/12. + - **New with this build, probably.** The ``/"figure this out" tell appears in `bench/eval_coldfusion_h300.json` (x2) and in **none** of `eval_aeon`, `eval_heresy`, `eval_mixed`, `eval_w4a16`. āš  **L35 was never evaled**, so this does NOT separate "a Cold-Fusion base trait" from "a Heretic-300 abliteration artifact" ā€” both are Cold-Fusion. Abliteration optimizes refusal count under a KL cap and has **no notion of format compliance**, so degrading the pre-closed-think-block convention is a plausible cost of the 8/100 result. + - **FIX, validated n=30 over 4 prompt types + a 3-turn conversation:** `chat_template_kwargs: {enable_thinking: true, reasoning_effort: low}` on `gen` ā†’ **0/30 leaks** (current config: **8/30**, worst on prose 4/6). Give the model a legitimately open `` and it closes it properly, the parser does its job, `content` comes out clean. Cost ~+27% completion tokens (257 vs 202 avg) and a residual **1/30 empty-content**. Semantic change: `gen` stops being a non-thinking deployment ā€” **operator's call, not applied.** + - **ā˜… PROCESS LESSON: the 7/7 alias smoke test structurally CANNOT catch this.** Trivial prompts ("Reply with exactly: OK-gen") never invite reasoning, so they never sample the leaking token. Same shape as the compose file's own warning that single-turn probes missed the xhigh budget bug. **Probe with a reasoning-inviting prompt at nā‰„12, and grep the raw `content` for `` ā€” never just check HTTP 200.** The tell was sitting in my own `eval_coldfusion_h300.json` output the night of the cutover and I read past it. + - **šŸŸ¢ GEN SEAT ā€” LIVE = COLD-FUSION HERETIC-300 (cut over 2026-08-20 23:05).** `GEN_MODEL=/tank/aimodels/qwen38-27b-coldfusion-h300-nvfp4-mixed` on `gen-seat`/`vllm-gen`, ana-ml2 GPU0 `:8015`, served-name unchanged (`qwen3.8-27b-uncensored` / `-thinking`) so all 7 LiteLLM aliases route without a gateway edit. **Verified end to end:** healthy in 5.5 min; KV **401,550 tok / 1.53Ɨ** (baseline 403k/1.54Ɨ ā€” within noise); **7/7 aliases green** through LiteLLM; **VISION INTACT** (correctly enumerated colour/form/position of 3 shapes ā€” the surface that had never been exercised after abliteration ā†’ MTP-dropping export ā†’ graft ā†’ quant); **MTP acceptance 59.7% median @ 118.37 tok/s** (`bench/mtp_coldfusion_h300.json`) ā€” statistically identical to L35's **59.1% @ 118.71** on the same instrument, so **the roadmap's "~47% for a pristine graft" prediction was WRONG ā€” a pristine graft accepts as well as an in-band one.** āš  A single long-prose sample read 47.5%; the 8-run spread is 49.0ā€“65.4%, so **one sample cannot characterize acceptance** ā€” always use `quickbench.py`. Deterministic quality gens all correct (heat-pump, primes=77, 14:20ā†’17:05, prose); abliteration survival **4/4 compliance**. **ā³ PPL NOT MEASURED** ā€” `eval_quality.py` aborts with "prompt_logprobs look uniform" because the seat runs `--speculative-config`; the documented workaround is a spec-decode-free probe seat (`bench/serve_probe.sh`, :8017), and **there is no VRAM for one** (GPU0 5.2 GB free, GPU1 2.4 GB free). Comparison target = heresy's **6.910 mean / 5.625 median**. **ROLLBACK (one line):** `sudo cp /opt/docker/compose/gen-seat/.env.bak-pre-h300-20260820 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen` ā†’ back to `qwen38-27b-coldfusion-L35-nvfp4-mixed`. **Do NOT delete** `-L35-nvfp4-mixed` or `qwen38-27b-heresy-nvfp4-mixed`. āš  **The self-harm guardrail is GONE on this build** (operator's own next work item). Also normalized the quant dir from root:0600 to `llmuser:llmuser` 0664 to match every other model dir. **Confirmed the right weights are mounted on TWO discriminating views** ā€” mtime and a 64 MB head-hash both match h300 and differ from L35; `config.json` sha256 is **identical** across both builds and therefore useless as a discriminator (it carries no weight-specific content ā€” don't reach for it again). - **āšŖ PRIOR GEN SEAT ā€” `absolute-heresy` 2026-08-17 (validated, promoted; superseded by L35 then h300 on 2026-08-20).** Live gen = `/tank/aimodels/qwen38-27b-heresy-nvfp4-mixed` ā€” **MuXodious/Qwen3.8-27B-absolute-heresy** (Heretic v1.4.0 + **SOMPOA**, trial T377, pin `c2374593`) put through our own mixed NVFP4+FP8 recipe. Chosen because it beats the incumbent on **both** axes at once: author refusals 2/101 vs 12/100, first-token KL 0.0759 vs 0.1191. **Gate (probe :8017, pinned nightly, seat-matched flags): MTP 47.2% (inc. 48.2%), decode 103.5 tok/s (96.4), prefill 6618/5403 @6.7k/27k (6334/5085), PPL 6.910 (7.059 ā€” 2.1% BETTER), surface 6/6, abliteration 4/4, and 0/55 refusals on our battery-instruct arm with ZERO EMPTY (no catatonia).** āš  speed deltas are **image-confounded** (probe on the pinned nightly, incumbent numbers from an earlier image) ā€” read as "not worse", not a clean win. All 7 LiteLLM aliases verified end-to-end; GPU0 at 91.3/97.9 GB with meromero healthy (more headroom than the old build's 96.8). āš  **RC1, 2 days old, ~348 downloads.** **Operator-confirmed "working very well" in real use 2026-08-17**, same evening as the cutover ā€” the signal the synthetic gates structurally cannot give (multi-turn degeneration is stochastic; four synthetic tests once validated three non-fixes). Not yet the 60k-token bar the prior seat cleared, so **keep watching and do NOT delete the rollback weights yet**. **ROLLBACK:** `sudo cp /opt/docker/compose/gen-seat/.env.bak-heresy-20260817 /opt/docker/compose/gen-seat/.env && cd /opt/docker/compose/gen-seat && sudo docker compose up -d vllm-gen`; incumbent weights UNTOUCHED at `qwen38-27b-uncensored-nvfp4-mixed` ā€” **do NOT delete** until this holds. Runbook `services/gen-seat-mixed-quant/RUNBOOK-heresy-swap.md`. diff --git a/services/gen-seat-mixed-quant/bench/think-leak/README.md b/services/gen-seat-mixed-quant/bench/think-leak/README.md new file mode 100644 index 0000000..2b50de1 --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/README.md @@ -0,0 +1,69 @@ +# think-leak ā€” reproducers for the h300 unterminated-`` defect + +`[2026-08-21]` The Cold-Fusion Heretic-300 gen seat emits an **unterminated +`` block into `content`** on any alias sampling at temperature > 0. +Operator-reported through Lobe as "sends CoT but never completes the turn". + +## What actually happens + +With `enable_thinking: false`, the Qwen3.8 chat template appends a *pre-closed* +`\n\n\n\n` to the prompt (`chat_template.jinja` L165-166). The +model opens a **fresh** `` anyway and never closes it. Because the prompt +already closed the block, vLLM's `qwen3` reasoning parser is not in reasoning +state, so the tag is passed through as ordinary text: `reasoning_content` is +empty, `reasoning_tokens` is 0, and the whole reasoning-plus-answer blob lands in +`content`. Any client that renders `ā€¦` as a collapsible trace then +shows an endless thought bubble and no answer. The client is behaving correctly. + +## The trigger is TEMPERATURE, not presence_penalty + +`arms_de.py`, n=12 per arm, same reproducer prompt: + +| arm | leaks | +|---|---| +| temp 0.7, presence_penalty 1.5 (current `gen`) | 4/12 | +| temp 0.7, presence_penalty 0.0 | 4/12 | +| temp 0.7, presence_penalty 0.5 | 3/12 | +| **temp 0**, presence_penalty 1.5 | **0/12** | + +This **falsifies** the standing hypothesis (in the litellm config comment, and +the operator's own 2026-08-16 note) that `presence_penalty: 1.5` is the first +dial to move. It is not this bug's cause. + +It also explains the blast radius exactly ā€” only the temp-0.7 aliases leak: + +| alias | temp | leak (n=12) | +|---|---|---| +| `gen` | 0.7 | 2 | +| `summarizer-large` | 0.7 | 2 | +| `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` | 0 | 0 | + +## The fix + +`final_validate.py`, n=30 across 4 prompt types plus a 3-turn conversation: + +| config | leaks | empty content | +|---|---|---| +| `{enable_thinking: false}` (current) | **8/30** | 0 | +| `{enable_thinking: true, reasoning_effort: low}` | **0/30** | 1 | + +Give the model a legitimately open `` and it closes it properly, the +parser does its job, and `content` comes out clean. Costs ~+27% completion +tokens and leaves a ~3% empty-content residual. + +## Scripts + +| script | what it does | +|---|---| +| `probe_gen.py ` | leak/finish/empty tally through the LiteLLM gateway | +| `blast.py` | leak rate across all seven seat aliases | +| `arms_de.py` | isolates temperature vs presence_penalty | +| `validate_fix.py` | thinking off vs low vs medium, n=12 | +| `final_validate.py` | the n=30 multi-prompt + multi-turn confirmation | +| `test_fix.py` | direct-to-vLLM A/B, bypasses the gateway | + +## Using these on any future seat + +**A "7/7 aliases return 200" smoke test cannot catch this.** Trivial prompts +never invite reasoning, so they never sample the leaking token. Probe with a +reasoning-inviting prompt at n>=12 and grep raw `content` for ``. diff --git a/services/gen-seat-mixed-quant/bench/think-leak/arms_de.py b/services/gen-seat-mixed-quant/bench/think-leak/arms_de.py new file mode 100644 index 0000000..eb5dbe8 --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/arms_de.py @@ -0,0 +1,21 @@ +import json, urllib.request, collections +P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain." +def run(label, params, n=12): + t=collections.Counter() + for i in range(n): + body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}], + "max_tokens":2048,"chat_template_kwargs":{"enable_thinking":False}, + "top_k":20,"min_p":0.0,"repetition_penalty":1.0} + body.update(params) + req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions", + data=json.dumps(body).encode(),headers={"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; cont=c["message"].get("content") or "" + t["n"]+=1 + if "" in cont: t["LEAK"]+=1 + t["ctok"]+=d.get("usage",{}).get("completion_tokens") or 0 + print(f" {label:<52} n={t['n']} LEAK={t.get('LEAK',0)} avg_ctok={t['ctok']//t['n']}") +run("A: temp0.7 top_p0.8 pp1.5 (CURRENT gen)", {"temperature":0.7,"top_p":0.8,"presence_penalty":1.5}) +run("D: temp0.7 top_p0.8 pp0.0 (drop presence_penalty)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.0}) +run("E: temp0.7 top_p0.8 pp0.5 (upstream low end)", {"temperature":0.7,"top_p":0.8,"presence_penalty":0.5}) +run("F: temp0 top_p0.8 pp1.5 (temp-0, like summarizer)", {"temperature":0,"top_p":0.8,"presence_penalty":1.5}) diff --git a/services/gen-seat-mixed-quant/bench/think-leak/blast.py b/services/gen-seat-mixed-quant/bench/think-leak/blast.py new file mode 100644 index 0000000..b9fb24d --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/blast.py @@ -0,0 +1,34 @@ +import json, urllib.request, collections +KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip() +URL="http://10.250.50.70:4000/v1/chat/completions" +PROMPTS=[ + ("reasoning","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."), + ("summarize","Summarise these in one sentence each:\n1. Fed holds rates, signals two cuts\n2. Port strike enters third week\n3. Study links microplastics to soil carbon loss"), + ("classify","Classify the sentiment of each line as POSITIVE, NEGATIVE or NEUTRAL:\n1. The deploy finally went clean.\n2. Third outage this week.\n3. Meeting moved to Thursday."), +] +def probe(model,n=4,maxtok=2048): + t=collections.Counter(); ex=None + for label,p in PROMPTS: + for i in range(n): + body={"model":model,"messages":[{"role":"user","content":p}],"max_tokens":maxtok} + req=urllib.request.Request(URL,data=json.dumps(body).encode(), + headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"}) + try: + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + except Exception as e: + t["error"]+=1; continue + c=d["choices"][0]; cont=c["message"].get("content") or "" + t["n"]+=1 + if "" in cont: + t["LEAK"]+=1; t[f"leak_{label}"]+=1 + if ex is None: ex=(label,cont[:120]) + if not cont.strip(): t["empty"]+=1 + if c.get("finish_reason")!="stop": t[f"finish_{c.get('finish_reason')}"]+=1 + return t,ex + +for m in ["gen","summarizer","summarizer-large","classifier","image-judge","qwen-image-bench","gen-reasoning"]: + t,ex=probe(m) + leak=t.get("LEAK",0); n=t.get("n",0) + pct=(100.0*leak/n) if n else 0 + print(f" {m:<20} n={n:<3} LEAK={leak:<3} ({pct:4.1f}%) empty={t.get('empty',0)} " + " ".join(f"{k}={v}" for k,v in t.items() if k.startswith(('leak_','finish_')))) + if ex: print(f" e.g. [{ex[0]}] {ex[1]!r}") diff --git a/services/gen-seat-mixed-quant/bench/think-leak/final_validate.py b/services/gen-seat-mixed-quant/bench/think-leak/final_validate.py new file mode 100644 index 0000000..48f0363 --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/final_validate.py @@ -0,0 +1,40 @@ +import json, urllib.request, collections +BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0} +PROMPTS=[ + ("math","A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain."), + ("tech","Compare NVMe raidz2 vs mirrored vdevs for a write-heavy Postgres workload. Which would you pick and why?"), + ("prose","Write a short scene: two engineers argue about shipping a known-flaky feature."), + ("openq","Why did the Bronze Age collapse happen? Give your best reasoning."), +] +CONVO=[{"role":"user","content":"Explain what a heat pump does."}, + {"role":"assistant","content":"A heat pump moves heat rather than generating it, using a refrigerant cycle to pull warmth from outside air into a house."}, + {"role":"user","content":"Why does its efficiency drop when it gets very cold out? Think it through."}] +def run(label, ctk, n=6): + t=collections.Counter() + for lbl,p in PROMPTS: + for i in range(n): + body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":p}], + "max_tokens":3072,"chat_template_kwargs":ctk}; body.update(BASE) + req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions", + data=json.dumps(body).encode(),headers={"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; cont=c["message"].get("content") or "" + t["n"]+=1 + if "" in cont: t["LEAK"]+=1; t["leak_"+lbl]+=1 + if not cont.strip(): t["EMPTY"]+=1 + if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1 + # multi-turn + for i in range(n): + body={"model":"qwen3.8-27b-uncensored","messages":CONVO,"max_tokens":3072,"chat_template_kwargs":ctk} + body.update(BASE) + req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions", + data=json.dumps(body).encode(),headers={"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; cont=c["message"].get("content") or "" + t["n"]+=1; t["mt"]+=1 + if "" in cont: t["LEAK"]+=1; t["leak_multiturn"]+=1 + if not cont.strip(): t["EMPTY"]+=1 + extra=" ".join(f"{k}={v}" for k,v in sorted(t.items()) if k.startswith(("leak_","finish_"))) + print(f" {label:<40} n={t['n']:<3} LEAK={t.get('LEAK',0):<3} EMPTY={t.get('EMPTY',0)} {extra}") +run("CURRENT: enable_thinking=false", {"enable_thinking":False}) +run("FIX B: thinking=true, effort=low", {"enable_thinking":True,"reasoning_effort":"low"}) diff --git a/services/gen-seat-mixed-quant/bench/think-leak/probe_gen.py b/services/gen-seat-mixed-quant/bench/think-leak/probe_gen.py new file mode 100644 index 0000000..23b0d18 --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/probe_gen.py @@ -0,0 +1,39 @@ +import json, sys, urllib.request, collections +KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip() +URL="http://10.250.50.70:4000/v1/chat/completions" + +PROMPTS = [ + "A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning.", + "Compare the trade-offs of NVMe RAIDZ2 versus mirrored vdevs for a write-heavy database workload.", + "Write a short scene: two engineers argue about whether to ship a known-flaky feature.", +] + +def call(model, prompt, max_tokens): + body={"model":model,"messages":[{"role":"user","content":prompt}],"max_tokens":max_tokens} + req=urllib.request.Request(URL,data=json.dumps(body).encode(), + headers={"Authorization":"Bearer "+KEY,"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; m=c["message"] + return {"finish":c.get("finish_reason"), + "content":m.get("content") or "", + "reasoning":m.get("reasoning_content") or "", + "ctok":d.get("usage",{}).get("completion_tokens"), + "rtok":(d.get("usage",{}).get("completion_tokens_details") or {}).get("reasoning_tokens")} + +model=sys.argv[1]; maxtok=int(sys.argv[2]); n=int(sys.argv[3]) +print(f"=== {model} | max_tokens={maxtok} | n={n} per prompt ===") +tally=collections.Counter() +for pi,p in enumerate(PROMPTS): + for i in range(n): + try: + r=call(model,p,maxtok) + except Exception as e: + print(f" p{pi} #{i}: ERROR {e}"); tally["error"]+=1; continue + empty = len(r["content"].strip())==0 + has_think = "" in r["content"] or "" in r["reasoning"] + flag = " <<< EMPTY CONTENT" if empty else "" + tally[r["finish"]]+=1 + if empty: tally["empty_content"]+=1 + if r["reasoning"]: tally["had_reasoning"]+=1 + print(f" p{pi} #{i}: finish={r['finish']:<8} ctok={r['ctok']:<5} content={len(r['content']):<5} reasoning={len(r['reasoning']):<6} rtok={r['rtok']} think_tag={has_think}{flag}") +print(" TALLY:", dict(tally)) diff --git a/services/gen-seat-mixed-quant/bench/think-leak/test_fix.py b/services/gen-seat-mixed-quant/bench/think-leak/test_fix.py new file mode 100644 index 0000000..339025a --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/test_fix.py @@ -0,0 +1,34 @@ +import json, urllib.request, collections, sys +KEY=open('/home/lkraven/.config/litellm/infra-ops-key').read().strip() +P="A farmer has 17 sheep. All but 9 run away. He then buys twice as many as he has left, and sells 4. How many does he have? Explain your reasoning." + +def raw(model, extra, n=6, maxtok=2048): + tally=collections.Counter() + for i in range(n): + body={"model":model,"messages":[{"role":"user","content":P}],"max_tokens":maxtok} + body.update(extra) + req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions", + data=json.dumps(body).encode(),headers={"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; m=c["message"] + cont=m.get("content") or ""; reas=m.get("reasoning_content") or "" + leak = "" in cont + tally["n"]+=1 + if leak: tally["LEAK_think_in_content"]+=1 + if not cont.strip(): tally["empty_content"]+=1 + if reas: tally["reasoning_captured"]+=1 + tally[f"finish_{c.get('finish_reason')}"]+=1 + print(f" #{i}: finish={c.get('finish_reason'):<7} content={len(cont):<5} reasoning={len(reas):<6} leak={leak}") + return tally + +print("=== ARM A: current gen config (enable_thinking=false), direct to vLLM ===") +a=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":False}, + "temperature":0.7,"top_p":0.8,"presence_penalty":1.5, + "top_k":20,"min_p":0.0,"repetition_penalty":1.0}) +print(" TALLY:", dict(a)); print() + +print("=== ARM B: enable_thinking=TRUE + reasoning_effort=low (proposed fix) ===") +b=raw("qwen3.8-27b-uncensored", {"chat_template_kwargs":{"enable_thinking":True,"reasoning_effort":"low"}, + "temperature":0.7,"top_p":0.8,"presence_penalty":1.5, + "top_k":20,"min_p":0.0,"repetition_penalty":1.0}) +print(" TALLY:", dict(b)) diff --git a/services/gen-seat-mixed-quant/bench/think-leak/validate_fix.py b/services/gen-seat-mixed-quant/bench/think-leak/validate_fix.py new file mode 100644 index 0000000..578a5b4 --- /dev/null +++ b/services/gen-seat-mixed-quant/bench/think-leak/validate_fix.py @@ -0,0 +1,23 @@ +import json, urllib.request, collections +P="A farmer has 17 sheep. All but 9 run away. He buys twice as many as he has left, then sells 4. How many now? Explain." +BASE={"temperature":0.7,"top_p":0.8,"presence_penalty":1.5,"top_k":20,"min_p":0.0,"repetition_penalty":1.0} +def run(label, ctk, n=12): + t=collections.Counter() + for i in range(n): + body={"model":"qwen3.8-27b-uncensored","messages":[{"role":"user","content":P}], + "max_tokens":2048,"chat_template_kwargs":ctk}; body.update(BASE) + req=urllib.request.Request("http://10.250.50.54:8015/v1/chat/completions", + data=json.dumps(body).encode(),headers={"Content-Type":"application/json"}) + d=json.loads(urllib.request.urlopen(req,timeout=300).read().decode(),strict=False) + c=d["choices"][0]; cont=c["message"].get("content") or ""; reas=c["message"].get("reasoning_content") or "" + t["n"]+=1 + if "" in cont: t["LEAK"]+=1 + if not cont.strip(): t["EMPTY"]+=1 + if reas: t["reasoning_captured"]+=1 + t["ctok_total"]+=d.get("usage",{}).get("completion_tokens") or 0 + if c.get("finish_reason")!="stop": t["finish_"+str(c.get("finish_reason"))]+=1 + n_=t["n"] + print(f" {label:<44} n={n_} LEAK={t.get('LEAK',0)} EMPTY={t.get('EMPTY',0)} reas_captured={t.get('reasoning_captured',0)} avg_ctok={t['ctok_total']//n_}") +run("A: enable_thinking=false (CURRENT gen)", {"enable_thinking":False}) +run("B: enable_thinking=true + effort=low", {"enable_thinking":True,"reasoning_effort":"low"}) +run("C: enable_thinking=true + effort=medium", {"enable_thinking":True,"reasoning_effort":"medium"})