# think-leak — reproducers for the h300 unterminated-`` defect `[2026-08-21]` The Cold-Fusion Heretic-300 gen seat emits an **unterminated `` block into `content`** on any alias sampling at temperature > 0. Operator-reported through Lobe as "sends CoT but never completes the turn". ## What actually happens With `enable_thinking: false`, the Qwen3.8 chat template appends a *pre-closed* `\n\n\n\n` to the prompt (`chat_template.jinja` L165-166). The model opens a **fresh** `` anyway and never closes it. Because the prompt already closed the block, vLLM's `qwen3` reasoning parser is not in reasoning state, so the tag is passed through as ordinary text: `reasoning_content` is empty, `reasoning_tokens` is 0, and the whole reasoning-plus-answer blob lands in `content`. Any client that renders `…` as a collapsible trace then shows an endless thought bubble and no answer. The client is behaving correctly. ## The trigger is TEMPERATURE, not presence_penalty `arms_de.py`, n=12 per arm, same reproducer prompt: | arm | leaks | |---|---| | temp 0.7, presence_penalty 1.5 (current `gen`) | 4/12 | | temp 0.7, presence_penalty 0.0 | 4/12 | | temp 0.7, presence_penalty 0.5 | 3/12 | | **temp 0**, presence_penalty 1.5 | **0/12** | This **falsifies** the standing hypothesis (in the litellm config comment, and the operator's own 2026-08-16 note) that `presence_penalty: 1.5` is the first dial to move. It is not this bug's cause. It also explains the blast radius exactly — only the temp-0.7 aliases leak: | alias | temp | leak (n=12) | |---|---|---| | `gen` | 0.7 | 2 | | `summarizer-large` | 0.7 | 2 | | `summarizer`, `classifier`, `image-judge`, `qwen-image-bench` | 0 | 0 | ## The fix `final_validate.py`, n=30 across 4 prompt types plus a 3-turn conversation: | config | leaks | empty content | |---|---|---| | `{enable_thinking: false}` (current) | **8/30** | 0 | | `{enable_thinking: true, reasoning_effort: low}` | **0/30** | 1 | Give the model a legitimately open `` and it closes it properly, the parser does its job, and `content` comes out clean. Costs ~+27% completion tokens and leaves a ~3% empty-content residual. ## Scripts | script | what it does | |---|---| | `probe_gen.py ` | leak/finish/empty tally through the LiteLLM gateway | | `blast.py` | leak rate across all seven seat aliases | | `arms_de.py` | isolates temperature vs presence_penalty | | `validate_fix.py` | thinking off vs low vs medium, n=12 | | `final_validate.py` | the n=30 multi-prompt + multi-turn confirmation | | `test_fix.py` | direct-to-vLLM A/B, bypasses the gateway | ## Using these on any future seat **A "7/7 aliases return 200" smoke test cannot catch this.** Trivial prompts never invite reasoning, so they never sample the leaking token. Probe with a reasoning-inviting prompt at n>=12 and grep raw `content` for ``. ## Did our abliteration cause it? No — the base did (~83% / ~17%) `think_prior.py ` measures P(``) at the **first generated token**, with the template rendered `enable_thinking=false` so the prompt already contains a *closed* think pair. That is the exact event behind the leak. Raw softmax over the full vocab, bf16, **CPU-only** — no GPU contention and no seat downtime, which is what makes this cheap enough to run on any future candidate. | build | abliteration | P(``) | rank | |---|---|---|---| | `coldfusion-bf16` | **none (stock)** | **0.1850** | 3 | | `coldfusion-abliterated-L35-bf16` | Robinson L35, mild | 0.2048 | 2 | | `coldfusion-h300-mtp-bf16` | Heretic-300, heavy | 0.2216 | 2 | The stock base already carries 18.5%. Abliteration adds a real, monotonic, dose-dependent **+3.7 points** — a nudge on a pre-existing base, not the cause. Cold-Fusion is a *reasoning-token-compression* finetune; it is a model trained to think, briefly, and it wants to open that block. **Consequence: rolling back does not fix the leak.** At 18.5% mass under temp 0.7 / top_p 0.8 the stock base leaks at nearly the same rate. Only leaving the Cold-Fusion family escapes it. The `chat_template_kwargs` fix is the right lever. **A forward-KL budget cannot catch this.** Heretic minimizes forward KL(stock||abliterated), which is near-blind to new mass on tokens stock barely used — that is reverse KL's job, and we measured that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL of 0.0136 is not evidence of innocence. For any "did the abliteration break behavior X" question, measure P(token) directly. Run it: sudo -u llmuser env OMP_NUM_THREADS=48 CUDA_VISIBLE_DEVICES="" \ /tank/aimodels/quant-work/.venv/bin/python think_prior.py /tank/aimodels/ ⚠ Abliteration and post-quant outputs are written root-owned `0600` and are unreadable to `llmuser`; normalize to `llmuser:llmuser 0664` first. The failure surfaces as a misleading `FileNotFoundError`, not a permission error. ## Resolution — Cold-Fusion abandoned, seat rolled back to `heresy` (2026-08-21) The operator's call, made before the result was in: *"If it's the base, we abandon h300 AND the base."* The measurement said base, so it fired. The gen seat is back on `qwen38-27b-heresy-nvfp4-mixed`. The clincher is the same instrument, pointed at heresy: | build | P(``) at first token | |---|---| | Cold-Fusion stock | 0.1850 | | Cold-Fusion L35 | 0.2048 | | Cold-Fusion h300 | 0.2216 | | **`heresy` (restored)** | **not in the top 20 — <0.002** | A >100x gap. The two families are categorically different here, which is exactly why no rollback *inside* Cold-Fusion would have helped. Verified after the rollback, same probes as before: - `final_validate.py` — **0/30 leaks, 0 empty**, with the *existing* `enable_thinking: false` config. h300 scored 8/30 on this same instrument. - KV pool 403,065 tok / 1.54x — heresy's exact documented baseline. - 7/7 gateway aliases 200; vision intact. - **No LiteLLM config change was needed.** The `chat_template_kwargs` fix developed above is left unapplied: it worked, but it was a workaround for a base we no longer serve. **The gate this earns: run `think_prior.py` on a candidate's STOCK weights before committing GPU time to it.** It is a ~10s CPU measurement, and it would have disqualified Cold-Fusion before the 300-trial Heretic study ever started.