vh
c8f128bdff
Pulled orcarouter/Qwen3.8-27B-Uncensored at rev 9878936b (55.5 GB, gated, our token has access) and built /tank/aimodels/qwen38-27b-orcarouter-nvfp4-mixed (23.4 GB, mixed NVFP4+FP8). Verified, not yet cut over. The operator asked whether we could apply the Robinson path to the MTP head. We cannot, because the author already did. compare_mtp_head.py against the verbatim base graft: 13 of 15 tensors byte-identical, exactly 2 differ -- mtp.layers.0.self_attn.o_proj.weight and mtp.layers.0.mlp.down_proj.weight, which are precisely the two residual writers our own abliterate.py targets (EXPECT_MTP_WRITERS = 2). Reverse-engineered the edit from the weights alone (mtp_delta.py, added here): sigma2/sigma1 = 0.0164 on BOTH tensors rank-1, a single-direction projection |cos| between the two recovered dirs = 1.0000 ONE shared direction ||delta||/||W|| = 1.42% and 1.41% a gentle, consistent projection sink energy dim 3994 = 0.0000% sink-clean; Heretic's was 6.18% That is the Robinson in-band MTP abliteration, already applied, with a direction that passes our sink screen outright. Nothing to do but preserve it, and the quant carries it byte-identically. This is the configuration the entire Cold-Fusion experiment was designed to test and never cleanly delivered. The new format screen paid for itself on its first real use: think_prior.py on the bf16 BEFORE any GPU time gave P(<think>) = 1.23e-06 at rank 52, against Cold-Fusion stock 0.1850 and h300 0.2216. Roughly 150,000x cleaner. Two durable findings about the pipeline itself: The quant needs ~17 GB, not a whole card. It ran entirely in GPU1's spare 16 GB with ZERO production seats stopped -- the h300 run's "stop BOTH GPU0 seats" was never necessary, it simply had a free card by coincidence. The first attempt OOM'd by 2.37 GiB at layer 64 of 65 with 3.57 GiB reserved-but-unallocated, which is fragmentation, and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True closed it. post_quant.py now builds a missing output index from the safetensors headers. A sub-23 GB quant saves one bare shard with no index, and post_quant needs one; this has broken three separate rounds and been hand-fixed every time. The header is read by struct-unpacking the u64 length and parsing the JSON -- never safe_open, which mmaps the whole 22 GB shard and ENOMEMs on ZFS. Artifact verified: mixed-precision, 1968 tensors, 15 mtp, 333 visual, re:^mtp.* present in the ignore list (llm-compressor pruned it as always), preproc restored. Imatrix deferred per operator; the log confirms the usual uniform-MSE fallback, so this build stays apples-to-apples with heresy's PPL 6.910.
think-leak — reproducers for the h300 unterminated-<think> defect
[2026-08-21] The Cold-Fusion Heretic-300 gen seat emits an unterminated
<think> block into content on any alias sampling at temperature > 0.
Operator-reported through Lobe as "sends CoT but never completes the turn".
What actually happens
With enable_thinking: false, the Qwen3.8 chat template appends a pre-closed
<think>\n\n</think>\n\n to the prompt (chat_template.jinja L165-166). The
model opens a fresh <think> anyway and never closes it. Because the prompt
already closed the block, vLLM's qwen3 reasoning parser is not in reasoning
state, so the tag is passed through as ordinary text: reasoning_content is
empty, reasoning_tokens is 0, and the whole reasoning-plus-answer blob lands in
content. Any client that renders <think>…</think> as a collapsible trace then
shows an endless thought bubble and no answer. The client is behaving correctly.
The trigger is TEMPERATURE, not presence_penalty
arms_de.py, n=12 per arm, same reproducer prompt:
| arm | leaks |
|---|---|
temp 0.7, presence_penalty 1.5 (current gen) |
4/12 |
| temp 0.7, presence_penalty 0.0 | 4/12 |
| temp 0.7, presence_penalty 0.5 | 3/12 |
| temp 0, presence_penalty 1.5 | 0/12 |
This falsifies the standing hypothesis (in the litellm config comment, and
the operator's own 2026-08-16 note) that presence_penalty: 1.5 is the first
dial to move. It is not this bug's cause.
It also explains the blast radius exactly — only the temp-0.7 aliases leak:
| alias | temp | leak (n=12) |
|---|---|---|
gen |
0.7 | 2 |
summarizer-large |
0.7 | 2 |
summarizer, classifier, image-judge, qwen-image-bench |
0 | 0 |
The fix
final_validate.py, n=30 across 4 prompt types plus a 3-turn conversation:
| config | leaks | empty content |
|---|---|---|
{enable_thinking: false} (current) |
8/30 | 0 |
{enable_thinking: true, reasoning_effort: low} |
0/30 | 1 |
Give the model a legitimately open <think> and it closes it properly, the
parser does its job, and content comes out clean. Costs ~+27% completion
tokens and leaves a ~3% empty-content residual.
Scripts
| script | what it does |
|---|---|
probe_gen.py <model> <max_tokens> <n> |
leak/finish/empty tally through the LiteLLM gateway |
blast.py |
leak rate across all seven seat aliases |
arms_de.py |
isolates temperature vs presence_penalty |
validate_fix.py |
thinking off vs low vs medium, n=12 |
final_validate.py |
the n=30 multi-prompt + multi-turn confirmation |
test_fix.py |
direct-to-vLLM A/B, bypasses the gateway |
Using these on any future seat
A "7/7 aliases return 200" smoke test cannot catch this. Trivial prompts
never invite reasoning, so they never sample the leaking token. Probe with a
reasoning-inviting prompt at n>=12 and grep raw content for <think>.
Did our abliteration cause it? No — the base did (~83% / ~17%)
think_prior.py <model-path> measures P(<think>) at the first generated
token, with the template rendered enable_thinking=false so the prompt already
contains a closed think pair. That is the exact event behind the leak. Raw
softmax over the full vocab, bf16, CPU-only — no GPU contention and no seat
downtime, which is what makes this cheap enough to run on any future candidate.
| build | abliteration | P(<think>) |
rank |
|---|---|---|---|
coldfusion-bf16 |
none (stock) | 0.1850 | 3 |
coldfusion-abliterated-L35-bf16 |
Robinson L35, mild | 0.2048 | 2 |
coldfusion-h300-mtp-bf16 |
Heretic-300, heavy | 0.2216 | 2 |
The stock base already carries 18.5%. Abliteration adds a real, monotonic, dose-dependent +3.7 points — a nudge on a pre-existing base, not the cause. Cold-Fusion is a reasoning-token-compression finetune; it is a model trained to think, briefly, and it wants to open that block.
Consequence: rolling back does not fix the leak. At 18.5% mass under temp 0.7
/ top_p 0.8 the stock base leaks at nearly the same rate. Only leaving the
Cold-Fusion family escapes it. The chat_template_kwargs fix is the right lever.
A forward-KL budget cannot catch this. Heretic minimizes forward KL(stock||abliterated), which is near-blind to new mass on tokens stock barely used — that is reverse KL's job, and we measured that asymmetry on L35 (reverse 1.43 vs forward 0.70). h300's KL of 0.0136 is not evidence of innocence. For any "did the abliteration break behavior X" question, measure P(token) directly.
Run it:
sudo -u llmuser env OMP_NUM_THREADS=48 CUDA_VISIBLE_DEVICES="" \
/tank/aimodels/quant-work/.venv/bin/python think_prior.py /tank/aimodels/<build>
⚠ Abliteration and post-quant outputs are written root-owned 0600 and are
unreadable to llmuser; normalize to llmuser:llmuser 0664 first. The failure
surfaces as a misleading FileNotFoundError, not a permission error.
Resolution — Cold-Fusion abandoned, seat rolled back to heresy (2026-08-21)
The operator's call, made before the result was in: "If it's the base, we abandon
h300 AND the base." The measurement said base, so it fired. The gen seat is back
on qwen38-27b-heresy-nvfp4-mixed.
The clincher is the same instrument, pointed at heresy:
| build | P(<think>) at first token |
|---|---|
| Cold-Fusion stock | 0.1850 |
| Cold-Fusion L35 | 0.2048 |
| Cold-Fusion h300 | 0.2216 |
heresy (restored) |
not in the top 20 — <0.002 |
A >100x gap. The two families are categorically different here, which is exactly why no rollback inside Cold-Fusion would have helped.
Verified after the rollback, same probes as before:
final_validate.py— 0/30 leaks, 0 empty, with the existingenable_thinking: falseconfig. h300 scored 8/30 on this same instrument.- KV pool 403,065 tok / 1.54x — heresy's exact documented baseline.
- 7/7 gateway aliases 200; vision intact.
- No LiteLLM config change was needed. The
chat_template_kwargsfix developed above is left unapplied: it worked, but it was a workaround for a base we no longer serve.
The gate this earns: run think_prior.py on a candidate's STOCK weights before
committing GPU time to it. It is a ~10s CPU measurement, and it would have
disqualified Cold-Fusion before the 300-trial Heretic study ever started.