feat(refusal-probe): A/B refusal harness + Fable-Fusion 711 probe seat
Dark-Scarlett v1.0 refuses too much on the char-rp-reasoning seat. Root cause is visible on its card: ReadyArt/Dark-Scarlett-v1.0-27B is a plain finetune of stock Qwen/Qwen3.6-27B, tagged unaligned/nsfw/erp but carrying no abliteration -- the base model's refusal machinery is intact, so off-distribution prompts revert to safety-tuned Qwen3.6 behaviour. Candidate kkuspa/Qwen3.6-27B-Fable-Fusion-711-...-NVFP4A16 is refusal-ablated (Heretic), a structural edit rather than a behavioural preference. Verified before pulling: Qwen3_5ForConditionalGeneration wrapper class, 15 mtp.* tensors in a separate bf16 shard AND individually enumerated in quantization_config.ignore, NVFP4A16 with null input_activations, FP8 KV scales shipped, 262K context, Apache-2.0. Staged byte-verified at /tank/aimodels/fable-fusion-711-nvfp4a16 (28.55 GB). services/refusal-probe: deterministic marker-based classifier (LLM judge only breaks AMBIGUOUS ties, never overrides), intensity-graded battery so the report renders a refusal curve rather than an average, benign controls that gate run validity, and explicit handling of the thinking-budget trap -- empty content with finish_reason=length is reasoning exhausting the budget, not a refusal, and is excluded from the denominator. stacks/fablefusion-charrp-probe: throwaway :8019 seat serving as char-rp-probe, never aliased to char-rp-reasoning. MTP depth 3 rather than the card's 5 -- its 1.56x was measured greedy, and acceptance degrades at the temp 1.0 this seat is probed at. GPU1 is zero-sum at 94.9/97.9 GB, so this seat takes Dark-Scarlett's vacated slot; the A/B is sequential.
This commit is contained in:
@@ -0,0 +1,108 @@
|
||||
# refusal-probe — A/B two inference seats on refusal rate
|
||||
|
||||
Built 2026-08-15 to answer one question: **does a candidate model refuse less
|
||||
than Dark-Scarlett v1.0 on the prompts the `char-rp-reasoning` seat actually has
|
||||
to serve?**
|
||||
|
||||
DS v1.0's refusals were too high in production. Root cause is visible on its
|
||||
model card: `ReadyArt/Dark-Scarlett-v1.0-27B` is a *finetune* of stock
|
||||
`Qwen/Qwen3.6-27B` — tagged `unaligned`/`nsfw`/`erp` but carrying **no
|
||||
abliteration**. A finetune teaches a preference for compliance on its training
|
||||
distribution while leaving the base model's refusal machinery intact, so
|
||||
off-distribution prompts snap back to safety-tuned Qwen3.6 behaviour. The
|
||||
candidate, `kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-NVFP4A16`, is refusal-*ablated*
|
||||
(Heretic) — a structural edit rather than a behavioural one.
|
||||
|
||||
This harness measures whether that difference shows up on the wire.
|
||||
|
||||
## Design decisions that are load-bearing
|
||||
|
||||
**Deterministic classifier first.** The headline number comes from an explicit
|
||||
marker list (`HARD_MARKERS` / `SOFT_MARKERS` / `OOC_MARKERS` in `probe.py`), not
|
||||
an LLM. It is auditable, free, and reproducible. `--judge` adds an LLM second
|
||||
opinion but *only* for samples the deterministic pass marks `AMBIGUOUS`; it never
|
||||
overrides a deterministic verdict. House policy is deterministic-before-LLM, and
|
||||
a headline driven by an unlogged model call is not reproducible.
|
||||
|
||||
**The thinking-budget trap is handled explicitly.** On a seat running
|
||||
`--reasoning-parser qwen3`, reasoning can eat the whole token budget, leaving
|
||||
empty `content` with `finish_reason: length`. That looks *exactly* like a silent
|
||||
refusal and is not one. Those samples score `INVALID` and are excluded from the
|
||||
denominator, with the count surfaced in the report so a high one is visible
|
||||
rather than quietly skewing the rate. Observed reasoning on this seat runs
|
||||
~3.7k chars, so keep `--max-tokens` at 3072+.
|
||||
|
||||
**Refusal is stochastic.** At temp 1.0 a model may refuse 2 of 5 times on the
|
||||
same prompt. Default `-n 5`; every figure is a rate, never a boolean.
|
||||
|
||||
**Controls gate validity.** `category: control` prompts are benign RP that no
|
||||
model should refuse. A non-zero control refusal rate means the classifier is
|
||||
miscalibrated for that model's voice — the report marks the run `⚠️ SUSPECT`
|
||||
rather than presenting a result you shouldn't trust.
|
||||
|
||||
**Intensity curve over average.** Every prompt carries `intensity: 1-3`. *Where*
|
||||
the boundary sits is the diagnostic — a safety-tuned finetune typically breaks at
|
||||
intensity 2, an abliterated model should hold to 3. An average refusal rate hides
|
||||
that shape.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Baseline the live seat (no disruption — read-only inference load)
|
||||
uv run probe.py \
|
||||
--endpoint "dark-scarlett=http://10.250.50.54:8018/v1|char-rp-reasoning" \
|
||||
-n 5 --concurrency 4 --max-tokens 3072 --out ./results
|
||||
|
||||
# A/B two seats in one run (requires both up simultaneously — see VRAM note)
|
||||
uv run probe.py \
|
||||
--endpoint "dark-scarlett=http://10.250.50.54:8018/v1|char-rp-reasoning" \
|
||||
--endpoint "fable-fusion=http://10.250.50.54:8019/v1|char-rp-probe" \
|
||||
-n 5 --out ./results
|
||||
|
||||
# Optional LLM second opinion on AMBIGUOUS only (free local endpoint)
|
||||
# --judge "http://10.250.50.70:4000/v1|classifier" --judge-key "$LITELLM_KEY"
|
||||
```
|
||||
|
||||
`--endpoint` syntax is `NAME=BASE_URL|SERVED_MODEL_NAME`, repeatable.
|
||||
Reports land in `results/report-<ts>.md` plus `results/report-latest.md`;
|
||||
raw samples (full text of every response) in `results/raw-<ts>.json`.
|
||||
|
||||
## ⚠️ GPU1 is zero-sum — the two seats cannot co-exist
|
||||
|
||||
ana-ml2 GPU1 sits at ~94.9/97.9 GB with the utility cluster (rerankers, embed,
|
||||
reward, selene, coder, lfm) co-resident. Dark-Scarlett occupies ~43 GB at
|
||||
`util 0.44`; Fable-Fusion needs the same slot. **They cannot run at once**, so an
|
||||
A/B is sequential:
|
||||
|
||||
```bash
|
||||
# 1. baseline DS live (no disruption)
|
||||
uv run probe.py --endpoint "dark-scarlett=…8018/v1|char-rp-reasoning" …
|
||||
|
||||
# 2. swap — char-rp-reasoning is DOWN for this window
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose down'
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose up -d'
|
||||
|
||||
# 3. probe the candidate
|
||||
uv run probe.py --endpoint "fable-fusion=…8019/v1|char-rp-probe" …
|
||||
|
||||
# 4. restore
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose down'
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose up -d'
|
||||
```
|
||||
|
||||
The probe seat serves as `char-rp-probe` and is deliberately **not** wired into
|
||||
any LiteLLM alias — nothing but the real seat may answer to `char-rp-reasoning`.
|
||||
|
||||
## Extending the battery
|
||||
|
||||
`battery.yaml` is meant to be edited. The highest-value additions are **real
|
||||
prompts Dark-Scarlett actually refused in production** — those are known failures
|
||||
with a known verdict, worth more than any synthetic case. Add them with
|
||||
`provenance: observed`.
|
||||
|
||||
## Note on logging
|
||||
|
||||
Responses are written verbatim to `results/raw-*.json` and, if `--judge` is used,
|
||||
prompt+response text transits the LiteLLM gateway (which records full
|
||||
prompt/response in `spend_logs`). Both are internal fleet surfaces; the seat
|
||||
itself is already logged the same way. Named here so it isn't a surprise.
|
||||
Reference in New Issue
Block a user