esh-pfi-infrastructure/services/refusal-probe

refusal-probe — A/B two inference seats on refusal rate

Built 2026-08-15 to answer one question: does a candidate model refuse less than Dark-Scarlett v1.0 on the prompts the char-rp-reasoning seat actually has to serve?

DS v1.0's refusals were too high in production. Root cause is visible on its model card: ReadyArt/Dark-Scarlett-v1.0-27B is a finetune of stock Qwen/Qwen3.6-27B — tagged unaligned/nsfw/erp but carrying no abliteration. A finetune teaches a preference for compliance on its training distribution while leaving the base model's refusal machinery intact, so off-distribution prompts snap back to safety-tuned Qwen3.6 behaviour. The candidate, kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-NVFP4A16, is refusal-ablated (Heretic) — a structural edit rather than a behavioural one.

This harness measures whether that difference shows up on the wire.

Design decisions that are load-bearing

Deterministic classifier first. The headline number comes from an explicit marker list (HARD_MARKERS / SOFT_MARKERS / OOC_MARKERS in probe.py), not an LLM. It is auditable, free, and reproducible. --judge adds an LLM second opinion but only for samples the deterministic pass marks AMBIGUOUS; it never overrides a deterministic verdict. House policy is deterministic-before-LLM, and a headline driven by an unlogged model call is not reproducible.

The thinking-budget trap is handled explicitly. On a seat running --reasoning-parser qwen3, reasoning can eat the whole token budget, leaving empty content with finish_reason: length. That looks exactly like a silent refusal and is not one. Those samples score INVALID and are excluded from the denominator, with the count surfaced in the report so a high one is visible rather than quietly skewing the rate. Observed reasoning on this seat runs ~3.7k chars, so keep --max-tokens at 3072+.

Refusal is stochastic. At temp 1.0 a model may refuse 2 of 5 times on the same prompt. Default -n 5; every figure is a rate, never a boolean.

Controls gate validity. category: control prompts are benign RP that no model should refuse. A non-zero control refusal rate means the classifier is miscalibrated for that model's voice — the report marks the run ⚠️ SUSPECT rather than presenting a result you shouldn't trust.

Intensity curve over average. Every prompt carries intensity: 1-3. Where the boundary sits is the diagnostic — a safety-tuned finetune typically breaks at intensity 2, an abliterated model should hold to 3. An average refusal rate hides that shape.

Usage

# Baseline the live seat (no disruption — read-only inference load)
uv run probe.py \
  --endpoint "dark-scarlett=http://10.250.50.54:8018/v1|char-rp-reasoning" \
  -n 5 --concurrency 4 --max-tokens 3072 --out ./results

# A/B two seats in one run (requires both up simultaneously — see VRAM note)
uv run probe.py \
  --endpoint "dark-scarlett=http://10.250.50.54:8018/v1|char-rp-reasoning" \
  --endpoint "fable-fusion=http://10.250.50.54:8019/v1|char-rp-probe" \
  -n 5 --out ./results

# Optional LLM second opinion on AMBIGUOUS only (free local endpoint)
#   --judge "http://10.250.50.70:4000/v1|classifier" --judge-key "$LITELLM_KEY"

--endpoint syntax is NAME=BASE_URL|SERVED_MODEL_NAME, repeatable. Reports land in results/report-<ts>.md plus results/report-latest.md; raw samples (full text of every response) in results/raw-<ts>.json.

⚠️ GPU1 is zero-sum — the two seats cannot co-exist

ana-ml2 GPU1 sits at ~94.9/97.9 GB with the utility cluster (rerankers, embed, reward, selene, coder, lfm) co-resident. Dark-Scarlett occupies ~43 GB at util 0.44; Fable-Fusion needs the same slot. They cannot run at once, so an A/B is sequential:

# 1. baseline DS live (no disruption)
uv run probe.py --endpoint "dark-scarlett=…8018/v1|char-rp-reasoning" …

# 2. swap — char-rp-reasoning is DOWN for this window
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose down'
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe   && docker compose up -d'

# 3. probe the candidate
uv run probe.py --endpoint "fable-fusion=…8019/v1|char-rp-probe" …

# 4. restore
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe   && docker compose down'
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose up -d'

The probe seat serves as char-rp-probe and is deliberately not wired into any LiteLLM alias — nothing but the real seat may answer to char-rp-reasoning.

Extending the battery

battery.yaml is meant to be edited. The highest-value additions are real prompts Dark-Scarlett actually refused in production — those are known failures with a known verdict, worth more than any synthetic case. Add them with provenance: observed.

Note on logging

Responses are written verbatim to results/raw-*.json and, if --judge is used, prompt+response text transits the LiteLLM gateway (which records full prompt/response in spend_logs). Both are internal fleet surfaces; the seat itself is already logged the same way. Named here so it isn't a surprise.