feat(refusal-probe): A/B refusal harness + Fable-Fusion 711 probe seat
Dark-Scarlett v1.0 refuses too much on the char-rp-reasoning seat. Root cause is visible on its card: ReadyArt/Dark-Scarlett-v1.0-27B is a plain finetune of stock Qwen/Qwen3.6-27B, tagged unaligned/nsfw/erp but carrying no abliteration -- the base model's refusal machinery is intact, so off-distribution prompts revert to safety-tuned Qwen3.6 behaviour. Candidate kkuspa/Qwen3.6-27B-Fable-Fusion-711-...-NVFP4A16 is refusal-ablated (Heretic), a structural edit rather than a behavioural preference. Verified before pulling: Qwen3_5ForConditionalGeneration wrapper class, 15 mtp.* tensors in a separate bf16 shard AND individually enumerated in quantization_config.ignore, NVFP4A16 with null input_activations, FP8 KV scales shipped, 262K context, Apache-2.0. Staged byte-verified at /tank/aimodels/fable-fusion-711-nvfp4a16 (28.55 GB). services/refusal-probe: deterministic marker-based classifier (LLM judge only breaks AMBIGUOUS ties, never overrides), intensity-graded battery so the report renders a refusal curve rather than an average, benign controls that gate run validity, and explicit handling of the thinking-budget trap -- empty content with finish_reason=length is reasoning exhausting the budget, not a refusal, and is excluded from the denominator. stacks/fablefusion-charrp-probe: throwaway :8019 seat serving as char-rp-probe, never aliased to char-rp-reasoning. MTP depth 3 rather than the card's 5 -- its 1.56x was measured greedy, and acceptance degrades at the temp 1.0 this seat is probed at. GPU1 is zero-sum at 94.9/97.9 GB, so this seat takes Dark-Scarlett's vacated slot; the A/B is sequential.
This commit is contained in:
@@ -0,0 +1,7 @@
|
||||
# ana-ml2 GPU1 THROWAWAY probe seat (Fable-Fusion 711). Real .env lives on the host.
|
||||
# Takes Dark-Scarlett's vacated slot — DS must be down first (GPU1 is zero-sum).
|
||||
FF_GPU_MEM_UTIL=0.44
|
||||
FF_MAX_MODEL_LEN=262144
|
||||
FF_GPU_ID=1
|
||||
FF_PORT=8019
|
||||
FF_MTP_DEPTH=3
|
||||
@@ -0,0 +1,66 @@
|
||||
# fablefusion-charrp-probe — THROWAWAY evaluation seat (ana-ml2 GPU1)
|
||||
|
||||
Evaluation-only seat for
|
||||
[`kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16`](https://huggingface.co/kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16),
|
||||
a candidate replacement for Dark-Scarlett v1.0 on the `char-rp-reasoning` seat.
|
||||
Serves as **`char-rp-probe`** on **:8019**.
|
||||
|
||||
**This is not a production seat.** It is deliberately not wired into any LiteLLM
|
||||
alias — per the no-false-aliases rule, nothing may answer to `char-rp-reasoning`
|
||||
except the real seat. `restart: "no"` so it never resurrects itself.
|
||||
|
||||
## Why this candidate
|
||||
|
||||
DS v1.0's refusals are too high. Its card explains why: `ReadyArt/Dark-Scarlett-v1.0-27B`
|
||||
is a *finetune* of stock `Qwen/Qwen3.6-27B`, tagged `unaligned`/`nsfw`/`erp` but
|
||||
carrying **no abliteration**. The refusal machinery of safety-tuned Qwen3.6 is
|
||||
still in there. Fable-Fusion 711 is refusal-*ablated* (Heretic, by trohrbaugh) —
|
||||
a structural edit rather than a behavioural preference.
|
||||
|
||||
Verified before pulling:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Loader class | `Qwen3_5ForConditionalGeneration` — the wrapper class DS also needs (a flat `Qwen3_5TextConfig` is rejected by both vLLM and SGLang) |
|
||||
| MTP | **15 `mtp.*` tensors present** in a separate bf16 shard AND individually enumerated in `quantization_config.ignore` — the exact bug that gave us 0% acceptance on the gen seat. Enumerated names are safer than our `re:^mtp.*` regex, which llm-compressor prunes when unmatched |
|
||||
| Scheme | NVFP4A16 compressed-tensors, `input_activations: null` — same as the DS seat runs today |
|
||||
| Held at bf16 | vision tower, all 48 DeltaNet linear-attn layers, MTP drafter, `lm_head` |
|
||||
| KV | FP8 k/v scales calibrated and shipped — the seat already runs `--kv-cache-dtype fp8` |
|
||||
| Context | 262,144 |
|
||||
| Fidelity | ARC-C / HellaSwag / Winogrande vs the BF16 parent all within standard error |
|
||||
| License | Apache-2.0 throughout |
|
||||
| On disk | `/tank/aimodels/fable-fusion-711-nvfp4a16` (28.55 GB, byte-verified against the HF tree) |
|
||||
|
||||
## ⚠️ GPU1 is zero-sum — this seat CANNOT run beside Dark-Scarlett
|
||||
|
||||
GPU1 sits at ~94.9/97.9 GB with the utility cluster co-resident. DS holds ~43 GB
|
||||
at `util 0.44`; this seat needs the same slot. The A/B is **sequential**, and the
|
||||
`char-rp-reasoning` seat is **down** for the probe window:
|
||||
|
||||
```bash
|
||||
# 1. baseline DS live first (no disruption)
|
||||
# 2. swap
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose down'
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose up -d'
|
||||
# 3. probe :8019, then
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/fablefusion-charrp-probe && docker compose down'
|
||||
ssh infra-ops@10.250.50.54 'cd /opt/docker/compose/darkscarlett-charrp-reasoning && docker compose up -d'
|
||||
```
|
||||
|
||||
## MTP depth: 3, not the card's 5
|
||||
|
||||
The card's headline 1.56x (56.2 → 87.4 tok/s on our exact GPU class) was measured
|
||||
on **greedy** 512-token generations. DavidAU states plainly that acceptance
|
||||
degrades above temp 1 — and temp 1.0 is the recommended creative setting this
|
||||
seat gets probed at. DS's own README already concluded spec-decode is net-negative
|
||||
at RP temps. Depth 3 matches the gen seat; treat any speed gain as unmeasured
|
||||
until benchmarked at real RP sampling settings.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
scripts/deploy-stack.sh ana-ml2 fablefusion-charrp-probe
|
||||
# on host: cp .env.example .env (defaults are fine)
|
||||
```
|
||||
|
||||
Probe harness: [`services/refusal-probe/`](../../services/refusal-probe/).
|
||||
@@ -0,0 +1,86 @@
|
||||
# fablefusion-charrp-probe — THROWAWAY evaluation seat for
|
||||
# kkuspa/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4A16
|
||||
# on ana-ml2 GPU1, served as `char-rp-probe` on :8019.
|
||||
#
|
||||
# ⚠️ THIS IS NOT A PRODUCTION SEAT. It exists to A/B against the live
|
||||
# Dark-Scarlett `char-rp-reasoning` seat (:8018) on refusal rate. It is
|
||||
# deliberately NOT wired into any LiteLLM alias — see the "no false aliases"
|
||||
# rule: nothing may answer to `char-rp-reasoning` except the real seat.
|
||||
#
|
||||
# ⚠️ GPU1 IS ZERO-SUM. The card sits at ~94.9/97.9 GB with the utility cluster
|
||||
# co-resident. This seat CANNOT run alongside darkscarlett-charrp-reasoning —
|
||||
# it takes DS's vacated slot at the same util. Sequence:
|
||||
# docker compose -p darkscarlett-charrp-reasoning down
|
||||
# docker compose -p fablefusion-charrp-probe up -d # probe
|
||||
# docker compose -p fablefusion-charrp-probe down
|
||||
# docker compose -p darkscarlett-charrp-reasoning up -d # restore
|
||||
#
|
||||
# Differences from the DS seat, and why:
|
||||
# --speculative-config : FF ships its 15 MTP tensors in bf16 and lists them
|
||||
# in quantization_config.ignore, so MTP actually
|
||||
# loads here (DS's quant dropped them). Depth 3 to
|
||||
# match the gen seat, not the card's 5 — the card's
|
||||
# 1.56x was measured GREEDY, and acceptance falls at
|
||||
# the temp 1.0 this seat is probed at.
|
||||
# --served-model-name : char-rp-probe, never char-rp-reasoning.
|
||||
|
||||
name: fablefusion-charrp-probe
|
||||
|
||||
services:
|
||||
vllm-fablefusion-probe:
|
||||
image: ${FF_IMAGE:-vllm/vllm-openai:latest}
|
||||
container_name: ${FF_CONTAINER:-vllm-fablefusion-probe}
|
||||
restart: "no" # throwaway: never resurrect itself across a reboot
|
||||
ipc: host
|
||||
ports:
|
||||
- "${FF_PORT:-8019}:8000"
|
||||
volumes:
|
||||
- /tank/aimodels:/tank/aimodels
|
||||
environment:
|
||||
- PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
|
||||
- VLLM_API_KEY=${API_KEY:-}
|
||||
command:
|
||||
- ${FF_MODEL:-/tank/aimodels/fable-fusion-711-nvfp4a16}
|
||||
- --quantization
|
||||
- compressed-tensors
|
||||
- --language-model-only
|
||||
- --served-model-name
|
||||
- char-rp-probe
|
||||
- --reasoning-parser
|
||||
- qwen3
|
||||
- --tool-call-parser
|
||||
- qwen3_coder
|
||||
- --enable-auto-tool-choice
|
||||
- --max-model-len
|
||||
- "${FF_MAX_MODEL_LEN:-262144}"
|
||||
- --max-num-seqs
|
||||
- "${FF_MAX_NUM_SEQS:-32}"
|
||||
- --gpu-memory-utilization
|
||||
- "${FF_GPU_MEM_UTIL:-0.44}"
|
||||
- --kv-cache-dtype
|
||||
- fp8
|
||||
- --speculative-config
|
||||
- '{"method":"qwen3_5_mtp","num_speculative_tokens":${FF_MTP_DEPTH:-3}}'
|
||||
- --trust-remote-code
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
device_ids:
|
||||
- "${FF_GPU_ID:-1}"
|
||||
capabilities:
|
||||
- gpu
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 600s
|
||||
networks:
|
||||
- tnet
|
||||
|
||||
networks:
|
||||
tnet:
|
||||
name: traefik-net
|
||||
external: true
|
||||
Reference in New Issue
Block a user