feat(cyberprev-seat): mixed-NVFP4 quant of the abliterated cyber-preview, displacing sentinel-r3

Third sec-seat candidate: hotdogs/Qwen3.8-27B-abliterated-cyber-preview, an abliteration (refusal-direction weight edit) of Qwen3.8-27B aimed at the cyber-offense refusal surface -- distinct from mog-sec (persona on stock weights) and sentinel-r3 (SFT finetune). Operator instruction: quant it, take sentinel down, serve it with mtp or dflash. Quantized to the house mixed recipe via services/gen-seat-mixed-quant/ (NVFP4 W4A4 on MLP layers 0-55 + FP8 W8A8 on attn/linear_attn/lm_head/MLP 56-63, FP8 KV). The prior attempt (2026-09-11/14) died with "Cannot determine num_attention_heads" because it ran from a bare .venv whose newer compressed-tensors reads that field at top level; quant_mixed_nvfp4.py already promotes text_config attention fields for exactly this reason, and the run through the canonical vLLM-image + llmcompressor 0.13.0 / compressed-tensors 0.18.0 path (versions recorded from the container) completed clean. post_quant.py did its job: grafted the 15 MTP tensors verbatim (BF16), and re-injected re:^mtp.* into the ignore list after llm-compressor pruned it for matching no module at quant time -- without which vLLM loads the grafted head uninitialised and speculative decoding runs at 0% acceptance. It also caught a missing preprocessor_config.json (absent from the abliterated source AND its hotdogs upstream); restored from Qwen/Qwen3.8-27B, verified byte-identical to the working sentinel-r3 build, so the vision tower preprocesses. Verified from the tensors, not the config: NVFP4 covers MLP 0-55, FP8 covers 56-63, no overlap, 168 weight_packed tensors (56x3), 15 BF16 MTP, 333 BF16 vision. 51.0 GiB bf16 -> 21.0 GiB. Served under its own name (cyberprev-27b / -thinking), NOT sentinel-r3 -- serving different weights under a retired name is silent substitution. Takes over :8025 and GPU 0 (co-resident with mog-sec). dflash k=7 is the default, carried from the sentinel measurement, but is re-measured against MTP on this ABLITERATED body before cutover, since abliteration is exactly what can desync an MTP head. Drafter (qwen38-27b-dflash2-drafter) shared with mog-sec. Context capped at 163840 not native 262K, inherited from mog-sec's hard-won depth lesson; raise only after a non-repeating deep-prefill probe on this checkpoint.
2026-09-14 03:32:09 -07:00
parent 4954cf6756
commit 1418edb03e
2 changed files with 195 additions and 0 deletions
@@ -0,0 +1,54 @@
+# cyberprev-seat — hotdogs/Qwen3.8-27B-abliterated-cyber-preview, fv-ml1 GPU 0, :8025.
+# Copy to .env on the host at /opt/docker/compose/cyberprev-seat/.env.
+
+# ── Image ───────────────────────────────────────────────────────────────────
+# Same pinned nightly mog-sec and sentinel-r3 run. Not :latest — pin it, so a seat
+# restart cannot silently change the engine under a measured configuration.
+CYBER_IMAGE=vllm/vllm-openai:nightly-e9d1398d9edfd90fcc1cf783805240e3effec013
+API_KEY=replace-me
+
+# ── Placement ───────────────────────────────────────────────────────────────
+# GPU 0, co-resident with mog-sec. Took over :8025 from the retired sentinel-r3 seat.
+CYBER_GPU_ID=0
+CYBER_PORT=8025
+CYBER_CONTAINER_NAME=vllm-cyberprev
+
+# ── Model ───────────────────────────────────────────────────────────────────
+# In-house mixed quant of /tank/aimodels/cyberprev-bf16 (51.0 GiB bf16 source),
+# built with services/gen-seat-mixed-quant/. compressed-tensors, NOT modelopt_fp4.
+CYBER_MODEL=/tank/aimodels/cyberprev-nvfp4-mixed
+CYBER_QUANT=compressed-tensors
+# ⚠ Its OWN name. Do not reuse `sentinel-r3` — that seat is retired and its gateway
+# aliases are deliberately left to 404 rather than repointed at different weights.
+CYBER_SERVED_NAME=cyberprev-27b
+CYBER_SERVED_NAME_THINK=cyberprev-27b-thinking
+
+# ── Memory ──────────────────────────────────────────────────────────────────
+# 0.40 of the card, sharing GPU 0 with mog-sec. KV pinned in bytes (8 GiB) so the
+# figure is reproducible regardless of what else is resident at start time.
+CYBER_GPU_MEM_UTIL=0.40
+CYBER_KV_CACHE_MEMORY=8589934592
+
+# ⚠ 163840, NOT native 262144. Inherited from mog-sec, which crashed five times
+# learning that the KV pool's capacity and the card's processing depth are different
+# numbers. Raising this requires a deep-prefill probe with a NON-REPEATING prompt on
+# THIS checkpoint — a repeated prompt hashes to cached blocks and never prefills deep.
+CYBER_MAX_MODEL_LEN=163840
+CYBER_MAX_NUM_SEQS=16
+CYBER_MAX_NUM_BATCHED_TOKENS=4096
+CYBER_KV_CACHE_DTYPE=fp8
+
+# ── Speculative decoding ────────────────────────────────────────────────────
+# One var, whole JSON — the two shapes are not interchangeable (dflash needs a
+# "model", MTP must not have one).
+#   DFlash2 : {"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
+#   MTP     : {"method": "qwen3_5_mtp", "num_speculative_tokens": 3}
+# dflash k=7 is the default, carried from the sentinel measurement (2.40 vs 2.18).
+# ⚠ Re-measure on THIS body before trusting it: cyberprev is ABLITERATED, and
+# abliteration is precisely what can desync an MTP head. Gate MTP on a measured
+# acceptance of >=~40%, never on an assumption.
+CYBER_SPEC_CONFIG={"method": "dflash", "model": "/drafter", "num_speculative_tokens": 7}
+
+CYBER_REASONING_PARSER=qwen3
+CYBER_REASONING_EFFORT=medium
+CYBER_ALLOC_CONF=