From b268f9303596a93129ccace999a9df72f8dcfc00 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 8 Jul 2026 02:43:00 -0700 Subject: [PATCH] feat(char-rp-gguf): replace broken Angel NVFP4 with dual GGUF RP seat on ana-ml2 GPU0 char-rp -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s, zero refusal, tight POV) via llama.cpp (:8016). char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning, ~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018). - New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that killed the Angel serve. Never Ollama. - Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is loose; Cydonia-R1 runs away; QwQ is template-managed). Pantheon-Reasoning-27B stays rejected (re-censors in ; RpR-v4 abliterated reasoning is the fix). - Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers, dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented. - Retired the ms32-24b-angel stack. --- persistent-memory.md | 208 +++++++++++++++++++++++++++---- stacks/char-rp-gguf/.env.example | 46 +++++++ stacks/char-rp-gguf/README.md | 95 ++++++++++++++ stacks/char-rp-gguf/compose.yaml | 149 ++++++++++++++++++++++ stacks/litellm/conf/config.yaml | 57 ++++----- 5 files changed, 500 insertions(+), 55 deletions(-) create mode 100644 stacks/char-rp-gguf/.env.example create mode 100644 stacks/char-rp-gguf/README.md create mode 100644 stacks/char-rp-gguf/compose.yaml diff --git a/persistent-memory.md b/persistent-memory.md index a73cf41..d30f7e2 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-07-07_ +_Last updated: 2026-07-08_ ## Repo purpose @@ -102,7 +102,66 @@ no longer deployed sidecars here. See Recent decisions.) ## Current state / in-flight -_As of 2026-07-07:_ +_As of 2026-07-08 — OFF-THE-SHELF INFERENCE STACK is the active work (home-training pivoted away):_ + +- **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:** + - **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` — LIVE + GOOD.** Off-the-shelf NVFP4 of + `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only` (MoE 256e/8-active = + qwen3_5_moe, vision, GDN-hybrid; **modelopt** format → `--quantization modelopt`; vision bf16; 24G at + `/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`). UNCENSORED (writes brutal fiction think-off, no system prompt). + "Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken). + General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`, + `AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now + `${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay + transparent). **util 0.40 (~38G) → ~55–58G FREE on GPU0 for the RP seat.** AEON + the intermediate Pantheon-27B + serves are RETIRED. + - **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral) + dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat: + elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence + (live-tested). Replaced the broken Angel NVFP4. Alt prose model (`.env` swap `CHARRP_MODEL`): + `MS3.2-PaintedFantasy-v4.1-24B` (more literary flair, looser POV). Both GGUFs pre-pulled at `/tank/aimodels/llm/rp/`. + - **char-rp-reasoning (:8018) = `QwQ-32B-ArliAI-RpR-v4-Q5_K_M` GGUF — LIVE.** QwQ reasoning RP tune (container + `llama-charrp-reasoning`). `--reasoning on` → managed CoT SURFACES in `reasoning_content`, content stays clean + prose, `--reasoning-budget 400` caps it. Reasoning data is QwQ-ABLITERATED → **no re-censor in ``** (the + Pantheon failure mode). ~52 tok/s @ Q5 (46 @ Q6). Best-of-breed-per-seat (NOT the same model as char-rp). + +- **🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08).** char-rp = **Magidonia-24B-v4.3** (Magistral prose, + 65tps); char-rp-reasoning = **QwQ-32B-ArliAI-RpR-v4** (abliterated managed reasoning, 52tps @ Q5). Both GGUF via + llama.cpp (`char-rp-gguf` stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo + `stacks/char-rp-gguf/`; gateway rewired (`char-rp`→:8016, `char-rp-reasoning`→:8018, Mistral/QwQ samplers, dropped + the Qwen `enable_thinking` kwarg). **KEY FINDINGS:** (a) no single dense 24-32B is BOTH an elite non-thinking prose + seat AND a clean *managed*-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral `[THINK]` + discipline is loose (won't close `[/THINK]` on substantive reasoning → prose bleeds into reasoning, content empties); + Cydonia-R1's `` is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template + opens `` natively → llama.cpp manages+caps it; (c) **Pantheon-Reasoning-27B stays REJECTED** (re-censors in + ``; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in `reasoning_content` (fixes the + old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack + header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7. + Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K. + +- **⏳ PENDING — wire LiteLLM gateway sampling defaults for GEN** (operator: "wire those settings in"). Config: + `ana-docker:/opt/docker/conf/litellm/config.yaml`. **gen (Qwen3.6)** — thinking + `temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / presence_penalty 1.5 / rep 1.0`; non-thinking + `temp 0.7 / top_p 0.8 / top_k 20 / pp 1.5` (NO rep-penalty in thinking mode). NOT yet applied to gen/gen-reasoning. + **char-rp / char-rp-reasoning samplers ARE now wired** (Magistral: temp 1.0 / top_p 0.95 / min_p 0.03; QwQ RpR: + temp 1.0 / top_k 40 / min_p 0.02, no rep/DRY — 2026-07-08). + +- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py` + (llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py` + (llm-compressor + qwen ignore + `trust_remote_code` + a torchvision-gate bypass). ModelOpt path: + `/home/lkraven/nvfp4-quant/.venv-mopt` (modelopt 0.44) + `mopt_w4a16.py` (= `mtq.quantize` + `export_hf_checkpoint` + → modelopt format w/ `hf_quant_config.json`). Downloads via a detached vLLM container running + `huggingface_hub.snapshot_download` to `/tank/aimodels/-{bf16,nvfp4}`. **Gotchas:** llm-compressor + `save_compressed` DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps + `re:.*visual.*` + `re:.*linear_attn.*`(GDN) + lm_head + embeds + `mlp.gate`(router) bf16. + +- **🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf).** DPO came back slop → operator killed + home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models, + and every other project ("ritual seppuku" otherwise). **Revival map:** adapters `data/spike/{sft,dpo}_adapter`, + harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G). + mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.) + +_Earlier (2026-07-07):_ - **T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving.** SFT ran clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final @@ -116,25 +175,37 @@ _As of 2026-07-07:_ it. **HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now** (static; reserved for a later RP-quant track only). -- **HOT-SWAP — LAST SHOT IN FLIGHT = SGLang (operator: no more formats after this).** SGLang 0.5.13 - (`lmsysorg/sglang:latest`, on ana-ml2) is the only engine that ATTEMPTS to apply the LoRA (vs vLLM's - silent drop): bf16 serves the qwen3_5 arch fine + enters the apply path, but wants its OWN packed-LoRA - layout (base r16 + `get_stacked_multiply=3` qkv stacking + TP-sliced qkv B — a THIRD format). **Engine - mirror-image finding was PARTLY WRONG — see next.** **NEXT ACTION (mtf-dev msg 17, from reading the - 0.5.13 source): smoke the EXISTING `sft_adapter_zc` on SGLang bf16 — do NOT build a new format.** SGLang's - `utils.py:139 params_mapping` wants SEPARATE q/k/v on disk and STACKS them itself into [r*3,hidden]=[48,5120]; - the `sft_adapter_fused` I already tested was the WRONG direction (SGLang ×3'd the already-fused weight → - [144,5120] → the :773 assert). `sft_adapter_zc` (separate q/k/v/o + in_proj_qkv/in_proj_z, canonical) was - NEVER smoked on SGLang and should clear the qkv assert. REMAINING UNKNOWN = the GDN `in_proj_qkvz` - (get_stacked_multiply=4; the default utils.py `get_hidden_dim` RAISES NotImplementedError for it, so its - real dims live in the qwen3_5 MODEL CLASS override — not in the ref file I sent). Run: `--lora-target-modules - all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`. ATTACHES → T1 hot-swaps; ASSERTS on - in_proj_qkvz → paste the assert + `cat` the qwen3_5 model-class `get_hidden_dim`/`get_stacked_multiply` to - mtf-dev (one targeted fix, or intractable-4-slice → cut to merge, NO more rounds — operator's LAST shot). - (SGLang GENERIC image can't LOAD our NVFP4 checkpoint — ModelOptModelLoader packing mismatch — so - NVFP4-on-SGLang later needs the dedicated dev image or a requant; the smoke is bf16.) Adapter variants - built: `sft_adapter` (separate), `sft_adapter_zc` (separate + zero `in_proj_z`, the SGLang + vLLM-canonical - one to smoke), `sft_adapter_fused` (fused — wrong for both engines). +- **HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup + validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a + one-line fix. MERGE RECOMMENDED (operator's call, pending).** Ran `sft_adapter_zc` on SGLang 0.5.13 bf16 + (`--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`, GPU0, gen+rp + stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted + fine. **Died at `lora_manager.init_lora_shapes` with a NAME-VALIDATION `ValueError` (NOT the feared + in_proj_qkvz shape-assert):** adapter's `in_proj_qkv`+`in_proj_z` "not included in --lora-target-modules + [...in_proj_qkvz...]". **ROOT CAUSE (pinned):** `srt/lora/utils.py:235 get_normalized_target_modules` uses + a HARDCODED `params_mapping` dict that maps `q/k/v_proj→qkv_proj` (why self-attn works) but OMITS + `in_proj_qkv→in_proj_qkvz` + `in_proj_z→in_proj_qkvz`; it does NOT consult the model's + `packed_modules_mapping` (which DOES declare `"in_proj_qkvz":["in_proj_qkv","in_proj_z"]` at + `models/qwen3_5.py:1010`). Everything else is ALREADY in place: `get_hidden_dim("in_proj_qkvz")` IS + implemented in the model class (`:1040`, → `(hidden_size, key_dim*2+value_dim*2)`; the "utils NotImplementedError" + was only the DEFAULT); `get_stacked_multiply("in_proj_qkvz")=4` in the utils default dict; base-weight + `stacked_params_mapping` fuses `in_proj_qkv→shards(0,1,2)`,`in_proj_z→shard 3`. **PATCH APPLIED + RE-SMOKED + (bind-mounted a patched utils.py overlay into the container):** the 2-line add cleared `init_lora_shapes`; + server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z + shrink — B-side math right per mtf-dev), **self-attn loaded end-to-end** ("loaded weights for [o_proj,out_proj, + qkv_proj]"). **BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable):** + (1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]"); + (2) inference buffer-fill `utils.py:300 get_target_module_name` RAISED — SUBSTRING match, `in_proj_qkvz` ⊄ + `...in_proj_qkv.` and `qkv_proj` ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" → + HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3). + **FINISH LINE (if ever revived):** teach `get_target_module_name` + the load-consolidation the + `in_proj_qkv/in_proj_z→in_proj_qkvz` map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer + shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one + targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied + + loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't + LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: `sft_adapter` (separate), `sft_adapter_zc` + (separate + zero `in_proj_z`, the canonical one — THIS smoke), `sft_adapter_fused` (fused — wrong), plus + `sft_adapter_attnonly`. All at `/home/lkraven/model-training-forge/data/spike/`. - **MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned.** peft merge_and_unload (~20min, LOW wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training @@ -149,8 +220,9 @@ _As of 2026-07-07:_ contract (shape A: `GET /published-loras` + `POST /gc/published {train_ids}`) CONFIRMED — worker impl DEFERRED to after the T1 hot-swap work. auto-memory `reference_lora_training_worker`. -- **Serving state:** AEON gen (:8015) + char-rp (:8016) NVFP4 on ana-ml2 GPU0, HEALTHY (repeatedly - stopped/restored for the hot-swap smokes via `docker start vllm-aeon-gen vllm-aeon-rp`, never compose up). +- **Serving state (SUPERSEDED — see top; AEON retired 2026-07-08):** during the T1 work, AEON gen (:8015) + + char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via `docker start vllm-aeon-gen + vllm-aeon-rp`. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced). - **worldtree-dev vh/pypi read token** delivered (bifrost==1.1.0 pin, #349); dedicated read:package token at `nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt`. The over-scoped write:package token the operator @@ -172,6 +244,72 @@ _As of 2026-07-07:_ ## Recent decisions +- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** + Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, + util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps + ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled + refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); + Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting + ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway + sampling-defaults wiring still PENDING. +- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 + epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass + `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs + + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the + off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a + ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. + FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, + float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. + GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min + train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; + train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 + banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): + ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is + COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral + literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, + arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep + on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO + adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). +- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** + Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path + finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied + VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before + `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does + `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers + 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the + DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED + un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO + confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% + util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. + gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. +- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** + Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the + SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo + --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src + -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank + -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml + --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is + `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. + Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` + (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ + immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ + **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ + (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate + 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) + but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. + Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, + expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. +- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN + in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the + feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z → + in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn + loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj + weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler + died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn + proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's + LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached. + Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed. - `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05 cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud, @@ -268,6 +406,30 @@ _118 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 + (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY + (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. + Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the + whole NVFP4-quant surface. +- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json` + for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the + `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply + PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has + no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral + tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native + mistral tokenizer + vision). +- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base + (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2** + (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the + `` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). + **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw + abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen. +- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head + (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the + quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors + main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16 + self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format + (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.) - `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200 but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc` adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key diff --git a/stacks/char-rp-gguf/.env.example b/stacks/char-rp-gguf/.env.example new file mode 100644 index 0000000..0ca3670 --- /dev/null +++ b/stacks/char-rp-gguf/.env.example @@ -0,0 +1,46 @@ +# char-rp-gguf stack tunables. Copy to `.env` on ana-ml2 before deploying. +# +# cp .env.example .env +# docker compose up -d +# +# Keep compose.yaml constant; edit values here. + +# llama.cpp server image (bundles /app/llama-server; already present on ana-ml2). +LLAMA_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda + +# GPU 0 = the on-demand large-model card (co-resident with `gen`). GPU 1 is the +# always-on vLLM trio (granite/embed/rerank/reward) — do NOT put this stack there. +CHARRP_GPU_ID=0 + +# GGUF store on the host (mounted read-only at /models). Candidate GGUFs live in +# /tank/aimodels/llm/rp/ (pre-pulled). +MODELS_DIR=/tank/aimodels/llm + +# ── PROSE seat (char-rp) ──────────────────────────────────────────────────── +CHARRP_CONTAINER=llama-charrp +CHARRP_PORT=8016 +# Model path relative to /models. Default = Magidonia-24B-v4.3 (Magistral RP tune): +# elite dark-romantasy prose, zero refusal, ~65 tok/s, tight POV adherence. +CHARRP_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf +# Alternate prose model (more literary flair, looser POV adherence) — uncomment to swap: +# CHARRP_MODEL=rp/MS3.2-PaintedFantasy-v4.1-24B-Q6_K.gguf +CHARRP_CTX=16384 + +# ── REASONING seat (char-rp-reasoning) ────────────────────────────────────── +CHARRP_REASONING_CONTAINER=llama-charrp-reasoning +CHARRP_REASONING_PORT=8018 +# Default = QwQ-32B-ArliAI-RpR-v4 Q5_K_M (abliterated reasoning → no re-censor; +# llama.cpp-managed CoT). ~50 tok/s @ Q5. Use Q6_K (~46 tok/s) for a touch more +# quality if speed is not binding. +CHARRP_REASONING_MODEL=rp/QwQ-32B-ArliAI-RpR-v4-Q5_K_M.gguf +CHARRP_REASONING_CTX=16384 +# Thinking-token cap (QwQ over-thinks otherwise → starves the prose). 300-500 = a +# concise, useful scene-plan before the response. +CHARRP_REASONING_BUDGET=400 + +# ── ONE-MODEL FALLBACK (consistent Mistral style, lighter reasoning) ───────── +# To collapse both seats onto Magidonia (drop QwQ): set +# CHARRP_REASONING_MODEL=rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf +# and remove the --reasoning* flags from the reasoning service in compose.yaml +# (Magistral reasons only when the caller's system prompt contains "/think"; +# managed but LIGHT — see the compose header for why QwQ is the default). diff --git a/stacks/char-rp-gguf/README.md b/stacks/char-rp-gguf/README.md new file mode 100644 index 0000000..6097748 --- /dev/null +++ b/stacks/char-rp-gguf/README.md @@ -0,0 +1,95 @@ +# char-rp-gguf + +Dedicated GGUF **character-RP seat** on ana-ml2 GPU 0, served via llama.cpp +(`llama-server`) — **never Ollama**. Replaces the broken `ms32-24b-angel` NVFP4 +serve (self-quant produced garbage even at greedy). + +Two co-located instances, alongside the 35B-A3B heretic `gen` (`qwen36-27b-aeon`, +:8015): + +| service | port | gateway alias | model | role | +|---|---|---|---|---| +| `llama-charrp` | 8016 | `char-rp` | TheDrummer **Magidonia-24B-v4.3** Q6_K | non-thinking **prose** seat | +| `llama-charrp-reasoning` | 8018 | `char-rp-reasoning` | ArliAI **QwQ-32B-RpR-v4** Q5_K_M | managed-**reasoning** seat | + +## Why these two models + +Requirements (operator, 2026-07-08): highest prose quality; ≥50 tok/s (single-seat); +very low refusal for dark explicit romantasy; dense; GGUF-via-llama.cpp; thinking +strongly preferred; must fit alongside the 35B-A3B on GPU 0. + +- **Magidonia-24B-v4.3** (`char-rp`) — Magistral (Mistral) dark-romantasy RP tune. + Live-tested: elite literary prose, **zero refusal** on explicit scenes, precise + POV/instruction adherence, **~65 tok/s** at Q6_K. The prose star. +- **QwQ-32B-RpR-v4** (`char-rp-reasoning`) — QwQ reasoning RP tune whose reasoning + data was generated with **QwQ-abliterated**, so it does **not re-censor inside the + think phase** — the failure mode that disqualified Pantheon-Reasoning-27B (its + DeepSeek-distilled traces reason themselves into refusals). llama.cpp **manages** + QwQ reasoning: `--reasoning on` surfaces the trace in `reasoning_content` (content + stays clean prose, no `` leak) and `--reasoning-budget` caps the CoT. + ~50 tok/s at Q5_K_M (46 at Q6). + +**Why not one model for both:** no single dense 24–32B is both an elite non-thinking +prose seat and a clean *managed*-reasoning seat on llama.cpp. Magidonia's Magistral +`[THINK]` discipline is loose (won't reliably close `[/THINK]` on substantive +reasoning → prose bleeds into `reasoning_content`, `content` empties). Cydonia-R1's +`` is emergent, so llama.cpp can't manage/cap it → runaway CoT that never +reaches the prose. QwQ's template opens `` natively → llama.cpp manages+caps +it. Best-of-breed per seat. See the compose header for the one-model fallback. + +**Alternate prose model:** `MS3.2-PaintedFantasy-v4.1-24B` (Magistral; more literary +flair, looser POV adherence) — swap via `CHARRP_MODEL` in `.env`. All candidate GGUFs +are pre-pulled to `/tank/aimodels/llm/rp/`. + +## Deploy + +Canonical copies live here; authoritative copies live on the host under +`/opt/docker/compose/char-rp-gguf/`. + +```bash +# On ana-ml2 (models already pre-pulled to /tank/aimodels/llm/rp/): +scripts/deploy-stack.sh ana-ml2 char-rp-gguf # from this workspace +# or, on the host: +cd /opt/docker/compose/char-rp-gguf && cp .env.example .env && docker compose up -d +``` + +GGUFs are pulled with a detached `huggingface_hub` container, e.g.: + +```bash +docker run -d --entrypoint python3 -v /tank:/tank vllm/vllm-openai:latest -c \ + 'from huggingface_hub import hf_hub_download as d; \ + d(repo_id="bartowski/TheDrummer_Magidonia-24B-v4.3-GGUF", \ + filename="TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf", local_dir="/tank/aimodels/llm/rp")' +``` + +## Gateway wiring (ana-docker LiteLLM `:4000`) + +`char-rp` → `http://10.250.50.54:8016/v1`, `char-rp-reasoning` → +`http://10.250.50.54:8018/v1`. Sampler defaults (per the model cards): + +- **char-rp (Magidonia / Mistral):** `temperature 1.0, top_p 0.95, min_p 0.03`; + no repetition penalty (Magistral is stable without one). Callers may override. +- **char-rp-reasoning (QwQ RpR-v4):** `temperature 1.0, top_k 40, min_p 0.02`; + **no repetition / DRY / XTC penalties** (RpR card is explicit about this). + Reasoning surfaces in `reasoning_content`. + +Do **not** carry the Qwen-specific `chat_template_kwargs.enable_thinking` on these +aliases — it is meaningless to Mistral/QwQ templates. Reasoning is controlled +server-side (`--reasoning on` on :8018). + +## Revert + +```bash +docker compose -f /opt/docker/compose/char-rp-gguf/compose.yaml down +``` + +The retired `ms32-24b-angel` stack is left staged for reference (its NVFP4 quant is +broken — do not re-serve it). + +## Verify + +```bash +curl -s http://10.250.50.54:8016/v1/models # char-rp up +curl -s http://10.250.50.54:8018/v1/models # char-rp-reasoning up +nvidia-smi --query-gpu=index,memory.free --format=csv # GPU 0 margin +``` diff --git a/stacks/char-rp-gguf/compose.yaml b/stacks/char-rp-gguf/compose.yaml new file mode 100644 index 0000000..cedd0ef --- /dev/null +++ b/stacks/char-rp-gguf/compose.yaml @@ -0,0 +1,149 @@ +# char-rp-gguf — dedicated GGUF character-RP seat on ana-ml2 GPU 0, REPLACING the +# broken ms32-24b-angel NVFP4 serve (garbage output — bad self-quant W4A4). +# +# Two co-located llama.cpp (llama-server) instances on GPU 0, served alongside the +# 35B-A3B heretic `gen` (qwen36-27b-aeon stack, :8015): +# +# llama-charrp (:8016, gateway char-rp) — TheDrummer Magidonia-24B-v4.3 Q6_K. +# Magistral (Mistral) dark-romantasy RP tune. NON-thinking PROSE seat: elite +# literary prose, zero refusal, ~65 tok/s, precise POV/instruction adherence. +# +# llama-charrp-reasoning (:8018, gateway char-rp-reasoning) — ArliAI QwQ-32B-RpR-v4 Q5_K_M. +# QwQ reasoning RP tune whose reasoning DATA was generated with QwQ-ABLITERATED +# → it does NOT re-censor in the think phase (the exact failure mode that killed +# the Pantheon/DeepSeek-distilled reasoners: they reason themselves into refusals +# inside ). llama.cpp MANAGES QwQ reasoning natively: --reasoning on +# surfaces the trace in reasoning_content (clean prose in content, no +# leak), --reasoning-budget caps the chain-of-thought. ~50 tok/s @ Q5_K_M. +# +# WHY GGUF/llama.cpp (not vLLM NVFP4): sidesteps BOTH traps that killed the Angel serve +# — the vLLM NVFP4 self-quant breakage AND the Mistral-tokenizer/vision crash. llama.cpp +# handles Mistral + QwQ tokenizers natively. NEVER Ollama (banned fleet-wide). +# +# WHY TWO models (not one): no single dense 24-32B is BOTH an elite non-thinking prose +# seat AND a clean managed-reasoning seat on llama.cpp. Magidonia's Magistral [THINK] +# discipline is loose (won't reliably close [/THINK] on substantive reasoning → prose +# bleeds into reasoning_content, content empties); Cydonia-R1's is emergent, so +# llama.cpp can't manage/cap it → runaway CoT that never reaches prose. QwQ's template +# opens natively → llama.cpp manages+caps it. So: best-of-breed per seat. +# ONE-MODEL FALLBACK (consistent Mistral style, lighter reasoning): point both services +# at Magidonia via CHARRP_REASONING_MODEL in .env and blank CHARRP_REASONING_EXTRA_*. +# +# ALTERNATE prose model: PaintedFantasy-v4.1-24B (also Magistral, more literary flair +# but looser POV adherence) — set CHARRP_MODEL in .env. All candidate GGUFs are +# pre-pulled to /tank/aimodels/llm/rp/. +# +# VRAM (GPU 0, co-resident with gen ~38G): Magidonia Q6 ~19G + RpR-v4 Q5 ~23G + KV/ +# compute ~6-8G = ~85-88G / 97G (~9-12G margin). Keep ctx modest; drop CHARRP_*_CTX +# to 8192 in .env if warmup bites. depends_on sequences char-rp first. +# +# API auth: blank (LAN-internal on the GPU host; matches API_KEY= in the AEON stack / +# gateway VLLM_API_KEY). llama-server ignores the gateway's api_key when none is set. +# +# All tunables live in .env — edit that, not this file. + +name: char-rp-gguf + +services: + # ── PROSE seat — non-thinking. gateway char-rp. ── + llama-charrp: + image: ${LLAMA_IMAGE:-ghcr.io/mostlygeek/llama-swap:cuda} + container_name: ${CHARRP_CONTAINER:-llama-charrp} + restart: unless-stopped + runtime: nvidia + ports: + - "${CHARRP_PORT:-8016}:8080" + volumes: + - ${MODELS_DIR:-/tank/aimodels/llm}:/models:ro + environment: + # Pin to GPU 0 (the on-demand large-model card; the always-on vLLM trio owns GPU 1). + - NVIDIA_VISIBLE_DEVICES=${CHARRP_GPU_ID:-0} + entrypoint: ["/app/llama-server"] + command: + - --model + - /models/${CHARRP_MODEL:-rp/TheDrummer_Magidonia-24B-v4.3-Q6_K.gguf} + - --host + - 0.0.0.0 + - --port + - "8080" + - --n-gpu-layers + - "999" + - --ctx-size + - "${CHARRP_CTX:-16384}" + - --flash-attn + - on + - --jinja + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 240s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=char-rp (Magidonia-24B GGUF) + - homepage.icon=mdi-drama-masks + - homepage.description=Dark-romantasy RP prose seat, non-thinking (llama.cpp, ana-ml2 GPU 0) + - homepage.href=http://10.250.50.54:${CHARRP_PORT:-8016} + + # ── REASONING seat — QwQ managed thinking. gateway char-rp-reasoning. ── + llama-charrp-reasoning: + image: ${LLAMA_IMAGE:-ghcr.io/mostlygeek/llama-swap:cuda} + container_name: ${CHARRP_REASONING_CONTAINER:-llama-charrp-reasoning} + restart: unless-stopped + runtime: nvidia + # Sequence AFTER the prose seat is healthy so the two GPU-0 allocations don't race. + depends_on: + llama-charrp: + condition: service_healthy + ports: + - "${CHARRP_REASONING_PORT:-8018}:8080" + volumes: + - ${MODELS_DIR:-/tank/aimodels/llm}:/models:ro + environment: + - NVIDIA_VISIBLE_DEVICES=${CHARRP_GPU_ID:-0} + entrypoint: ["/app/llama-server"] + command: + - --model + - /models/${CHARRP_REASONING_MODEL:-rp/QwQ-32B-ArliAI-RpR-v4-Q5_K_M.gguf} + - --host + - 0.0.0.0 + - --port + - "8080" + - --n-gpu-layers + - "999" + - --ctx-size + - "${CHARRP_REASONING_CTX:-16384}" + - --flash-attn + - on + - --jinja + # QwQ reasoning is template-native → llama.cpp manages it. --reasoning on surfaces + # the trace in reasoning_content (content stays clean prose); --reasoning-budget + # caps the CoT so it can't run away and starve the prose (QwQ over-thinks otherwise). + - --reasoning + - on + - --reasoning-format + - deepseek + - --reasoning-budget + - "${CHARRP_REASONING_BUDGET:-400}" + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health >/dev/null || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 300s + networks: + - tnet + labels: + - homepage.group=AI Systems + - homepage.name=char-rp-reasoning (QwQ-32B RpR-v4 GGUF) + - homepage.icon=mdi-brain + - homepage.description=Dark-romantasy RP reasoning seat, managed CoT (llama.cpp, ana-ml2 GPU 0) + - homepage.href=http://10.250.50.54:${CHARRP_REASONING_PORT:-8018} + +networks: + tnet: + name: traefik-net + external: true diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index a41ed22..f1fbfc5 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -148,49 +148,42 @@ model_list: enable_thinking: true model_info: mode: chat - # char-rp -> the native-MTP single-seat RP twin (:8016, served qwen3.6-27b-aeon-rp). - # SAME weights as gen, MTP ON (qwen3_5_mtp n=3) for single-stream RP latency. RP-shaped - # sampler defaults (Worldtree character-rp role / callers override); thinking OFF. - # ⚠️ MTP silently drops min_p/logit_bias — don't rely on those through char-rp. + # char-rp -> GGUF dark-romantasy PROSE seat (:8016, llama.cpp, char-rp-gguf stack on + # ana-ml2 GPU 0). TheDrummer Magidonia-24B-v4.3 Q6_K — Magistral (Mistral) RP tune. + # NON-thinking: elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, + # tight POV/instruction adherence (live-tested 2026-07-08). Replaced the broken Angel + # NVFP4 serve AND the earlier AEON-rp MTP twin. Magistral is stable WITHOUT a repetition + # penalty (dropped the old 1.05); min_p 0.03 is the anti-slop knob. No enable_thinking + # kwarg — meaningless to the Mistral template. Alt prose model (swap via the stack .env): + # MS3.2-PaintedFantasy-v4.1-24B. Callers may override the sampler. - model_name: char-rp litellm_params: - model: hosted_vllm/qwen3.6-27b-aeon-rp + model: hosted_vllm/magidonia-24b-v4.3 api_base: http://10.250.50.54:8016/v1 api_key: os.environ/VLLM_API_KEY - # Qwen3.x non-thinking RP recs (operator 2026-07-05, adapted from Qwen + community - # RP testing). presence_penalty light (0.1) to reduce topic drift; repetition_penalty - # 1.05. min_p skipped (Qwen rec + MTP drops it anyway). DRY off (Qwen3.x artifacts). - temperature: 0.7 - top_p: 0.8 - presence_penalty: 0.1 + temperature: 1.0 + top_p: 0.95 extra_body: - top_k: 20 - repetition_penalty: 1.05 - chat_template_kwargs: - enable_thinking: false + min_p: 0.03 model_info: mode: chat - # char-rp-reasoning -> same RP seat (:8016, MTP), thinking ON. Reasoning-profile - # sampler (lower temp than char-rp for coherent thought); tunable. ⚠️ The thinking - # TRACE does not yet surface in reasoning_content (chat_template injects in the - # prompt → parser drops the span); pending a template fix, NOT a parser swap. + # char-rp-reasoning -> GGUF managed-REASONING seat (:8018, llama.cpp, char-rp-gguf stack). + # ArliAI QwQ-32B-ArliAI-RpR-v4 Q5_K_M — QwQ reasoning RP tune. Reasoning is ON server-side + # (--reasoning on): the CoT SURFACES in reasoning_content and content stays clean prose + # (fixes the old trace-not-surfacing gap), CoT budget-capped so it can't starve the prose. + # Its reasoning data is QwQ-ABLITERATED → no re-censor inside (the failure mode + # that disqualified Pantheon-Reasoning-27B). ~52 tok/s @ Q5_K_M. RpR card: temp 1.0, + # top_k 40, min_p 0.02, and NO repetition / DRY / XTC penalties. NOT the same model as + # char-rp (best-of-breed per seat) — see stacks/char-rp-gguf/README.md. - model_name: char-rp-reasoning litellm_params: - # Distinct served-name so a thinking-off char-rp request can't clobber this to - # enable_thinking:false (the bug that broke it). Same backend :8016, different model id. - model: hosted_vllm/qwen3.6-27b-aeon-rp-thinking - api_base: http://10.250.50.54:8016/v1 + model: hosted_vllm/qwq-32b-rpr-v4 + api_base: http://10.250.50.54:8018/v1 api_key: os.environ/VLLM_API_KEY - # Same RP profile as char-rp but thinking-mode top_p 0.95 (Qwen thinking rec) + focused - # temp 0.6. (Reasoning-trace surfacing still pending the parser/template fix.) - temperature: 0.6 - top_p: 0.95 - presence_penalty: 0.1 + temperature: 1.0 extra_body: - top_k: 20 - repetition_penalty: 1.05 - chat_template_kwargs: - enable_thinking: true + top_k: 40 + min_p: 0.02 model_info: mode: chat