feat(char-rp-gguf): swap reasoning seat Deckard-PKD → NEO-CODE (Heretic2-Thinking Qwen3.6-27B)
R36 gate (2026-07-14) validated NEO-CODE ships on all axes: tool-calling 0.967 (attach_tool 1.00, 0 runaways — #355 eliminated), prose genre-artifact-fine (less clichéd than gen), refusal uncensored-as-spec + CSAM-clean. #355 root cause was MODEL-level, not the reasoning-budget-forcing bug: Deckard emitted Qwen's native qwen3_coder XML tool format malformed -> llama.cpp leaked the closing tags into the arg value -> Bifrost attach_tool schema error -> retry -> reasoning runaway to max_tokens. NEO-CODE emits the same native format cleanly on the same seat/parser -> no schema error -> no runaway. The fix was the model swap; there was never a wrong parser (the XML is Qwen3.5/3.6-native). - reasoning seat: Deckard-PKD (Qwen3.5) -> NEO-CODE=Heretic2-Thinking (Qwen3.6-27B) Q5 - samplers: card defaults (temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0), DRY dropped - ctx: 256K max; custom llama.cpp kept (qwen3_coder parse + PR#25544 belt-and-suspenders) - persistent-memory ACTIVE 1 marked resolved
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-07-13_
|
||||
_Last updated: 2026-07-14_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -104,7 +104,13 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._
|
||||
|
||||
### 🔺 ACTIVE 1 — #355-residual ROOT CAUSE FOUND: seat reasoning-loop to max_tokens (routed to brokkr)
|
||||
### ✅ #355-residual — RESOLVED 2026-07-14 via MODEL SWAP (Deckard→NEO-CODE); was ACTIVE 1
|
||||
|
||||
**UPDATE 2026-07-14 — FULLY RESOLVED via MODEL SWAP.** The custom llama.cpp (PR #25544 multi-terminator) was necessary but **INSUFFICIENT alone** (Deckard still ran away ~5% of tool-turns). Root cause was **MODEL-LEVEL**: Deckard emitted Qwen's NATIVE qwen3_coder XML tool format **MALFORMED** → llama.cpp leaked the closing tags into the arg value (`{"name":"\n</parameter>\n</function>..."}`) → Bifrost `attach_tool` schema error → retry → reasoning-runaway. **Swapped the model to NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** (GGUF base_model `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`; "NEO-CODE" = DavidAU's imatrix quant recipe, not a code finetune) which emits the same native format **CLEANLY** (`{"name":"Sindra"}`) on the same seat/parser. **R36 gate (brokkr 2026-07-14): NEO-CODE ships on ALL axes** — tool-calling **0.967** (attach_tool 1.00, **0 runaways**, beats gen 0.856 + Deckard 0.08/0.80), prose genre-artifact-fine (D2-slop repetition = dialogue-tags/anatomical genre convention, LESS clichéd than gen — not degradation), refusal uncensored-as-spec (creative-dark 38/40) + CSAM-gate-clean (0/100 operational-harm → serve-layer guardrails are the NSFW-RP consumer's job). **LIVE in production** (Q5, 256K ctx, card samplers temp 1.0 — temp 1.0 IMPROVED tools; DRY dropped); Deckard = rollback (`.env`/compose `.bak-pre-neocode-20260714`). **Custom llama.cpp KEPT** (now needed for qwen3_coder parsing — stock b8840 predates it, so stock is NOT a valid NEO-CODE rollback — + #25544 belt-and-suspenders). There was **NEVER a "wrong parser"** — the XML IS Qwen3.5/3.6-native (byte-identical to `Qwen/Qwen3.6-27B`'s template); an earlier "misconfigured OpenHands template" config-override reframe was WRONG and retracted. **OPEN:** seat is ~50 tok/s GGUF → soong's 45s client-timeout + the reasoning-budget-latency squeeze remain the constraint; the DFlash spec-decode recipe (z-lab Qwen3.6-27B drafter, ~4× on vLLM) applied to an NVFP4 Heretic2 vLLM seat is the future faster-seat path (NOT AEON-7-the-model — a non-RP-tuned base abliteration whose 4× speed is DGX-Spark-locked). Full settled record: auto-memory [[charrp-custom-llamacpp-pr25544]]. Historical mid-investigation saga kept below for context.
|
||||
|
||||
**BOTTOM LINE (historical):** char-rp-reasoning ran away to `max_tokens=32768` (~22 min, ~13% of turns) because stock llama.cpp **b8840's reasoning-budget forcing can't match Qwen3.5's `<tool_call>` reasoning terminator** (it only knows `</think>`) → on the tool-retry path forcing never fires. **FIXED** by a custom build (master `6eddde0` + unmerged upstream **PR #25544**, multi-terminator) now **LIVE** on the seat (`llamacpp-charrp:custom-latest`). **PARTIALLY VALIDATED**: the fix cut the runaway RATE hugely (soong smoke — most tool-turns now complete ~90 tokens, reasoning ~40 tokens, clean) BUT has a **RESIDUAL GAP** — task 5674 (2026-07-13 ~21:43) still ran away to 32768 (~9min; reasoned >45s before any content → the cap didn't fire on that turn). b61's 300s = client backstop but does NOT propagate to the seat → seat still burns to 32768 (confirmed). **⚠️ DO NOT retire the custom build as "0-gain" — CORRECTED: the seat is meant to be an RPG-GM (tools + prose + NSFW), so the tool-path #355 fix is the TOOL-PATH ENABLER for any reasoning-RP model here → KEEP it.** The REAL problem is the MODEL: **Deckard (RP-abliterated) is a poor tool-caller** (rambles/loops on tool-turns even with the fix — abliteration + heavy RP-tuning damages tool-calling/instruction-following). **MODEL-SWAP underway:** brokkr running a hunt for an NSFW-abliterated + **RELIABLE-tool-calling** + GM-prose **~24-27B** replacement (thread `01KXEQWEN2…`; ★ tool-calling is now a DISQUALIFYING A/B axis — the original A/B that picked Deckard scored prose/loops/refusals but NEVER tested tools). **PLAN: leave soong on char-rp-reasoning; swap the MODEL under it when brokkr delivers (no double-push).** Still **REMOVE-WHEN-#25544-MERGES** for the custom llama.cpp itself (retire custom→stock llama.cpp when the fix ships upstream; independent of the model swap). Records: auto-memory `charrp-custom-llamacpp-pr25544` + eshpfi `stacks/char-rp-gguf/llamacpp-custom/`; rollback = `.env` `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda`. Deploy-speed side-quest (#359+#360, ~11m→~4m) + soong-lab CI/CD (green-gated push-to-deploy + red-relay) also landed this session. Full diagnostic saga below (kept for the retire-decision context).
|
||||
|
||||
### 🔺 #355-residual ROOT CAUSE (historical saga): seat reasoning-loop to max_tokens
|
||||
- **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** → ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle — the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all → no semaphore to leak).
|
||||
- **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits `<think>`.
|
||||
- **MECHANISM — RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≤0) → so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate `</think>`, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) → brokkr's watchdog needs the bump too.
|
||||
|
||||
@@ -94,13 +94,14 @@ services:
|
||||
- homepage.description=Dark-romantasy RP prose seat, non-thinking (llama.cpp, ana-ml2 GPU 0)
|
||||
- homepage.href=http://10.250.50.54:${CHARRP_PORT:-8016}
|
||||
|
||||
# ── REASONING seat — Deckard-PKD (Qwen3.5) managed thinking + DRY. gateway char-rp-reasoning. ──
|
||||
# ── REASONING seat — NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) managed thinking. gateway char-rp-reasoning. ──
|
||||
llama-charrp-reasoning:
|
||||
# ⚠️ CUSTOM llama.cpp build (master 6eddde0 + unmerged PR #25544) — the Worldtree
|
||||
# #355 reasoning-budget-forcing fix (multi-terminator: handles Qwen3.5's <tool_call>
|
||||
# reasoning end-tag, which stock b8840's single-</think> budget can't match → the seat
|
||||
# loops in reasoning to max_tokens ≈ 22 min). DO NOT revert to a stock image until
|
||||
# #25544 merges upstream. Build recipe + why + rollback: ./llamacpp-custom/README.md.
|
||||
# ⚠️ CUSTOM llama.cpp build (master 6eddde0 + unmerged PR #25544). Needed for two reasons:
|
||||
# (1) recent master parses Qwen3.6's native qwen3_coder tool-call format (stock b8840 predates it —
|
||||
# the <tool_call><function=..><parameter=..> XML is Qwen3.5/3.6-native, NOT an OpenHands quirk);
|
||||
# (2) PR #25544 multi-terminator reasoning-budget fix (Worldtree #355) — belt-and-suspenders now that
|
||||
# NEO-CODE shows 0.0 runaway (R36 gate), but keep it. DO NOT revert to stock until #25544 merges.
|
||||
# Build recipe + why + rollback: ./llamacpp-custom/README.md.
|
||||
# Rollback: set LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda in .env + recreate.
|
||||
image: ${LLAMA_REASONING_IMAGE:-llamacpp-charrp:custom-latest}
|
||||
container_name: ${CHARRP_REASONING_CONTAINER:-llama-charrp-reasoning}
|
||||
@@ -119,7 +120,7 @@ services:
|
||||
entrypoint: ["/app/llama-server"]
|
||||
command:
|
||||
- --model
|
||||
- /models/${CHARRP_REASONING_MODEL:-rp/Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M.gguf}
|
||||
- /models/${CHARRP_REASONING_MODEL:-rp/Qwen3.6-27B-NEO-CODE-HERE-2T-OT-Q5_K_M.gguf}
|
||||
- --host
|
||||
- 0.0.0.0
|
||||
- --port
|
||||
@@ -130,33 +131,35 @@ services:
|
||||
- "${CHARRP_REASONING_CTX:-40960}"
|
||||
- --flash-attn
|
||||
- on
|
||||
# Deckard = Qwen3.5-27B GDN-hybrid (16 KV-caching layers → KV cheap); native ctx 262144
|
||||
# (256K, no YaRN). Full 256K @ q8_0 KV ≈ 8.5G. q8_0 verified coherent; f16 in .env if gibberish.
|
||||
# NEO-CODE = Qwen3.6-27B GDN-hybrid (16 of 64 layers cache KV → KV cheap); native ctx 262144
|
||||
# (256K). Full 256K @ q8_0 KV ≈ 8.6G, fits GPU0 w/ ~3.8G margin. q8_0 coherent; f16 in .env if gibberish.
|
||||
- --cache-type-k
|
||||
- ${CHARRP_REASONING_KV_TYPE:-q8_0}
|
||||
- --cache-type-v
|
||||
- ${CHARRP_REASONING_KV_TYPE:-q8_0}
|
||||
- --jinja
|
||||
# Deckard's Qwen3.5 template natively opens <think> + has enable_thinking → llama.cpp
|
||||
# manages the reasoning (trace to reasoning_content, content stays clean prose);
|
||||
# --reasoning-budget caps the CoT. (A/B 2026-07-08: Deckard 0/30 loops + 0/30 refusals,
|
||||
# beat RpR-v4 (1/30 loop), Pantheon (7/30 refusals), Snowdrop + Gembrain (template-incompat).)
|
||||
# NEO-CODE's Qwen3.6 template natively opens <think> → llama.cpp manages the reasoning
|
||||
# (trace to reasoning_content, content stays clean prose); --reasoning-budget caps the CoT.
|
||||
# (R36 gate 2026-07-14: NEO-CODE composite 0.922 tool-calling + 0.0 runaway — beat Deckard
|
||||
# 0.08/0.80 and gen-reasoning 0.856. Budget held at 400: latency-coupled to soong's client timeout.)
|
||||
- --reasoning
|
||||
- on
|
||||
- --reasoning-format
|
||||
- deepseek
|
||||
- --reasoning-budget
|
||||
- "${CHARRP_REASONING_BUDGET:-400}"
|
||||
# DRY anti-repetition (Deckard/Qwen3.5 tolerates it; sampler ORDER = dry AFTER temperature,
|
||||
# per the QwQ-family finding that a repetition PENALTY worsens looping but DRY fixes it).
|
||||
- --samplers
|
||||
- "top_k;top_p;min_p;temperature;dry"
|
||||
- --dry-multiplier
|
||||
- "${CHARRP_REASONING_DRY:-0.8}"
|
||||
- --dry-base
|
||||
- "1.75"
|
||||
- --dry-allowed-length
|
||||
- "2"
|
||||
# Sampler defaults per the DavidAU/Qwen3.6 model card (thinking-mode, general tasks): temp 1.0,
|
||||
# top_p 0.95, top_k 20, min_p 0.0, no rep-penalty, no DRY (DRY was a QwQ/Deckard looping band-aid
|
||||
# NEO-CODE doesn't need). All tunable via .env. NOTE: 0.922 tool-gate was on the OLD Deckard
|
||||
# samplers (effective temp~0.8 + DRY); re-validate tools + slop on these card samplers.
|
||||
- --temp
|
||||
- "${CHARRP_REASONING_TEMP:-1.0}"
|
||||
- --top-p
|
||||
- "${CHARRP_REASONING_TOP_P:-0.95}"
|
||||
- --top-k
|
||||
- "${CHARRP_REASONING_TOP_K:-20}"
|
||||
- --min-p
|
||||
- "${CHARRP_REASONING_MIN_P:-0.0}"
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health >/dev/null || exit 1"]
|
||||
interval: 30s
|
||||
|
||||
Reference in New Issue
Block a user