diff --git a/persistent-memory.md b/persistent-memory.md index 3cddd8c..0503675 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory β€” eshpfi-management -_Last updated: 2026-07-13_ +_Last updated: 2026-07-14_ ## Repo purpose @@ -104,7 +104,13 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-07-13 β€” two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._ -### πŸ”Ί ACTIVE 1 β€” #355-residual ROOT CAUSE FOUND: seat reasoning-loop to max_tokens (routed to brokkr) +### βœ… #355-residual β€” RESOLVED 2026-07-14 via MODEL SWAP (Deckardβ†’NEO-CODE); was ACTIVE 1 + +**UPDATE 2026-07-14 β€” FULLY RESOLVED via MODEL SWAP.** The custom llama.cpp (PR #25544 multi-terminator) was necessary but **INSUFFICIENT alone** (Deckard still ran away ~5% of tool-turns). Root cause was **MODEL-LEVEL**: Deckard emitted Qwen's NATIVE qwen3_coder XML tool format **MALFORMED** β†’ llama.cpp leaked the closing tags into the arg value (`{"name":"\n\n..."}`) β†’ Bifrost `attach_tool` schema error β†’ retry β†’ reasoning-runaway. **Swapped the model to NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** (GGUF base_model `DavidAU/Qwen3.6-27B-Heretic2-Uncensored-Finetune-Thinking`; "NEO-CODE" = DavidAU's imatrix quant recipe, not a code finetune) which emits the same native format **CLEANLY** (`{"name":"Sindra"}`) on the same seat/parser. **R36 gate (brokkr 2026-07-14): NEO-CODE ships on ALL axes** β€” tool-calling **0.967** (attach_tool 1.00, **0 runaways**, beats gen 0.856 + Deckard 0.08/0.80), prose genre-artifact-fine (D2-slop repetition = dialogue-tags/anatomical genre convention, LESS clichΓ©d than gen β€” not degradation), refusal uncensored-as-spec (creative-dark 38/40) + CSAM-gate-clean (0/100 operational-harm β†’ serve-layer guardrails are the NSFW-RP consumer's job). **LIVE in production** (Q5, 256K ctx, card samplers temp 1.0 β€” temp 1.0 IMPROVED tools; DRY dropped); Deckard = rollback (`.env`/compose `.bak-pre-neocode-20260714`). **Custom llama.cpp KEPT** (now needed for qwen3_coder parsing β€” stock b8840 predates it, so stock is NOT a valid NEO-CODE rollback β€” + #25544 belt-and-suspenders). There was **NEVER a "wrong parser"** β€” the XML IS Qwen3.5/3.6-native (byte-identical to `Qwen/Qwen3.6-27B`'s template); an earlier "misconfigured OpenHands template" config-override reframe was WRONG and retracted. **OPEN:** seat is ~50 tok/s GGUF β†’ soong's 45s client-timeout + the reasoning-budget-latency squeeze remain the constraint; the DFlash spec-decode recipe (z-lab Qwen3.6-27B drafter, ~4Γ— on vLLM) applied to an NVFP4 Heretic2 vLLM seat is the future faster-seat path (NOT AEON-7-the-model β€” a non-RP-tuned base abliteration whose 4Γ— speed is DGX-Spark-locked). Full settled record: auto-memory [[charrp-custom-llamacpp-pr25544]]. Historical mid-investigation saga kept below for context. + +**BOTTOM LINE (historical):** char-rp-reasoning ran away to `max_tokens=32768` (~22 min, ~13% of turns) because stock llama.cpp **b8840's reasoning-budget forcing can't match Qwen3.5's `` reasoning terminator** (it only knows ``) β†’ on the tool-retry path forcing never fires. **FIXED** by a custom build (master `6eddde0` + unmerged upstream **PR #25544**, multi-terminator) now **LIVE** on the seat (`llamacpp-charrp:custom-latest`). **PARTIALLY VALIDATED**: the fix cut the runaway RATE hugely (soong smoke β€” most tool-turns now complete ~90 tokens, reasoning ~40 tokens, clean) BUT has a **RESIDUAL GAP** β€” task 5674 (2026-07-13 ~21:43) still ran away to 32768 (~9min; reasoned >45s before any content β†’ the cap didn't fire on that turn). b61's 300s = client backstop but does NOT propagate to the seat β†’ seat still burns to 32768 (confirmed). **⚠️ DO NOT retire the custom build as "0-gain" β€” CORRECTED: the seat is meant to be an RPG-GM (tools + prose + NSFW), so the tool-path #355 fix is the TOOL-PATH ENABLER for any reasoning-RP model here β†’ KEEP it.** The REAL problem is the MODEL: **Deckard (RP-abliterated) is a poor tool-caller** (rambles/loops on tool-turns even with the fix β€” abliteration + heavy RP-tuning damages tool-calling/instruction-following). **MODEL-SWAP underway:** brokkr running a hunt for an NSFW-abliterated + **RELIABLE-tool-calling** + GM-prose **~24-27B** replacement (thread `01KXEQWEN2…`; β˜… tool-calling is now a DISQUALIFYING A/B axis β€” the original A/B that picked Deckard scored prose/loops/refusals but NEVER tested tools). **PLAN: leave soong on char-rp-reasoning; swap the MODEL under it when brokkr delivers (no double-push).** Still **REMOVE-WHEN-#25544-MERGES** for the custom llama.cpp itself (retire customβ†’stock llama.cpp when the fix ships upstream; independent of the model swap). Records: auto-memory `charrp-custom-llamacpp-pr25544` + eshpfi `stacks/char-rp-gguf/llamacpp-custom/`; rollback = `.env` `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda`. Deploy-speed side-quest (#359+#360, ~11mβ†’~4m) + soong-lab CI/CD (green-gated push-to-deploy + red-relay) also landed this session. Full diagnostic saga below (kept for the retire-decision context). + +### πŸ”Ί #355-residual ROOT CAUSE (historical saga): seat reasoning-loop to max_tokens - **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** β†’ ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle β€” the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all β†’ no semaphore to leak). - **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits ``. - **MECHANISM β€” RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≀0) β†’ so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate ``, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) β†’ brokkr's watchdog needs the bump too. diff --git a/stacks/char-rp-gguf/compose.yaml b/stacks/char-rp-gguf/compose.yaml index c1fc2de..48008eb 100644 --- a/stacks/char-rp-gguf/compose.yaml +++ b/stacks/char-rp-gguf/compose.yaml @@ -94,13 +94,14 @@ services: - homepage.description=Dark-romantasy RP prose seat, non-thinking (llama.cpp, ana-ml2 GPU 0) - homepage.href=http://10.250.50.54:${CHARRP_PORT:-8016} - # ── REASONING seat β€” Deckard-PKD (Qwen3.5) managed thinking + DRY. gateway char-rp-reasoning. ── + # ── REASONING seat β€” NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) managed thinking. gateway char-rp-reasoning. ── llama-charrp-reasoning: - # ⚠️ CUSTOM llama.cpp build (master 6eddde0 + unmerged PR #25544) β€” the Worldtree - # #355 reasoning-budget-forcing fix (multi-terminator: handles Qwen3.5's - # reasoning end-tag, which stock b8840's single- budget can't match β†’ the seat - # loops in reasoning to max_tokens β‰ˆ 22 min). DO NOT revert to a stock image until - # #25544 merges upstream. Build recipe + why + rollback: ./llamacpp-custom/README.md. + # ⚠️ CUSTOM llama.cpp build (master 6eddde0 + unmerged PR #25544). Needed for two reasons: + # (1) recent master parses Qwen3.6's native qwen3_coder tool-call format (stock b8840 predates it β€” + # the XML is Qwen3.5/3.6-native, NOT an OpenHands quirk); + # (2) PR #25544 multi-terminator reasoning-budget fix (Worldtree #355) β€” belt-and-suspenders now that + # NEO-CODE shows 0.0 runaway (R36 gate), but keep it. DO NOT revert to stock until #25544 merges. + # Build recipe + why + rollback: ./llamacpp-custom/README.md. # Rollback: set LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda in .env + recreate. image: ${LLAMA_REASONING_IMAGE:-llamacpp-charrp:custom-latest} container_name: ${CHARRP_REASONING_CONTAINER:-llama-charrp-reasoning} @@ -119,7 +120,7 @@ services: entrypoint: ["/app/llama-server"] command: - --model - - /models/${CHARRP_REASONING_MODEL:-rp/Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M.gguf} + - /models/${CHARRP_REASONING_MODEL:-rp/Qwen3.6-27B-NEO-CODE-HERE-2T-OT-Q5_K_M.gguf} - --host - 0.0.0.0 - --port @@ -130,33 +131,35 @@ services: - "${CHARRP_REASONING_CTX:-40960}" - --flash-attn - on - # Deckard = Qwen3.5-27B GDN-hybrid (16 KV-caching layers β†’ KV cheap); native ctx 262144 - # (256K, no YaRN). Full 256K @ q8_0 KV β‰ˆ 8.5G. q8_0 verified coherent; f16 in .env if gibberish. + # NEO-CODE = Qwen3.6-27B GDN-hybrid (16 of 64 layers cache KV β†’ KV cheap); native ctx 262144 + # (256K). Full 256K @ q8_0 KV β‰ˆ 8.6G, fits GPU0 w/ ~3.8G margin. q8_0 coherent; f16 in .env if gibberish. - --cache-type-k - ${CHARRP_REASONING_KV_TYPE:-q8_0} - --cache-type-v - ${CHARRP_REASONING_KV_TYPE:-q8_0} - --jinja - # Deckard's Qwen3.5 template natively opens + has enable_thinking β†’ llama.cpp - # manages the reasoning (trace to reasoning_content, content stays clean prose); - # --reasoning-budget caps the CoT. (A/B 2026-07-08: Deckard 0/30 loops + 0/30 refusals, - # beat RpR-v4 (1/30 loop), Pantheon (7/30 refusals), Snowdrop + Gembrain (template-incompat).) + # NEO-CODE's Qwen3.6 template natively opens β†’ llama.cpp manages the reasoning + # (trace to reasoning_content, content stays clean prose); --reasoning-budget caps the CoT. + # (R36 gate 2026-07-14: NEO-CODE composite 0.922 tool-calling + 0.0 runaway β€” beat Deckard + # 0.08/0.80 and gen-reasoning 0.856. Budget held at 400: latency-coupled to soong's client timeout.) - --reasoning - on - --reasoning-format - deepseek - --reasoning-budget - "${CHARRP_REASONING_BUDGET:-400}" - # DRY anti-repetition (Deckard/Qwen3.5 tolerates it; sampler ORDER = dry AFTER temperature, - # per the QwQ-family finding that a repetition PENALTY worsens looping but DRY fixes it). - - --samplers - - "top_k;top_p;min_p;temperature;dry" - - --dry-multiplier - - "${CHARRP_REASONING_DRY:-0.8}" - - --dry-base - - "1.75" - - --dry-allowed-length - - "2" + # Sampler defaults per the DavidAU/Qwen3.6 model card (thinking-mode, general tasks): temp 1.0, + # top_p 0.95, top_k 20, min_p 0.0, no rep-penalty, no DRY (DRY was a QwQ/Deckard looping band-aid + # NEO-CODE doesn't need). All tunable via .env. NOTE: 0.922 tool-gate was on the OLD Deckard + # samplers (effective temp~0.8 + DRY); re-validate tools + slop on these card samplers. + - --temp + - "${CHARRP_REASONING_TEMP:-1.0}" + - --top-p + - "${CHARRP_REASONING_TOP_P:-0.95}" + - --top-k + - "${CHARRP_REASONING_TOP_K:-20}" + - --min-p + - "${CHARRP_REASONING_MIN_P:-0.0}" healthcheck: test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health >/dev/null || exit 1"] interval: 30s