From 85792f4b55f10f7b98d4b454bdc32cda8b30433e Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 13 Jul 2026 13:47:42 -0700 Subject: [PATCH] feat(char-rp-gguf): swap reasoning seat to custom llama.cpp (master 6eddde0 + PR #25544) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The char-rp-reasoning (Deckard) seat now runs llamacpp-charrp:custom-latest via a new LLAMA_REASONING_IMAGE var (Magidonia char-rp stays on stock — no reasoning bug). Fixes Worldtree #355 at the source (budget multi-terminator handles Qwen3.5's reasoning end-tag). Live 2026-07-13: Deckard loads on Blackwell, serves coherent, reasoning bounds at the 400 budget. Rollback via .env LLAMA_REASONING_IMAGE. --- persistent-memory.md | 2 +- stacks/char-rp-gguf/compose.yaml | 8 +++++++- stacks/char-rp-gguf/llamacpp-custom/README.md | 8 +++++--- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index d34e75a..3cddd8c 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -108,7 +108,7 @@ _As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway - **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** → ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle — the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all → no semaphore to leak). - **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits ``. - **MECHANISM — RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≤0) → so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate ``, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) → brokkr's watchdog needs the bump too. -- **FIX (agreed with brokkr, NO max_tokens ceiling per operator):** (1) IMMEDIATE relief = push worldtree's **b61 270s wall-clock** — bounds the user-facing hang regardless of the seat loop (so the seat bump is NOT urgent). (2) DIAGNOSE-THEN-FIX (Eitri refinement, may SAVE the bump) — first capture the detailed tuple `tokens/start/end/forced` (needs a **verbose / low-budget-matrix seat restart**; 8840's default log has only the simpler activated/deactivated line — no `forced=`/`start=…end=`, confirmed by grep): **`end=0` → the FORCED-injection end_tag isn't reaching the budget path (Deckard template / reasoning-format mismatch) → CHEAP CONFIG fix = explicit budget start/end tags, a restart NOT a bump**; `end>0,forced=0` → end-tag tokenization bug; `end>0 + no forcing across arms` → real state bug → build bump. (NB: 110 `deactivated (natural end)` prove the DETECTION end-tag works, so an `end=0` would be specifically the FORCED-inject tag, a narrow config miss.) Only if the tuple says end>0 → **DURABLE = stage + validate a post-#22740** llama.cpp build OFF the hot path (⚠️ #22717 garbled-forcing regression ~b9018 fixed by #22740 → don't land in the b8840→b9018 window; validate vs `tests/test-reasoning-budget.cpp` + that it loads Deckard Q5_K_M), then surface the seat-image swap to Vuong; `--reasoning-budget-message` additive WITH the bump. Getting the tuple = the disruptive step (bounce the prod RP seat to --verbose/low-budget; a separate test instance won't fit — GPU0 has only ~4G margin). **brokkr FINAL rec (thread wrapped 2026-07-13): go STRAIGHT to the staged bump — the tuple-diagnostic / tag-probe is a LONG-SHOT (the 110 natural closes prove the parser knows the delimiter → more likely a build/state bug than a tag-carry miss). **RESOLVED WITHOUT the bounce (brokkr msg16 deduction): the `deactivated (natural end)` log is the budget SAMPLER's namespace → the sampler matched its `reasoning_budget_end` token → end_tag NON-empty (110×); and per source `forced = tokenize(message+end_tag)` is built in the SAME block → forced also NON-empty. So the tuple is logically end>0/forced>0 → forced tokens EXIST, forcing just never FIRES = STATE-MACHINE bug → the post-#22740 BUILD BUMP. The `end=0` cheap-config branch is RULED OUT. NO diagnostic bounce needed (would only confirm) → no prod-seat-restart go required from Vuong for diagnosis.** XTC HELD. b61 = the immediate relief → NO rush. Deliverable owed to brokkr = the validated bump-target (loads Deckard Q5_K_M + passes tests/test-reasoning-budget.cpp AND — the real acceptance test — a **live low-budget check on the Qwen3.5/Deckard arch showing an ACTUAL forced-close, not just natural ends**, since a forced-close-that-never-fires is exactly what 8840 gets wrong); Dvalin standing by for a 2nd read, Eitri if arch specifics get hairy.** **VERIFICATION (2026-07-13, WebFetch ggml-org/llama.cpp — the build-bump has NO released target; the dwarf "bump to post-#22740" was WRONG, #22740 fixed a DIFFERENT regression #22717):** the real fixes are UNMERGED — #22684 (Qwen3.5/3.6 tool-call-in-`reasoning_content` = our TRIGGER) is **CLOSED-AS-NOT-PLANNED**; #24202 (Qwen tool-parsing) is **OPEN**; **#25544** (reasoning-budget sampler → MULTIPLE terminating sequences) is **OPEN, awaiting review** = OUR forcing fix. MECHANISM NAILED: 8840's budget sampler knows only a SINGLE end tag (``), but Qwen3.5's tool-path terminates reasoning with `` which the sampler can't match → forcing never fires on the tool path (explains 110 natural-`` closes working + 0 forced + tool-retry specificity). **RE-FRAMED PLAN: no bump target exists yet → b61's 270s wall-clock is the ACTUAL mitigation (not just interim); TRACK PR #25544 and bump only when it merges into a release + validate the forced-close empirically. Seat XTC = optional loop-frequency fallback, not a fix.** **BUILT 2026-07-13 (operator directed the custom build after we verified NO released fix exists): image `llamacpp-charrp:6eddde0-pr25544`/`:custom-latest` on ana-ml2 (reports b10001, merged HEAD `6ce128d0b` = master 6eddde0 + PR #25544, CUDA 12.8.1 sm_120, server target `/app/llama-server`), binary smoke-tested OK. Reproducible recipe + records: eshpfi `stacks/char-rp-gguf/llamacpp-custom/` (build.sh + README) + auto-memory `charrp-custom-llamacpp-pr25544`. **SEAT SWAP PENDING operator go on timing** (interrupts live Worldtree-personal RP ~2-3min; rollback = stock 8840 `ghcr.io/mostlygeek/llama-swap:cuda` imageID d6c39f55, preserved). On-seat acceptance = Deckard loads + serves + a FORCED close fires at low budget (not just natural ends). #355 DEPLOY-SPEED task separately CONFIRMED landed: #359 merged→build-and-deploy 11m→~5m (~2×, wt-dev corroborated 5m/4m vs 11m); b61 pushed (v1.0.0b61 747d7a0) = 300s client backstop live on deploy.** (3) Optional interim = mild XTC on the seat (behavior-shaping, not enforcement) — holding unless the loop-frequency reduction is wanted pre-bump. Seat config: canonical `stacks/char-rp-gguf/compose.yaml` (service `llama-charrp-reasoning`), live `/opt/docker/compose/char-rp-gguf/` on ana-ml2. **Cheap `--reasoning-budget-message`-alone test is DEAD (source-confirmed won't fix it) — did NOT waste a restart on it.** +- **FIX (agreed with brokkr, NO max_tokens ceiling per operator):** (1) IMMEDIATE relief = push worldtree's **b61 270s wall-clock** — bounds the user-facing hang regardless of the seat loop (so the seat bump is NOT urgent). (2) DIAGNOSE-THEN-FIX (Eitri refinement, may SAVE the bump) — first capture the detailed tuple `tokens/start/end/forced` (needs a **verbose / low-budget-matrix seat restart**; 8840's default log has only the simpler activated/deactivated line — no `forced=`/`start=…end=`, confirmed by grep): **`end=0` → the FORCED-injection end_tag isn't reaching the budget path (Deckard template / reasoning-format mismatch) → CHEAP CONFIG fix = explicit budget start/end tags, a restart NOT a bump**; `end>0,forced=0` → end-tag tokenization bug; `end>0 + no forcing across arms` → real state bug → build bump. (NB: 110 `deactivated (natural end)` prove the DETECTION end-tag works, so an `end=0` would be specifically the FORCED-inject tag, a narrow config miss.) Only if the tuple says end>0 → **DURABLE = stage + validate a post-#22740** llama.cpp build OFF the hot path (⚠️ #22717 garbled-forcing regression ~b9018 fixed by #22740 → don't land in the b8840→b9018 window; validate vs `tests/test-reasoning-budget.cpp` + that it loads Deckard Q5_K_M), then surface the seat-image swap to Vuong; `--reasoning-budget-message` additive WITH the bump. Getting the tuple = the disruptive step (bounce the prod RP seat to --verbose/low-budget; a separate test instance won't fit — GPU0 has only ~4G margin). **brokkr FINAL rec (thread wrapped 2026-07-13): go STRAIGHT to the staged bump — the tuple-diagnostic / tag-probe is a LONG-SHOT (the 110 natural closes prove the parser knows the delimiter → more likely a build/state bug than a tag-carry miss). **RESOLVED WITHOUT the bounce (brokkr msg16 deduction): the `deactivated (natural end)` log is the budget SAMPLER's namespace → the sampler matched its `reasoning_budget_end` token → end_tag NON-empty (110×); and per source `forced = tokenize(message+end_tag)` is built in the SAME block → forced also NON-empty. So the tuple is logically end>0/forced>0 → forced tokens EXIST, forcing just never FIRES = STATE-MACHINE bug → the post-#22740 BUILD BUMP. The `end=0` cheap-config branch is RULED OUT. NO diagnostic bounce needed (would only confirm) → no prod-seat-restart go required from Vuong for diagnosis.** XTC HELD. b61 = the immediate relief → NO rush. Deliverable owed to brokkr = the validated bump-target (loads Deckard Q5_K_M + passes tests/test-reasoning-budget.cpp AND — the real acceptance test — a **live low-budget check on the Qwen3.5/Deckard arch showing an ACTUAL forced-close, not just natural ends**, since a forced-close-that-never-fires is exactly what 8840 gets wrong); Dvalin standing by for a 2nd read, Eitri if arch specifics get hairy.** **VERIFICATION (2026-07-13, WebFetch ggml-org/llama.cpp — the build-bump has NO released target; the dwarf "bump to post-#22740" was WRONG, #22740 fixed a DIFFERENT regression #22717):** the real fixes are UNMERGED — #22684 (Qwen3.5/3.6 tool-call-in-`reasoning_content` = our TRIGGER) is **CLOSED-AS-NOT-PLANNED**; #24202 (Qwen tool-parsing) is **OPEN**; **#25544** (reasoning-budget sampler → MULTIPLE terminating sequences) is **OPEN, awaiting review** = OUR forcing fix. MECHANISM NAILED: 8840's budget sampler knows only a SINGLE end tag (``), but Qwen3.5's tool-path terminates reasoning with `` which the sampler can't match → forcing never fires on the tool path (explains 110 natural-`` closes working + 0 forced + tool-retry specificity). **RE-FRAMED PLAN: no bump target exists yet → b61's 270s wall-clock is the ACTUAL mitigation (not just interim); TRACK PR #25544 and bump only when it merges into a release + validate the forced-close empirically. Seat XTC = optional loop-frequency fallback, not a fix.** **BUILT 2026-07-13 (operator directed the custom build after we verified NO released fix exists): image `llamacpp-charrp:6eddde0-pr25544`/`:custom-latest` on ana-ml2 (reports b10001, merged HEAD `6ce128d0b` = master 6eddde0 + PR #25544, CUDA 12.8.1 sm_120, server target `/app/llama-server`), binary smoke-tested OK. Reproducible recipe + records: eshpfi `stacks/char-rp-gguf/llamacpp-custom/` (build.sh + README) + auto-memory `charrp-custom-llamacpp-pr25544`. **SWAPPED + LIVE 2026-07-13 (operator: swap now)** — reasoning seat recreated on `llamacpp-charrp:custom-latest` (via new `LLAMA_REASONING_IMAGE` compose var; Magidonia char-rp stays stock); Deckard loads clean on Blackwell, serves coherent output, reasoning bounds at 400 (non-tool forcing intact). Per-request `reasoning_budget` override IGNORED (CLI-400 pins it, expected). New build's log DROPPED the `reasoning-budget: activated` line (b10001 format change) → can't watch forcing via seat log; DEFINITIVE tool-path proof = spend_logs `completion_tokens=32768` runaways STOP (baseline 4 today pre-swap). Rollback = set `LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda` in `/opt/docker/compose/char-rp-gguf/.env` + `docker compose up -d --no-deps llama-charrp-reasoning` (8840 image d6c39f55 preserved). On-seat acceptance = Deckard loads + serves + a FORCED close fires at low budget (not just natural ends). #355 DEPLOY-SPEED task separately CONFIRMED landed: #359 merged→build-and-deploy 11m→~5m (~2×, wt-dev corroborated 5m/4m vs 11m); b61 pushed (v1.0.0b61 747d7a0) = 300s client backstop live on deploy.** (3) Optional interim = mild XTC on the seat (behavior-shaping, not enforcement) — holding unless the loop-frequency reduction is wanted pre-bump. Seat config: canonical `stacks/char-rp-gguf/compose.yaml` (service `llama-charrp-reasoning`), live `/opt/docker/compose/char-rp-gguf/` on ana-ml2. **Cheap `--reasoning-budget-message`-alone test is DEAD (source-confirmed won't fix it) — did NOT waste a restart on it.** - **Operator direction:** find the SERVER-side fix (params/config), do NOT ceiling max_tokens (legit long outputs must survive). Routed to brokkr-smithy-dev (thread 01KXEEQ0TD6D…) → he accepted, pulled dvalin for the known-bug search, running the dwarf consult. I offered to run the live-seat with/without-tools split-test on his signal (own the seat; mildly disruptive → off-peak). - **Wasted-GPU side issue + b61 re-measure:** LiteLLM logged these `success` at full 1255–1390s → on b60 it did NOT abort at WT's cancel (whatever WT did, no upstream abort → ~18min wasted GPU/runaway). worldtree-dev's **unpushed b61** adds a **270s provider wall-clock** that runs `_bounded_stream_close`→httpx FIN (force-closes WT's upstream side). So post-b61 the FIN-check flips to "WT closes at ~270s"; the only open question is whether **LiteLLM propagates that client-disconnect to the seat** (my gateway domain). **RE-MEASURE after b61 deploys** — don't assume b60 behavior. - **Data on hand:** pcaps at `corviduo-dev:~infra-ops/py-spy-355/wedge_*/` are only ~99s windows triggered on the attach_tool failure (too short for the 300s cancel — a limitation, not usable for the FIN-check). LiteLLM config `/opt/docker/conf/litellm/config.yaml` → `/app/config.yaml`; spend_logs = `litellm-db` (`docker exec litellm-db psql -U litellm -d litellm`; camelCase cols double-quoted; request body in `proxy_server_request::jsonb->>'max_tokens'` top-level). diff --git a/stacks/char-rp-gguf/compose.yaml b/stacks/char-rp-gguf/compose.yaml index 46e4a3f..c1fc2de 100644 --- a/stacks/char-rp-gguf/compose.yaml +++ b/stacks/char-rp-gguf/compose.yaml @@ -96,7 +96,13 @@ services: # ── REASONING seat — Deckard-PKD (Qwen3.5) managed thinking + DRY. gateway char-rp-reasoning. ── llama-charrp-reasoning: - image: ${LLAMA_IMAGE:-ghcr.io/mostlygeek/llama-swap:cuda} + # ⚠️ CUSTOM llama.cpp build (master 6eddde0 + unmerged PR #25544) — the Worldtree + # #355 reasoning-budget-forcing fix (multi-terminator: handles Qwen3.5's + # reasoning end-tag, which stock b8840's single- budget can't match → the seat + # loops in reasoning to max_tokens ≈ 22 min). DO NOT revert to a stock image until + # #25544 merges upstream. Build recipe + why + rollback: ./llamacpp-custom/README.md. + # Rollback: set LLAMA_REASONING_IMAGE=ghcr.io/mostlygeek/llama-swap:cuda in .env + recreate. + image: ${LLAMA_REASONING_IMAGE:-llamacpp-charrp:custom-latest} container_name: ${CHARRP_REASONING_CONTAINER:-llama-charrp-reasoning} restart: unless-stopped runtime: nvidia diff --git a/stacks/char-rp-gguf/llamacpp-custom/README.md b/stacks/char-rp-gguf/llamacpp-custom/README.md index cb818c6..eb2795b 100644 --- a/stacks/char-rp-gguf/llamacpp-custom/README.md +++ b/stacks/char-rp-gguf/llamacpp-custom/README.md @@ -2,9 +2,11 @@ > **STATUS (2026-07-13):** image `llamacpp-charrp:6eddde0-pr25544` (+ `:custom-latest`) > BUILT on ana-ml2 (reports `version: 10001 (6ce128d0b)`, CUDA 12.8.1, sm_120) and -> binary-smoke-tested OK. **Seat swap PENDING** — the live `char-rp-gguf` compose -> still points the reasoning seat at stock 8840 until the swap + on-seat forced-close -> validation. Once swapped, update this line to DEPLOYED. +> binary-smoke-tested OK. **DEPLOYED 2026-07-13** — the live `char-rp-gguf` +> reasoning seat runs `llamacpp-charrp:custom-latest` (via `LLAMA_REASONING_IMAGE`); +> Deckard loads + serves coherent output + reasoning bounds at the 400 budget. +> Definitive tool-path proof = production monitoring (spend_logs `completion_tokens=32768` +> runaways should stop; baseline was ~13%/~4 today pre-swap). b61's 300s wall-clock backstops meanwhile. **The `llama-charrp-reasoning` seat (Deckard-PKD, ana-ml2:8018) runs a CUSTOM-BUILT llama.cpp, NOT a stock upstream image.** If you are upgrading,