vh
85792f4b55
The char-rp-reasoning (Deckard) seat now runs llamacpp-charrp:custom-latest via a new LLAMA_REASONING_IMAGE var (Magidonia char-rp stays on stock — no reasoning bug). Fixes Worldtree #355 at the source (budget multi-terminator handles Qwen3.5's <tool_call> reasoning end-tag). Live 2026-07-13: Deckard loads on Blackwell, serves coherent, reasoning bounds at the 400 budget. Rollback via .env LLAMA_REASONING_IMAGE.
103 lines
5.0 KiB
Markdown
103 lines
5.0 KiB
Markdown
# ⚠️ CUSTOM llama.cpp build for the char-rp-reasoning seat
|
|
|
|
> **STATUS (2026-07-13):** image `llamacpp-charrp:6eddde0-pr25544` (+ `:custom-latest`)
|
|
> BUILT on ana-ml2 (reports `version: 10001 (6ce128d0b)`, CUDA 12.8.1, sm_120) and
|
|
> binary-smoke-tested OK. **DEPLOYED 2026-07-13** — the live `char-rp-gguf`
|
|
> reasoning seat runs `llamacpp-charrp:custom-latest` (via `LLAMA_REASONING_IMAGE`);
|
|
> Deckard loads + serves coherent output + reasoning bounds at the 400 budget.
|
|
> Definitive tool-path proof = production monitoring (spend_logs `completion_tokens=32768`
|
|
> runaways should stop; baseline was ~13%/~4 today pre-swap). b61's 300s wall-clock backstops meanwhile.
|
|
|
|
**The `llama-charrp-reasoning` seat (Deckard-PKD, ana-ml2:8018) runs a
|
|
CUSTOM-BUILT llama.cpp, NOT a stock upstream image.** If you are upgrading,
|
|
rebuilding, or "cleaning up" the seat, read this first — a naive bump to a
|
|
stock image will silently reintroduce Worldtree #355.
|
|
|
|
## What it is
|
|
|
|
`llamacpp-charrp:custom-latest` = **llama.cpp master `6eddde0`** (2026-07-13,
|
|
~b9990 era) **+ unmerged upstream PR [#25544]** (`aldehir:reasoning-budget-multi-seq`),
|
|
built for Blackwell (sm_120) via `build.sh` in this directory.
|
|
|
|
Merged tree HEAD (on ana-ml2 `/home/lkraven/llamacpp-build`): `6ce128d0b`.
|
|
|
|
[#25544]: https://github.com/ggml-org/llama.cpp/pull/25544
|
|
|
|
## Why it exists (Worldtree #355 root cause)
|
|
|
|
Stock llama.cpp **b8840** (the previous seat build, `ghcr.io/mostlygeek/llama-swap:cuda`)
|
|
— and every *released* build to date — has a reasoning-budget sampler that
|
|
recognises only **one** end-of-thinking tag, `</think>`. But Qwen3.5's
|
|
tool-call path ends its reasoning block with **`<tool_call>`**, a terminator
|
|
the single-tag sampler can't match. So on the agentic tool-retry path,
|
|
`--reasoning-budget 400`'s forced-close **never fires**, and the model loops
|
|
in reasoning all the way to `max_tokens` (32768) ≈ **22 min per turn** (~13%
|
|
of char-rp-reasoning turns). That is #355's residual hang.
|
|
|
|
Evidence that pinned it (all three agree): LiteLLM spend_logs (4 requests at
|
|
exactly `completion_tokens=32768`), the seat's own log (`158 activated / 0
|
|
forced-close / 110 natural-end`), and the WT↔gateway pcap (runaway stream is
|
|
100 % `reasoning_content`, coherent tool-retry planning, never exits `<think>`).
|
|
|
|
**PR #25544 is the fix**: it extends the budget sampler to **multiple
|
|
terminating sequences** (`</think>` *or* `<tool_call>`) and replays the matched
|
|
end-sequence when the budget expires. It is **OPEN / unmerged** upstream
|
|
(the underlying trigger #22684 is closed-as-not-planned; the tool-parsing PR
|
|
#24202 is also open), so there is **no released build that fixes our bug** —
|
|
hence this local build.
|
|
|
|
## Conflict resolution (the one thing to re-verify on any rebump)
|
|
|
|
The PR (based on master of 2026-07-10) is 25 commits behind `6eddde0`; the
|
|
merge has a single conflict in `tools/server/server-common.cpp`. Resolution
|
|
(baked into `build.sh`): keep the **PR's plural `reasoning_budget_end_tags`**
|
|
(the fix) **and master's per-request body-read of `reasoning_budget_message`**
|
|
(a newer master feature). Both improvements retained.
|
|
|
|
## Build / rebuild
|
|
|
|
```bash
|
|
# on ana-ml2 (has the Blackwell GPUs + docker + disk):
|
|
scp stacks/char-rp-gguf/llamacpp-custom/build.sh ana-ml2:/home/lkraven/
|
|
ssh ana-ml2 'bash /home/lkraven/build-charrp.sh' # ~20-40 min, sm_120 compile
|
|
```
|
|
|
|
Then deploy via the seat compose (`stacks/char-rp-gguf/compose.yaml`, service
|
|
`llama-charrp-reasoning`), whose `image:` is pinned to `llamacpp-charrp:custom-latest`.
|
|
|
|
## Acceptance test before trusting a build (do NOT skip)
|
|
|
|
The bug is specifically that forcing *never fires*. A build is only good if it
|
|
demonstrably **force-closes** — not just "loads + serves":
|
|
|
|
1. `tests/test-reasoning-budget.cpp` passes (`ctest -R reasoning-budget`).
|
|
2. Live low-budget check on Deckard: launch with `--reasoning-budget 20`, send a
|
|
tools-bearing request that induces a tool-retry loop, and confirm the seat
|
|
logs a **forced** close (budget-reached), not just `deactivated (natural end)`,
|
|
and the response stops near the budget instead of running to `max_tokens`.
|
|
|
|
Then restore `--reasoning-budget 400` for production.
|
|
|
|
## Rollback
|
|
|
|
The previous stock image `ghcr.io/mostlygeek/llama-swap:cuda` (build 8840) is
|
|
preserved on ana-ml2 (imageID `d6c39f55…`). To roll back: point the seat's
|
|
`image:` back to it and recreate. 8840 works (RP is fine) — it only has the
|
|
forcing bug.
|
|
|
|
## 🔔 Retiring this custom build (REMOVE-WHEN-MERGED)
|
|
|
|
This is a temporary local build carrying an unmerged PR. **When PR #25544
|
|
merges upstream and lands in a stock release:**
|
|
|
|
1. Confirm the released build's `tests/test-reasoning-budget.cpp` covers the
|
|
`<tool_call>` terminator and passes on Qwen3.5.
|
|
2. Switch the seat `image:` to that stock image (e.g. a
|
|
`ghcr.io/ggml-org/llama.cpp:server-cuda` tag ≥ the merge) and run the
|
|
acceptance test above.
|
|
3. Delete this directory + the compose `⚠️ CUSTOM BUILD` note, and update
|
|
`persistent-memory.md` + the `reference_charrp_custom_llamacpp_pr25544`
|
|
auto-memory.
|
|
|
|
Check status: <https://github.com/ggml-org/llama.cpp/pull/25544>.
|