docs(char-rp-gguf): record custom llama.cpp build (master 6eddde0 + unmerged PR #25544)
Durable record of the custom llama.cpp the char-rp-reasoning seat will run to fix Worldtree #355 (reasoning-budget forcing broken in stock b8840 — single end-tag </think> can't match Qwen3.5's <tool_call> reasoning terminator, so the budget never force-closes and reasoning runs away to max_tokens). PR #25544 adds multiple terminating sequences; unmerged upstream, so we build it. - build.sh: reproducible recipe (clone master@6eddde0 + merge PR #25544 + resolve the 1 server-common.cpp conflict + CUDA build for Blackwell sm_120) - README.md: why + acceptance test + rollback + REMOVE-WHEN-MERGED tracking Image llamacpp-charrp:6eddde0-pr25544 BUILT + smoke-tested on ana-ml2; seat swap pending. See also auto-memory reference_charrp_custom_llamacpp_pr25544.
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
# ⚠️ CUSTOM llama.cpp build for the char-rp-reasoning seat
|
||||
|
||||
> **STATUS (2026-07-13):** image `llamacpp-charrp:6eddde0-pr25544` (+ `:custom-latest`)
|
||||
> BUILT on ana-ml2 (reports `version: 10001 (6ce128d0b)`, CUDA 12.8.1, sm_120) and
|
||||
> binary-smoke-tested OK. **Seat swap PENDING** — the live `char-rp-gguf` compose
|
||||
> still points the reasoning seat at stock 8840 until the swap + on-seat forced-close
|
||||
> validation. Once swapped, update this line to DEPLOYED.
|
||||
|
||||
**The `llama-charrp-reasoning` seat (Deckard-PKD, ana-ml2:8018) runs a
|
||||
CUSTOM-BUILT llama.cpp, NOT a stock upstream image.** If you are upgrading,
|
||||
rebuilding, or "cleaning up" the seat, read this first — a naive bump to a
|
||||
stock image will silently reintroduce Worldtree #355.
|
||||
|
||||
## What it is
|
||||
|
||||
`llamacpp-charrp:custom-latest` = **llama.cpp master `6eddde0`** (2026-07-13,
|
||||
~b9990 era) **+ unmerged upstream PR [#25544]** (`aldehir:reasoning-budget-multi-seq`),
|
||||
built for Blackwell (sm_120) via `build.sh` in this directory.
|
||||
|
||||
Merged tree HEAD (on ana-ml2 `/home/lkraven/llamacpp-build`): `6ce128d0b`.
|
||||
|
||||
[#25544]: https://github.com/ggml-org/llama.cpp/pull/25544
|
||||
|
||||
## Why it exists (Worldtree #355 root cause)
|
||||
|
||||
Stock llama.cpp **b8840** (the previous seat build, `ghcr.io/mostlygeek/llama-swap:cuda`)
|
||||
— and every *released* build to date — has a reasoning-budget sampler that
|
||||
recognises only **one** end-of-thinking tag, `</think>`. But Qwen3.5's
|
||||
tool-call path ends its reasoning block with **`<tool_call>`**, a terminator
|
||||
the single-tag sampler can't match. So on the agentic tool-retry path,
|
||||
`--reasoning-budget 400`'s forced-close **never fires**, and the model loops
|
||||
in reasoning all the way to `max_tokens` (32768) ≈ **22 min per turn** (~13%
|
||||
of char-rp-reasoning turns). That is #355's residual hang.
|
||||
|
||||
Evidence that pinned it (all three agree): LiteLLM spend_logs (4 requests at
|
||||
exactly `completion_tokens=32768`), the seat's own log (`158 activated / 0
|
||||
forced-close / 110 natural-end`), and the WT↔gateway pcap (runaway stream is
|
||||
100 % `reasoning_content`, coherent tool-retry planning, never exits `<think>`).
|
||||
|
||||
**PR #25544 is the fix**: it extends the budget sampler to **multiple
|
||||
terminating sequences** (`</think>` *or* `<tool_call>`) and replays the matched
|
||||
end-sequence when the budget expires. It is **OPEN / unmerged** upstream
|
||||
(the underlying trigger #22684 is closed-as-not-planned; the tool-parsing PR
|
||||
#24202 is also open), so there is **no released build that fixes our bug** —
|
||||
hence this local build.
|
||||
|
||||
## Conflict resolution (the one thing to re-verify on any rebump)
|
||||
|
||||
The PR (based on master of 2026-07-10) is 25 commits behind `6eddde0`; the
|
||||
merge has a single conflict in `tools/server/server-common.cpp`. Resolution
|
||||
(baked into `build.sh`): keep the **PR's plural `reasoning_budget_end_tags`**
|
||||
(the fix) **and master's per-request body-read of `reasoning_budget_message`**
|
||||
(a newer master feature). Both improvements retained.
|
||||
|
||||
## Build / rebuild
|
||||
|
||||
```bash
|
||||
# on ana-ml2 (has the Blackwell GPUs + docker + disk):
|
||||
scp stacks/char-rp-gguf/llamacpp-custom/build.sh ana-ml2:/home/lkraven/
|
||||
ssh ana-ml2 'bash /home/lkraven/build-charrp.sh' # ~20-40 min, sm_120 compile
|
||||
```
|
||||
|
||||
Then deploy via the seat compose (`stacks/char-rp-gguf/compose.yaml`, service
|
||||
`llama-charrp-reasoning`), whose `image:` is pinned to `llamacpp-charrp:custom-latest`.
|
||||
|
||||
## Acceptance test before trusting a build (do NOT skip)
|
||||
|
||||
The bug is specifically that forcing *never fires*. A build is only good if it
|
||||
demonstrably **force-closes** — not just "loads + serves":
|
||||
|
||||
1. `tests/test-reasoning-budget.cpp` passes (`ctest -R reasoning-budget`).
|
||||
2. Live low-budget check on Deckard: launch with `--reasoning-budget 20`, send a
|
||||
tools-bearing request that induces a tool-retry loop, and confirm the seat
|
||||
logs a **forced** close (budget-reached), not just `deactivated (natural end)`,
|
||||
and the response stops near the budget instead of running to `max_tokens`.
|
||||
|
||||
Then restore `--reasoning-budget 400` for production.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous stock image `ghcr.io/mostlygeek/llama-swap:cuda` (build 8840) is
|
||||
preserved on ana-ml2 (imageID `d6c39f55…`). To roll back: point the seat's
|
||||
`image:` back to it and recreate. 8840 works (RP is fine) — it only has the
|
||||
forcing bug.
|
||||
|
||||
## 🔔 Retiring this custom build (REMOVE-WHEN-MERGED)
|
||||
|
||||
This is a temporary local build carrying an unmerged PR. **When PR #25544
|
||||
merges upstream and lands in a stock release:**
|
||||
|
||||
1. Confirm the released build's `tests/test-reasoning-budget.cpp` covers the
|
||||
`<tool_call>` terminator and passes on Qwen3.5.
|
||||
2. Switch the seat `image:` to that stock image (e.g. a
|
||||
`ghcr.io/ggml-org/llama.cpp:server-cuda` tag ≥ the merge) and run the
|
||||
acceptance test above.
|
||||
3. Delete this directory + the compose `⚠️ CUSTOM BUILD` note, and update
|
||||
`persistent-memory.md` + the `reference_charrp_custom_llamacpp_pr25544`
|
||||
auto-memory.
|
||||
|
||||
Check status: <https://github.com/ggml-org/llama.cpp/pull/25544>.
|
||||
Reference in New Issue
Block a user