esh-pfi-infrastructure/stacks/char-rp-gguf/llamacpp-custom

⚠️ CUSTOM llama.cpp build for the char-rp-reasoning seat

STATUS (2026-07-13): image llamacpp-charrp:6eddde0-pr25544 (+ :custom-latest) BUILT on ana-ml2 (reports version: 10001 (6ce128d0b), CUDA 12.8.1, sm_120) and binary-smoke-tested OK. DEPLOYED 2026-07-13 — the live char-rp-gguf reasoning seat runs llamacpp-charrp:custom-latest (via LLAMA_REASONING_IMAGE); Deckard loads + serves coherent output + reasoning bounds at the 400 budget. Definitive tool-path proof = production monitoring (spend_logs completion_tokens=32768 runaways should stop; baseline was ~13%/~4 today pre-swap). b61's 300s wall-clock backstops meanwhile.

The llama-charrp-reasoning seat (Deckard-PKD, ana-ml2:8018) runs a CUSTOM-BUILT llama.cpp, NOT a stock upstream image. If you are upgrading, rebuilding, or "cleaning up" the seat, read this first — a naive bump to a stock image will silently reintroduce Worldtree #355.

What it is

llamacpp-charrp:custom-latest = llama.cpp master 6eddde0 (2026-07-13, ~b9990 era) + unmerged upstream PR #25544 (aldehir:reasoning-budget-multi-seq), built for Blackwell (sm_120) via build.sh in this directory.

Merged tree HEAD (on ana-ml2 /home/lkraven/llamacpp-build): 6ce128d0b.

Why it exists (Worldtree #355 root cause)

Stock llama.cpp b8840 (the previous seat build, ghcr.io/mostlygeek/llama-swap:cuda) — and every released build to date — has a reasoning-budget sampler that recognises only one end-of-thinking tag, </think>. But Qwen3.5's tool-call path ends its reasoning block with <tool_call>, a terminator the single-tag sampler can't match. So on the agentic tool-retry path, --reasoning-budget 400's forced-close never fires, and the model loops in reasoning all the way to max_tokens (32768) ≈ 22 min per turn (~13% of char-rp-reasoning turns). That is #355's residual hang.

Evidence that pinned it (all three agree): LiteLLM spend_logs (4 requests at exactly completion_tokens=32768), the seat's own log (158 activated / 0 forced-close / 110 natural-end), and the WT↔gateway pcap (runaway stream is 100 % reasoning_content, coherent tool-retry planning, never exits <think>).

PR #25544 is the fix: it extends the budget sampler to multiple terminating sequences (</think> or <tool_call>) and replays the matched end-sequence when the budget expires. It is OPEN / unmerged upstream (the underlying trigger #22684 is closed-as-not-planned; the tool-parsing PR #24202 is also open), so there is no released build that fixes our bug — hence this local build.

Conflict resolution (the one thing to re-verify on any rebump)

The PR (based on master of 2026-07-10) is 25 commits behind 6eddde0; the merge has a single conflict in tools/server/server-common.cpp. Resolution (baked into build.sh): keep the PR's plural reasoning_budget_end_tags (the fix) and master's per-request body-read of reasoning_budget_message (a newer master feature). Both improvements retained.

Build / rebuild

# on ana-ml2 (has the Blackwell GPUs + docker + disk):
scp stacks/char-rp-gguf/llamacpp-custom/build.sh ana-ml2:/home/lkraven/
ssh ana-ml2 'bash /home/lkraven/build-charrp.sh'   # ~20-40 min, sm_120 compile

Then deploy via the seat compose (stacks/char-rp-gguf/compose.yaml, service llama-charrp-reasoning), whose image: is pinned to llamacpp-charrp:custom-latest.

Acceptance test before trusting a build (do NOT skip)

The bug is specifically that forcing never fires. A build is only good if it demonstrably force-closes — not just "loads + serves":

1. tests/test-reasoning-budget.cpp passes (ctest -R reasoning-budget).
2. Live low-budget check on Deckard: launch with --reasoning-budget 20, send a tools-bearing request that induces a tool-retry loop, and confirm the seat logs a forced close (budget-reached), not just deactivated (natural end), and the response stops near the budget instead of running to max_tokens.

Then restore --reasoning-budget 400 for production.

Rollback

The previous stock image ghcr.io/mostlygeek/llama-swap:cuda (build 8840) is preserved on ana-ml2 (imageID d6c39f55…). To roll back: point the seat's image: back to it and recreate. 8840 works (RP is fine) — it only has the forcing bug.

🔔 Retiring this custom build (REMOVE-WHEN-MERGED)

This is a temporary local build carrying an unmerged PR. When PR #25544 merges upstream and lands in a stock release:

1. Confirm the released build's tests/test-reasoning-budget.cpp covers the <tool_call> terminator and passes on Qwen3.5.
2. Switch the seat image: to that stock image (e.g. a ghcr.io/ggml-org/llama.cpp:server-cuda tag ≥ the merge) and run the acceptance test above.
3. Delete this directory + the compose ⚠️ CUSTOM BUILD note, and update persistent-memory.md + the reference_charrp_custom_llamacpp_pr25544 auto-memory.

Check status: https://github.com/ggml-org/llama.cpp/pull/25544.