docs(char-rp-gguf): record custom llama.cpp build (master 6eddde0 + unmerged PR #25544)
Durable record of the custom llama.cpp the char-rp-reasoning seat will run to fix Worldtree #355 (reasoning-budget forcing broken in stock b8840 — single end-tag </think> can't match Qwen3.5's <tool_call> reasoning terminator, so the budget never force-closes and reasoning runs away to max_tokens). PR #25544 adds multiple terminating sequences; unmerged upstream, so we build it. - build.sh: reproducible recipe (clone master@6eddde0 + merge PR #25544 + resolve the 1 server-common.cpp conflict + CUDA build for Blackwell sm_120) - README.md: why + acceptance test + rollback + REMOVE-WHEN-MERGED tracking Image llamacpp-charrp:6eddde0-pr25544 BUILT + smoke-tested on ana-ml2; seat swap pending. See also auto-memory reference_charrp_custom_llamacpp_pr25544.
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
# ⚠️ CUSTOM llama.cpp build for the char-rp-reasoning seat
|
||||
|
||||
> **STATUS (2026-07-13):** image `llamacpp-charrp:6eddde0-pr25544` (+ `:custom-latest`)
|
||||
> BUILT on ana-ml2 (reports `version: 10001 (6ce128d0b)`, CUDA 12.8.1, sm_120) and
|
||||
> binary-smoke-tested OK. **Seat swap PENDING** — the live `char-rp-gguf` compose
|
||||
> still points the reasoning seat at stock 8840 until the swap + on-seat forced-close
|
||||
> validation. Once swapped, update this line to DEPLOYED.
|
||||
|
||||
**The `llama-charrp-reasoning` seat (Deckard-PKD, ana-ml2:8018) runs a
|
||||
CUSTOM-BUILT llama.cpp, NOT a stock upstream image.** If you are upgrading,
|
||||
rebuilding, or "cleaning up" the seat, read this first — a naive bump to a
|
||||
stock image will silently reintroduce Worldtree #355.
|
||||
|
||||
## What it is
|
||||
|
||||
`llamacpp-charrp:custom-latest` = **llama.cpp master `6eddde0`** (2026-07-13,
|
||||
~b9990 era) **+ unmerged upstream PR [#25544]** (`aldehir:reasoning-budget-multi-seq`),
|
||||
built for Blackwell (sm_120) via `build.sh` in this directory.
|
||||
|
||||
Merged tree HEAD (on ana-ml2 `/home/lkraven/llamacpp-build`): `6ce128d0b`.
|
||||
|
||||
[#25544]: https://github.com/ggml-org/llama.cpp/pull/25544
|
||||
|
||||
## Why it exists (Worldtree #355 root cause)
|
||||
|
||||
Stock llama.cpp **b8840** (the previous seat build, `ghcr.io/mostlygeek/llama-swap:cuda`)
|
||||
— and every *released* build to date — has a reasoning-budget sampler that
|
||||
recognises only **one** end-of-thinking tag, `</think>`. But Qwen3.5's
|
||||
tool-call path ends its reasoning block with **`<tool_call>`**, a terminator
|
||||
the single-tag sampler can't match. So on the agentic tool-retry path,
|
||||
`--reasoning-budget 400`'s forced-close **never fires**, and the model loops
|
||||
in reasoning all the way to `max_tokens` (32768) ≈ **22 min per turn** (~13%
|
||||
of char-rp-reasoning turns). That is #355's residual hang.
|
||||
|
||||
Evidence that pinned it (all three agree): LiteLLM spend_logs (4 requests at
|
||||
exactly `completion_tokens=32768`), the seat's own log (`158 activated / 0
|
||||
forced-close / 110 natural-end`), and the WT↔gateway pcap (runaway stream is
|
||||
100 % `reasoning_content`, coherent tool-retry planning, never exits `<think>`).
|
||||
|
||||
**PR #25544 is the fix**: it extends the budget sampler to **multiple
|
||||
terminating sequences** (`</think>` *or* `<tool_call>`) and replays the matched
|
||||
end-sequence when the budget expires. It is **OPEN / unmerged** upstream
|
||||
(the underlying trigger #22684 is closed-as-not-planned; the tool-parsing PR
|
||||
#24202 is also open), so there is **no released build that fixes our bug** —
|
||||
hence this local build.
|
||||
|
||||
## Conflict resolution (the one thing to re-verify on any rebump)
|
||||
|
||||
The PR (based on master of 2026-07-10) is 25 commits behind `6eddde0`; the
|
||||
merge has a single conflict in `tools/server/server-common.cpp`. Resolution
|
||||
(baked into `build.sh`): keep the **PR's plural `reasoning_budget_end_tags`**
|
||||
(the fix) **and master's per-request body-read of `reasoning_budget_message`**
|
||||
(a newer master feature). Both improvements retained.
|
||||
|
||||
## Build / rebuild
|
||||
|
||||
```bash
|
||||
# on ana-ml2 (has the Blackwell GPUs + docker + disk):
|
||||
scp stacks/char-rp-gguf/llamacpp-custom/build.sh ana-ml2:/home/lkraven/
|
||||
ssh ana-ml2 'bash /home/lkraven/build-charrp.sh' # ~20-40 min, sm_120 compile
|
||||
```
|
||||
|
||||
Then deploy via the seat compose (`stacks/char-rp-gguf/compose.yaml`, service
|
||||
`llama-charrp-reasoning`), whose `image:` is pinned to `llamacpp-charrp:custom-latest`.
|
||||
|
||||
## Acceptance test before trusting a build (do NOT skip)
|
||||
|
||||
The bug is specifically that forcing *never fires*. A build is only good if it
|
||||
demonstrably **force-closes** — not just "loads + serves":
|
||||
|
||||
1. `tests/test-reasoning-budget.cpp` passes (`ctest -R reasoning-budget`).
|
||||
2. Live low-budget check on Deckard: launch with `--reasoning-budget 20`, send a
|
||||
tools-bearing request that induces a tool-retry loop, and confirm the seat
|
||||
logs a **forced** close (budget-reached), not just `deactivated (natural end)`,
|
||||
and the response stops near the budget instead of running to `max_tokens`.
|
||||
|
||||
Then restore `--reasoning-budget 400` for production.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous stock image `ghcr.io/mostlygeek/llama-swap:cuda` (build 8840) is
|
||||
preserved on ana-ml2 (imageID `d6c39f55…`). To roll back: point the seat's
|
||||
`image:` back to it and recreate. 8840 works (RP is fine) — it only has the
|
||||
forcing bug.
|
||||
|
||||
## 🔔 Retiring this custom build (REMOVE-WHEN-MERGED)
|
||||
|
||||
This is a temporary local build carrying an unmerged PR. **When PR #25544
|
||||
merges upstream and lands in a stock release:**
|
||||
|
||||
1. Confirm the released build's `tests/test-reasoning-budget.cpp` covers the
|
||||
`<tool_call>` terminator and passes on Qwen3.5.
|
||||
2. Switch the seat `image:` to that stock image (e.g. a
|
||||
`ghcr.io/ggml-org/llama.cpp:server-cuda` tag ≥ the merge) and run the
|
||||
acceptance test above.
|
||||
3. Delete this directory + the compose `⚠️ CUSTOM BUILD` note, and update
|
||||
`persistent-memory.md` + the `reference_charrp_custom_llamacpp_pr25544`
|
||||
auto-memory.
|
||||
|
||||
Check status: <https://github.com/ggml-org/llama.cpp/pull/25544>.
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Reproducible build of the CUSTOM llama.cpp that backs the char-rp-reasoning
|
||||
# seat (Deckard-PKD, ana-ml2:8018).
|
||||
#
|
||||
# = llama.cpp latest master (pinned 6eddde0, 2026-07-13, ~b9990 era)
|
||||
# + UNMERGED PR #25544 (aldehir:reasoning-budget-multi-seq)
|
||||
#
|
||||
# WHY THIS EXISTS (do not "clean this up" without reading README.md):
|
||||
# Stock llama.cpp b8840 (and every released build to date) has a reasoning-
|
||||
# budget sampler that knows only ONE end-tag (</think>). Qwen3.5's tool-call
|
||||
# path terminates reasoning with <tool_call>, which the single-tag sampler
|
||||
# cannot match, so --reasoning-budget forcing NEVER fires on the tool-retry
|
||||
# path → the model loops in reasoning to max_tokens (32768) ≈ 22 min. That is
|
||||
# Worldtree #355's residual. PR #25544 teaches the budget MULTIPLE terminating
|
||||
# sequences (</think> OR <tool_call>) — the exact fix — but it is UNMERGED
|
||||
# upstream, so we build it ourselves.
|
||||
#
|
||||
# ⚠️ REMOVE-WHEN-MERGED: once #25544 merges upstream and lands in a release,
|
||||
# retire this custom build and switch the seat back to a stock image
|
||||
# (see README.md → "Retiring this custom build").
|
||||
#
|
||||
# Run on ana-ml2 (Blackwell RTX PRO 6000, sm_120; docker + buildx; ~120G free).
|
||||
# Produces: llamacpp-charrp:6eddde0-pr25544 (+ :custom-latest)
|
||||
# Runtime ~20-40 min (CUDA compile for sm_120).
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
MASTER_SHA="${MASTER_SHA:-6eddde0}" # llama.cpp master pin
|
||||
PR="${PR:-25544}" # aldehir:reasoning-budget-multi-seq
|
||||
BUILDDIR="${BUILDDIR:-/home/lkraven/llamacpp-build}"
|
||||
IMAGE="${IMAGE:-llamacpp-charrp}"
|
||||
CUDA_VERSION="${CUDA_VERSION:-12.8.1}" # 12.8+ required for Blackwell sm_120
|
||||
CUDA_ARCH="${CUDA_ARCH:-120}" # RTX PRO 6000 = compute 12.0
|
||||
|
||||
echo "== fetch master@${MASTER_SHA} + PR #${PR} =="
|
||||
rm -rf "$BUILDDIR"
|
||||
git clone https://github.com/ggml-org/llama.cpp "$BUILDDIR"
|
||||
cd "$BUILDDIR"
|
||||
git checkout "$MASTER_SHA"
|
||||
git fetch origin "pull/${PR}/head:pr${PR}"
|
||||
git config user.email infra-ops@phasefinal.com
|
||||
git config user.name infra-ops
|
||||
|
||||
echo "== merge PR #${PR} (expect 1 conflict in server-common.cpp) =="
|
||||
git merge --no-commit --no-ff "pr${PR}" || true
|
||||
# Resolve the single conflict: keep the PR's PLURAL reasoning_budget_end_tags
|
||||
# (the whole point of the fix) AND master's per-request body-read of
|
||||
# reasoning_budget_message (a newer master feature the PR's base predates).
|
||||
python3 - <<'PY'
|
||||
p = "tools/server/server-common.cpp"
|
||||
s = open(p).read()
|
||||
i = s.index("<<<<<<< HEAD")
|
||||
j = s.index(">>>>>>> pr25544") + len(">>>>>>> pr25544")
|
||||
new = (' llama_params["reasoning_budget_end_tags"] = chat_params.thinking_end_tags;\n'
|
||||
' llama_params["reasoning_budget_message"] = json_value(body, "reasoning_budget_message", opt.reasoning_budget_message);')
|
||||
open(p, "w").write(s[:i] + new + s[j:])
|
||||
assert open(p).read().count("<<<<<<<") == 0, "unresolved conflict markers remain"
|
||||
print("resolved server-common.cpp")
|
||||
PY
|
||||
git add -A
|
||||
git commit -m "merge PR #${PR} (reasoning-budget multi-seq) onto master ${MASTER_SHA} — char-rp-reasoning seat custom build"
|
||||
|
||||
echo "== docker build (server target, CUDA ${CUDA_VERSION}, sm_${CUDA_ARCH}) =="
|
||||
docker build -f .devops/cuda.Dockerfile --target server \
|
||||
--build-arg CUDA_VERSION="${CUDA_VERSION}" \
|
||||
--build-arg CUDA_DOCKER_ARCH="${CUDA_ARCH}" \
|
||||
--build-arg APP_VERSION="${MASTER_SHA}-pr${PR}" \
|
||||
--build-arg APP_REVISION="$(git rev-parse HEAD)" \
|
||||
-t "${IMAGE}:${MASTER_SHA}-pr${PR}" \
|
||||
-t "${IMAGE}:custom-latest" \
|
||||
.
|
||||
|
||||
echo "== built ${IMAGE}:${MASTER_SHA}-pr${PR} (entrypoint /app/llama-server, drop-in for the seat) =="
|
||||
Reference in New Issue
Block a user