diff --git a/persistent-memory.md b/persistent-memory.md index f951ba4..d34e75a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -104,20 +104,20 @@ no longer deployed sidecars here. See Recent decisions.) _As of 2026-07-13 β€” two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._ -### πŸ”Ί ACTIVE 1 β€” LiteLLM gateway #355-residual investigation (our domain) -- **The data-confirmed finding:** WT #355's residual 300s hang is NOT the seat and NOT WT-pre-send β€” it's OUR **LiteLLM gateway** (ana-docker `10.250.50.70:4000`) holding two char-rp-reasoning requests **~21 min** (spend_logs dur: **1255.9s + 1357.4s**, startTimes 16:54:44 + 16:56:26Z = the two wedge onsets) while the **Deckard seat sat IDLE**. The seat (ana-ml2:8018 llama.cpp, `--reasoning-budget 400` β€” forecloses an indefinite mid-thinking hang) completes every request **≀72s** and received no request at the wedge onset β†’ **EXONERATED**. This CONTRADICTS worldtree-dev's "seat wedging mid-thinking" premise, which I'd relayed unconfirmed until the operator challenged it. -- **DECISIVE next step (brokkr's dwarf-panel, read-only pcap analysis):** the **FIN-check** β€” in the WT↔gateway pcap for the WEDGED streams, did WT send FIN/RST to the gateway at its 302s stall-cancel? - - WT closed but gateway held anyway β†’ **pure LiteLLM bug** (brokkr's lead: a concurrency-slot LEAK β€” cancelled requests don't free LiteLLM's internal slot β†’ new invokes queue ~21 min behind leaked slots while the seat idles). Our fix. - - WT did NOT close β†’ WT's un-propagated cancel (earlier httpcore shielded-`aclose` can't-kill bug) β†’ worldtree's already-planned force-close fix also releases the hold. -- **Then:** pull LiteLLM router / `max_parallel_requests` config for deployment `hosted_vllm/deckard-pkd-27b`; confirm whether the seat EVER `launch_slot`'d the 1255s request (never-served vs delayed). pcap-only tools: no tshark on corviduo β€” use `tcpdump -r`. -- **Data on hand:** WT-netns pcaps + py-spy dumps at `corviduo-dev:~infra-ops/py-spy-355/wedge_20260713_095626/` (+ 100039/100341/100543); the 095626 capture overlaps turn-2064's wedge (16:54:46–16:59:46Z). **py-spy + tcpdump are installed on corviduo-dev** (`~/.local/bin/py-spy`, hand-carried musl binary; no pip/uv on that box). LiteLLM spend_logs = `litellm-db` on ana-docker (`docker exec litellm printenv DATABASE_URL`; SpendLogs cols are camelCase β†’ double-quote them). -- **Status:** WT-code panel DONE (seat-innocent accepted); residual is LiteLLM-forensic (mine, brokkr reasons alongside). **Awaiting operator go on the FIN-check.** +### πŸ”Ί ACTIVE 1 β€” #355-residual ROOT CAUSE FOUND: seat reasoning-loop to max_tokens (routed to brokkr) +- **DEFINITIVE root cause (3 independent sources, 2026-07-13):** char-rp-reasoning (Deckard-PKD, seat ana-ml2:8018 `llama-charrp-reasoning`) intermittently enters a **non-terminating REASONING loop** and runs to the request's **`max_tokens=32768`** β†’ ~22 min at 24.7 tok/s. ~13% of requests (4 of ~30 in the wedge window). It is NOT the gateway holding, NOT a slot leak, NOT the seat idle β€” the seat genuinely GENERATES 32768 tokens. My earlier "LiteLLM holds ~21 min while seat idles / slot-leak" read was **DISPROVEN** (the seat wasn't idle; there's no `max_parallel_requests`/`router_settings` in the config at all β†’ no semaphore to leak). +- **Evidence:** (a) spend_logs: 4 requests each `completion_tokens = EXACTLY 32768`, status=success (hit the length cap cleanly); normal requests EOS at 730–2250. (b) seat llama.cpp log: `eval time = 1325064ms / 32768 tokens (24.73 tok/s)`, `truncated=0`. (c) WT↔gateway pcap: runaway stream is **100% `reasoning_content` deltas**, coherent, planning tool-call retries ("The tool call failed due to input schema errors… Let me make these calls one at a time"). NEVER exits ``. +- **MECHANISM β€” RESOLVED (Step-1 non-disruptive + Dvalin source read, 2026-07-13):** the budget **ARMS but its force-close NEVER FIRES** in the seat's build. Seat log aggregate: 158 `reasoning-budget: activated, budget=400` / 110 `deactivated (natural end)` / **0 forced closes** / 48 never-closed arms (= the runaways). Build = **llama.cpp 8840 (`9e5647aff`)**, image `ghcr.io/mostlygeek/llama-swap:cuda`. Dvalin read commit 9e5647aff: force-inject is **NOT gated on `--reasoning-budget-message`** (server-task.cpp:493-495 `if (!end_tag.empty())`, no message guard; reasoning-budget.cpp:99-108 fires unconditionally at remaining≀0) β†’ so it's genuinely broken in b8840, NOT the missing flag. NOT a re-arm loop (9e5647aff's DONE is a plain break; re-arm landed later on master). Every request carries 8 tools; runaways = the tool-retry-FAILURE subset (model won't self-terminate ``, forcing doesn't save it). `reasoning_control` API is ABSENT in 8840 (`--help` has no flag; binary has no `/control` strings) β†’ brokkr's watchdog needs the bump too. +- **FIX (agreed with brokkr, NO max_tokens ceiling per operator):** (1) IMMEDIATE relief = push worldtree's **b61 270s wall-clock** β€” bounds the user-facing hang regardless of the seat loop (so the seat bump is NOT urgent). (2) DIAGNOSE-THEN-FIX (Eitri refinement, may SAVE the bump) β€” first capture the detailed tuple `tokens/start/end/forced` (needs a **verbose / low-budget-matrix seat restart**; 8840's default log has only the simpler activated/deactivated line β€” no `forced=`/`start=…end=`, confirmed by grep): **`end=0` β†’ the FORCED-injection end_tag isn't reaching the budget path (Deckard template / reasoning-format mismatch) β†’ CHEAP CONFIG fix = explicit budget start/end tags, a restart NOT a bump**; `end>0,forced=0` β†’ end-tag tokenization bug; `end>0 + no forcing across arms` β†’ real state bug β†’ build bump. (NB: 110 `deactivated (natural end)` prove the DETECTION end-tag works, so an `end=0` would be specifically the FORCED-inject tag, a narrow config miss.) Only if the tuple says end>0 β†’ **DURABLE = stage + validate a post-#22740** llama.cpp build OFF the hot path (⚠️ #22717 garbled-forcing regression ~b9018 fixed by #22740 β†’ don't land in the b8840β†’b9018 window; validate vs `tests/test-reasoning-budget.cpp` + that it loads Deckard Q5_K_M), then surface the seat-image swap to Vuong; `--reasoning-budget-message` additive WITH the bump. Getting the tuple = the disruptive step (bounce the prod RP seat to --verbose/low-budget; a separate test instance won't fit β€” GPU0 has only ~4G margin). **brokkr FINAL rec (thread wrapped 2026-07-13): go STRAIGHT to the staged bump β€” the tuple-diagnostic / tag-probe is a LONG-SHOT (the 110 natural closes prove the parser knows the delimiter β†’ more likely a build/state bug than a tag-carry miss). **RESOLVED WITHOUT the bounce (brokkr msg16 deduction): the `deactivated (natural end)` log is the budget SAMPLER's namespace β†’ the sampler matched its `reasoning_budget_end` token β†’ end_tag NON-empty (110Γ—); and per source `forced = tokenize(message+end_tag)` is built in the SAME block β†’ forced also NON-empty. So the tuple is logically end>0/forced>0 β†’ forced tokens EXIST, forcing just never FIRES = STATE-MACHINE bug β†’ the post-#22740 BUILD BUMP. The `end=0` cheap-config branch is RULED OUT. NO diagnostic bounce needed (would only confirm) β†’ no prod-seat-restart go required from Vuong for diagnosis.** XTC HELD. b61 = the immediate relief β†’ NO rush. Deliverable owed to brokkr = the validated bump-target (loads Deckard Q5_K_M + passes tests/test-reasoning-budget.cpp AND β€” the real acceptance test β€” a **live low-budget check on the Qwen3.5/Deckard arch showing an ACTUAL forced-close, not just natural ends**, since a forced-close-that-never-fires is exactly what 8840 gets wrong); Dvalin standing by for a 2nd read, Eitri if arch specifics get hairy.** **VERIFICATION (2026-07-13, WebFetch ggml-org/llama.cpp β€” the build-bump has NO released target; the dwarf "bump to post-#22740" was WRONG, #22740 fixed a DIFFERENT regression #22717):** the real fixes are UNMERGED β€” #22684 (Qwen3.5/3.6 tool-call-in-`reasoning_content` = our TRIGGER) is **CLOSED-AS-NOT-PLANNED**; #24202 (Qwen tool-parsing) is **OPEN**; **#25544** (reasoning-budget sampler β†’ MULTIPLE terminating sequences) is **OPEN, awaiting review** = OUR forcing fix. MECHANISM NAILED: 8840's budget sampler knows only a SINGLE end tag (``), but Qwen3.5's tool-path terminates reasoning with `` which the sampler can't match β†’ forcing never fires on the tool path (explains 110 natural-`` closes working + 0 forced + tool-retry specificity). **RE-FRAMED PLAN: no bump target exists yet β†’ b61's 270s wall-clock is the ACTUAL mitigation (not just interim); TRACK PR #25544 and bump only when it merges into a release + validate the forced-close empirically. Seat XTC = optional loop-frequency fallback, not a fix.** **BUILT 2026-07-13 (operator directed the custom build after we verified NO released fix exists): image `llamacpp-charrp:6eddde0-pr25544`/`:custom-latest` on ana-ml2 (reports b10001, merged HEAD `6ce128d0b` = master 6eddde0 + PR #25544, CUDA 12.8.1 sm_120, server target `/app/llama-server`), binary smoke-tested OK. Reproducible recipe + records: eshpfi `stacks/char-rp-gguf/llamacpp-custom/` (build.sh + README) + auto-memory `charrp-custom-llamacpp-pr25544`. **SEAT SWAP PENDING operator go on timing** (interrupts live Worldtree-personal RP ~2-3min; rollback = stock 8840 `ghcr.io/mostlygeek/llama-swap:cuda` imageID d6c39f55, preserved). On-seat acceptance = Deckard loads + serves + a FORCED close fires at low budget (not just natural ends). #355 DEPLOY-SPEED task separately CONFIRMED landed: #359 mergedβ†’build-and-deploy 11mβ†’~5m (~2Γ—, wt-dev corroborated 5m/4m vs 11m); b61 pushed (v1.0.0b61 747d7a0) = 300s client backstop live on deploy.** (3) Optional interim = mild XTC on the seat (behavior-shaping, not enforcement) β€” holding unless the loop-frequency reduction is wanted pre-bump. Seat config: canonical `stacks/char-rp-gguf/compose.yaml` (service `llama-charrp-reasoning`), live `/opt/docker/compose/char-rp-gguf/` on ana-ml2. **Cheap `--reasoning-budget-message`-alone test is DEAD (source-confirmed won't fix it) β€” did NOT waste a restart on it.** +- **Operator direction:** find the SERVER-side fix (params/config), do NOT ceiling max_tokens (legit long outputs must survive). Routed to brokkr-smithy-dev (thread 01KXEEQ0TD6D…) β†’ he accepted, pulled dvalin for the known-bug search, running the dwarf consult. I offered to run the live-seat with/without-tools split-test on his signal (own the seat; mildly disruptive β†’ off-peak). +- **Wasted-GPU side issue + b61 re-measure:** LiteLLM logged these `success` at full 1255–1390s β†’ on b60 it did NOT abort at WT's cancel (whatever WT did, no upstream abort β†’ ~18min wasted GPU/runaway). worldtree-dev's **unpushed b61** adds a **270s provider wall-clock** that runs `_bounded_stream_close`β†’httpx FIN (force-closes WT's upstream side). So post-b61 the FIN-check flips to "WT closes at ~270s"; the only open question is whether **LiteLLM propagates that client-disconnect to the seat** (my gateway domain). **RE-MEASURE after b61 deploys** β€” don't assume b60 behavior. +- **Data on hand:** pcaps at `corviduo-dev:~infra-ops/py-spy-355/wedge_*/` are only ~99s windows triggered on the attach_tool failure (too short for the 300s cancel β€” a limitation, not usable for the FIN-check). LiteLLM config `/opt/docker/conf/litellm/config.yaml` β†’ `/app/config.yaml`; spend_logs = `litellm-db` (`docker exec litellm-db psql -U litellm -d litellm`; camelCase cols double-quoted; request body in `proxy_server_request::jsonb->>'max_tokens'` top-level). -### πŸ”Ί ACTIVE 2 β€” Worldtree deploy-speed: wire the runner-side persistent BuildKit cache -- **Measured (gitea Actions API, `vh/worldtree`, claude-bot token):** full code deploy β‰ˆ**12 min**; bottleneck = `build-and-deploy` **~11 min** (685/614/659s recent). `lint-test` ~7.5 min runs PARALLEL (off critical path). `deploy`/recreate β‰ˆ50s (fine β€” not the problem). -- **Root cause (Dockerfile at repo root):** the dep install `RUN --mount=type=secret,... uv sync --frozen --no-dev --no-install-project --no-cache` uses `--no-cache` AND has **no BuildKit `--mount=type=cache`** β†’ re-downloads+rebuilds ALL Python deps cold every build. Layer ordering is CORRECT (`COPY pyproject.toml uv.lock` + `packages/` before `uv sync`, app code after) β€” the problem is purely caching. -- **Fix (split by domain):** (a) worldtree-dev's Dockerfile diff β€” add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`; (b) **MINE, runner-side** β€” persist the BuildKit layer cache between CI runs (`--cache-to/--cache-from` registry or runner-local). Config-only changes already skip the pipeline via a pinned recreate. -- **Confirm first:** pull the build-STEP log to verify the dep-install is THE dominant sub-step (vs the apt layer or the registry push) before handing worldtree-dev the diff β€” don't assume. **Awaiting operator go.** +### πŸ”Ί ACTIVE 2 β€” Worldtree deploy-speed: DELIVERED as PR #359 (green-lit, awaiting merge) +- **Real bottleneck was NOT uv sync (memory's assumption was WRONG).** Pulled the buildx STEP log (gitea `actions_log` on ana-docker, `docker exec gitea` β†’ `/data/gitea/actions_log/vh/Worldtree/*/{task}.log.zst`, zstd). Sub-steps of the ~7min build: `#24 RUN useradd … chown -R /app /data … = **251.2s**` ← the monster; `#25 export image+push = 123.7s`; `#13 uv sync --no-cache = **only 35.6s**`; `#26 cache-export mode=max = 18.4s`. The runner-side registry LAYER cache (`--cache-from/--cache-to type=registry:buildcache,mode=max`) is **already wired in deploy.yml** β€” my prior "runner cache missing" note was STALE. +- **Root cause:** `chown -R worldtree:worldtree /app` forces BuildKit to copy-up the entire root-owned `.venv` (torch + 125 pkgs) into a fresh layer β†’ ~4min + a duplicate venv baked into the image (also slows export/push). +- **Fix (PR #359, branch `infra/build-cache` off origin/main@b60):** (1) drop `/app` from the chown (keep `/data`); (2) add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`. **VALIDATED against BOTH live containers** (demo+personal): zero files written under `/app` post-boot (PID1-start ref) β€” all runtime writes β†’ `/data/state`, `/data/kb`, `/home/worldtree`; `/app` is read-only for uid 1000, `/app/config` chowned at first boot by the root entrypoint shim. So the chown was pure waste. Expected **~5min off** (~11β†’~6min). +- **Status:** worldtree-dev GREEN-LIT (confirmed `--no-cache` not deliberate; b61 is theirs, core/llm-only, zero Dockerfile overlap). PR open + mergeable; they review, mergeβ†’main triggers the real build that measures the actual speedup. Deploy trigger unaffected (Dockerfile change β†’ builds). **Done from my side.** ### Landed this session (2026-07-13) β€” context for the fresh session - **WT #355 turn-lifecycle fix VALIDATED** (worldtree b60): wedged turns self-terminate cancelled/stalled at the 300s watchdog (turns 2064/2065 = 302s/360s) vs pre-b60 turn 2061's 16-min no-terminal silence. The residual (ACTIVE 1) is a separate LiteLLM matter. @@ -132,7 +132,11 @@ _As of 2026-07-13 β€” two active tasks (per /snapshot args): the LiteLLM gateway ## Recent decisions -- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat β€” Deckard seat EXONERATED (completes ≀72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviduo). See in-flight ACTIVE 1. +- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below β€” that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 Γ—4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) β†’ routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding β€” the seat-idle claim came from reading only the ≀73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. + +- `[2026-07-13]` **Deploy-speed real bottleneck β‰  uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11β†’~6min). Runner-side BuildKit cache task (b) was already done β†’ moot. + +- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat β€” Deckard seat EXONERATED (completes ≀72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 β€” see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** - `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 β€” wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume β€” the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. diff --git a/stacks/char-rp-gguf/llamacpp-custom/README.md b/stacks/char-rp-gguf/llamacpp-custom/README.md new file mode 100644 index 0000000..cb818c6 --- /dev/null +++ b/stacks/char-rp-gguf/llamacpp-custom/README.md @@ -0,0 +1,100 @@ +# ⚠️ CUSTOM llama.cpp build for the char-rp-reasoning seat + +> **STATUS (2026-07-13):** image `llamacpp-charrp:6eddde0-pr25544` (+ `:custom-latest`) +> BUILT on ana-ml2 (reports `version: 10001 (6ce128d0b)`, CUDA 12.8.1, sm_120) and +> binary-smoke-tested OK. **Seat swap PENDING** β€” the live `char-rp-gguf` compose +> still points the reasoning seat at stock 8840 until the swap + on-seat forced-close +> validation. Once swapped, update this line to DEPLOYED. + +**The `llama-charrp-reasoning` seat (Deckard-PKD, ana-ml2:8018) runs a +CUSTOM-BUILT llama.cpp, NOT a stock upstream image.** If you are upgrading, +rebuilding, or "cleaning up" the seat, read this first β€” a naive bump to a +stock image will silently reintroduce Worldtree #355. + +## What it is + +`llamacpp-charrp:custom-latest` = **llama.cpp master `6eddde0`** (2026-07-13, +~b9990 era) **+ unmerged upstream PR [#25544]** (`aldehir:reasoning-budget-multi-seq`), +built for Blackwell (sm_120) via `build.sh` in this directory. + +Merged tree HEAD (on ana-ml2 `/home/lkraven/llamacpp-build`): `6ce128d0b`. + +[#25544]: https://github.com/ggml-org/llama.cpp/pull/25544 + +## Why it exists (Worldtree #355 root cause) + +Stock llama.cpp **b8840** (the previous seat build, `ghcr.io/mostlygeek/llama-swap:cuda`) +β€” and every *released* build to date β€” has a reasoning-budget sampler that +recognises only **one** end-of-thinking tag, ``. But Qwen3.5's +tool-call path ends its reasoning block with **``**, a terminator +the single-tag sampler can't match. So on the agentic tool-retry path, +`--reasoning-budget 400`'s forced-close **never fires**, and the model loops +in reasoning all the way to `max_tokens` (32768) β‰ˆ **22 min per turn** (~13% +of char-rp-reasoning turns). That is #355's residual hang. + +Evidence that pinned it (all three agree): LiteLLM spend_logs (4 requests at +exactly `completion_tokens=32768`), the seat's own log (`158 activated / 0 +forced-close / 110 natural-end`), and the WT↔gateway pcap (runaway stream is +100 % `reasoning_content`, coherent tool-retry planning, never exits ``). + +**PR #25544 is the fix**: it extends the budget sampler to **multiple +terminating sequences** (`` *or* ``) and replays the matched +end-sequence when the budget expires. It is **OPEN / unmerged** upstream +(the underlying trigger #22684 is closed-as-not-planned; the tool-parsing PR +#24202 is also open), so there is **no released build that fixes our bug** β€” +hence this local build. + +## Conflict resolution (the one thing to re-verify on any rebump) + +The PR (based on master of 2026-07-10) is 25 commits behind `6eddde0`; the +merge has a single conflict in `tools/server/server-common.cpp`. Resolution +(baked into `build.sh`): keep the **PR's plural `reasoning_budget_end_tags`** +(the fix) **and master's per-request body-read of `reasoning_budget_message`** +(a newer master feature). Both improvements retained. + +## Build / rebuild + +```bash +# on ana-ml2 (has the Blackwell GPUs + docker + disk): +scp stacks/char-rp-gguf/llamacpp-custom/build.sh ana-ml2:/home/lkraven/ +ssh ana-ml2 'bash /home/lkraven/build-charrp.sh' # ~20-40 min, sm_120 compile +``` + +Then deploy via the seat compose (`stacks/char-rp-gguf/compose.yaml`, service +`llama-charrp-reasoning`), whose `image:` is pinned to `llamacpp-charrp:custom-latest`. + +## Acceptance test before trusting a build (do NOT skip) + +The bug is specifically that forcing *never fires*. A build is only good if it +demonstrably **force-closes** β€” not just "loads + serves": + +1. `tests/test-reasoning-budget.cpp` passes (`ctest -R reasoning-budget`). +2. Live low-budget check on Deckard: launch with `--reasoning-budget 20`, send a + tools-bearing request that induces a tool-retry loop, and confirm the seat + logs a **forced** close (budget-reached), not just `deactivated (natural end)`, + and the response stops near the budget instead of running to `max_tokens`. + +Then restore `--reasoning-budget 400` for production. + +## Rollback + +The previous stock image `ghcr.io/mostlygeek/llama-swap:cuda` (build 8840) is +preserved on ana-ml2 (imageID `d6c39f55…`). To roll back: point the seat's +`image:` back to it and recreate. 8840 works (RP is fine) β€” it only has the +forcing bug. + +## πŸ”” Retiring this custom build (REMOVE-WHEN-MERGED) + +This is a temporary local build carrying an unmerged PR. **When PR #25544 +merges upstream and lands in a stock release:** + +1. Confirm the released build's `tests/test-reasoning-budget.cpp` covers the + `` terminator and passes on Qwen3.5. +2. Switch the seat `image:` to that stock image (e.g. a + `ghcr.io/ggml-org/llama.cpp:server-cuda` tag β‰₯ the merge) and run the + acceptance test above. +3. Delete this directory + the compose `⚠️ CUSTOM BUILD` note, and update + `persistent-memory.md` + the `reference_charrp_custom_llamacpp_pr25544` + auto-memory. + +Check status: . diff --git a/stacks/char-rp-gguf/llamacpp-custom/build.sh b/stacks/char-rp-gguf/llamacpp-custom/build.sh new file mode 100644 index 0000000..4f90858 --- /dev/null +++ b/stacks/char-rp-gguf/llamacpp-custom/build.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# ───────────────────────────────────────────────────────────────────────────── +# Reproducible build of the CUSTOM llama.cpp that backs the char-rp-reasoning +# seat (Deckard-PKD, ana-ml2:8018). +# +# = llama.cpp latest master (pinned 6eddde0, 2026-07-13, ~b9990 era) +# + UNMERGED PR #25544 (aldehir:reasoning-budget-multi-seq) +# +# WHY THIS EXISTS (do not "clean this up" without reading README.md): +# Stock llama.cpp b8840 (and every released build to date) has a reasoning- +# budget sampler that knows only ONE end-tag (). Qwen3.5's tool-call +# path terminates reasoning with , which the single-tag sampler +# cannot match, so --reasoning-budget forcing NEVER fires on the tool-retry +# path β†’ the model loops in reasoning to max_tokens (32768) β‰ˆ 22 min. That is +# Worldtree #355's residual. PR #25544 teaches the budget MULTIPLE terminating +# sequences ( OR ) β€” the exact fix β€” but it is UNMERGED +# upstream, so we build it ourselves. +# +# ⚠️ REMOVE-WHEN-MERGED: once #25544 merges upstream and lands in a release, +# retire this custom build and switch the seat back to a stock image +# (see README.md β†’ "Retiring this custom build"). +# +# Run on ana-ml2 (Blackwell RTX PRO 6000, sm_120; docker + buildx; ~120G free). +# Produces: llamacpp-charrp:6eddde0-pr25544 (+ :custom-latest) +# Runtime ~20-40 min (CUDA compile for sm_120). +# ───────────────────────────────────────────────────────────────────────────── +set -euo pipefail + +MASTER_SHA="${MASTER_SHA:-6eddde0}" # llama.cpp master pin +PR="${PR:-25544}" # aldehir:reasoning-budget-multi-seq +BUILDDIR="${BUILDDIR:-/home/lkraven/llamacpp-build}" +IMAGE="${IMAGE:-llamacpp-charrp}" +CUDA_VERSION="${CUDA_VERSION:-12.8.1}" # 12.8+ required for Blackwell sm_120 +CUDA_ARCH="${CUDA_ARCH:-120}" # RTX PRO 6000 = compute 12.0 + +echo "== fetch master@${MASTER_SHA} + PR #${PR} ==" +rm -rf "$BUILDDIR" +git clone https://github.com/ggml-org/llama.cpp "$BUILDDIR" +cd "$BUILDDIR" +git checkout "$MASTER_SHA" +git fetch origin "pull/${PR}/head:pr${PR}" +git config user.email infra-ops@phasefinal.com +git config user.name infra-ops + +echo "== merge PR #${PR} (expect 1 conflict in server-common.cpp) ==" +git merge --no-commit --no-ff "pr${PR}" || true +# Resolve the single conflict: keep the PR's PLURAL reasoning_budget_end_tags +# (the whole point of the fix) AND master's per-request body-read of +# reasoning_budget_message (a newer master feature the PR's base predates). +python3 - <<'PY' +p = "tools/server/server-common.cpp" +s = open(p).read() +i = s.index("<<<<<<< HEAD") +j = s.index(">>>>>>> pr25544") + len(">>>>>>> pr25544") +new = (' llama_params["reasoning_budget_end_tags"] = chat_params.thinking_end_tags;\n' + ' llama_params["reasoning_budget_message"] = json_value(body, "reasoning_budget_message", opt.reasoning_budget_message);') +open(p, "w").write(s[:i] + new + s[j:]) +assert open(p).read().count("<<<<<<<") == 0, "unresolved conflict markers remain" +print("resolved server-common.cpp") +PY +git add -A +git commit -m "merge PR #${PR} (reasoning-budget multi-seq) onto master ${MASTER_SHA} β€” char-rp-reasoning seat custom build" + +echo "== docker build (server target, CUDA ${CUDA_VERSION}, sm_${CUDA_ARCH}) ==" +docker build -f .devops/cuda.Dockerfile --target server \ + --build-arg CUDA_VERSION="${CUDA_VERSION}" \ + --build-arg CUDA_DOCKER_ARCH="${CUDA_ARCH}" \ + --build-arg APP_VERSION="${MASTER_SHA}-pr${PR}" \ + --build-arg APP_REVISION="$(git rev-parse HEAD)" \ + -t "${IMAGE}:${MASTER_SHA}-pr${PR}" \ + -t "${IMAGE}:custom-latest" \ + . + +echo "== built ${IMAGE}:${MASTER_SHA}-pr${PR} (entrypoint /app/llama-server, drop-in for the seat) =="