From d3727dee53406d4717317d7f4a261cbc27143564 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Mon, 10 Aug 2026 07:27:36 -0700 Subject: [PATCH] =?UTF-8?q?feat(vllm):=20lfm2.5=20reasoning-parser=20(deep?= =?UTF-8?q?seek=5Fr1)=20=E2=80=94=20scoreable=20JSON=20for=20brokkr's=20ba?= =?UTF-8?q?ke-off?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LFM2.5 is -delimited (opening tag in prompt); deepseek_r1 splits reasoning into reasoning_content so content is the clean post- answer. Re-smoke: content valid JSON + reasoning_content populated. License production-cleared (operator <$10M ruling), still out of routing per the measurement gate. --- persistent-memory.md | 2 +- stacks/vllm/compose.yaml | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/persistent-memory.md b/persistent-memory.md index 5533a70..7876314 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -119,7 +119,7 @@ _As of 2026-08-08 — long session; all major arcs LANDED (full detail per arc i - **EVIDENCE HOLD (partial):** WT #394 index-row half LIFTED + swept (kb 8,230→2,876); the **FILE half STILL STANDS** — do NOT delete the on-disk generation dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`). Also un-actioned worldtree-side: the `reconcile --repair` rc=1 git-staging bug (phantom `docs/specs/saga-taxonomy-redesign-plan.md`) — flagged, their code. -- **OPEN LOOPS (2026-08-10):** (1) **dots-tts ratatoskr cutover — COMPLETE.** Operator ear-confirmed on the live RP surface (:8765); **dots-tts v0.22.2 is ratatoskr's PRIMARY RP voice** (tts.py seam → OpenAI `/v1/audio/speech`, SR→48k, default voice→glados, donut carries; 520 tests green; their one-synth-per-turn lock DEC-5 matches the server's serialized model). ratatoskr RELEASED chatterbox-fast :8197. **chatterbox-fast RECLAIMED (operator go 2026-08-10):** `docker compose stop` on irv-ml1 freed ~7GB on the 3090 (GPU0 16.0→9.1GB); container Exited(0) — `restart: unless-stopped` honors the manual stop so it stays down across reboots; revive with `docker start chatterbox-fast`. dots-tts unaffected. **dots.tts migration FULLY CLOSED.** thread `01KZNZF1…`. (2) **LFM2.5-2.6b bake-off for brokkr — DONE:** `vllm-lfm25` LIVE on ana-ml2 GPU1 :8021 (LiquidAI/LFM2.5-2.6B, Lfm2ForCausalLM/vLLM 0.24.0, BF16, util 0.09 ~8.7GB into GPU1's unreserved slack — no prod reservation touched; max-len 16384; NO `--reasoning-parser` so content non-empty). LiteLLM alias `lfm2.5-2.6b` (vendor sampling temp0.1 + topk50/reppen1.1 via `extra_body` so drop_params doesn't strip them). Smoke: content_len 636, reasoning_content 0, finish stop. brokkr notified (`01KZP08NB7…`). EVAL-ONLY pending operator production ruling (LFM Open License); NOT in any default/fallback chain. Backups: litellm `config.yaml.bak-pre-lfm25-20260810`; targeted `up -d vllm-lfm25` (shared-.env → never bare up-d). Repo: `stacks/vllm/` + `stacks/litellm/conf/config.yaml`. +- **OPEN LOOPS (2026-08-10):** (1) **dots-tts ratatoskr cutover — COMPLETE.** Operator ear-confirmed on the live RP surface (:8765); **dots-tts v0.22.2 is ratatoskr's PRIMARY RP voice** (tts.py seam → OpenAI `/v1/audio/speech`, SR→48k, default voice→glados, donut carries; 520 tests green; their one-synth-per-turn lock DEC-5 matches the server's serialized model). ratatoskr RELEASED chatterbox-fast :8197. **chatterbox-fast RECLAIMED (operator go 2026-08-10):** `docker compose stop` on irv-ml1 freed ~7GB on the 3090 (GPU0 16.0→9.1GB); container Exited(0) — `restart: unless-stopped` honors the manual stop so it stays down across reboots; revive with `docker start chatterbox-fast`. dots-tts unaffected. **dots.tts migration FULLY CLOSED.** thread `01KZNZF1…`. (2) **LFM2.5-2.6b bake-off for brokkr — DONE:** `vllm-lfm25` LIVE on ana-ml2 GPU1 :8021 (LiquidAI/LFM2.5-2.6B, Lfm2ForCausalLM/vLLM 0.24.0, BF16, util 0.09 ~8.7GB into GPU1's unreserved slack — no prod reservation touched; max-len 16384; NO `--reasoning-parser` so content non-empty). LiteLLM alias `lfm2.5-2.6b` (vendor sampling temp0.1 + topk50/reppen1.1 via `extra_body` so drop_params doesn't strip them). **PRODUCTION-CLEARED 2026-08-10** (operator ruled they're `-delimited: opening `` injected by chat template into the prompt, output = `reasoning…answer`; deepseek_r1 splits on ``). Re-smoke: content=valid JSON (51 chars, `json.loads` passes) + reasoning_content=824 chars → both halves of brokkr's gate met (scoreable + non-empty). ⚠ model wraps its answer in a ```json fence — a bare `json.loads(content)` chokes; scorer must strip the fence. brokkr notified + clear to run (`01KZP142…`). Backups: litellm `config.yaml.bak-pre-lfm25-20260810`; targeted `up -d vllm-lfm25` (shared-.env → never bare up-d). Repo: `stacks/vllm/` + `stacks/litellm/conf/config.yaml`. - **OPEN FOLLOW-UPS:** chatterbox-fast **deployed flat-build-context vs package-repo divergence** (reconcile so a repo rebuild matches deploy — repo commit `6bc7bf0` has cap+norm_loudness in package layout; deployed is flat cap-only). Standing/parked: CI-flip runner-auth research, gitea `REQUIRE_SIGNIN_VIEW=false`, #363 research-wing ingest (no deadline), zonos-gateway CI-wire. diff --git a/stacks/vllm/compose.yaml b/stacks/vllm/compose.yaml index 0b39196..f0038c9 100644 --- a/stacks/vllm/compose.yaml +++ b/stacks/vllm/compose.yaml @@ -385,6 +385,12 @@ services: - --kv-cache-dtype - ${LFM25_KV_CACHE_DTYPE} - --enable-prefix-caching + # Split the reasoning (delimited by ; the opening tag is + # injected by the chat template into the prompt) into reasoning_content, so + # `content` is the clean post- answer — scoreable JSON for brokkr's + # structured-extraction bake-off (raw-served, reasoning prepended, is not). + - --reasoning-parser + - deepseek_r1 deploy: resources: reservations: