esh-pfi-infrastructure/persistent-memory.d/2026-09-08-erp-seat-nvfp4-trial-and-toolcalling.md

erp-tune-v6-nvfp4a16 on ana-ml2 :8021, aliased to trial, and the Gemma-4 tool-calling findings (2026-09-08)

Seat. Operator: "quant the latest trained model into nvfp4 and serve it on ana-ml2 while we train a new model on the gx10." Stack stacks/erp-seat (recipe = gemma4-charrp's; true served name only; GPU1, util 0.35, 32K ctx). Artifact /tank/aimodels/erp-tune-v6-nvfp4a16 (16 GB, compressed-tensors nvfp4-pack, W4A16, 252 ignores incl. 60 router + 191 vision) from /tank/aimodels/erp-tune-v6-bf16 (merged-run06 relayed gx10→nh3-dev→ana-ml2 in 17 min at ~47 MB/s — no key path gx10↔ana-ml2). Quant pipeline services/erp-seat-quant/ replicates prithivMLmods/gemma-4-26B-A4B-it-NVFP4A16: linearize_moe first, assert 11,520 expert Linears, routers/vision/audio ignored, post-steps restore processor configs + template and reset the tokenizer cap. DATA-FREE (~90 s) — playbook §3.16. Commits 911ff20, 8512dd4.

Alias. Operator: "alias erp-tune-v6-nvfp4 to trial, please. no gate." The trial model lives in stacks/litellm/conf/config.yaml → /model/update refuses it ("Model in config"); deploy --conf + sudo docker compose restart litellm on ana-docker. Verified 3/3. Commit 48aa2c6. Block states the seat as unrated on every safety axis.

Tool calling (operator: "fix toolcalling with the trial seat").

1. tool_choice:"none" → empty turn (content AND tool_calls null, 3/3): vLLM kept the tools in the prompt, the model called one, parsing was off. Fix --exclude-tools-when-tool-choice-none (7f6be8a). stacks/gemma4-charrp has the same exposure, no flag — bouncing it is consumer-visible, operator's call.
2. Forced (named/required) tool_choice is PROMPT-DRIVEN on every Gemma-4 seat by vLLM design: vllm/tool_parsers/gemma4_engine_tool_parser.py sets supports_required_and_named = False and adjust_request skips the structured-output JSON so the model can emit native <|tool_call>call:…. A tune that weakened that syntax honours forced calls only sometimes. Measured (3 conv × 3 turns, real system prompt): v0.26.0 1/9; nightly 311b3513 (v0.27.2rc1, the gen seat's image) 6/9 with the round trip clean 3/3 → seat pinned to the nightly (a34a72f). tts-dev: 0/18 on v0.26.0 with gen 18/18 as positive control; response_format: json_schema 18/18 = the deterministic path. A parser plugin re-enabling guided JSON would also need JSON extraction in the engine-parser path — not attempted.
3. ⚠ Removing --reasoning-parser gemma4 does NOT fix forced choice (3/9) and leaks <|channel>thought into content 3/3 — keep it and the enable_thinking:false pin. Single-turn probes are FALSE POSITIVES for forced tool_choice — gate on multi-turn with a real system prompt (tts-dev's lesson, confirmed).