92 lines
5.9 KiB
Markdown
92 lines
5.9 KiB
Markdown
# eRP dual-seat overhaul — MeroMero-v2 + Dark-Scarlett, NVFP4A16 @ 256K on ana-ml2
|
||
|
||
`[2026-08-12]` Replaced the two legacy char-rp seats with home-quantized NVFP4A16 vLLM
|
||
seats. Operator-driven, end to end this session.
|
||
|
||
## What landed
|
||
|
||
| Seat (LiteLLM alias) | Model | Role | GPU | Context |
|
||
|---|---|---|---|---|
|
||
| `char-rp` (:8016) | **G4-MeroMero-v2-31B** (Gemma-4) | non-thinking PROSE, **multimodal (vision)** | GPU0 | 256K @ 2.07× (util 0.52) |
|
||
| `char-rp-reasoning` (:8018) | **Dark-Scarlett-v1.0-27B** (Qwen3.6) | THINKING (default) | GPU1 | 256K @ 1.62× (util 0.44) |
|
||
|
||
- Both **NVFP4A16 weight-only** (llm-compressor, `compressed-tensors`), `--kv-cache-dtype fp8`.
|
||
- Replace: `char-rp-gguf` (Magidonia-24B GGUF/llama.cpp, :8016) + `heretic2-charrp-reasoning`
|
||
(DavidAU Qwen3.6-27B-Heretic2 modelopt NVFP4+MTP, :8018). Old stacks/containers **stopped +
|
||
retained** for rollback.
|
||
- Compose-ified: `stacks/meromero-charrp` + `stacks/darkscarlett-charrp-reasoning` (ana-ml2
|
||
`/opt/docker/compose/`, mirrored to eshpfi, commit **`f08b6cb`**) → survive reboot.
|
||
- Research that drove picks: `docs/pfi/erp-thinking-finetunes-2026.md` (from the `gecko-65` Booth).
|
||
|
||
## Load-bearing lessons (the whole point of this file)
|
||
|
||
1. **Load via the ConditionalGeneration WRAPPER class, never `AutoModelForCausalLM`.** For a
|
||
multimodal-capable base (Gemma-4, Qwen3.6), `AutoModelForCausalLM.from_pretrained` +
|
||
`save_pretrained` writes a FLAT text config (`Qwen3_5TextConfig`, `model.layers.*`) that
|
||
**both vLLM AND SGLang reject** (SGLang: "Qwen3_5ForCausalLM has no SGLang implementation";
|
||
vLLM wants `Qwen3_5ForConditionalGeneration`). Loading via `Qwen3_5ForConditionalGeneration` /
|
||
`Gemma4ForConditionalGeneration` keeps the wrapper config they accept. **This was the DS
|
||
blocker** — re-quant via the wrapper fixed it (`Dark-Scarlett-...-NVFP4A16-wrapper`).
|
||
2. **NVFP4A16 is weight-only → DATA-FREE.** llm-compressor infers `DataFreePipeline`; calibration
|
||
data is unused (only matters for W4A4 activation quant). W4A16 chosen per NVIDIA's sm_120
|
||
long-context guidance (W4A4 KLD 2-4× worse past ~10k ctx).
|
||
3. **Load on CPU (`device_map=None`)** so llm-compressor onloads one layer at a time. `device_map=
|
||
"auto"` packs the whole model onto the GPU and OOMs when the card isn't fully free.
|
||
4. **Both models are KV-EFFICIENT — the "dense = KV-hungry" worry was WRONG.** MeroMero (Gemma-4)
|
||
uses **sliding-window attention** (most layers cache only a bounded window); DS (Qwen3.6) uses
|
||
**hybrid GatedDeltaNet linear-attention** (3:1 linear:full, linear layers carry no KV). Both
|
||
hit full native 256K easily. (MeroMero KV pool ~542K tokens at util 0.52.)
|
||
5. **MeroMero vision reconstruction.** The finetune ships `processor_config.json` (image_processor
|
||
inline, `Gemma4ImageProcessor`) but NOT `preprocessor_config.json` — the old-format file vLLM's
|
||
feature-extractor loader wants. **Even google/gemma-4-31B-it (ungated!) ships only
|
||
processor_config.json.** FIX: extract the `image_processor` section → write
|
||
`preprocessor_config.json` verbatim, serve WITHOUT `--language-model-only`. Verified (model
|
||
correctly ID'd a red circle). Audio is config-declared but WEIGHTLESS (0 audio tensors).
|
||
6. **GPU placement.** Match the KV-heavier model to the roomier GPU. GPU0 (gen neighbor, ~54GB
|
||
free) > GPU1 (utility cluster, ~45GB free). Swapped MeroMero→GPU0, DS→GPU1. Pins via compose
|
||
`deploy.resources.reservations.devices`.
|
||
|
||
## Dead ends (tried + abandoned)
|
||
|
||
- **DS via llm-compressor `AutoModelForCausalLM`** → flat config vLLM/SGLang reject. → wrapper class.
|
||
- **DS via NVIDIA ModelOpt** → modelopt↔transformers **version deadlock**: current transformers
|
||
supports `qwen3_5` but crashes modelopt's sparse-moe plugin (`issubclass()` on a non-class);
|
||
modelopt 0.43.0 pulls an old transformers that can't load `qwen3_5` at all. Abandoned.
|
||
- **DS via SGLang** → `Qwen3_5ForCausalLM has no SGLang implementation`. Abandoned, but it REVEALED
|
||
that both engines need the wrapper (→ the fix in lesson 1).
|
||
- **`device_map="auto"` for the quant** → CUDA OOM in the weight observer. → `device_map=None`.
|
||
|
||
## granite retired + gateway repoint
|
||
|
||
- `vllm-granite` (granite-4.1-8b, fleet summarizer, GPU1) **`docker stop`ped** (reversible) to
|
||
reclaim ~13.6GB GPU1 for RP context.
|
||
- LiteLLM (`ana-docker:/opt/docker/conf/litellm/config.yaml`, backed up
|
||
`.bak-pre-granite-down-*`): **`granite-4.1-8b` alias RETIRED** — commented out, now 404s cleanly
|
||
(the `*` wildcard→llama-swap was decommissioned 2026-06-20, so no fallthrough). **`summarizer` +
|
||
`classifier` REPOINTED to gen** (`hosted_vllm/qwen3.6-35b-a3b-heretic` @ :8015,
|
||
`enable_thinking:false`) — both verified. ⚠ This LiteLLM change is **server-only / not
|
||
version-controlled** (a follow-up).
|
||
|
||
## MTP — deferred
|
||
|
||
DS's MTP heads were dropped by the CausalLM loader; **deferred, not restored** (spec-decode is
|
||
net-negative at RP temps: ~38-52% accept at temp 0.8-1.25, below vLLM's 0.5 cutoff). The
|
||
splice-back path (`splice_mtp.py` in the heretic2 work dir) exists if ever wanted. MeroMero
|
||
(Gemma-4) has no MTP by architecture.
|
||
|
||
## On-disk / where things live
|
||
|
||
- Quant pipelines: `ana-ml2:/tank/aimodels/meromero-v2-nvfp4-work/` +
|
||
`/tank/aimodels/darkscarlett-nvfp4-work/` (scripts, BF16 source, NVFP4 outputs).
|
||
- Compose stacks: `ana-ml2:/opt/docker/compose/{meromero-charrp,darkscarlett-charrp-reasoning}/`.
|
||
- Gateway aliases (unchanged, port-based): `char-rp`→:8016, `char-rp-reasoning`→:8018. (char-rp was
|
||
also fixed from the stale `magidonia-24b-v4.3` backend model name → `char-rp`.)
|
||
|
||
## Open follow-ups
|
||
|
||
1. LiteLLM granite/repoint change NOT version-controlled (server + backup only).
|
||
2. eshpfi unpushed (many commits this session incl. `f08b6cb`, `7bd7375`, `398b58a`).
|
||
3. MTP deferred (see above).
|
||
4. DS thinks verbosely (~13:1 reasoning:content) — eval item; consumers need generous `max_tokens`.
|
||
5. MeroMero full 256K needs util 0.55 (GPU0 ~1.8GB free, tight); ran at 0.52 for headroom (~4.6GB).
|