memory: snapshot — LiteLLM #355-residual investigation + BuildKit deploy-speed cache (both in-flight); #355 fix validated, Ledger tier-3 provisioned, assistant/thoughtful-assistant roles added, 3 zombie sessions retired
This commit is contained in:
+32
-256
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-07-11_
|
||||
_Last updated: 2026-07-13_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -102,268 +102,41 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-07-11 — the R17 TTS audition CONCLUDED and **Zonos was productionized** (won the expressive tier →
|
||||
shipped as the `zonos-gateway` stack, below). Fleet changes this session: **omnivoice RETIRED**, **vllm-morpheus
|
||||
KILLED** (operator: unimpressive), **ComfyUI TAKEN DOWN** (A6000 freed for the audition; comfy-dev PARKED — the
|
||||
v0.27.1 bump it validated stays LANDED, just not running). **Higgs** auditioned well + streams (room in the zoo)
|
||||
but is PARKING-LOT, not shipped. Immediate fresh-context pickup:_
|
||||
_As of 2026-07-13 — two active tasks (per /snapshot args): the LiteLLM gateway #355-residual investigation and wiring the runner-side persistent BuildKit cache. The big WT #355 validation saga + Ledger provisioning + the assistant-role adds all LANDED this session; the concluded TTS/Zonos/RP-seat/ComfyUI/T1 in-flight was rolled out of this section (it lives in git + auto-memory)._
|
||||
|
||||
- **🗣️ ZONOS-GATEWAY TTS — SHIPPED + ONLINE (irv-ml1), the R17 expressive winner.** Two pieces. (1) **Zonos
|
||||
engine** — native zonos2 server (`Zyphra/ZONOS2`, Apache), bare `python3` on irv-ml1 **:1920**, on the **3090**
|
||||
(operator moved it here off the retired omnivoice); streams float32 PCM @44.1kHz via `POST /tts/generate`;
|
||||
emotion = **embedding-direction conditioning (NOT tags)** + **clone-gated** (no-ops without a speaker ref);
|
||||
vendored copy kept **STOCK**. (2) **zonos-gateway wrapper** — OpenAI `/v1/audio/speech` facade over the native
|
||||
endpoint, **containerized** at `/opt/docker/compose/zonos-gateway/` (image `zonos-gateway:0.2.0`,
|
||||
`restart: unless-stopped`, reaches host Zonos via `host.docker.internal:1920`), **:8890**, default voice **Cora**
|
||||
(voices = `*.wav` baked into the image). Full **24-dial** expressive surface + presets
|
||||
(neutral/warm/excited/sad/intense/whisper) via `GET /v1/dials`; `response_format` pcm|wav. Repo
|
||||
`~/development/zonos-gateway` (v0.2.0, LOCAL git only). **In asset-engine catalog**
|
||||
(`docs/asset-engine/services.yaml`, id `zonos-gateway`, status experimental) → shows **online** (background poller
|
||||
auto-adopts the running container, NO VRAM gate, NO `/start`) → experiment at
|
||||
`http://10.250.50.70:8200/ui/stream-audition/zonos-gateway`. **gateway-chat** (ana-docker:8091) repointed DIRECT
|
||||
to it (OpenAI body, Cora, streamed, quotes joined not chunked). ⚠️ **NEVER expose the rep-penalty dials**
|
||||
(50/1.2/8 — LOAD-BEARING; OFF → 70s-silence footgun). Auto-memory `reference_zonos_tts_stack`. **PENDING
|
||||
close-outs:** (a) repoint LiteLLM `ext-tts` alias → the wrapper (generic hot-swap point; NOTE LiteLLM buffers
|
||||
audio → streaming needs a DIRECT wrapper hit, not through `ext-tts`); (b) revert vendored Zonos `api_server.py`
|
||||
OpenAI-endpoint patches to STOCK (rep-penalty + `_voice_to_speaker_id` + Cora reg; `.bak-*-20260710` on irv-ml1 —
|
||||
the wrapper made them redundant); (c) cosmetic: catalog `model.image` reads `local/zonos-gateway:0.1.0`, reality
|
||||
`zonos-gateway:0.2.0` — reconcile on next catalog touch (a resync bounces the live lab). Old bare-deploy dir
|
||||
`/home/lkraven/zonos-gateway` on irv-ml1 = idle cruft.
|
||||
### 🔺 ACTIVE 1 — LiteLLM gateway #355-residual investigation (our domain)
|
||||
- **The data-confirmed finding:** WT #355's residual 300s hang is NOT the seat and NOT WT-pre-send — it's OUR **LiteLLM gateway** (ana-docker `10.250.50.70:4000`) holding two char-rp-reasoning requests **~21 min** (spend_logs dur: **1255.9s + 1357.4s**, startTimes 16:54:44 + 16:56:26Z = the two wedge onsets) while the **Deckard seat sat IDLE**. The seat (ana-ml2:8018 llama.cpp, `--reasoning-budget 400` — forecloses an indefinite mid-thinking hang) completes every request **≤72s** and received no request at the wedge onset → **EXONERATED**. This CONTRADICTS worldtree-dev's "seat wedging mid-thinking" premise, which I'd relayed unconfirmed until the operator challenged it.
|
||||
- **DECISIVE next step (brokkr's dwarf-panel, read-only pcap analysis):** the **FIN-check** — in the WT↔gateway pcap for the WEDGED streams, did WT send FIN/RST to the gateway at its 302s stall-cancel?
|
||||
- WT closed but gateway held anyway → **pure LiteLLM bug** (brokkr's lead: a concurrency-slot LEAK — cancelled requests don't free LiteLLM's internal slot → new invokes queue ~21 min behind leaked slots while the seat idles). Our fix.
|
||||
- WT did NOT close → WT's un-propagated cancel (earlier httpcore shielded-`aclose` can't-kill bug) → worldtree's already-planned force-close fix also releases the hold.
|
||||
- **Then:** pull LiteLLM router / `max_parallel_requests` config for deployment `hosted_vllm/deckard-pkd-27b`; confirm whether the seat EVER `launch_slot`'d the 1255s request (never-served vs delayed). pcap-only tools: no tshark on corviduo — use `tcpdump -r`.
|
||||
- **Data on hand:** WT-netns pcaps + py-spy dumps at `corviduo-dev:~infra-ops/py-spy-355/wedge_20260713_095626/` (+ 100039/100341/100543); the 095626 capture overlaps turn-2064's wedge (16:54:46–16:59:46Z). **py-spy + tcpdump are installed on corviduo-dev** (`~/.local/bin/py-spy`, hand-carried musl binary; no pip/uv on that box). LiteLLM spend_logs = `litellm-db` on ana-docker (`docker exec litellm printenv DATABASE_URL`; SpendLogs cols are camelCase → double-quote them).
|
||||
- **Status:** WT-code panel DONE (seat-innocent accepted); residual is LiteLLM-forensic (mine, brokkr reasons alongside). **Awaiting operator go on the FIN-check.**
|
||||
|
||||
- **🔊 R17 TTS AUDITION — CONCLUDED (operator ranked live).** 3-model expressive audition
|
||||
(Chatterbox-fast / ZONOS2 / Higgs-v3) on irv-ml1's A6000 (ComfyUI taken down to free it). **Latency smoke**
|
||||
(`~/tts-audition/records/latency.csv`): Chatterbox TTFA 0.47–0.80s / RTF 0.28–0.38 (24kHz streamed — fastest,
|
||||
stays the low-latency front-end); ZONOS2 warm TTFA ~8s / RTF 2.79 (44.1kHz, clone-gated emotion); Higgs whole-file
|
||||
TTFA≈gen 2.6–2.8s / RTF 0.98 (NOT streamed — footnote, not comparable to the streamed two). **Outcome: Zonos
|
||||
productionized** for the expressive tier (see zonos-gateway above); **Higgs = parking-lot** (streams, room in the
|
||||
zoo, not shipped — Path-A in-process build preserved at `~/tts-audition/`: venv `venvs/higgs`, transformers 5.5.0,
|
||||
`harness/higgs_native.py`, Qwen3-4B backbone + baked codec; revisit if wanted). **Chatterbox-fast** stays the
|
||||
low-latency front-end. Protocol: brokkr-smithy R17 `tts-audition-protocol.md`.
|
||||
### 🔺 ACTIVE 2 — Worldtree deploy-speed: wire the runner-side persistent BuildKit cache
|
||||
- **Measured (gitea Actions API, `vh/worldtree`, claude-bot token):** full code deploy ≈**12 min**; bottleneck = `build-and-deploy` **~11 min** (685/614/659s recent). `lint-test` ~7.5 min runs PARALLEL (off critical path). `deploy`/recreate ≈50s (fine — not the problem).
|
||||
- **Root cause (Dockerfile at repo root):** the dep install `RUN --mount=type=secret,... uv sync --frozen --no-dev --no-install-project --no-cache` uses `--no-cache` AND has **no BuildKit `--mount=type=cache`** → re-downloads+rebuilds ALL Python deps cold every build. Layer ordering is CORRECT (`COPY pyproject.toml uv.lock` + `packages/` before `uv sync`, app code after) — the problem is purely caching.
|
||||
- **Fix (split by domain):** (a) worldtree-dev's Dockerfile diff — add `--mount=type=cache,target=/root/.cache/uv` + drop `--no-cache`; (b) **MINE, runner-side** — persist the BuildKit layer cache between CI runs (`--cache-to/--cache-from` registry or runner-local). Config-only changes already skip the pipeline via a pinned recreate.
|
||||
- **Confirm first:** pull the build-STEP log to verify the dep-install is THE dominant sub-step (vs the apt layer or the registry push) before handing worldtree-dev the diff — don't assume. **Awaiting operator go.**
|
||||
|
||||
- **🖼️ COMFYUI — v0.27.1 LANDED + comfy-dev-VALIDATED, now TAKEN DOWN (A6000 freed; comfy-dev PARKED).** The
|
||||
v0.27.1 bump stays LANDED (validated clean by comfy-dev, thread 01KX6D3C — 3-tier sweep 189 graphs / 105
|
||||
node-types, ZERO breakage, all 9 templates + smoke renders OK: Z-Image / FLUX.1-dev / Illustrious-SDXL /
|
||||
FLUX.2-GGUF / Qwen-edit sage-AttentionSelector / LTX i2v). ComfyUI is just **not RUNNING** — operator took it down
|
||||
to give the R17 audition the A6000. **Restart when comfy-dev un-parks.** Landed on the EXISTING **torch
|
||||
2.12.1+cu129, SageAttention 2.2.0 intact** (deliberately did NOT bump torch — the 2.13 bump was the mmartial
|
||||
boot-upgrade's ABI crash-loop cause, never an LTX requirement; **SETTLED = don't bump**). Mechanics: code
|
||||
`/worktank/comfyui/run/ComfyUI` (=/comfy/mnt/ComfyUI in-container), `docker exec` lands as uid **1025** → use
|
||||
`-u 1000` + venv python `/comfy/mnt/venv/bin/python` (venv owned lkraven:1000), `DISABLE_UPGRADES=true` pinned,
|
||||
NVIDIA_VISIBLE_DEVICES=1 (A6000). Rollback net `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar`. LTX
|
||||
Phase-2 node-install = its own future thread (installs uid-1000).
|
||||
### Landed this session (2026-07-13) — context for the fresh session
|
||||
- **WT #355 turn-lifecycle fix VALIDATED** (worldtree b60): wedged turns self-terminate cancelled/stalled at the 300s watchdog (turns 2064/2065 = 302s/360s) vs pre-b60 turn 2061's 16-min no-terminal silence. The residual (ACTIVE 1) is a separate LiteLLM matter.
|
||||
- **Ledger tier-3 provisioned** (`ledger:miranda`) on personal `:8081`: key minted (user `ledger`, key_id `b38932f5`), GPG-delivered + verified + cleartext shredded; scope confirmed (user tier has `bifrost:invoke`+`agents.define`); allowlist `10.100.10.50:8770` activated by the b60 recreate.
|
||||
- **`assistant`+`thoughtful-assistant` roles** added (bind `gen`/`gen-reasoning`, kind capability, grants `[foundational]`) on personal+demo; canonicalized by worldtree-dev (`d8bd497`).
|
||||
- **3 zombie soong sessions retired** (2ad34ec6+7d23052f+0775c9b1 via `POST /admin/sessions/<id>/retire`); pre-b60 stale-active backlog HELD (worldtree's #356 rehydrate fix reclaims that class).
|
||||
|
||||
- **📅 BIWEEKLY LANDSCAPE-SCAN CRON — SETTLED.** Live systemd --user timer on nh3-dev (next auto-run 2026-07-15
|
||||
09:00 PT), dedicated `brokkr-scan-dev` handle, notify = `post --to brokkr-smithy-dev` (Vuong ruled inbox-post,
|
||||
no phone-push). Fully validated end-to-end. Nothing owed; watch for a brokkr-scan-dev first-dwarf-arm routing
|
||||
hiccup flag on the 15th.
|
||||
|
||||
- **📬 althing:** a harness-tracked wake-listener was armed for comfy-dev's reply. **A /clear kills the monitor
|
||||
loop → the fresh session must re-arm (`/althing:monitor`)**; the 📬 banner surfaces new peer mail on next prompt
|
||||
regardless.
|
||||
|
||||
_Earlier (2026-07-09) context preserved below:_
|
||||
|
||||
_As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp +
|
||||
char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped
|
||||
end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_
|
||||
|
||||
- **⚠️ SUPERSEDED 2026-07-11: vllm-morpheus KILLED (operator: unimpressive); gateway-chat is now voiced by
|
||||
zonos-gateway (above), NOT mOrpheus. Block kept for the SNAC/clone-voice detail only.**
|
||||
- **🗣️ mOrpheus TTS — PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat.** Off-the-shelf
|
||||
uncensored Orpheus TTS (`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`; Llama-3.2-3B LLM
|
||||
→ SNAC 24kHz audio). Canonical `stacks/mOrpheus/`; deployed at `/home/lkraven/morpheus/` on irv-ml1.
|
||||
**Two containers:** `vllm-morpheus` (engine — **bf16 REQUIRED** (FP8 breaks audio); **image PINNED
|
||||
`vllm/vllm-openai:v0.23.0`** (`latest` crashes on Ampere import); on the **3090** (A6000 is
|
||||
comfy-contended, spikes to ~41GB); `--enforce-eager` + util 0.32 ≈ 7.9GB) + `morpheus-tts` (CPU
|
||||
FastAPI — tokenizer + SNAC decode; `/tts` whole-clip WAV, `/tts/stream` windowed-chunk PCM stream,
|
||||
zero-shot cloning; CORS-enabled). **RTF ~1.0 end-to-end** (enforce-eager caps it; vLLM single-3090
|
||||
concurrency is near-linear to `--max-num-seqs 8` = 8× at 707 tok/s). **gateway-chat** (ana-docker:8091,
|
||||
the zero-dep `tools/gateway-chat.html`) auto-voices assistant **quoted** dialogue: chunk by QUOTED
|
||||
SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), 🔊 sidebar
|
||||
toggle + endpoint/voice fields. **Clone voices** (drop `<name>.wav` + `<name>.txt` in the voices dir →
|
||||
`voice="<name>"` clones it): `baddy` (native speaker), `beatrice`, `whisper` (kokoro **af_nicole**, a
|
||||
genuine whisper voiced 0.24; mOrpheus partially re-voices it → clone voiced 0.42, breathy not pure).
|
||||
Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/
|
||||
sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at `stacks/mOrpheus/agent-voice-prompt.md`.
|
||||
**OPEN:** whisper re-voices (checkpoint limit, not the reference); throwaway audition `:8210` +
|
||||
`/home/lkraven/orpheus-audition/` still up on irv-ml1 (teardown pending, ~6.3G weights moved out already);
|
||||
production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen).
|
||||
|
||||
- **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:**
|
||||
- **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` — LIVE + GOOD.** Off-the-shelf NVFP4 of
|
||||
`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only` (MoE 256e/8-active =
|
||||
qwen3_5_moe, vision, GDN-hybrid; **modelopt** format → `--quantization modelopt`; vision bf16; 24G at
|
||||
`/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`). UNCENSORED (writes brutal fiction think-off, no system prompt).
|
||||
"Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken).
|
||||
General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`,
|
||||
`AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now
|
||||
`${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay
|
||||
transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B
|
||||
serves are RETIRED.
|
||||
- **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral)
|
||||
dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat:
|
||||
elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence
|
||||
(live-tested). Replaced the broken Angel NVFP4. Alt prose model (`.env` swap `CHARRP_MODEL`):
|
||||
`MS3.2-PaintedFantasy-v4.1-24B` (more literary flair, looser POV). Both GGUFs pre-pulled at `/tank/aimodels/llm/rp/`.
|
||||
- **char-rp-reasoning (:8018) = `Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M` GGUF — LIVE
|
||||
(A/B WINNER 2026-07-08, REPLACED RpR-v4).** DavidAU creative (PKD) tune on Qwen3.5-27B (container
|
||||
`llama-charrp-reasoning`). Native Qwen3.5 template → `--reasoning on` managed CoT SURFACES in `reasoning_content`,
|
||||
content stays clean prose, `--reasoning-budget 400` caps it; **DRY server-side** (sampler order = dry-after-temp)
|
||||
tames looping. A/B: **0/30 loops + 0/30 refusals**, ~57 tok/s. Runs on the base llama-swap b8840 image (Qwen3.5).
|
||||
Best-of-breed-per-seat (NOT the same model as char-rp).
|
||||
|
||||
- **🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08).** char-rp = **Magidonia-24B-v4.3** (Magistral prose,
|
||||
65tps); char-rp-reasoning = **Deckard-PKD (Qwen3.5-27B)** (managed reasoning + DRY, ~57tps; A/B winner over RpR-v4 — see re-A/B note below). Both GGUF via
|
||||
llama.cpp (`char-rp-gguf` stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo
|
||||
`stacks/char-rp-gguf/`; gateway rewired (`char-rp`→:8016, `char-rp-reasoning`→:8018, Mistral/QwQ samplers, dropped
|
||||
the Qwen `enable_thinking` kwarg). **KEY FINDINGS:** (a) no single dense 24-32B is BOTH an elite non-thinking prose
|
||||
seat AND a clean *managed*-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral `[THINK]`
|
||||
discipline is loose (won't close `[/THINK]` on substantive reasoning → prose bleeds into reasoning, content empties);
|
||||
Cydonia-R1's `<think>` is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template
|
||||
opens `<think>` natively → llama.cpp manages+caps it; (c) **Pantheon-Reasoning-27B stays REJECTED** (re-censors in
|
||||
`<think>`; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in `reasoning_content` (fixes the
|
||||
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
|
||||
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
|
||||
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
|
||||
**REASONING-SEAT RE-A/B (2026-07-08, later — RpR→Deckard):** operator wanted a reasoning model that TAKES DRY
|
||||
(RpR forbids rep/DRY → 1/30 loop tail). Full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored):
|
||||
**Deckard-PKD (Qwen3.5-27B) WON** (0/30 loops, 0/30 refusals, clean) → NOW the char-rp-reasoning seat.
|
||||
RpR-v4: 0 refusals but 1/30 loop (no-DRY). **Pantheon-Reasoning-27B: 7/30 explicit refusals** (DeepSeek-distilled
|
||||
re-censor IS real across the batch, milder than feared; NOT wholesale-rejected) — kept on disk as alt.
|
||||
Snowdrop-v0.5-Type-S + Gembrain-31B (Gemma-4): FLOORED — llama.cpp can't do MANAGED reasoning on them (Snowdrop's
|
||||
ChatML template has no `<think>`/`enable_thinking` hook; Gemma-4's reasoning parser splits content wrong). GATE for a
|
||||
llama.cpp reasoning seat = STOCK template natively opens `<think>` or has `enable_thinking` (Qwen3.x/QwQ do; ChatML +
|
||||
Gemma-4 don't). **INFRA:** llama-swap b8840 image can't load Qwen3.6/Gemma-4 archs → use
|
||||
`ghcr.io/ggml-org/llama.cpp:server-cuda` (newer, pulled on ana-ml2) for those; Deckard (Qwen3.5) runs on b8840.
|
||||
**brokkr frozen-scorer CONFIRMED (median composite, content-only; lower=cleaner):** Deckard **2.176** (0 loop/0
|
||||
refuse — DEPLOYED, big win over RpR **3.716** + its loop-tail); Pantheon-Reasoning **1.383** (cleanest prose of the
|
||||
field — beats even Magidonia char-rp 1.774 — BUT 7/30 explicit refusals → rejected: clean-but-refuses loses to
|
||||
clean-and-uncensored). **MEMORY-LAYOUT note:** Deckard (Qwen3.5-27B) is GDN-hybrid → only **16 KV-caching layers** →
|
||||
KV is TINY (1.4G @ 40K vs Magidonia's 10.9G @ 128K, 40 dense layers); Deckard native ctx = 256K, so the reasoning
|
||||
seat's ctx scales nearly free. **Deckard BUMPED to full 256K (262144) 2026-07-08** (KV 8.7G q8_0). GPU0 layout now:
|
||||
gen ~35G (256K/16-seq/util0.37), char-rp Magidonia ~30G (128K/4-slot), char-rp-reasoning Deckard ~28G (256K/4-slot),
|
||||
**~4G free** (all static allocations → stable, no OOM). **Canonical Deckard RP-reasoning samplers CONFIRMED by dvalin 2026-07-08**
|
||||
(thread 01KX1Y7P): the live empirical set IS canonical — temp1.0/top_p0.95/top_k40/min_p0.05, NO presence/rep
|
||||
penalty, DRY0.8 server-side (base1.75/len2, dry-after-temp), reasoning-budget400. Delta from card base-thinking:
|
||||
top_k20→40, min_p0→0.05, presence1.5→0, DRY-on. Tuning ladder: flat prose→min_p0.08; loops→DRY0.9; over-damped→
|
||||
DRY0.6/off. Do NOT import RpR/QwQ rules (QwQ hated DRY; Qwen3.5 benefits from DRY over rep_penalty).
|
||||
**MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ
|
||||
native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen
|
||||
gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%,
|
||||
so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util +
|
||||
static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots
|
||||
share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.
|
||||
|
||||
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
|
||||
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
|
||||
top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 /
|
||||
presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake;
|
||||
verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** /
|
||||
top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief
|
||||
scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY.
|
||||
Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`.
|
||||
|
||||
- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py`
|
||||
(llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py`
|
||||
(llm-compressor + qwen ignore + `trust_remote_code` + a torchvision-gate bypass). ModelOpt path:
|
||||
`/home/lkraven/nvfp4-quant/.venv-mopt` (modelopt 0.44) + `mopt_w4a16.py` (= `mtq.quantize` + `export_hf_checkpoint`
|
||||
→ modelopt format w/ `hf_quant_config.json`). Downloads via a detached vLLM container running
|
||||
`huggingface_hub.snapshot_download` to `/tank/aimodels/<name>-{bf16,nvfp4}`. **Gotchas:** llm-compressor
|
||||
`save_compressed` DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps
|
||||
`re:.*visual.*` + `re:.*linear_attn.*`(GDN) + lm_head + embeds + `mlp.gate`(router) bf16.
|
||||
|
||||
- **🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf).** DPO came back slop → operator killed
|
||||
home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models,
|
||||
and every other project ("ritual seppuku" otherwise). **Revival map:** adapters `data/spike/{sft,dpo}_adapter`,
|
||||
harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G).
|
||||
mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.)
|
||||
|
||||
_Earlier (2026-07-07):_
|
||||
|
||||
- **T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving.** SFT ran
|
||||
clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final
|
||||
loss 2.445); adapter at `ana-ml2:~/model-training-forge/data/spike/sft_adapter` (124MB, r16 α32, targets
|
||||
`self_attn.q/k/v/o_proj` + `linear_attn.in_proj_qkv/out_proj`, NO `in_proj_z`). DPO leg deferred (needs
|
||||
mergekit resolved + on-policy `rejected` regen). **THE WALL: the adapter is INERT on the vLLM NVFP4
|
||||
hot-load** — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent
|
||||
no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4
|
||||
AND FP8 both inert) AND adapter-format-agnostic (separate-key `sft_adapter_zc` loads clean via vLLM's
|
||||
allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix
|
||||
it. **HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now** (static;
|
||||
reserved for a later RP-quant track only).
|
||||
|
||||
- **HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup
|
||||
validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a
|
||||
one-line fix. MERGE RECOMMENDED (operator's call, pending).** Ran `sft_adapter_zc` on SGLang 0.5.13 bf16
|
||||
(`--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`, GPU0, gen+rp
|
||||
stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted
|
||||
fine. **Died at `lora_manager.init_lora_shapes` with a NAME-VALIDATION `ValueError` (NOT the feared
|
||||
in_proj_qkvz shape-assert):** adapter's `in_proj_qkv`+`in_proj_z` "not included in --lora-target-modules
|
||||
[...in_proj_qkvz...]". **ROOT CAUSE (pinned):** `srt/lora/utils.py:235 get_normalized_target_modules` uses
|
||||
a HARDCODED `params_mapping` dict that maps `q/k/v_proj→qkv_proj` (why self-attn works) but OMITS
|
||||
`in_proj_qkv→in_proj_qkvz` + `in_proj_z→in_proj_qkvz`; it does NOT consult the model's
|
||||
`packed_modules_mapping` (which DOES declare `"in_proj_qkvz":["in_proj_qkv","in_proj_z"]` at
|
||||
`models/qwen3_5.py:1010`). Everything else is ALREADY in place: `get_hidden_dim("in_proj_qkvz")` IS
|
||||
implemented in the model class (`:1040`, → `(hidden_size, key_dim*2+value_dim*2)`; the "utils NotImplementedError"
|
||||
was only the DEFAULT); `get_stacked_multiply("in_proj_qkvz")=4` in the utils default dict; base-weight
|
||||
`stacked_params_mapping` fuses `in_proj_qkv→shards(0,1,2)`,`in_proj_z→shard 3`. **PATCH APPLIED + RE-SMOKED
|
||||
(bind-mounted a patched utils.py overlay into the container):** the 2-line add cleared `init_lora_shapes`;
|
||||
server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z
|
||||
shrink — B-side math right per mtf-dev), **self-attn loaded end-to-end** ("loaded weights for [o_proj,out_proj,
|
||||
qkv_proj]"). **BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable):**
|
||||
(1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]");
|
||||
(2) inference buffer-fill `utils.py:300 get_target_module_name` RAISED — SUBSTRING match, `in_proj_qkvz` ⊄
|
||||
`...in_proj_qkv.` and `qkv_proj` ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" →
|
||||
HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3).
|
||||
**FINISH LINE (if ever revived):** teach `get_target_module_name` + the load-consolidation the
|
||||
`in_proj_qkv/in_proj_z→in_proj_qkvz` map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer
|
||||
shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one
|
||||
targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied +
|
||||
loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't
|
||||
LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: `sft_adapter` (separate), `sft_adapter_zc`
|
||||
(separate + zero `in_proj_z`, the canonical one — THIS smoke), `sft_adapter_fused` (fused — wrong), plus
|
||||
`sft_adapter_attnonly`. All at `/home/lkraven/model-training-forge/data/spike/`.
|
||||
|
||||
- **MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned.** peft merge_and_unload (~20min, LOW
|
||||
wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training
|
||||
is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h,
|
||||
MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain
|
||||
base (no `--enable-lora`). Static, NOT hot-swappable.
|
||||
|
||||
- **comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session).** Built/deployed the worker
|
||||
(`services/lora-training-worker/`, irv-ml1 `0.0.0.0:8203`, runs as llmuser, fixed-invocation sd-scripts,
|
||||
systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra
|
||||
fast/lean). Phase 2 publish-step (`published_lora_name` + optional `train_id` field). Phase 2.5 GC
|
||||
contract (shape A: `GET /published-loras` + `POST /gc/published {train_ids}`) CONFIRMED — worker impl
|
||||
DEFERRED to after the T1 hot-swap work. auto-memory `reference_lora_training_worker`.
|
||||
|
||||
- **Serving state (SUPERSEDED — see top; AEON retired 2026-07-08):** during the T1 work, AEON gen (:8015) +
|
||||
char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via `docker start vllm-aeon-gen
|
||||
vllm-aeon-rp`. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced).
|
||||
|
||||
- **worldtree-dev vh/pypi read token** delivered (bifrost==1.1.0 pin, #349); dedicated read:package token
|
||||
at `nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt`. The over-scoped write:package token the operator
|
||||
pasted was revoked + verified dead.
|
||||
|
||||
- **Backups — STILL OPEN:** rotate the 5 disclosed rest-server creds (operator, offline); confirm
|
||||
esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`.
|
||||
|
||||
- **Standing / parked:** inert `mood.decay_rate`/`stale_hours` keys in deployed WT config; Mac Pro
|
||||
migration (hw-gated); R22 gateway-key at `/home/lkraven/.r22-gateway-key` (mode 600, paid GLM, do NOT
|
||||
delete); Deckard staged on ana-ml2 as T1's writing benchmark; `/books` transient NFS mount on nh3-dev;
|
||||
**dvalin sampler re-tune pass** for char-rp / char-rp-reasoning once they accumulate live Worldtree/ST
|
||||
character-role traffic (dvalin offered 2026-07-08 to do a follow-up after more live sessions; the current
|
||||
paper defaults + 2-prompt A/B are the starting point — real long-session usage is where repetition/drift
|
||||
surfaces; re-open thread `01KX1DS6…` to take them up on it).
|
||||
|
||||
- **Worldtree config-propagation (reference):** demo+personal bind-mount config from
|
||||
`/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload via
|
||||
`docker restart <container>`, NEVER `compose up` (stale-`:latest` footgun). Role registry loaded ONCE +
|
||||
CACHED at startup → a bind-mount `model_roles.yaml` change needs a restart; pre-stage BEFORE the
|
||||
activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image.
|
||||
### Still-open standing items (carried)
|
||||
- **T1 SFT LoRA (AEON-27B E-RP)** — adapter banked but BLOCKED on hot-swap serving; DORMANT (operator pivoted to off-the-shelf). Merge fallback primed (~2-3h) if hot-swap abandoned.
|
||||
- **Backups** — rotate the 5 disclosed rest-server creds (operator, offline); confirm cross-site restic mirror.
|
||||
- **althing monitor dies on /clear** — re-arm `/althing:monitor` in the fresh session (open await-reply loops: brokkr on the #355-residual, worldtree-dev on the deploy-resilience follow-ons #356/#357/#358).
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviduo). See in-flight ACTIVE 1.
|
||||
- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize.
|
||||
- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate).
|
||||
- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated).
|
||||
|
||||
- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev
|
||||
R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml`
|
||||
(`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347
|
||||
@@ -666,6 +439,9 @@ _125 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
|
||||
- `[2026-07-13]` `althing-cli reply <THREAD_id>` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to <peer>`. Bit me several times this session.
|
||||
|
||||
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
|
||||
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
|
||||
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
|
||||
|
||||
Reference in New Issue
Block a user