713 lines
65 KiB
Markdown
713 lines
65 KiB
Markdown
# Persistent memory — eshpfi-management
|
||
|
||
_Last updated: 2026-07-10_
|
||
|
||
## Repo purpose
|
||
|
||
Reference workspace for PFI infrastructure: server inventory, canonical
|
||
Docker Compose stacks, ops playbooks, and conventions. Authoritative
|
||
copies of compose files live on the servers under
|
||
`/opt/docker/compose/<stack>/`; this repo mirrors them for version
|
||
control, editing, planning, and CI-driven deploys. **It was originally
|
||
spun up to handle the fleet backups** — keep that lens when triaging
|
||
backup/storage issues.
|
||
|
||
## Tools and conventions
|
||
|
||
Sister repos (separate gitea repos, deployed by playbooks here):
|
||
|
||
| Repo | Role | CI status |
|
||
|---|---|---|
|
||
| `vh/task-board` | MCP + web dashboard for assistant task state (port 7878) | push-to-main → CI deploys (2026-04-29) |
|
||
| `vh/vor` | Inquisitor UI sidecar (port 7879) | push-to-main → CI deploys (2026-04-29) |
|
||
| `vh/nevermore` | Twice-daily LLM-curated briefing (port 8181, replaces news-digest) | push-to-main → CI deploys (2026-04-30) |
|
||
| `vh/asset-engine` | Internal control plane over inference services (port 8200, LAN-direct) | push-to-main → CI deploys (2026-05-12) |
|
||
| `vh/althing` | Lean trusted inter-agent message bus — **v2 "email model" (v2.0.0b2, 2026-07)**: per-box local-SQLite bus + courier/receiver for P2P over the 10.x net; pillars = open-loops / per-box herald + wake-listener / roaming owner API `/owner/*` / `althing-mcp` stdio surface. The v0.15 lean-bus cut RIPPED moderation / chamber / forseti-daemon / agent-runner / redis-valkey. | per-box `uv tool install` (NOT CI-deploy); **nh3-dev = the DEV box** (editable install of `~/development/althing`, gets new versions first); **nh3-extdev** a mesh peer (model B: althing-svc + shared `/srv/althing`) |
|
||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||
| `vh/Worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration. **gitea-runner builds on ana-docker**; claude-bot ADMIN collaborator (2026-06-20). Now v1.0.0b19. | push-to-main → CI build-and-deploy (runner on ana-docker) |
|
||
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
|
||
| `vh/arbo` | Catalog-driven ComfyUI engine (irv-ml1 :8201, comfy-dev owns engine/catalog/image) | push-to-main → gitea Actions CI (deploy-engine.sh, build-local, health-gated) now LIVE; catalog via :9009 webhook |
|
||
| `model-training-forge` (mtf-dev) | Fine-tuning recipe forge; **T1 = E-RP writing LoRA, retargeted qwopus-122B→AEON-27B (2026-07-06)** (SFT→DPO, LitBench-RM reward) | training runs, not a deployed sidecar |
|
||
|
||
(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
|
||
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
|
||
no longer deployed sidecars here. See Recent decisions.)
|
||
|
||
- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
|
||
fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
|
||
VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
|
||
cross-site restic targets — see `docs/runbooks/disaster-recovery.md`
|
||
for the blast-radius matrix. **⚠️ The restic file+DB layer routes
|
||
through TWO rest-servers** (`rest-server-ana` @ ana-docker:8000 →
|
||
ana-docker/ana-ml2/esh-docker-vm/vm-esh-nas; `rest-server-nh3` @
|
||
nh3-nas:8000 → irv-ml1/nh3-docker). Both depend on their NAS's NFS
|
||
export of `/mnt/backup`. (rest-server-ana recovered 2026-06-20.)
|
||
|
||
- **`pull-hf-repo.yaml`** is the canonical "get a HuggingFace
|
||
model/dataset onto ana-ml2's shared cache at
|
||
`/tank/aimodels/huggingface/`" playbook. Supports `--var repo_type=model|dataset|space`. Replaces ad-hoc `huggingface_hub.snapshot_download` patterns.
|
||
|
||
- **Worldtree admin auth — per-instance.** Each Worldtree deployment
|
||
(demo :8080, personal :8081, pinned :8082) has its own Heimdall
|
||
registry and its own bootstrap admin key. Infra-ops's stored
|
||
long-lived admin key (`key_id 61419c92`) at
|
||
`ana-docker:/opt/docker/conf/.secrets/worldtree-infra-ops-admin`
|
||
auths against **demo only**. Personal-instance admin (the
|
||
`~/.config/worldtree/personal-admin-token`, mode 600) POSTs
|
||
`/admin/keys` (mints per-project keys; takes `user_id`+`label`, **no
|
||
scope param** — scopes are tier-derived). **On-instance mint recipe
|
||
(cleaner than DB-manip):** `docker exec worldtree-worldtree-api-1` POST
|
||
`/admin/keys` with the in-container `WORLDTREE_BOOTSTRAP_ADMIN_KEY`; cleartext
|
||
once in `.key`=`wt_live_+16hex`. auto-memory `reference_worldtree_demo_key_mint`.
|
||
|
||
- **Per-project user keys against personal Worldtree** (issued
|
||
2026-05-19): `skaldsong:79744637`, `skaldsong:7c1dbbbe`,
|
||
`althing:50d85460`, `mead-hall:a360822d`. Mint via `/admin/keys`, drop
|
||
value to `/tmp/wt-personal-<name>.key` mode 600, dev collects +
|
||
shreds (DO NOT cat to chat transcript).
|
||
|
||
- **Skaldsong CD pattern (registry-pull).** vh/skaldsong's CI builds and
|
||
pushes `gitea.phasefinal.com/vh/skaldsong:<sha>` + `:latest`;
|
||
`playbooks/deploy-skaldsong.yaml` on ana-docker pulls + recreates.
|
||
SHA-pin only. Prereq: host needs `docker login gitea.phasefinal.com` once.
|
||
|
||
- **gitea internal route for fleet hosts.** gitea is a container on
|
||
**ana-docker** — git-SSH `10.250.50.70:222`, HTTP `:3000`. Fleet/colo
|
||
hosts must use this internal route, NOT public `gitea.phasefinal.com`
|
||
(`38.120.12.44`) — the public path fail2bans the host egress IP. Full
|
||
gotcha in `docs/orientation.md` → Git/gitea.
|
||
|
||
- **docker-as-root pattern** (for ops with no admin API, or to edit
|
||
deploy-owned/root-owned files without sudo): `docker run --rm -v
|
||
<target-dir>:/wt docker:cli sh -c "..."`. docker-group membership is
|
||
effectively root via bind-mount. **Foot-gun: relative paths in compose.yaml
|
||
resolve against the sandbox CWD but the daemon interprets them against the
|
||
HOST fs — always pass `-e VAR=/abs/path` for any relative-default config dir.**
|
||
|
||
- **`scripts/elway` sudo handling** — elway prompts for the sudo password
|
||
ONCE via `getpass` before the first `sudo: true` step → can't run
|
||
unattended from a non-TTY tool if any step needs sudo. Sudo-free
|
||
playbooks run fully non-interactive over key SSH.
|
||
|
||
- **Per-host SSH identity matters for sudo.** infra-ops has NOPASSWD sudo
|
||
on most PFI Linux boxes (corviduo-dev included since 2026-06-15). On
|
||
ana-docker: **default `ssh ana-docker` = `lkraven`** (docker-group, NO
|
||
passwordless sudo); **`ssh infra-ops@ana-docker` HAS NOPASSWD root**. **→
|
||
For any sudo op on ana-docker, use `ssh infra-ops@ana-docker`.** `ssh
|
||
infra-ops@10.100.10.50` (nh3-dev) ALSO NOPASSWD sudo; on **nh3-extdev** infra-ops
|
||
is sudo-LESS by design (`ssh lkraven@10.100.50.42` is the NOPASSWD path). **irv-ml1:
|
||
`ssh irv-ml1` = lkraven, docker-group (plain docker) but sudo needs a PASSWORD
|
||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||
|
||
## Current state / in-flight
|
||
|
||
_As of 2026-07-10 — session ran three Vuong-authorized peer tasks: brokkr TTS audition (DELEGATED, in-flight),
|
||
comfy-dev ComfyUI bump (attempted → ROLLED BACK, blocked on comfy-dev), biweekly scan cron (SETTLED). Full detail
|
||
in the 2026-07-10 Recent-decisions entries. Immediate fresh-context pickup:_
|
||
|
||
- **🔊 TTS AUDITION (research-slot) — env+harness build DELEGATED to a background agent (IN-FLIGHT).** Stand up
|
||
Higgs TTS 3 (`bosonai/higgs-tts-3-4b`, ungated despite NC) + ZONOS2 (`Zyphra/ZONOS2`, Apache) + Chatterbox
|
||
baseline (already running `chatterbox-fast` :8197) on irv-ml1's **A6000** (~30G free; 3090 full w/ mOrpheus),
|
||
SEQUENTIAL load. Harness = text→WAV per model + **TTFA/RTF** logging. Protocol (READ IT):
|
||
`~/development/brokkr-smithy/research/R17-nvv-extraction/erp-voice-nvv-corpus/tts-audition-protocol.md`. Decider =
|
||
NVV/expressive naturalness; deliver blind-A/B web-listen + latency table → **operator ranks** (not me). brokkr
|
||
delivers the ~15-20 prompt set AFTER the env's up. **⚠️ FRESH-CONTEXT PICKUP: a background general-purpose agent
|
||
(spawned 2026-07-10) is doing the build; its WORK persists on irv-ml1 but its completion-ping/agent-ID likely
|
||
won't route post-/clear → on resume VERIFY the build state DIRECTLY on irv-ml1 (`ssh irv-ml1`; look for a
|
||
`tts-audition` work area under /worktank or /home/lkraven, pulled models, the harness) instead of waiting for the
|
||
agent.** irv-ml1 = lkraven, docker-group, NO sudo; A6000 = docker device 1.
|
||
|
||
- **🖼️ COMFYUI BUMP → v0.27.1 — comfy-dev GAVE THE GO + FULL RECIPE (2026-07-10); re-attempt on fresh context.**
|
||
Target CORRECTED to **v0.27.1** (the ACTUAL latest; operator said "v25" but wants latest + Krea-2 needs ≥0.25 so
|
||
0.27.1 covers it; braced for breakage). irv-ml1 `comfyui` currently healthy on rolled-back 0.24.1. Prior attempt
|
||
crash-looped (SageAttention ABI vs torch 2.13; DISABLE_UPGRADES bumped the venv but NOT the ComfyUI git checkout
|
||
— see the 2026-07-10 Recent-decisions entry). **comfy-dev's atomic recipe (althing thread 01KX655V — run "your
|
||
way", bundle to avoid a broken intermediate):** (1) git-advance `/comfy/mnt/ComfyUI` → tag **v0.27.1** +
|
||
`pip install -r requirements.txt` **as uid 1000** for new deps; (2) torch 2.13.0 rides along (expected, for LTX);
|
||
(3) **rebuild SageAttention vs torch 2.13.0** so `--use-sage-attention` works — IF the recompile fights, DROP the
|
||
flag as fallback + **FLAG comfy-dev** (SageAttn is load-bearing for WAN speed + tied to their per-workflow
|
||
AttentionSelector workaround for the Qwen-black bug, so they re-validate); (4) re-pin `DISABLE_UPGRADES=true`. Net
|
||
= `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` (crash-loop → roll back like last time + ping).
|
||
**PING comfy-dev when healthy on 0.27.1** → they run their 3-tier validation (node-schema diff vs their saved
|
||
105-node baseline → validate_graph_templates.py → smoke renders across families) + own the workflow-breakage
|
||
triage. Optional: add `decord` (uid 1000) for the pre-existing ComfyUI-RMBG SAM3Segment warning if trivial, else
|
||
skip. **⚠️ comfy-dev's GO is currently UNACKED; it's a RISKY re-attempt (crash-looped once) and the operator said
|
||
"wait for comfy… before WE go back" → the fresh context should SURFACE to the operator (comfy-dev GO'd + v0.27.1
|
||
recipe ready) and confirm before executing, NOT silently auto-run.**
|
||
|
||
- **📅 BIWEEKLY LANDSCAPE-SCAN CRON — SETTLED.** Live systemd --user timer on nh3-dev (next auto-run 2026-07-15
|
||
09:00 PT), dedicated `brokkr-scan-dev` handle, notify = `post --to brokkr-smithy-dev` (Vuong ruled inbox-post,
|
||
no phone-push). Fully validated end-to-end. Nothing owed; watch for a brokkr-scan-dev first-dwarf-arm routing
|
||
hiccup flag on the 15th.
|
||
|
||
- **📬 althing:** a harness-tracked wake-listener was armed for comfy-dev's reply. **A /clear kills the monitor
|
||
loop → the fresh session must re-arm (`/althing:monitor`)**; the 📬 banner surfaces new peer mail on next prompt
|
||
regardless.
|
||
|
||
_Earlier (2026-07-09) context preserved below:_
|
||
|
||
_As of 2026-07-09 — RP-seat 3-seat inference stack still COMPLETE + SETTLED (gen + char-rp +
|
||
char-rp-reasoning, detail below). NEW this session: a **mOrpheus TTS voice pipeline** shipped
|
||
end-to-end (deployed on irv-ml1, voicing gateway-chat). Home-training dormant. Live state:_
|
||
|
||
- **🗣️ mOrpheus TTS — PERMANENT STACK DEPLOYED (irv-ml1), voicing gateway-chat.** Off-the-shelf
|
||
uncensored Orpheus TTS (`MrDragonFox/mOrpheus_3B-1Base_early_preview-v1-25000`; Llama-3.2-3B LLM
|
||
→ SNAC 24kHz audio). Canonical `stacks/mOrpheus/`; deployed at `/home/lkraven/morpheus/` on irv-ml1.
|
||
**Two containers:** `vllm-morpheus` (engine — **bf16 REQUIRED** (FP8 breaks audio); **image PINNED
|
||
`vllm/vllm-openai:v0.23.0`** (`latest` crashes on Ampere import); on the **3090** (A6000 is
|
||
comfy-contended, spikes to ~41GB); `--enforce-eager` + util 0.32 ≈ 7.9GB) + `morpheus-tts` (CPU
|
||
FastAPI — tokenizer + SNAC decode; `/tts` whole-clip WAV, `/tts/stream` windowed-chunk PCM stream,
|
||
zero-shot cloning; CORS-enabled). **RTF ~1.0 end-to-end** (enforce-eager caps it; vLLM single-3090
|
||
concurrency is near-linear to `--max-num-seqs 8` = 8× at 707 tok/s). **gateway-chat** (ana-docker:8091,
|
||
the zero-dep `tools/gateway-chat.html`) auto-voices assistant **quoted** dialogue: chunk by QUOTED
|
||
SECTION (prosody), Web-Audio streaming player (TTFA ~0.8s, shared clock, resume-on-gesture), 🔊 sidebar
|
||
toggle + endpoint/voice fields. **Clone voices** (drop `<name>.wav` + `<name>.txt` in the voices dir →
|
||
`voice="<name>"` clones it): `baddy` (native speaker), `beatrice`, `whisper` (kokoro **af_nicole**, a
|
||
genuine whisper voiced 0.24; mOrpheus partially re-voices it → clone voiced 0.42, breathy not pure).
|
||
Honored tags = learned text-patterns (canonical Orpheus laugh/chuckle/giggle/sigh/groan/gasp/yawn/cough/
|
||
sniffle + NVV pant/grunt/kiss/moan); agent voicing prompt at `stacks/mOrpheus/agent-voice-prompt.md`.
|
||
**OPEN:** whisper re-voices (checkpoint limit, not the reference); throwaway audition `:8210` +
|
||
`/home/lkraven/orpheus-audition/` still up on irv-ml1 (teardown pending, ~6.3G weights moved out already);
|
||
production concurrent-gen + a SNAC-decode worker pool NOT built (CPU decode is the scale bottleneck, not gen).
|
||
|
||
- **SERVING TOPOLOGY, ana-ml2 GPU0 (co-resident), served via the `qwen36-27b-aeon` + `ms32-24b-angel` compose stacks:**
|
||
- **gen (:8015) = `qwen36-35b-a3b-heretic-nvfp4` — LIVE + GOOD.** Off-the-shelf NVFP4 of
|
||
`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-NVFP4-Experts-Only` (MoE 256e/8-active =
|
||
qwen3_5_moe, vision, GDN-hybrid; **modelopt** format → `--quantization modelopt`; vision bf16; 24G at
|
||
`/tank/aimodels/qwen36-35b-a3b-heretic-nvfp4`). UNCENSORED (writes brutal fiction think-off, no system prompt).
|
||
"Blathers" → needs adequate max_tokens (thinking-on looks EMPTY under ~600 tok, fine at 3k — NOT broken).
|
||
General-assistant seat. Served by `/opt/docker/compose/qwen36-27b-aeon/` (`.env` repointed: `AEON_GEN_MODEL`,
|
||
`AEON_GEN_QUANT=modelopt`, `AEON_GEN_GPU_MEM_UTIL=0.40`; the `--quantization` line is now
|
||
`${AEON_GEN_QUANT}`-parameterized). Served-names `qwen3.6-27b-aeon`/`-thinking` (gateway gen/gen-reasoning stay
|
||
transparent). **util 0.37 (~35G; dropped from 0.40 on 2026-07-08 to fund char-rp 128K — the RP seats are now co-resident, ~4.4G GPU0 margin).** AEON + the intermediate Pantheon-27B
|
||
serves are RETIRED.
|
||
- **char-rp (:8016) = `TheDrummer_Magidonia-24B-v4.3-Q6_K` GGUF — LIVE + EXCELLENT.** Magistral (Mistral)
|
||
dark-romantasy RP tune via llama.cpp (`char-rp-gguf` stack, container `llama-charrp`). NON-thinking PROSE seat:
|
||
elite literary prose, zero refusal on dark/explicit scenes, ~65 tok/s, tight POV/instruction adherence
|
||
(live-tested). Replaced the broken Angel NVFP4. Alt prose model (`.env` swap `CHARRP_MODEL`):
|
||
`MS3.2-PaintedFantasy-v4.1-24B` (more literary flair, looser POV). Both GGUFs pre-pulled at `/tank/aimodels/llm/rp/`.
|
||
- **char-rp-reasoning (:8018) = `Qwen3.5-27B-Deckard-PKD-Heretic-Uncensored-Thinking.i1-Q5_K_M` GGUF — LIVE
|
||
(A/B WINNER 2026-07-08, REPLACED RpR-v4).** DavidAU creative (PKD) tune on Qwen3.5-27B (container
|
||
`llama-charrp-reasoning`). Native Qwen3.5 template → `--reasoning on` managed CoT SURFACES in `reasoning_content`,
|
||
content stays clean prose, `--reasoning-budget 400` caps it; **DRY server-side** (sampler order = dry-after-temp)
|
||
tames looping. A/B: **0/30 loops + 0/30 refusals**, ~57 tok/s. Runs on the base llama-swap b8840 image (Qwen3.5).
|
||
Best-of-breed-per-seat (NOT the same model as char-rp).
|
||
|
||
- **🦄 RP-SEAT UNICORN — RESOLVED + DEPLOYED (2026-07-08).** char-rp = **Magidonia-24B-v4.3** (Magistral prose,
|
||
65tps); char-rp-reasoning = **Deckard-PKD (Qwen3.5-27B)** (managed reasoning + DRY, ~57tps; A/B winner over RpR-v4 — see re-A/B note below). Both GGUF via
|
||
llama.cpp (`char-rp-gguf` stack, ana-ml2 GPU0, ~86/97G co-resident with gen, ~11G margin). Canonical stack in repo
|
||
`stacks/char-rp-gguf/`; gateway rewired (`char-rp`→:8016, `char-rp-reasoning`→:8018, Mistral/QwQ samplers, dropped
|
||
the Qwen `enable_thinking` kwarg). **KEY FINDINGS:** (a) no single dense 24-32B is BOTH an elite non-thinking prose
|
||
seat AND a clean *managed*-reasoning seat on llama.cpp → best-of-breed per seat; (b) Magidonia's Magistral `[THINK]`
|
||
discipline is loose (won't close `[/THINK]` on substantive reasoning → prose bleeds into reasoning, content empties);
|
||
Cydonia-R1's `<think>` is emergent (llama.cpp can't manage/cap → runaway CoT never reaches prose); QwQ's template
|
||
opens `<think>` natively → llama.cpp manages+caps it; (c) **Pantheon-Reasoning-27B stays REJECTED** (re-censors in
|
||
`<think>`; RpR-v4's abliterated-reasoning is the fix); (d) reasoning now SURFACES in `reasoning_content` (fixes the
|
||
old AEON trace-not-surfacing gap). One-model fallback (Magidonia both, lighter reasoning) documented in the stack
|
||
header/README. Requirements met: prose#1, ≥50tps#2, low-refusal#3, dense#4, GGUF-not-Ollama#5, fit-GPU0#6, thinking#7.
|
||
Candidate GGUFs also on disk for A/B: Cydonia-R1-24B-v4.1, PaintedFantasy-v4.1-24B, RpR-v4 Q6_K.
|
||
**REASONING-SEAT RE-A/B (2026-07-08, later — RpR→Deckard):** operator wanted a reasoning model that TAKES DRY
|
||
(RpR forbids rep/DRY → 1/30 loop tail). Full A/B on brokkr's 30-prompt D1 suite (content-only, slop-scored):
|
||
**Deckard-PKD (Qwen3.5-27B) WON** (0/30 loops, 0/30 refusals, clean) → NOW the char-rp-reasoning seat.
|
||
RpR-v4: 0 refusals but 1/30 loop (no-DRY). **Pantheon-Reasoning-27B: 7/30 explicit refusals** (DeepSeek-distilled
|
||
re-censor IS real across the batch, milder than feared; NOT wholesale-rejected) — kept on disk as alt.
|
||
Snowdrop-v0.5-Type-S + Gembrain-31B (Gemma-4): FLOORED — llama.cpp can't do MANAGED reasoning on them (Snowdrop's
|
||
ChatML template has no `<think>`/`enable_thinking` hook; Gemma-4's reasoning parser splits content wrong). GATE for a
|
||
llama.cpp reasoning seat = STOCK template natively opens `<think>` or has `enable_thinking` (Qwen3.x/QwQ do; ChatML +
|
||
Gemma-4 don't). **INFRA:** llama-swap b8840 image can't load Qwen3.6/Gemma-4 archs → use
|
||
`ghcr.io/ggml-org/llama.cpp:server-cuda` (newer, pulled on ana-ml2) for those; Deckard (Qwen3.5) runs on b8840.
|
||
**brokkr frozen-scorer CONFIRMED (median composite, content-only; lower=cleaner):** Deckard **2.176** (0 loop/0
|
||
refuse — DEPLOYED, big win over RpR **3.716** + its loop-tail); Pantheon-Reasoning **1.383** (cleanest prose of the
|
||
field — beats even Magidonia char-rp 1.774 — BUT 7/30 explicit refusals → rejected: clean-but-refuses loses to
|
||
clean-and-uncensored). **MEMORY-LAYOUT note:** Deckard (Qwen3.5-27B) is GDN-hybrid → only **16 KV-caching layers** →
|
||
KV is TINY (1.4G @ 40K vs Magidonia's 10.9G @ 128K, 40 dense layers); Deckard native ctx = 256K, so the reasoning
|
||
seat's ctx scales nearly free. **Deckard BUMPED to full 256K (262144) 2026-07-08** (KV 8.7G q8_0). GPU0 layout now:
|
||
gen ~35G (256K/16-seq/util0.37), char-rp Magidonia ~30G (128K/4-slot), char-rp-reasoning Deckard ~28G (256K/4-slot),
|
||
**~4G free** (all static allocations → stable, no OOM). **Canonical Deckard RP-reasoning samplers CONFIRMED by dvalin 2026-07-08**
|
||
(thread 01KX1Y7P): the live empirical set IS canonical — temp1.0/top_p0.95/top_k40/min_p0.05, NO presence/rep
|
||
penalty, DRY0.8 server-side (base1.75/len2, dry-after-temp), reasoning-budget400. Delta from card base-thinking:
|
||
top_k20→40, min_p0→0.05, presence1.5→0, DRY-on. Tuning ladder: flat prose→min_p0.08; loops→DRY0.9; over-damped→
|
||
DRY0.6/off. Do NOT import RpR/QwQ rules (QwQ hated DRY; Qwen3.5 benefits from DRY over rep_penalty).
|
||
**MAX CONTEXT (2026-07-08):** char-rp **128K** (Magidonia FULL native 131072), char-rp-reasoning **40K** (QwQ
|
||
native 40960, YaRN-free max), **q8_0 KV cache both** (near-lossless, ~2× ctx/GB; verified coherent, no Qwen
|
||
gibberish). **Funded by gen util 0.40→0.37** (freed ~2.9G of gen's IDLE KV headroom — gen KV usage runs 0-2%,
|
||
so its 4.74×→3.66× concurrency-at-256K cut is invisible). GPU0 ~93/97G, **~4.4G margin** (gen fixed-util +
|
||
static KV → stable, won't OOM). 64/50 tps. `kv_unified` → a single conversation gets the FULL n_ctx (4 slots
|
||
share the pool, not a /4 split). Tunables: CHARRP_CTX / CHARRP_REASONING_CTX / *_KV_TYPE / AEON_GEN_GPU_MEM_UTIL.
|
||
|
||
- **✅ LiteLLM gateway sampling defaults WIRED for all 4 seats (2026-07-08)** — dvalin-smithy canonical, infra-ops
|
||
triaged + char-rp A/B-validated. **gen** (+summarizer-large twin, shared :8015 deployment): temp 0.7 / top_p 0.8 /
|
||
top_k 20 / presence_penalty **1.5** (non-thinking). **gen-reasoning**: temp **1.0** / top_p 0.95 / top_k 20 /
|
||
presence_penalty **1.5** (Qwen general-thinking profile — the old temp 0.6 was the CODING sub-profile by mistake;
|
||
verbose ~9k-char CoT → callers need big max_tokens). **char-rp**: temp **1.1** / top_p 0.95 / min_p **0.10** /
|
||
top_k 0, NO rep (A/B: min_p 0.10 richened imagery vs 0.03; rep 1.05 REJECTED — injected a markdown title in a grief
|
||
scene, matches Drummer-no-rep). **char-rp-reasoning**: temp 1.0 / top_p **0.95** / top_k 40 / min_p 0.02, no rep/DRY.
|
||
Canonical doc `docs/pfi/model-sampler-defaults.md`; dvalin derivation `dvalin-smithy/hoard-drafts/pfi-gateway-sampler-defaults-20260708.md`.
|
||
|
||
- **NVFP4/GGUF quant tooling (proven, on ana-ml2):** `/tank/aimodels/quant-work/` — `nvfp4_quant.py`
|
||
(llm-compressor, MoE-aware `load_quantizable_moe`, Mistral-tuned ignore) + `.venv`; `qwen35_vision_nvfp4.py`
|
||
(llm-compressor + qwen ignore + `trust_remote_code` + a torchvision-gate bypass). ModelOpt path:
|
||
`/home/lkraven/nvfp4-quant/.venv-mopt` (modelopt 0.44) + `mopt_w4a16.py` (= `mtq.quantize` + `export_hf_checkpoint`
|
||
→ modelopt format w/ `hf_quant_config.json`). Downloads via a detached vLLM container running
|
||
`huggingface_hub.snapshot_download` to `/tank/aimodels/<name>-{bf16,nvfp4}`. **Gotchas:** llm-compressor
|
||
`save_compressed` DROPS tokenizer/chat/preprocessor files → stage from the bf16 source; the qwen ignore keeps
|
||
`re:.*visual.*` + `re:.*linear_attn.*`(GDN) + lm_head + embeds + `mlp.gate`(router) bf16.
|
||
|
||
- **🪦 T1 / home-trained LoRA — DORMANT (pivoted to off-the-shelf).** DPO came back slop → operator killed
|
||
home-training (2026-07-08). ONLY this project dormant; LEAVE arbo's LoRA-worker (irv-ml1 :8203), ALL /tank models,
|
||
and every other project ("ritual seppuku" otherwise). **Revival map:** adapters `data/spike/{sft,dpo}_adapter`,
|
||
harness canonical v0.0.44, full saga in Recent decisions + mtf-dev auto-memory; aeon-trainer image left (28G).
|
||
mtf-dev stood down. (Detailed 2026-07-07 hot-swap findings preserved below + in Recent decisions.)
|
||
|
||
_Earlier (2026-07-07):_
|
||
|
||
- **T1 SFT COMPLETE (AEON-27B E-RP LoRA) — adapter banked, but BLOCKED on hot-swap serving.** SFT ran
|
||
clean on ana-ml2 GPU0 (all-TRL torch-GDN, batch1/accum8/GC-off, 1 epoch, ~9h @ 0.25 s/s steady, final
|
||
loss 2.445); adapter at `ana-ml2:~/model-training-forge/data/spike/sft_adapter` (124MB, r16 α32, targets
|
||
`self_attn.q/k/v/o_proj` + `linear_attn.in_proj_qkv/out_proj`, NO `in_proj_z`). DPO leg deferred (needs
|
||
mergekit resolved + on-policy `rejected` regen). **THE WALL: the adapter is INERT on the vLLM NVFP4
|
||
hot-load** — loads HTTP 200 but zero deltas reach inference (vLLM 0.24.0 qwen3_5 LoRA-APPLICATION silent
|
||
no-op; upstream #47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4
|
||
AND FP8 both inert) AND adapter-format-agnostic (separate-key `sft_adapter_zc` loads clean via vLLM's
|
||
allowlist but inert; the fused-key rekey rejected outright). The #47640 guard-patch overlay did NOT fix
|
||
it. **HARD REQUIREMENT (operator): hot-swappable or ABANDON — merge is NOT a solution now** (static;
|
||
reserved for a later RP-quant track only).
|
||
|
||
- **HOT-SWAP — SGLang zc smoke + 2-line patch RAN (2026-07-07). PARTIAL PROGRESS: patch cleared startup
|
||
validation + self-attn LoRA loads end-to-end, but GDN in_proj hit 2 MORE name-routing gaps → NOT a
|
||
one-line fix. MERGE RECOMMENDED (operator's call, pending).** Ran `sft_adapter_zc` on SGLang 0.5.13 bf16
|
||
(`--lora-target-modules all --max-lora-rank 16 --lora-paths t1=/adapters/sft_adapter_zc`, GPU0, gen+rp
|
||
stopped for the smoke then restored). Base weights loaded clean; self-attn q/k/v/o normalized + accepted
|
||
fine. **Died at `lora_manager.init_lora_shapes` with a NAME-VALIDATION `ValueError` (NOT the feared
|
||
in_proj_qkvz shape-assert):** adapter's `in_proj_qkv`+`in_proj_z` "not included in --lora-target-modules
|
||
[...in_proj_qkvz...]". **ROOT CAUSE (pinned):** `srt/lora/utils.py:235 get_normalized_target_modules` uses
|
||
a HARDCODED `params_mapping` dict that maps `q/k/v_proj→qkv_proj` (why self-attn works) but OMITS
|
||
`in_proj_qkv→in_proj_qkvz` + `in_proj_z→in_proj_qkvz`; it does NOT consult the model's
|
||
`packed_modules_mapping` (which DOES declare `"in_proj_qkvz":["in_proj_qkv","in_proj_z"]` at
|
||
`models/qwen3_5.py:1010`). Everything else is ALREADY in place: `get_hidden_dim("in_proj_qkvz")` IS
|
||
implemented in the model class (`:1040`, → `(hidden_size, key_dim*2+value_dim*2)`; the "utils NotImplementedError"
|
||
was only the DEFAULT); `get_stacked_multiply("in_proj_qkvz")=4` in the utils default dict; base-weight
|
||
`stacked_params_mapping` fuses `in_proj_qkv→shards(0,1,2)`,`in_proj_z→shard 3`. **PATCH APPLIED + RE-SMOKED
|
||
(bind-mounted a patched utils.py overlay into the container):** the 2-line add cleared `init_lora_shapes`;
|
||
server came up "fired up and ready", LoRA kernels compiled (K=16384 in_proj_qkvz expand + K=6144 in_proj_z
|
||
shrink — B-side math right per mtf-dev), **self-attn loaded end-to-end** ("loaded weights for [o_proj,out_proj,
|
||
qkv_proj]"). **BUT GDN in_proj died at TWO MORE name-resolution points (name-routing, NOT shape/intractable):**
|
||
(1) adapter-load SKIPPED 192 in_proj_qkv/in_proj_z weights ("did not match any target module [...in_proj_qkvz...]");
|
||
(2) inference buffer-fill `utils.py:300 get_target_module_name` RAISED — SUBSTRING match, `in_proj_qkvz` ⊄
|
||
`...in_proj_qkv.` and `qkv_proj` ⊄ it → "Cannot find target module name for ...in_proj_qkv.lora_A.weight" →
|
||
HTTP 500 + scheduler killed. So SGLang's GDN LoRA name-routing is incomplete in ≥3 code paths (fixed 1/3).
|
||
**FINISH LINE (if ever revived):** teach `get_target_module_name` + the load-consolidation the
|
||
`in_proj_qkv/in_proj_z→in_proj_qkvz` map (mirror qkv_proj, which works E2E), THEN mtf-dev's buffer
|
||
shard-placement (0,1,2)+3 question goes live again (still UNPROVEN, deepest step). Per operator's "one
|
||
targeted fix else merge" rule → this is multi-point, NOT one fix → MERGE recommended. mtf-dev replied +
|
||
loop closed (thread 01KWZDZFYN…). (SGLang GENERIC image still can't
|
||
LOAD our NVFP4 checkpoint — bf16-only smoke.) Adapter variants: `sft_adapter` (separate), `sft_adapter_zc`
|
||
(separate + zero `in_proj_z`, the canonical one — THIS smoke), `sft_adapter_fused` (fused — wrong), plus
|
||
`sft_adapter_attnonly`. All at `/home/lkraven/model-training-forge/data/spike/`.
|
||
|
||
- **MERGE FALLBACK — primed (~2-3h) if hot-swap is fully abandoned.** peft merge_and_unload (~20min, LOW
|
||
wall — separate keys merge directly, no rekey) → sanity greedy-diff (finally reveals if the ~10h training
|
||
is any good — we've NEVER seen the LoRA apply) → ModelOpt NVFP4 re-quant of the merged bf16 (~1-2h,
|
||
MEDIUM-HIGH wall = the GDN-hybrid requant + calibration + verify GDN modules quantize) → serve as plain
|
||
base (no `--enable-lora`). Static, NOT hot-swappable.
|
||
|
||
- **comfy-dev in-arbo LoRA-training worker — DONE (all phases, this session).** Built/deployed the worker
|
||
(`services/lora-training-worker/`, irv-ml1 `0.0.0.0:8203`, runs as llmuser, fixed-invocation sd-scripts,
|
||
systemd, elway deploy playbook; commits 888ba6a→b617a8b). Phase 1 first real train SUCCEEDED (Sindra
|
||
fast/lean). Phase 2 publish-step (`published_lora_name` + optional `train_id` field). Phase 2.5 GC
|
||
contract (shape A: `GET /published-loras` + `POST /gc/published {train_ids}`) CONFIRMED — worker impl
|
||
DEFERRED to after the T1 hot-swap work. auto-memory `reference_lora_training_worker`.
|
||
|
||
- **Serving state (SUPERSEDED — see top; AEON retired 2026-07-08):** during the T1 work, AEON gen (:8015) +
|
||
char-rp (:8016) NVFP4 were HEALTHY, stopped/restored for hot-swap smokes via `docker start vllm-aeon-gen
|
||
vllm-aeon-rp`. Now replaced: gen = 35B-A3B heretic, char-rp = (broken Angel, being replaced).
|
||
|
||
- **worldtree-dev vh/pypi read token** delivered (bifrost==1.1.0 pin, #349); dedicated read:package token
|
||
at `nh3-dev:~/.config/gitea-tokens/vh-pypi-read.txt`. The over-scoped write:package token the operator
|
||
pasted was revoked + verified dead.
|
||
|
||
- **Backups — STILL OPEN:** rotate the 5 disclosed rest-server creds (operator, offline); confirm
|
||
esh-vm-db's resticprofile includes DB dumps. `docs/runbooks/backups.md`.
|
||
|
||
- **Standing / parked:** inert `mood.decay_rate`/`stale_hours` keys in deployed WT config; Mac Pro
|
||
migration (hw-gated); R22 gateway-key at `/home/lkraven/.r22-gateway-key` (mode 600, paid GLM, do NOT
|
||
delete); Deckard staged on ana-ml2 as T1's writing benchmark; `/books` transient NFS mount on nh3-dev;
|
||
**dvalin sampler re-tune pass** for char-rp / char-rp-reasoning once they accumulate live Worldtree/ST
|
||
character-role traffic (dvalin offered 2026-07-08 to do a follow-up after more live sessions; the current
|
||
paper defaults + 2-prompt A/B are the starting point — real long-session usage is where repetition/drift
|
||
surfaces; re-open thread `01KX1DS6…` to take them up on it).
|
||
|
||
- **Worldtree config-propagation (reference):** demo+personal bind-mount config from
|
||
`/opt/worldtree{,-personal}/config` (infra-ops-deployable, byte-identical from canonical); reload via
|
||
`docker restart <container>`, NEVER `compose up` (stale-`:latest` footgun). Role registry loaded ONCE +
|
||
CACHED at startup → a bind-mount `model_roles.yaml` change needs a restart; pre-stage BEFORE the
|
||
activating deploy's restart. config REMOVALS are NOT backward-compatible with the running image.
|
||
|
||
## Recent decisions
|
||
|
||
- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev
|
||
requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`,
|
||
0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the
|
||
VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` =
|
||
`/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump
|
||
broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch
|
||
2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke
|
||
the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot
|
||
`/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention),
|
||
re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at
|
||
`/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):**
|
||
comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention
|
||
rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY
|
||
the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) →
|
||
rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1),
|
||
lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved.
|
||
|
||
- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable
|
||
systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00`
|
||
America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh`
|
||
runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in
|
||
`~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH
|
||
footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS
|
||
new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode
|
||
ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0):
|
||
web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input +
|
||
caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle
|
||
brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor);
|
||
registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired
|
||
step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default
|
||
+ `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled
|
||
1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending.
|
||
**NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start
|
||
brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session
|
||
handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 +
|
||
ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token
|
||
feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo).
|
||
|
||
- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker
|
||
docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB
|
||
registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the
|
||
2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to
|
||
remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because
|
||
daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never
|
||
reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd
|
||
opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`,
|
||
`rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the
|
||
frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime
|
||
(zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares
|
||
`live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start,
|
||
never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers
|
||
once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon;
|
||
log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running
|
||
containers).
|
||
|
||
- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed,
|
||
#335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides
|
||
intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on
|
||
`summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit
|
||
both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen**
|
||
(b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted
|
||
`worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL
|
||
NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1`
|
||
after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config
|
||
INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven
|
||
safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed).
|
||
StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON:
|
||
the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never
|
||
updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) —
|
||
pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass`
|
||
parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider
|
||
description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix
|
||
(operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING
|
||
defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs
|
||
when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway
|
||
config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD;
|
||
live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09):
|
||
character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving
|
||
character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping
|
||
principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the
|
||
dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent
|
||
but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44,
|
||
UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly
|
||
from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) +
|
||
mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's
|
||
`load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character +
|
||
character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window
|
||
VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev
|
||
to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked
|
||
config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly
|
||
re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard
|
||
+ character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning
|
||
downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying
|
||
(both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37**
|
||
(char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09):
|
||
mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx
|
||
131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45
|
||
canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push
|
||
of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity
|
||
block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev),
|
||
monitor armed. See [[reference_corviduo_dev_emergency_ops]].
|
||
|
||
- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into
|
||
gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17,
|
||
vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack
|
||
(01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829);
|
||
max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and
|
||
`repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture
|
||
no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone
|
||
voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514).
|
||
**Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not
|
||
A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s).
|
||
- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).**
|
||
Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty
|
||
idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching).
|
||
Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08).
|
||
**This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the
|
||
operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread
|
||
01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]].
|
||
- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD
|
||
Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.**
|
||
Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0
|
||
KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf);
|
||
(4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a
|
||
DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over
|
||
RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp
|
||
template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the
|
||
live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively
|
||
opens `<think>`/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap
|
||
b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).**
|
||
- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` +
|
||
`envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal),
|
||
YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my
|
||
parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.)
|
||
- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.**
|
||
Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision,
|
||
util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps
|
||
ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled
|
||
refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path);
|
||
Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting
|
||
("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway
|
||
sampling-defaults wiring still PENDING.
|
||
- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3
|
||
epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass
|
||
`num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs
|
||
+ _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the
|
||
off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a
|
||
~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run.
|
||
FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig,
|
||
float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`.
|
||
GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min
|
||
train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step;
|
||
train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45
|
||
banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0):
|
||
ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is
|
||
COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral
|
||
literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift,
|
||
arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep
|
||
on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval +
|
||
the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO
|
||
adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed).
|
||
- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.**
|
||
Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path
|
||
finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied
|
||
VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before
|
||
`from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does
|
||
`model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers
|
||
5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the
|
||
DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED
|
||
un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO
|
||
confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93%
|
||
util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit.
|
||
gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter.
|
||
- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.**
|
||
Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the
|
||
SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo
|
||
--entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src
|
||
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank
|
||
-w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml
|
||
--stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is
|
||
`["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash.
|
||
Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer`
|
||
(backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→
|
||
immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→
|
||
**llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→
|
||
(more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate
|
||
1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0)
|
||
but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image.
|
||
Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX,
|
||
expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image.
|
||
- `[2026-07-07]` **SGLang zc hot-swap: 2-line patch RAN → PARTIAL. Cleared startup + self-attn E2E, GDN
|
||
in_proj hit 2 MORE name-routing gaps → MERGE recommended.** Root cause was a NAME-routing gap (not the
|
||
feared shape-assert): `lora/utils.py:235` hardcoded `params_mapping` lacked `in_proj_qkv/in_proj_z →
|
||
in_proj_qkvz`. Patched those 2 lines (overlay bind-mount) → `init_lora_shapes` passed, server up, self-attn
|
||
loaded end-to-end. But the SAME missing GDN mapping bit 2 MORE resolvers: adapter-load skipped 192 in_proj
|
||
weights, and inference `utils.py:300 get_target_module_name` (substring match) RAISED → 500 + scheduler
|
||
died. So GDN LoRA name-routing is incomplete in ≥3 SGLang paths (fixed 1). NOT shape-intractable (self-attn
|
||
proves the separate-key→fused-buffer machinery works), but NOT "one targeted fix" either → per operator's
|
||
LAST-shot rule → MERGE. mtf-dev looped + closed (thread 01KWZDZFYN…); B/A shard-math never reached.
|
||
Supersedes the "SGLang wants a THIRD format" framing — no new adapter format needed.
|
||
- `[2026-07-07]` **T1 (retargeted qwopus-122B→AEON-27B) SFT COMPLETE on ana-ml2** — all-TRL torch-GDN,
|
||
batch1/accum8/GC-off, 1 epoch ~9h @ 0.25 s/s, loss 2.445, adapter banked. Supersedes the 2026-07-05
|
||
cloud-vs-smoke venue decision: AEON-27B's 52G bf16 fits one Blackwell → ran on-prem in ~9h (no cloud,
|
||
no full-fleet outage). GC is NOT the throughput lever (batch1-no-GC == batch4+GC at 0.25; dvalin's 0.565
|
||
was warmup-noise from a 30-sample smoke); ~0.25 s/s is the torch-GDN floor (only FLA beats it, sm_120-
|
||
blocked #913). `reference_t1_cloud_train_plan`.
|
||
- `[2026-07-07]` **T1 hot-swap on vLLM NVFP4 is DEAD — vLLM 0.24.0 qwen3_5 LoRA-application silent no-op**
|
||
(#47639 open / fix PR #47640 open / regression from #37912). Proven quant-agnostic (NVFP4+FP8) AND
|
||
adapter-format-agnostic (separate `zc` loads-but-inert; fused rejected); #47640 guard-patch overlay didn't
|
||
fix. Root-caused by a 3-peer dwarf panel (eitri/regin/dvalin); their vLLM issue refs verified real via
|
||
GitHub API. Engine mirror-image: vLLM wants SEPARATE keys, SGLang wants FUSED.
|
||
- `[2026-07-07]` **HARD REQUIREMENT restated: hot-swappable LoRA or ABANDON.** Merge is NOT a solution for
|
||
this requirement (static; reserved for a later RP-quant track). SGLang is the last hot-swap shot; if it
|
||
can't apply, park the adapter + file upstream repros. `reference_lora_training_worker` (adjacent).
|
||
- `[2026-07-07]` **comfy-dev in-arbo LoRA-training worker DONE** — built/deployed/validated Phases 1+2
|
||
(first Sindra train succeeded, publish-step + `train_id`), Phase 2.5 GC contract (shape A) confirmed +
|
||
impl deferred. `services/lora-training-worker/`, commits 888ba6a→b617a8b. `reference_lora_training_worker`.
|
||
- `[2026-07-06]` **infra-ops NOPASSWD sudo confirmed on irv-ml1 + ana-ml2** (the arbo memory's "NOT
|
||
infra-ops on irv-ml1" was stale) — enabled the worker deploy + every ana-ml2 hot-swap smoke.
|
||
- `[2026-07-05]` **T1 training venue: CLOUD recommended; operator chose smoke-first on ana-ml2.**
|
||
On-prem ruled out (ana-ml2 full — both 96G GPUs ~93G used): keep-serving = NVMe offload ~6-8 DAYS;
|
||
full ana-ml2 shutdown = CPU offload ~1 DAY but a whole-fleet outage. Cloud Vast.ai 8×80GB (no
|
||
offload → ~3-6h, ~$60-500, zero fleet impact) is the clean alt (mtf-dev + infra-ops both rec;
|
||
Vast for its no-content-AUP marketplace + likely-existing VastBlue account). Operator's next step
|
||
= the ana-ml2 CPU-offload SMOKE (~60 steps) to get real samples/sec before the full-outage-vs-cloud
|
||
call. HF base verified public (zero-auth pull). Runbook + gotchas in `reference_t1_cloud_train_plan`.
|
||
|
||
- `[2026-07-05]` **glm-5.2 canonical limits recorded** (probed live vs z.ai): **1,048,576 (1M) input
|
||
context / 131,072 (128K) max output**; NO gateway-side cap (pure passthrough → z.ai's limits are
|
||
effective). Written to the config comment (commit `624a07e`) + `reference_litellm_gateway`.
|
||
|
||
- `[2026-07-04]` **character-rp: gateway-forwarding proven + role pre-staged + #344 shipped.**
|
||
Empirically confirmed per-request `extra_body` (top_k/repetition_penalty) forwards through the
|
||
`gen-reasoning` LiteLLM alias to vLLM + standard params override the alias defaults — no gateway
|
||
cap needed (I over-built a dedicated alias, operator corrected, reverted with zero fleet impact).
|
||
Pre-staged the `character-rp` role into demo+personal bind-mount `model_roles.yaml` (byte-verified
|
||
on b18; caught the cached-registry ordering). worldtree-dev shipped **#344 (v1.0.0b19)** for the
|
||
durable-agent override-drop. spend_logs spot-check queued (ratatoskr's timestamp ping).
|
||
|
||
- `[2026-07-04]` **althing v2 herald+receiver formalized as systemd on nh3-dev.** `althing-herald.service`
|
||
(Restart=always, **Environment=PATH incl ~/.cargo/bin** — the pane-dispatch fix) + `althing-receiver.service`
|
||
(v2 → pillar-3 `/owner/*` live); stale forseti unit removed; both on v2.0.0b2, canonicalized by
|
||
forseti. `reference_nh3_dev_althing_herald`.
|
||
|
||
- `[2026-07-04]` **LitBench-RM respun (irv-ml1 A6000, comfyui displaced)** for T1's reward ensemble;
|
||
operator sole comfyui consumer, holding image-gen until LitBench done. `reference_litbench_rm_irv_ml1`.
|
||
|
||
- `[2026-07-03]` **ratatoskr-dev DEMO Heimdall key provisioned (R30 φ0).** Minted a tier-user key on
|
||
the demo via `POST /admin/keys` (bootstrap admin key), mirroring their personal base consumer (no
|
||
character-binding); base-agent affect reads work ungated. `reference_worldtree_demo_key_mint`.
|
||
|
||
- `[2026-07-02]` **mtf-dev granite harness-spike ran GREEN — MECHANICAL only, efficacy DEFERRED to
|
||
the T1 run.** Trainer TRL SFT→DPO→eval seam proven end-to-end on a synthetic fixture (not the E-RP
|
||
corpus); operator DECIDED no intermediate real-efficacy granite spike (uninterpretable proxy —
|
||
arch gap + abliteration axis). `reference_gen_qwopus_122b`.
|
||
|
||
- `[2026-07-01]` **Worldtree #332 embed-recall diagnosed + scoped-log view/tunnel provisioned + fix
|
||
verified** (15×→1.01× re-embed). `reference_wt_gateway_scoped_log_view`.
|
||
|
||
- `[2026-07-01]` **qwopus native MTP speculative-decode tested on `gen` → NOT kept** (+12% single-stream,
|
||
−15–20% aggregate at concurrency, silently drops min_p/logit_bias). Banked for T1. `reference_gen_qwopus_122b`.
|
||
|
||
- `[2026-07-01]` **Deckard trial → reverted to qwopus (`gen`)** (won writing "in every way" but ~36 vs
|
||
~90 tok/s; spec-decode rescue ruled out). git `b63c48b`→`681eb70`. Deckard kept staged as T1's
|
||
writing benchmark.
|
||
|
||
- `[2026-06-14]` **STANDING DIRECTIVE: migrate ALL infra access to Claude-specific credentials.**
|
||
(auto-memory `project_migrate_infra_access_to_claude_credentials`)
|
||
|
||
_125 older entries archived to archival-memory.md._
|
||
|
||
## Tried and abandoned
|
||
|
||
- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC
|
||
frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks-
|
||
TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the
|
||
bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real.
|
||
- `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter,
|
||
`has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs.
|
||
(`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.)
|
||
- `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized
|
||
→ RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx]
|
||
window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s).
|
||
- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that
|
||
spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning
|
||
(penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path).
|
||
- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8
|
||
(not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the
|
||
cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) —
|
||
that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting.
|
||
- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
|
||
(compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
|
||
(temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
|
||
Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
|
||
whole NVFP4-quant surface.
|
||
- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
|
||
for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
|
||
`tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
|
||
PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
|
||
no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
|
||
tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
|
||
mistral tokenizer + vision).
|
||
- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
|
||
(`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
|
||
(safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
|
||
`<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
|
||
**Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
|
||
abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
|
||
- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
|
||
(`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
|
||
quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
|
||
main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
|
||
self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
|
||
(like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
|
||
- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
|
||
but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
|
||
adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
|
||
rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
|
||
`expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
|
||
version-bump helps. Merge bakes deltas in (bypasses this) but is static.
|
||
- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
|
||
packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
|
||
`qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
|
||
quant-loader-specific).
|
||
- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
|
||
(invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
|
||
its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
|
||
shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
|
||
- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
|
||
--rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
|
||
`--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
|
||
break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
|
||
whose crash logs you need, BEFORE bouncing a production serve.
|
||
- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
|
||
per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
|
||
param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
|
||
caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
|
||
a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
|
||
flushes residual carryover. `feedback_litellm_shared_param_mutation`.
|
||
|
||
- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools
|
||
needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The
|
||
althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO
|
||
routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`.
|
||
|
||
- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe
|
||
ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast
|
||
on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the
|
||
566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`.
|
||
|
||
- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized"
|
||
is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but
|
||
fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push
|
||
finishes. FIX: re-run once built, or gate on `workflow_run: completed`.
|
||
|
||
- `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS
|
||
moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,
|
||
N=4 −20%). Reserve for dedicated/interactive deployments.
|
||
|
||
- `[2026-07-02]` **irv-ml1 `/worktank` ROOT is root-owned — lkraven can't write there (irv-ml1 sudo
|
||
needs a password) → stage model pulls to `/home`.** PIN THE A6000 BY UUID for training (native-CUDA
|
||
ordering differs vs docker; the 3090 index 0 is usually near-full → OOM). `CUDA_VISIBLE_DEVICES=GPU-<uuid>`.
|
||
|
||
_101 older entries archived to archival-memory.md._
|