memory: snapshot — uncensored gen seat landed (Qwen3.8-27B-Uncensored, gen-seat/vllm-gen); next = chase W4A8
This commit is contained in:
-6
@@ -1,6 +0,0 @@
|
||||
- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on
|
||||
per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a
|
||||
param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT
|
||||
caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at
|
||||
a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm`
|
||||
flushes residual carryover. `feedback_litellm_shared_param_mutation`.
|
||||
-5
@@ -1,5 +0,0 @@
|
||||
- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run
|
||||
--rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`);
|
||||
`--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"`
|
||||
break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container
|
||||
whose crash logs you need, BEFORE bouncing a production serve.
|
||||
@@ -1,4 +0,0 @@
|
||||
- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/
|
||||
packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated
|
||||
`qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was
|
||||
quant-loader-specific).
|
||||
@@ -1,4 +0,0 @@
|
||||
- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for**
|
||||
(invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants
|
||||
its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144]
|
||||
shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`).
|
||||
@@ -1,6 +0,0 @@
|
||||
- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200
|
||||
but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc`
|
||||
adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key
|
||||
rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of
|
||||
`expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no
|
||||
version-bump helps. Merge bakes deltas in (bypasses this) but is static.
|
||||
@@ -1,5 +0,0 @@
|
||||
- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4
|
||||
(compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY
|
||||
(temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models.
|
||||
Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the
|
||||
whole NVFP4-quant surface.
|
||||
@@ -1,7 +0,0 @@
|
||||
- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json`
|
||||
for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the
|
||||
`tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply
|
||||
PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has
|
||||
no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral
|
||||
tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native
|
||||
mistral tokenizer + vision).
|
||||
@@ -1,6 +0,0 @@
|
||||
- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head
|
||||
(`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the
|
||||
quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors
|
||||
main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16
|
||||
self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format
|
||||
(like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.)
|
||||
-6
@@ -1,6 +0,0 @@
|
||||
- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base
|
||||
(`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2**
|
||||
(safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the
|
||||
`<think>` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified).
|
||||
**Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw
|
||||
abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny '<ip>'` line) AND an instrumented receiver.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`.
|
||||
-1
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe.
|
||||
@@ -1 +0,0 @@
|
||||
- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.)
|
||||
@@ -1,15 +0,0 @@
|
||||
- `[2026-07-17]` **Zonos2 `:1920` engine → self-contained container (stays on 3090); prosody-priming is a SERVING-LAYER change (engine stays stock).**
|
||||
|
||||
**Context.** The production Zonos TTS engine (irv-ml1 `:1920`, feeds asset-engine + gateway-chat via `zonos-gateway` :8890) was a bare native process — its real launch config existed ONLY in the running process argv (the committed `~/tts-audition/harness/zonos_server.sh` was STALE: said A6000/:1919/no perf flags; live is 3090/:1920 with `--cuda-graph-max-bs 1 --num-pages 16384 --max-running-requests 2 --memory-ratio 0.3`). Captured to eshpfi `stacks/zonos-engine/` (README + corrected `zonos2-server.sh` + `.env.example`), commit **14a0004** (UNPUSHED as of the snapshot).
|
||||
|
||||
**Decision 1 — containerize as a SELF-CONTAINED image** (not systemd — operator rejected; not a thin bind-mount wrapper — I walked that back: bind-mounting the host's CUDA-compiled `.venv` couples to the host's exact CUDA/glibc and is fragile + not reproducible). Shape: `FROM` a CUDA 12.8 base → `uv sync` against the repo's committed `uv.lock` (deterministic env) → mount the ~15 GB HF weights (`~/.cache/huggingface/hub/models--Zyphra--ZONOS2`, do NOT bake) → pin the **3090** (`NVIDIA_VISIBLE_DEVICES=0`) → `restart: unless-stopped` → CMD = the captured invocation. **Engine stays STOCK** Zyphra/Zonos2 @ commit `194c0a3` (no fork — the `zonos2` package ships its own server). **Build risk:** heavy compiled-CUDA deps (flashinfer / sgl_kernel / cutlass-dsl / apache-tvm-ffi / pynini) on torch 2.9.1+cu128 — mostly prebuilt wheels + the `uv.lock` make it tractable, expect a couple build iterations. **Cutover (in place on the 3090):** stop the native process (frees ~17 GB) → `docker compose up -d` (re-allocates ~17 GB, same footprint) → repoint `zonos-gateway`'s `ZONOS_URL` at the container (or keep the `:1920` host-port publish). One brief prod-TTS blip.
|
||||
|
||||
**GPU = 3090 (operator 2026-07-17).** Keep it OFF the A6000 — the A6000 already OOMs under ComfyUI load (idle ~19 GB but spikes far higher during gen), so it can't host Zonos too. The 3090 already runs Zonos, so the containerize-in-place cutover changes nothing about placement.
|
||||
|
||||
**Decision 2 — the prosody-priming hypothesis (operator's test; the reason for building fresh).** PRIME the autoregressive engine with an emotional sentence, then TRUNCATE it from delivery: prepend a primer → **generate "primer + real text" as ONE continuous utterance** (the AR model carries prosody forward across the boundary) → ASR-timestamp the primer's end (**parakeet**, already up on irv-ml1 `:8765`, word timestamps) → **clip the primer in the inter-sentence silence gap** (+ ~15 ms fade-in, no click) → deliver only the real text, now wearing the primed prosody. Examples: primer "I'm so EXCITED about this." → "This will be a lot of fun!" spoken excited; primer "I'm whispering this to you right now." → "I'm so glad to see you baby." whispered. **This is PURE serving-layer orchestration — the engine is untouched; it lives in the gateway adapter `stacks/zonos/adapter/server.py`.** Only fork the engine if the black-box approach fails.
|
||||
|
||||
**THE CRUX the test resolves:** does AR prosody actually **carry across the sentence boundary**, or does Zonos reset at the period? → the harness A/Bs the **JOIN punctuation**: period (operator's examples) vs comma vs ellipsis vs none ("…excited about this, this will be…"). Everything else is plumbing.
|
||||
|
||||
**Plan / design recs.** (a) Build the stock engine image (parallel track). (b) Stand up a priming TEST HARNESS against the NATIVE engine (fast iteration, seconds) + parakeet ASR: prime→generate→timestamp→gap-clip→out; compare primed-clipped vs plain on the two cases (subjective + a cheap objective proxy: pitch/energy variance for "excited", spectral-tilt/low-energy for "whisper"). Iterate on the join, then bake the winner into the gateway adapter. **Primer source:** caller-supplied for the harness (test arbitrary primers) → a curated emotion→primer library (`excited`/`whisper`/…) + optional caller override for production. **ASR:** parakeet primary; WhisperX forced-align fallback if parakeet word timestamps are coarse.
|
||||
|
||||
See eshpfi `stacks/zonos-engine/README.md` + `stacks/zonos/` (the gateway adapter).
|
||||
@@ -1,57 +0,0 @@
|
||||
- `[2026-07-18]` **Fleet Gitea-Actions build recipe + the `vh`-is-a-user package-write constraint** (learned the hard way across 3 failed soong-lab validation builds; reusable for ANY fleet CI image build or package publish).
|
||||
|
||||
**The runner.** One `act_runner` (`gitea/act_runner`) on ana-docker, labels
|
||||
`pfi-fleet` / `ana-docker` → both map to job image **`node:20-bookworm-slim`**,
|
||||
which has **NO docker and NO git**. Config `/opt/docker/conf/gitea-runner/data/config.yaml`:
|
||||
`valid_volumes: []` (no socket propagated to job containers). So:
|
||||
- `actions/checkout@v4` fails (needs git); `docker/*` marketplace actions fail
|
||||
(need docker) — a workflow built on those dies at the first step (~15s).
|
||||
|
||||
**The working recipe (mirror Worldtree `deploy.yml`).** Run the job in a
|
||||
docker-capable image + drive docker with RAW commands, not the JS actions:
|
||||
```yaml
|
||||
runs-on: pfi-fleet
|
||||
container:
|
||||
image: docker:24.0.7-cli # has docker+buildx; add git+node
|
||||
steps:
|
||||
- run: apk add --no-cache git nodejs # so actions/checkout@v4 works
|
||||
- uses: actions/checkout@v4
|
||||
- name: login # RAW, not docker/login-action
|
||||
run: echo "$REGISTRY_TOKEN" | docker login gitea.phasefinal.com -u "$REGISTRY_USER" --password-stdin
|
||||
- name: buildx builder
|
||||
run: docker buildx create --name X --driver docker-container --use; docker buildx inspect --bootstrap
|
||||
- name: build+push # RAW, not docker/build-push-action
|
||||
run: docker buildx build --secret id=<name>,env=<TOKEN> -t <img>:latest --push .
|
||||
```
|
||||
The runner mounts the host docker socket into ITSELF; the docker:cli job reaches
|
||||
the daemon through that. The `docker/*` JS actions are unreliable on act_runner —
|
||||
raw commands are the fleet convention.
|
||||
|
||||
**`vh` is a USER account, not an org.** Consequences that bit repeatedly:
|
||||
1. `GET /api/v1/orgs/vh` → 404 "user redirect"; there are **no org teams** to add
|
||||
a service account to.
|
||||
2. **User-owned packages are OWNER-WRITE-ONLY.** claude-bot (even repo
|
||||
admin-*collaborator* on `vh/soong-lab`, even with `write:package` scope + full
|
||||
basic-auth) gets **`401 unauthorized`** on `docker push` to `vh/soong-lab`, and
|
||||
`npm publish` to `vh/npm/` would 401 too. Only `vh` itself can write vh packages.
|
||||
→ CI must authenticate AS `vh` for the push (a vh-owned `write:package` PAT as
|
||||
`REGISTRY_TOKEN` + `REGISTRY_USER=vh`), exactly how WT pushes `vh/worldtree`.
|
||||
claude-bot CAN still: clone/read repos, READ packages (pulled the image fine),
|
||||
dispatch workflows, mint demo Worldtree keys.
|
||||
3. **Repo Actions secrets are OWNER-ONLY too** — `PUT .../actions/secrets/X` as
|
||||
claude-bot (repo admin-collab) → 403 "user should be the owner of the repo".
|
||||
Only `vh` can set a repo's secrets.
|
||||
|
||||
**Other gotchas:**
|
||||
- Gitea **reserves the `GITEA_` secret-name prefix** — a secret named
|
||||
`GITEA_PYPI_TOKEN` is illegal; use e.g. `PYPI_TOKEN`.
|
||||
- Gitea **package auth is token-based / username-lenient** — `docker login` /
|
||||
PyPI basic-auth authenticate via the token; the username is nominal (tested
|
||||
`-u gitea` and `-u claude-bot` both 200 against the vh PyPI). So a Dockerfile
|
||||
hardcoding `UV_INDEX_GITEA_USERNAME=gitea` is fine with any valid token.
|
||||
- Homepage (esh-docker-vm) docker-label auto-discovery only covers the 5 endpoints
|
||||
in its `docker.yaml` (esh-vm-docker, ana-docker, ana-ml2, nh3-docker, irv-ml1);
|
||||
**corviduo-dev is NOT watched** → services there need a manual `services.yaml`
|
||||
entry, not labels.
|
||||
|
||||
Applied in the soong-lab CI: [[2026-07-18-soong-lab-containerize-cutover]].
|
||||
@@ -1,46 +0,0 @@
|
||||
- `[2026-07-18]` **soong-lab containerize cutover — COMPLETE + LIVE on corviduo-dev.**
|
||||
|
||||
Migrated soong-lab (Noonien Soong character-design studio) from a hand-built
|
||||
`soong-lab-studio.service` (systemd + git-pull-on-webhook) to a containerized
|
||||
deploy, image built by CI + pushed to the Gitea registry. soong-dev owns the
|
||||
in-repo artifacts (Dockerfile/compose/workflow/`docs/DEPLOY.md` = checklist);
|
||||
infra-ops owned the host cutover. Operator confirmed functional ("Soong works
|
||||
great" — a real Soong turn round-trips + saves) → cutover 100% closed.
|
||||
|
||||
**Final state (corviduo-dev, 10.250.50.152):**
|
||||
- Container `soong-lab-soong-lab-1` LIVE + healthy on `0.0.0.0:8443`, image
|
||||
`gitea.phasefinal.com/vh/soong-lab:latest` (v0.3.24), `restart:unless-stopped`
|
||||
(survives reboot; no systemd unit needed — docker restart policy handles boot).
|
||||
- Deploy dir **`/home/infra-ops/soong-lab-deploy/`** — pull-based `compose.yaml`
|
||||
(image + env_file + `8443:8443` + named volumes; NO build/secrets stanza) +
|
||||
`.env` (copied from the live `soong-lab.env`, STRIPPED of the `SOONG_LAB_*_DIR`
|
||||
overrides so the container uses image defaults `/data/library` + `/data/portraits`
|
||||
+ `/app/web` → the volumes).
|
||||
- Named volumes `soong-lab_soong-library` + `soong-lab_soong-portraits`, migrated
|
||||
from `/home/infra-ops/soong-lab-data/{library,portraits}` (2 saved designs incl.
|
||||
**Sindra** + 27 portraits), **chowned `10001:999`** (the container `soong` user)
|
||||
so it can read AND write new designs.
|
||||
- Old `soong-lab-studio.service` + `soong-webhook.service` (the `:9010` git-pull
|
||||
redeploy listener) both **stopped + disabled**.
|
||||
|
||||
**Topology reality (≠ what DEPLOY.md assumed):** there is **NO TLS proxy**.
|
||||
WT-personal (`:8081`) and soong-lab are **co-located on corviduo-dev**, and the
|
||||
Bifrost callback is **plain-HTTP same-host** `http://10.250.50.152:8443` — the
|
||||
value of `SOONG_LAB_BIFROST_ENDPOINT_URL`, unchanged by the move, so the WT
|
||||
Bifrost host-allowlist stayed valid as-is. Nothing on the WT side needed touching.
|
||||
|
||||
**Safety net:** data backup `/home/infra-ops/soong-lab-data-backup-20260718-091831.tar.gz`
|
||||
(35M) taken BEFORE migration. Verified pre-retire: `/api/version` 200 (0.3.24),
|
||||
SPA `/` 200, `POST /bifrost/tool-call` → 401 (route present + auth-gated),
|
||||
bidirectional WT↔soong reachability, container healthcheck green.
|
||||
|
||||
**Ops commands:**
|
||||
- Redeploy a new image: `cd /home/infra-ops/soong-lab-deploy && sudo docker compose pull && sudo docker compose up -d`.
|
||||
(Auto-pull-on-`:latest` — watchtower or a deploy hook — is an open follow-up.)
|
||||
- Rollback: `sudo docker compose down` + `sudo systemctl enable --now soong-lab-studio.service soong-webhook.service`.
|
||||
- Homepage tile: manual `- Apps:` entry "Soong Lab" (href http://10.250.50.152:8443)
|
||||
in esh-docker-vm `/opt/docker/conf/homepage/services.yaml` — corviduo-dev isn't
|
||||
a Homepage-watched docker endpoint, so docker-label auto-discovery can't surface
|
||||
it (see [[2026-07-18-fleet-gitea-runner-build-recipe]] for the CI half).
|
||||
|
||||
See [[reference_corviduo_dev_emergency_ops]], [[reference_claude_bot_gitea_creds]].
|
||||
@@ -1,72 +0,0 @@
|
||||
- `[2026-07-18]` **Zonos2 emotion CANONICAL from an empirical sweep + the voice-cloning pipeline.**
|
||||
|
||||
**Voice-cloning pipeline (established this session).** Source zips at
|
||||
`/mnt/smithy/voice_clones/<name>.zip` (irv-ml1 NFS from nh3-nas; remount
|
||||
post-reboot) — each = diarized single-speaker podcast clips + `manifest.jsonl`
|
||||
(per-clip WhisperX `mean_score`, word timestamps, text) + `metadata.csv`.
|
||||
`~/development/zonos-tools/assemble_voice.py <dir>` ranks by mean_score and
|
||||
concatenates top clips to ~15–24s (Zyphra's blessed clone-ref length; single
|
||||
clip if already ≥15s). Drop the assembled `<Name>.wav` into the gateway voices
|
||||
dir → `voice:"name"`. 4 characters cloned: **Emmie, Penny, Natalie, Miranda**
|
||||
(+ Zyphra defaults AmericanFemale/Male/British/Cora) = 8 voices in
|
||||
`zonos-gateway`. Clone is inline `speaker_audio_base64` (text-independent Qwen3
|
||||
speaker embedding — NO transcript); `/tts/speakers` registration is
|
||||
session-scoped (needs `X-TTS-Session-ID`), so the gateway holds the ref wav and
|
||||
clones per-call.
|
||||
|
||||
**Gateway voices are host-managed (bind-mount, added this session).** Added
|
||||
`./voices:/app/voices:ro` to `/opt/docker/compose/zonos-gateway/compose.yaml`
|
||||
(committed to `vh/zonos-gateway` + eshpfi mirror `438cd35`). So adding a voice =
|
||||
drop the wav + `docker compose restart zonos-gateway` (registry rebuilds at
|
||||
boot; NO image rebuild). This also un-stranded the other voices (deploy build
|
||||
context had only Cora before). Voice wavs committed to the repo for backup.
|
||||
|
||||
**Emotion mechanism (Zyphra canonical, from their README @194c0a3).** Additive
|
||||
direction vectors: 4 named (happy/sad/angry/surprised) + valence/arousal axes.
|
||||
`emotion_strength` 1.0 = per-voice calibrated (calibration.json optimizes
|
||||
emotion2vec recognizability only, NOT identity). `accurate_mode` is THE trade-off:
|
||||
`true` = closer voice match (identity), `false` = expressive mode (emotion lands,
|
||||
identity drifts). Zyphra's strong recipe: `accurate_mode:false` + `cfg~1.5`.
|
||||
Single-emotion is blessed; mixing is unblessed (and degrades the clone — operator
|
||||
confirmed by ear). "deaf by 1.5" — cfg past 1.5 distorts + costs ~2× compute.
|
||||
|
||||
**THE SWEEP (`~/development/zonos-tools/emotion_sweep.py`).** 4 cloned voices × 4
|
||||
named emotions × {accurate,expressive}×{cfg 1.0,1.3,1.5} @ strength 1.0,
|
||||
single-emotion, neutral sentence + a neutral baseline per voice (~100 clips).
|
||||
Scored on TWO axes: **emotion-landing** = emotion2vec `iic/emotion2vec_plus_large`
|
||||
target-emotion prob [0-1]; **identity** = resemblyzer speaker-embedding cosine vs
|
||||
the clone reference (neutral baseline ~0.85). Scoring env:
|
||||
`uv run --with resemblyzer --with funasr --with "numpy<2" --with soundfile
|
||||
--with requests --with "setuptools<80" --with torchaudio` (setuptools<80 for
|
||||
webrtcvad's pkg_resources; torchaudio for funasr).
|
||||
|
||||
**RESULTS (mean across the 4 voices) — emotion, best setting, emo/id:**
|
||||
- happy — **exp cfg1.5** 0.80/0.68 (soft: exp cfg1.0 0.76/0.69) → WORKS
|
||||
- sad — **exp cfg1.5** 0.53/0.57 (only working cell; id below the ~0.65 floor) → modest
|
||||
- angry — acc cfg1.3 / exp cfg1.5 tied at ~0.25 emo → WEAK (named ceiling ~0.25)
|
||||
- surprised — max ~0.015 across ALL settings → NON-FUNCTIONAL on the named direction
|
||||
Accurate + low cfg = identity/suppress regime (emo→0); expressive REQUIRED for
|
||||
emotion to land, at ~0.15–0.28 identity cost.
|
||||
|
||||
**dvalin-smithy-dev synthesis (adopted, triaged genuine-adds; thread
|
||||
`01KXT12FN0AS5A3WMKEK06BVPS`):**
|
||||
1. Treat **identity as a hard FLOOR (~0.65)**, not a free variable in emo×id.
|
||||
2. **Two-regime policy** — Regime A (default, identity-critical dialogue):
|
||||
`accurate_mode:true, cfg 1.0, emotion off` (text carries it) or soft-happy
|
||||
(exp cfg1.0). Regime B (tagged drama beats): `accurate_mode:false, cfg 1.5`,
|
||||
single emotion or axes. Line-type→regime heuristic (exposition→A, grief→B+sad,
|
||||
confrontation→B+axes-angry, shock→B+axes-arousal).
|
||||
3. **Axes-first for the broken emotions** — angry ≈ valence −0.6..−0.8 / arousal
|
||||
+0.5..+0.8; surprised ≈ valence +0.2..+0.4 / arousal +0.7..+1.0 (exp cfg1.5);
|
||||
or "startled-happy" (happy + high arousal) as a surprised stand-in. These are
|
||||
PROVISIONAL — the sweep did NOT test axes.
|
||||
|
||||
**NEXT (highest VoI, operator to green-light):** an **axes sweep** for
|
||||
angry/surprised (valence×arousal grid) — the only path to rescue the two broken
|
||||
named emotions; then a strength ladder at the best cells + emotion-congruent text
|
||||
(neutral content understates landing) + per-voice tables + a 2nd emotion judge /
|
||||
human pairwise. Then bake the happy/sad canonical into gateway presets. I owe
|
||||
dvalin the axes-sweep numbers.
|
||||
|
||||
See [[reference_zonos_tts_stack]]; dials-first spec at `vh/zonos-gateway`
|
||||
`docs/EMOTION-DIALS-SPEC.md`.
|
||||
@@ -1,41 +0,0 @@
|
||||
- `[2026-07-18]` **zonos-gateway 0.2.1 — voice-resolved emotion presets baked (provisional) from the axes sweep.**
|
||||
|
||||
After the axes sweep ([[reference_zonos_tts_stack]] + the `[2026-07-18] axes sweep`
|
||||
Recent-decisions entry) rescued angry and confirmed startled-happy, the operator
|
||||
green-lit baking the results as **provisional** gateway presets + docs. Shipped
|
||||
`vh/zonos-gateway` **0.2.1** (main `8f1885b`, tag `v0.2.1`, PUSHED; deployed live
|
||||
on irv-ml1 `:8890`).
|
||||
|
||||
**Design — voice-resolved, NOT global.** `resolve_preset(name, voice)` picks the
|
||||
per-voice measured cell, because a single global preset is unsafe (dvalin ruling;
|
||||
BritishFemale's *named* angry misfires as fear). Presets:
|
||||
- `angry`, `happy`, `startled_happy` (+ aliases `surprised`, `startled` →
|
||||
startled_happy). All expressive (`accurate_mode:false`), cfg 1.5, pure-axes
|
||||
(no named sliders).
|
||||
- Calibrated cells (the 3 default voices):
|
||||
- angry: AmF v-0.4/a+1.0 s1.0 (emo0.53/id0.685); BrF v-0.4/a+0.8 s1.0
|
||||
(emo0.99/id0.725, metric fear-clean); AmM **two-tier** — soft v-0.6/a+0.8 s1.0
|
||||
(0.23/id0.654) + drama v-0.6/a+0.8 s1.2 (1.0/id0.616 clean; strength is NOT a
|
||||
smooth knob on AmM, 1.0→1.2 is the window, past that flips to disgust).
|
||||
- happy / startled_happy: AmF v+0.6/a+0.8; AmM v+0.3/a+1.0; BrF v+0.6/a+1.0
|
||||
(happy~1.0, id 0.74-0.80; axes-happy keeps +0.15 id over the named happy slider).
|
||||
- `sad` = unchanged named-slider preset (not axes-tested).
|
||||
- Uncalibrated voices (Cora + the 4 clones) → mid-region fallback until measured.
|
||||
- Docs surface: `/v1/dials` exposes `voice_emotion_presets`; the FastAPI `/docs`
|
||||
description documents it; durable spec `docs/EMOTION-DIALS-SPEC.md` (moved INTO
|
||||
the repo — was mirror-only); README table. 44 tests green.
|
||||
|
||||
**Repo-hygiene gotcha (fixed).** The local clone `~/development/zonos-gateway` and
|
||||
gitea `vh/zonos-gateway` had **TWO UNRELATED git histories** (no merge-base) — gitea
|
||||
held the voice-wav commits, the local clone held the code + no remote. Reconciled
|
||||
by resetting local→origin/main, overlaying the 7 bake files, `uv lock`, commit,
|
||||
push (fast-forward). Voices stay tracked; local now shares gitea's lineage + has
|
||||
origin wired. **The deployed irv-ml1 tree `/opt/docker/compose/zonos-gateway` is
|
||||
still NON-git** (hand-updated build context) — CI-wire remains an open follow-up.
|
||||
|
||||
**Provisional pending** ear-validation on emotion-congruent text (the neutral-text
|
||||
audition was inconclusive: "they all sound different, hard to tell"). Follow-ups:
|
||||
sad axes/text pass on the 3 voices; congruent-text pass; clone-char emotion rows.
|
||||
Tools `~/development/zonos-tools/{axes_sweep,strength_ladder,gen_auditions,dial-in-studio}.py`
|
||||
(run ON irv-ml1; scoring env `uv run --with resemblyzer --with funasr --with "numpy<2"
|
||||
--with soundfile --with requests --with "setuptools<80" --with torchaudio`).
|
||||
@@ -0,0 +1,52 @@
|
||||
# [2026-08-15] Uncensored gen seat: Qwen3.8-27B-Uncensored deployed; the definitive MTP-graft fix
|
||||
|
||||
**Outcome.** The fleet `gen` seat is now **`JonathanColetti/Qwen3.8-27B-Uncensored`** (Heretic
|
||||
abliteration, KL 0.12 vs base, bench Δ −0.5 within noise, refusals 98→12/100), quantized in-house
|
||||
to **NVFP4 W4A16** (llm-compressor / compressed-tensors) with a **grafted bf16 MTP head**,
|
||||
vision-intact, **262K** ctx, MTP n=3 (**~42% accept, ~68 tok/s**), coherent. Live at ana-ml2 `:8015`
|
||||
(project `gen-seat` / container `vllm-gen`), backing all 7 gateway aliases.
|
||||
|
||||
**THE definitive lesson (resolved 3 failed attempts + one premature 50 GB delete).** A grafted bf16
|
||||
MTP scored **0% on the quant but 83% at bf16** — for TWO different abliterated models. Root cause was
|
||||
NEITHER the abliteration NOR the quant scheme: it was **the grafted `mtp.*` tensors missing from
|
||||
`config.json` → `quantization_config.ignore`.** The wrapper-class quant DROPS the MTP before
|
||||
llm-compressor sees it, so nothing gets added to `ignore`; vLLM then tries to load the bf16 MTP as
|
||||
*quantized* format → "Parameter … not found in params_dict, skip loading" → uninitialized head → 0%.
|
||||
**FIX: after grafting, add `re:^mtp.*` to `quantization_config.ignore`** (one line — all unsloth's
|
||||
working checkpoint has). MTP jumped 0%→83% (bf16-identical). Full lesson in auto-memory
|
||||
`reference_abliteration_mtp_lessons`.
|
||||
|
||||
**The pipeline that works (for the next VL+MTP quant, incl. the W4A8 chase):**
|
||||
1. Pull bf16 (kept at `ana-ml2:/tank/aimodels/qwen38-27b-uncensored-bf16`).
|
||||
2. Quant via `quant_nvfp4_qwen.py` (darkscarlett dir) = the **wrapper-class** loader
|
||||
(`Qwen3_5ForConditionalGeneration`, keeps the vLLM-serveable config); container = `vllm-openai`
|
||||
+ `pip install llmcompressor==0.13.0` (drags in a transformers with `qwen3_5`).
|
||||
3. **Graft** the author's `model-mtp.safetensors` verbatim into the output + merge the index.
|
||||
4. **Reconstruct** `preprocessor_config.json` from `processor_config.json`'s `image_processor`
|
||||
sub-dict (the repo omits it → else "Can't load image processor" crash-loop).
|
||||
5. **Add `re:^mtp.*` to the output config's `quantization_config.ignore`.** ← the fix.
|
||||
6. Serve: `--quantization compressed-tensors --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'`
|
||||
`--mamba-cache-dtype float32 --kv-cache-dtype fp8 --reasoning-parser qwen3`.
|
||||
|
||||
**VRAM / full-context budget (measured).** Weights ~27 GB; hybrid attention → **only 16 of 64 layers
|
||||
carry KV** → 32 KiB/token → **262K KV = 8.6 GB** (vs ~60–70 GB for a normal dense 27B). Full 262K fits
|
||||
GPU0 at **util 0.45** (~43 GB) alongside meromero (~49 GB used, it's a 31B) — pre-flight rejects util
|
||||
0.48 (wants 45.6 GB, only 45.5 free). `max-num-seqs 16` keeps cudagraph modest (an ad-hoc serve with
|
||||
no cap OOM'd — cudagraph captured to batch-512).
|
||||
|
||||
**Why unsloth's `qwen3.8-27b` (the prior gen model) was faster (97 vs 68 tok/s).** ~half = quant kernel
|
||||
(unsloth native NVFP4+FP8 tensor cores vs our W4A16 → Marlin dequant, ~20% even on decode — I'd
|
||||
under-stated this); ~half = MTP acceptance (unsloth 55% un-ablated head vs our 42% — inherent to the
|
||||
ablation, no quant fixes it). **W4A8 recovers the first ~20% (→~82 tok/s) + prefill; not the MTP half.**
|
||||
|
||||
**modelopt dead-end (for W4A8, avoid).** `nvidia-modelopt[hf]==0.43.0` is too old for qwen3_5's
|
||||
transformers: (a) its `NVFP4_DEFAULT_CFG.quant_cfg` is a LIST but 0.43 wants a DICT (pydantic reject);
|
||||
(b) it warns transformers 5.15 untested. Use **llm-compressor** for W4A8 instead (custom recipe: NVFP4
|
||||
weights + FP8 input_quantizer + calibration on `heretic2-nvfp4-work/production_calib_512.jsonl`).
|
||||
|
||||
**Deleted (premature — the delete I owned).** `windowsxp811203/Qwen3.8-27B-Abliterated` (~79 GB) — I
|
||||
declared it desync-dead off a 0% that was actually this ignore bug. Lesson: **test MTP on bf16 first;
|
||||
isolate before deleting.**
|
||||
|
||||
Commits: eshpfi `680c30e` (deploy + rename + litellm + README), dotfiles `1d1970f` (CLAUDE.md roster) —
|
||||
both UNPUSHED. Related: [[reference_abliteration_mtp_lessons]], [[reference_verify_hf_repo_ids_before_pull]].
|
||||
Reference in New Issue
Block a user