From d710e56acae0e1e07e47cc9e8b119bfe60bcbca1 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 15 Jul 2026 13:33:42 -0700 Subject: [PATCH] memory: migrate persistent-memory.md to two-tier index (53 detail files) Split the 53 over-threshold dated log entries (Recent decisions, Tried and abandoned) into per-entry persistent-memory.d/.md detail files, leaving one-line pointers in the index; the 10 short entries stay inline. Startup index drops 60,527 -> 25,256 bytes (492 -> 270 lines); entry bodies move verbatim to on-demand detail files, so a fresh session loads ~25 KB instead of ~60 KB and pulls a detail file only when its pointer is relevant. Top matter (Repo purpose, Tools & conventions, Current state) is unchanged; both archival back-references preserved. CLAUDE.md persistent-memory section now documents the index<->detail read discipline (read the index, pull details on demand, never bulk-read the dir, commit both together). Auto-archival still held every dated entry back (all <30 days old); the July burst begins aging past the 30-day guard ~2026-07-31. --- CLAUDE.md | 9 + ...onal-worldtree-ci-deploy-that-fails-85s.md | 4 + ...-systemd-user-daemon-that-shells-out-to.md | 4 + ...y-version-mutates-the-shared-deployment.md | 6 + ...-04-on-prem-t1-train-that-keeps-any-ana.md | 4 + ...tguns-cost-several-wasted-serve-bounces.md | 5 + ...lang-generic-image-can-t-load-our-nvfp4.md | 4 + ...ora-target-modules-cli-enum-rejects-the.md | 4 + ...07-vllm-0-24-0-qwen3-5-lora-application.md | 6 + ...8-angel-allura-org-ms3-2-24b-angel-self.md | 5 + ...s-silently-running-3-epochs-harness-gap.md | 19 + ...vllm-tokenizer-vision-traps-serve-ms3-2.md | 7 + ...-inference-pivot-executed-serve-curated.md | 8 + ...on-27b-mtp-on-vllm-compressed-tensors-0.md | 6 + ...ing-27b-refuses-dark-fiction-despite-an.md | 6 + ...t-campaign-closed-char-rp-magidonia-24b.md | 11 + ...t1-dpo-leg-is-running-unblocked-2-fixes.md | 12 + ...dpo-leg-launch-prior-block-now-resolved.md | 17 + ...mir-deploy-blocker-resolved-mid-session.md | 4 + ...-breaks-morpheus-audio-token-generation.md | 4 + ...gen-memory-extractor-bind-green-lit-for.md | 7 + ...en-memory-extractor-bind-host-synced-on.md | 46 ++ ...sper-datasets-aren-t-actually-whispered.md | 4 + ...ts-off-the-shelf-voice-pipeline-shipped.md | 10 + ...-09-sentence-chunking-tts-loses-prosody.md | 3 + ...parked-items-closed-phantom-qwen3-6-35b.md | 16 + ...y-open-weight-releases-scan-cron-set-up.md | 20 + ...26-07-10-comfyui-0-25-x-bump-on-irv-ml1.md | 16 + ...7-10-comfyui-v0-27-1-success-on-irv-ml1.md | 14 + ...toskr-affect-full-on-personal-worldtree.md | 13 + ...ot-cause-supersedes-the-litellm-gateway.md | 1 + ...-speed-real-bottleneck-uv-sync-memory-s.md | 1 + ...-consumer-ledger-miranda-provisioned-on.md | 1 + ...ying-a-peer-s-diagnosis-as-fact-without.md | 1 + ...eploy-bottleneck-the-image-build-11-min.md | 1 + ...355-residual-300s-hang-localized-to-our.md | 1 + ...rn-lifecycle-fix-validated-on-worldtree.md | 1 + ...14-aeon-s-working-nvfp4-mtp-rp-seat-was.md | 1 + ...-rp-reasoning-seat-deckard-pkd-neo-code.md | 1 + ...-gitea-test-delivery-204-is-not-proof-a.md | 1 + ...tp-graft-via-top-level-mtp-tensor-names.md | 1 + ...modelopt-no-checkpoint-config-skips-the.md | 1 + ...essor-compressed-tensors-gives-no-batch.md | 1 + ...-mtp-fast-char-rp-reasoning-seat-landed.md | 1 + ...-resolved-gibberish-pivoted-to-modelopt.md | 1 + ...ke-built-the-full-mtp-serve-scaffolding.md | 1 + ...ue-the-nvfp4-mtp-fast-char-rp-reasoning.md | 1 + ...lab-webhook-auto-deploy-real-root-cause.md | 1 + ...lly-switched-off-image-judge-qwen-image.md | 1 + ...-code-statusline-cost-total-cost-usd-is.md | 1 + ...service-after-remote-fs-target-does-not.md | 1 + ...5-esh-docker-vm-nfs-fstab-fix-x-systemd.md | 1 + ...esh-docker-vm-d-state-phantom-container.md | 1 + ...15-vllm-max-model-len-does-not-free-gpu.md | 1 + persistent-memory.md | 396 ++++-------------- 55 files changed, 405 insertions(+), 309 deletions(-) create mode 100644 persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md create mode 100644 persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md create mode 100644 persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md create mode 100644 persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md create mode 100644 persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md create mode 100644 persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md create mode 100644 persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md create mode 100644 persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md create mode 100644 persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md create mode 100644 persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md create mode 100644 persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md create mode 100644 persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md create mode 100644 persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md create mode 100644 persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md create mode 100644 persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md create mode 100644 persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md create mode 100644 persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md create mode 100644 persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md create mode 100644 persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md create mode 100644 persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md create mode 100644 persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md create mode 100644 persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md create mode 100644 persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md create mode 100644 persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md create mode 100644 persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md create mode 100644 persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md create mode 100644 persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md create mode 100644 persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md create mode 100644 persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md create mode 100644 persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md create mode 100644 persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md create mode 100644 persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md create mode 100644 persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md create mode 100644 persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md create mode 100644 persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md create mode 100644 persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md create mode 100644 persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md create mode 100644 persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md create mode 100644 persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md create mode 100644 persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md create mode 100644 persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md create mode 100644 persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md create mode 100644 persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md create mode 100644 persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md create mode 100644 persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md create mode 100644 persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md create mode 100644 persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md create mode 100644 persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md create mode 100644 persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md create mode 100644 persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md create mode 100644 persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md create mode 100644 persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md create mode 100644 persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md diff --git a/CLAUDE.md b/CLAUDE.md index 2692845..6b935ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,6 +10,15 @@ state) across context resets. Read it at session start; treat it as one input alongside this CLAUDE.md and the auto-memory system, not as the single source of truth. +It is a lean **index**: the dated log sections (Recent decisions, +Tried and abandoned) keep each over-threshold entry's full body in +`persistent-memory.d/.md`. Read the index at session start; +pull a detail file only when its index line is relevant to your work — +never bulk-read `persistent-memory.d/`. When you commit, stage any +pending `persistent-memory.md` and `persistent-memory.d/` updates in +the same commit as the work that prompted them — durable memory that +lags the code defeats its own purpose. + **New session starting here?** Read [`docs/orientation.md`](docs/orientation.md) first — fleet topology, backup architecture, governing principles, and all the NFS/DSM/naming gotchas that have cost past sessions time. **For SSH-driven work: use `scripts/elway`.** Write a playbook under diff --git a/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md b/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md new file mode 100644 index 0000000..aaffaaa --- /dev/null +++ b/persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md @@ -0,0 +1,4 @@ +- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" + is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but + fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push + finishes. FIX: re-run once built, or gate on `workflow_run: completed`. diff --git a/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md b/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md new file mode 100644 index 0000000..f391305 --- /dev/null +++ b/persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md @@ -0,0 +1,4 @@ +- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools + needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The + althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO + routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. diff --git a/persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md b/persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md new file mode 100644 index 0000000..5837c47 --- /dev/null +++ b/persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md @@ -0,0 +1,6 @@ +- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on + per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a + param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT + caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at + a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm` + flushes residual carryover. `feedback_litellm_shared_param_mutation`. diff --git a/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md b/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md new file mode 100644 index 0000000..c5a2fcb --- /dev/null +++ b/persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md @@ -0,0 +1,4 @@ +- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe + ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast + on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the + 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. diff --git a/persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md b/persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md new file mode 100644 index 0000000..26159ec --- /dev/null +++ b/persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md @@ -0,0 +1,5 @@ +- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run + --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`); + `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"` + break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container + whose crash logs you need, BEFORE bouncing a production serve. diff --git a/persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md b/persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md new file mode 100644 index 0000000..56d7e8d --- /dev/null +++ b/persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md @@ -0,0 +1,4 @@ +- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/ + packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated + `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was + quant-loader-specific). diff --git a/persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md b/persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md new file mode 100644 index 0000000..ca64f10 --- /dev/null +++ b/persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md @@ -0,0 +1,4 @@ +- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for** + (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants + its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] + shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`). diff --git a/persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md b/persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md new file mode 100644 index 0000000..a4a5b53 --- /dev/null +++ b/persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md @@ -0,0 +1,6 @@ +- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200 + but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc` + adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key + rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of + `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no + version-bump helps. Merge bakes deltas in (bypasses this) but is static. diff --git a/persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md b/persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md new file mode 100644 index 0000000..243f207 --- /dev/null +++ b/persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md @@ -0,0 +1,5 @@ +- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 + (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY + (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. + Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the + whole NVFP4-quant surface. diff --git a/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md b/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md new file mode 100644 index 0000000..a3bd8ef --- /dev/null +++ b/persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md @@ -0,0 +1,19 @@ +- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 + epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass + `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs + + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the + off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a + ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. + FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, + float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. + GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min + train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; + train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 + banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): + ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is + COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral + literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, + arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep + on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + + the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO + adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). diff --git a/persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md b/persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md new file mode 100644 index 0000000..09ba115 --- /dev/null +++ b/persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md @@ -0,0 +1,7 @@ +- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json` + for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the + `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply + PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has + no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral + tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native + mistral tokenizer + vision). diff --git a/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md b/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md new file mode 100644 index 0000000..5b77505 --- /dev/null +++ b/persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md @@ -0,0 +1,8 @@ +- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** + Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, + util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps + ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled + refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); + Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting + ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway + sampling-defaults wiring still PENDING. diff --git a/persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md b/persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md new file mode 100644 index 0000000..dd87d90 --- /dev/null +++ b/persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md @@ -0,0 +1,6 @@ +- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head + (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the + quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors + main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16 + self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format + (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.) diff --git a/persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md b/persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md new file mode 100644 index 0000000..ead399a --- /dev/null +++ b/persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md @@ -0,0 +1,6 @@ +- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base + (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2** + (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the + `` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). + **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw + abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen. diff --git a/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md b/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md new file mode 100644 index 0000000..58d4f02 --- /dev/null +++ b/persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md @@ -0,0 +1,11 @@ +- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD + Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** + Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 + KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); + (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a + DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over + RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp + template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the + live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively + opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap + b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** diff --git a/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md b/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md new file mode 100644 index 0000000..510dec4 --- /dev/null +++ b/persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md @@ -0,0 +1,12 @@ +- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** + Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path + finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied + VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before + `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does + `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers + 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the + DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED + un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO + confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% + util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. + gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. diff --git a/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md b/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md new file mode 100644 index 0000000..3025923 --- /dev/null +++ b/persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md @@ -0,0 +1,17 @@ +- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** + Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the + SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo + --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src + -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank + -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml + --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is + `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. + Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` + (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ + immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ + **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ + (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate + 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) + but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. + Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, + expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. diff --git a/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md b/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md new file mode 100644 index 0000000..ddd10c6 --- /dev/null +++ b/persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md @@ -0,0 +1,4 @@ +- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + + `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), + YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my + parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) diff --git a/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md b/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md new file mode 100644 index 0000000..4ed80d1 --- /dev/null +++ b/persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md @@ -0,0 +1,4 @@ +- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC + frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- + TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the + bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. diff --git a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md new file mode 100644 index 0000000..c7fae06 --- /dev/null +++ b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md @@ -0,0 +1,7 @@ +- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** + Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty + idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). + Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). + **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the + operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread + 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. diff --git a/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md new file mode 100644 index 0000000..f000c4e --- /dev/null +++ b/persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md @@ -0,0 +1,46 @@ +- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, + #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides + intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on + `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit + both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** + (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted + `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL + NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` + after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config + INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven + safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). + StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: + the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never + updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — + pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` + parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider + description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix + (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING + defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs + when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway + config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; + live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): + character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving + character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping + principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the + dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent + but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, + UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly + from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + + mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's + `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + + character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window + VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev + to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked + config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly + re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard + + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning + downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying + (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** + (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): + mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx + 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 + canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push + of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity + block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), + monitor armed. See [[reference_corviduo_dev_emergency_ops]]. diff --git a/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md b/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md new file mode 100644 index 0000000..4c546ac --- /dev/null +++ b/persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md @@ -0,0 +1,4 @@ +- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 + (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the + cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — + that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. diff --git a/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md b/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md new file mode 100644 index 0000000..7450511 --- /dev/null +++ b/persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md @@ -0,0 +1,10 @@ +- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into + gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, + vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack + (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); + max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and + `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture + no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone + voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). + **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not + A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). diff --git a/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md b/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md new file mode 100644 index 0000000..0dec87f --- /dev/null +++ b/persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md @@ -0,0 +1,3 @@ +- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that + spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning + (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). diff --git a/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md b/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md new file mode 100644 index 0000000..ac18f7f --- /dev/null +++ b/persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md @@ -0,0 +1,16 @@ +- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker + docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB + registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the + 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to + remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because + daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never + reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd + opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, + `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the + frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime + (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares + `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, + never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers + once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; + log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running + containers). diff --git a/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md b/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md new file mode 100644 index 0000000..024ebbc --- /dev/null +++ b/persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md @@ -0,0 +1,20 @@ +- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable + systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` + America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` + runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in + `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH + footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS + new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode + ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): + web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + + caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle + brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); + registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired + step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default + + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled + 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. + **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start + brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session + handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + + ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token + feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). diff --git a/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md b/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md new file mode 100644 index 0000000..9e903ea --- /dev/null +++ b/persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md @@ -0,0 +1,16 @@ +- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev + requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, + 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the + VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = + `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump + broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch + 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke + the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot + `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), + re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at + `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** + comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention + rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY + the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → + rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), + lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. diff --git a/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md b/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md new file mode 100644 index 0000000..4a77776 --- /dev/null +++ b/persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md @@ -0,0 +1,14 @@ +- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, + SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch + 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` + FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came + only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` + as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any + transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). + `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB + migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 + needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a + mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE + without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid + 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See + [[reference_irv_ml1_comfyui_mmartial]]. diff --git a/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md b/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md new file mode 100644 index 0000000..9f0b553 --- /dev/null +++ b/persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md @@ -0,0 +1,13 @@ +- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev + R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` + (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 + `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's + personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based + `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. + R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe + → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string + insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own + yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows + baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo + untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they + want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. diff --git a/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md b/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md new file mode 100644 index 0000000..c51ff11 --- /dev/null +++ b/persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md @@ -0,0 +1 @@ +- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. diff --git a/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md b/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md new file mode 100644 index 0000000..c32ffcc --- /dev/null +++ b/persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md @@ -0,0 +1 @@ +- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. diff --git a/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md b/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md new file mode 100644 index 0000000..406ffe6 --- /dev/null +++ b/persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md @@ -0,0 +1 @@ +- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). diff --git a/persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md b/persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md new file mode 100644 index 0000000..cca7d59 --- /dev/null +++ b/persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md @@ -0,0 +1 @@ +- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction. diff --git a/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md b/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md new file mode 100644 index 0000000..3a1efd5 --- /dev/null +++ b/persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md @@ -0,0 +1 @@ +- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). diff --git a/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md b/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md new file mode 100644 index 0000000..6a75d9f --- /dev/null +++ b/persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md @@ -0,0 +1 @@ +- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** diff --git a/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md b/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md new file mode 100644 index 0000000..dd7d71a --- /dev/null +++ b/persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md @@ -0,0 +1 @@ +- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. diff --git a/persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md b/persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md new file mode 100644 index 0000000..f434a47 --- /dev/null +++ b/persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md @@ -0,0 +1 @@ +- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance. diff --git a/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md b/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md new file mode 100644 index 0000000..54d02d3 --- /dev/null +++ b/persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md @@ -0,0 +1 @@ +- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. diff --git a/persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md b/persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md new file mode 100644 index 0000000..89a1b85 --- /dev/null +++ b/persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md @@ -0,0 +1 @@ +- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny ''` line) AND an instrumented receiver. diff --git a/persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md b/persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md new file mode 100644 index 0000000..f11b692 --- /dev/null +++ b/persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md @@ -0,0 +1 @@ +- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip. diff --git a/persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md b/persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md new file mode 100644 index 0000000..efadc05 --- /dev/null +++ b/persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md @@ -0,0 +1 @@ +- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`. diff --git a/persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md b/persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md new file mode 100644 index 0000000..433a4fe --- /dev/null +++ b/persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md @@ -0,0 +1 @@ +- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. diff --git a/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md b/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md new file mode 100644 index 0000000..6f3ca43 --- /dev/null +++ b/persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md @@ -0,0 +1 @@ +- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. diff --git a/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md b/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md new file mode 100644 index 0000000..5ac3ee2 --- /dev/null +++ b/persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md @@ -0,0 +1 @@ +- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. diff --git a/persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md b/persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md new file mode 100644 index 0000000..06e37be --- /dev/null +++ b/persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md @@ -0,0 +1 @@ +- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. diff --git a/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md b/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md new file mode 100644 index 0000000..ecd85e1 --- /dev/null +++ b/persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md @@ -0,0 +1 @@ +- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. diff --git a/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md b/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md new file mode 100644 index 0000000..2113020 --- /dev/null +++ b/persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md @@ -0,0 +1 @@ +- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. diff --git a/persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md b/persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md new file mode 100644 index 0000000..69c494c --- /dev/null +++ b/persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md @@ -0,0 +1 @@ +- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. diff --git a/persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md b/persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md new file mode 100644 index 0000000..e6885b8 --- /dev/null +++ b/persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md @@ -0,0 +1 @@ +- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement. diff --git a/persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md b/persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md new file mode 100644 index 0000000..bca801f --- /dev/null +++ b/persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md @@ -0,0 +1 @@ +- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm. diff --git a/persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md b/persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md new file mode 100644 index 0000000..8dbcd92 --- /dev/null +++ b/persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md @@ -0,0 +1 @@ +- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned. diff --git a/persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md b/persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md new file mode 100644 index 0000000..eeff991 --- /dev/null +++ b/persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md @@ -0,0 +1 @@ +- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe. diff --git a/persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md b/persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md new file mode 100644 index 0000000..a9ada3d --- /dev/null +++ b/persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md @@ -0,0 +1 @@ +- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.) diff --git a/persistent-memory.md b/persistent-memory.md index 004633f..8f60a3e 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -128,357 +128,135 @@ _As of 2026-07-15 — a long fleet-ops + infra session. Everything below either ## Recent decisions -- `[2026-07-15]` **arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake.** Operator-directed full switch (comfy-dev executed, live in prod). Established: gen (`qwen3.6-35b-a3b-heretic`) is vision-enabled and was image-bench's predecessor as arbo's hero-judge; image-judge actually serves 4 roles (vision quality-scoring + identity-scoring + bbox grounding + an uncensored text tier), not just grounding. comfy-dev spot-check: gen faster on every task, grounding within ~3px, uncensoring preserved, and it FIXED a bug (image-judge's reasoning preamble broke json_object + stalled the router). Sequencing = short prod bake then evict (~30 GB GPU1 reclaim); revert = flip `ARBO_VISION_MODEL`. Full record: auto-memory `project_arbo_gen_switch_imagebench_evict`. -- `[2026-07-15]` **esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service`** (the prior `After=remote-fs.target` drop-in was silently defeated by `nofail`). Reached only after a REBOOT (D-state phantom containers uptime-kuma + paperless-web that no `docker`/`ctr`/daemon-restart could clear). Committed `21d9a07` + playbook updated. See Tried and abandoned. +- `[2026-07-15]` arbo fully switched off image-judge (qwen-image-bench) -> gen; image-bench pending eviction post-bake → `persistent-memory.d/2026-07-15-arbo-fully-switched-off-image-judge-qwen-image.md` + +- `[2026-07-15]` esh-docker-vm NFS fstab fix = `x-systemd.before=docker.service` → `persistent-memory.d/2026-07-15-esh-docker-vm-nfs-fstab-fix-x-systemd.md` + - `[2026-07-15]` **Homepage AI-tab revamp** — flat "AI Systems" group -> dedicated AI tab, 6 role-based groups + AI-Dormant; committed `569e1af`, pushed. (Also caught + pushed a ~100-commit unpushed eshpfi backlog.) + - `[2026-07-15]` **Home Assistant config repo created** (`vh/home-assistant-config`, private). UI-managed HA -> allowlist model (YAML + curated secret-free `.storage` subset). git-in-place in `/config` on esh-docker-vm + scoped deploy key + local clone `~/development/home-assistant-config`. + - `[2026-07-15]` **char-rp-reasoning OOM rescue** — solo-restart on the packed GPU0 crash-looped; fixed via `expandable_segments:True` + util 0.39->0.38 + max-model-len 192K->150K. LESSON (Tried): `max-model-len` does NOT free vLLM VRAM (util-pinned KV pool). ~4.5 GB GPU0 headroom now. + - `[2026-07-15]` **soong-lab `SOONG_LAB_LIBRARY_DIR` made persistent** (corviduo-dev) — was on the redeploy-wiped code default; set to `/home/infra-ops/soong-lab-data/library` (mirrors PORTRAIT_DIR), restarted. Closed a queued no-rush item; unblocked the operator. + - `[2026-07-15]` **Statusline overhauled** (`~/.claude/statusline-command.sh`) — git state / 🔔🔕 monitor-armed / project tag / abs tokens / per-session cost (`.cost.total_cost_usd`) / threshold-colored ctx+rate (green<60 / yellow60-90 / red>90). -- `[2026-07-14]` **NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned.** Modelopt-format re-quant made MTP work. The load-crash root cause = **vLLM 0.24 does NOT propagate modelopt `exclude_modules` to the spec-decode DRAFT model** → the bf16 mtp head gets quantized → shape crash; NO checkpoint config fixes it (`is_layer_skipped` is exact-membership, and the drafter never sees the mtp excludes) → **workaround = a mounted `sitecustomize` that force-skips `mtp.*` in `is_layer_skipped`** (upstream vLLM bug to file). Productionized as compose stack `heretic2-charrp-reasoning` (:8018, workaround baked in). Gateway `char-rp-reasoning` alias fixed: repointed off the stale GGUF served-name `deckard-pkd-27b`, added `enable_thinking:true`, **dropped `min_p`** (MTP-incompatible), canonical samplers temp1.0/top_p0.95/top_k20. Rebalanced GPU0 (gen 0.37→0.30/16-seq/256K + reasoning 0.39/16-seq/192K+MTP + char-rp 128K, 2.7GB free). All 4 gateway roles verified; vLLM reasoning-parser confirmed **leak-free** (unlike the GGUF budget-forcing). Full record + the 4 quant landmines in `docs/runbooks/heretic2-nvfp4-mtp-seat.md`; committed `982c319`. Open (non-blocking): brokkr P00 (seat is live ahead of it), retire the stopped GGUF reasoning seat, file the vLLM bug. -- `[2026-07-14]` **NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP.** One ~40-min GPU0 window. Root-caused the `!!!!` to the quant NAMESPACE (text-only `AutoModelForCausalLM`→`model.layers.*` keys; vLLM serves only `Qwen3_5ForConditionalGeneration`, which needs `model.language_model.*`) — found from config diffs + vLLM source with ZERO GPU time; fixed by loading as `AutoModelForImageTextToText`. NVFP4 now serves COHERENT (validated greedy). BUT base NVFP4 ≈53 tok/s ≈ GGUF's 59.5 at batch-1 (no single-stream win) AND MTP = 0% acceptance on compressed-tensors (bf16 mtp head only loads on the modelopt format). Operator chose to **pursue a modelopt-format re-quant** (the only path to the 2-4× MTP goal; AEON-proven on this exact Qwen3.6-27B arch). Scoped + de-risked: AEON `/tank/aimodels/qwen36-27b-aeon-nvfp4` = the modelopt reference (quant_method modelopt, 1967 tensors, 15 bf16 mtp keys identical to graft); nvidia-modelopt 0.45.0 installs + `mtq.quantize`/`NVFP4_DEFAULT_CFG`/`export_hf_checkpoint` API confirmed; pipeline unchanged except swap llm-compressor→modelopt. Seats restored; char-rp-reasoning stays GGUF. Full plan in Current state ★ section. -- `[2026-07-14]` **Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion** (Vuong-directed via /snapshot: "chase the nvfp4 quant, we know it works, write down the recipe"). Full recipe + diagnostic ladder in Current state / in-flight above. Artifacts on ana-ml2 `/tank/aimodels/heretic2-nvfp4-work/` + scripts committed in eshpfi `services/heretic2-nvfp4-quant/`. -- `[2026-07-14]` **char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B)** — R36 gate PASSED (tools 0.967, #355 runaway ELIMINATED). #355 was MODEL-level (Deckard emitted qwen3_coder XML malformed → mangled args → retry-runaway), NOT the reasoning-budget bug; NEO-CODE emits it clean. Custom llama.cpp KEPT (qwen3_coder parse — stock b8840 predates it — + PR#25544). Committed f960a73; full record auto-memory [[charrp-custom-llamacpp-pr25544]]. -- `[2026-07-14]` **soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST`** (was `external, 10.100.0.0/16` = NH3-only; blocked corviduo-dev's Anaheim `10.250.x` → gitea refused to deliver, never opened the connection). Fixed to fleet-wide `10.0.0.0/8` (app.ini `[webhook]`) + gitea restart; listener now logs every delivery. The ufw `10/8` open (also this session) was a real-but-secondary gap. Committed 462d528. -- `[2026-07-13]` **#355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN).** char-rp-reasoning enters a non-terminating REASONING loop (tool-call-retry planning) and runs to `max_tokens=32768` (~22 min @ 24.7 tok/s, ~13% of requests); the seat GENERATES all 32768 tokens (not idle), and `--reasoning-budget 400` is NOT enforced. 3-source-confirmed (spend_logs completion_tokens=32768 ×4; seat eval-time log; pcap 100%-`reasoning_content` deltas). Server-side fix wanted (operator: no max_tokens ceiling) → routed to brokkr (accepted, pulled dvalin). Lesson (again): confirm before concluding — the seat-idle claim came from reading only the ≤73s requests + missing the concurrent 32768-token slots. See ACTIVE 1. +- `[2026-07-14]` NVFP4+MTP fast char-rp-reasoning seat LANDED + LIVE + gateway-repointed + VRAM-tuned → `persistent-memory.d/2026-07-14-nvfp4-mtp-fast-char-rp-reasoning-seat-landed.md` -- `[2026-07-13]` **Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong).** Buildx step log: `chown -R /app` = 251s (copy-up of the root-owned venv into a fresh layer), uv sync only 35.6s, registry layer cache already wired. Fix = drop `/app` from the chown (validated safe: zero /app runtime writes on both live instances) + uv cache-mount. Shipped as PR #359 (branch off origin/main@b60), worldtree-dev green-lit. Expected ~5min off (~11→~6min). Runner-side BuildKit cache task (b) was already done → moot. +- `[2026-07-14]` NVFP4 quant chase RESOLVED (gibberish) + PIVOTED to modelopt for MTP → `persistent-memory.d/2026-07-14-nvfp4-quant-chase-resolved-gibberish-pivoted-to-modelopt.md` -- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONERATED (completes ≤72s; `--reasoning-budget 400` forecloses a mid-thinking hang). Corrects worldtree-dev's "seat wedging" diagnosis. Decisive next = the FIN-check (pcap on corviuo). See in-flight ACTIVE 1. **[SUPERSEDED 2026-07-13 — see the ROOT CAUSE entry above; the gateway-hold/slot-leak theory was disproven, the seat was generating 32768 tokens.]** +- `[2026-07-14]` Pursue the NVFP4+MTP fast char-rp-reasoning seat to completion → `persistent-memory.d/2026-07-14-pursue-the-nvfp4-mtp-fast-char-rp-reasoning.md` -- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 2061's 16-min no-terminal). worldtree-dev filed follow-ons #356 (rehydrate Tier-3 ctx on resume — the recreate-durability gap), #357 (reclaim orphaned active-turn locks), #358 (LLM-provider read-timeout audit); surfacing to Vuong to prioritize. +- `[2026-07-14]` char-rp-reasoning seat: Deckard-PKD → NEO-CODE = Heretic2-Thinking (Qwen3.6-27B) → `persistent-memory.d/2026-07-14-char-rp-reasoning-seat-deckard-pkd-neo-code.md` -- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync ... --no-cache` + no BuildKit cache-mount (re-downloads all deps cold every build). Fix split: worldtree-dev Dockerfile cache-mount diff + infra-ops runner-side persistent BuildKit cache. Config-only changes skip the build entirely (pinned recreate). +- `[2026-07-14]` soong-lab webhook auto-deploy real root cause = gitea `webhook.ALLOWED_HOST_LIST` → `persistent-memory.d/2026-07-14-soong-lab-webhook-auto-deploy-real-root-cause.md` -- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); `assistant`+`thoughtful-assistant` capability roles added (gen/gen-reasoning) on personal+demo, canonical d8bd497. Chosen instance = personal (the tier-3-consumer instance, ratatoskr+soong-lab colocated). +- `[2026-07-13]` #355-residual ROOT CAUSE (supersedes the "LiteLLM gateway holds while seat idles" entry below — that was DISPROVEN) → `persistent-memory.d/2026-07-13-355-residual-root-cause-supersedes-the-litellm-gateway.md` -- `[2026-07-10]` **Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev - R34-v1 request).** Added allow-rule `ratatoskr-affect-full-allow` to `/opt/worldtree-personal/config/policies.yaml` - (`principal.user_ids:["ratatoskr"]`, action `affect.full`, resource `*`, effect allow), mirroring the #347 - `session-history-write-ratatoskr` rule exactly + placed right after it. **WHY user_ids-based (not tier):** ratatoskr's - personal key is the minimal **readonly-admin** observability tier, which is NOT in the tier-based - `affect-render-baseline-allow` (anonymous/user/free/pro/admin) → needs an explicit user_id grant, same as #347. - R34-v1 (b46, committed UNPUSHED) gates `affect.emit` `dominant_emotion` egress by exposure ceiling (affect.full|safe - → present; neither → null); this grant keeps ratatoskr's view alive across the b46 deploy. Surgical exact-string - insert (preserves comments), backup `policies.yaml.bak-pre-affectfull-20260710`, validated via the CONTAINER's own - yaml parser (35 rules, +1, payload confirmed). **NOT restarted — deliberate:** rule is on the bind-mount (shadows - baked), INERT until b46 gating ships, so the b46 CI/CD deploy restart activates it (no live-session blip now). Demo - untouched (personal-only per key scope). Replied to wt-dev (thread 01KX6DB3…) offering an immediate restart if they - want it live for pre-b46 testing. See [[reference_corviduo_dev_emergency_ops]]. +- `[2026-07-13]` Deploy-speed real bottleneck ≠ uv sync (memory's assumption was wrong) → `persistent-memory.d/2026-07-13-deploy-speed-real-bottleneck-uv-sync-memory-s.md` -- `[2026-07-10]` **ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, - SageAttention preserved, crash-loop AVOIDED.** The prior attempt (entry below) crash-looped because a torch - 2.12.1→2.13 bump broke SageAttention's ABI. This time I checked `git diff v0.24.1 v0.27.1 -- requirements.txt` - FIRST and found **core v0.27.1 leaves `torch` UNPINNED** → the version bump does NOT require torch 2.13 (that came - only from the mmartial boot-upgrade). So: `git checkout v0.27.1` (clean tree) → `pip install -r requirements.txt` - as **uid 1000** with a **torch-pin constraint file** (torch/vision/audio pinned to current +cu129) to block any - transitive bump → torch stayed 2.12.1, SageAttention 2.2.0 untouched. Added decord 0.6.0 (fixed SAM3Segment). - `docker restart comfyui` → healthy, `/system_stats` comfyui_version=0.27.1, "Using sage attention", HTTP 200, DB - migrated 0003→0004. Reported the divergence to comfy-dev (thread 01KX6D3C…, reply pending) + asked whether LTXV 2.3 - needs a separate torch-2.13 follow-up (their domain; Krea-2's ≥0.25 need is met by 0.27.1). **LESSON: before a - mmartial ComfyUI version bump, `git diff -- requirements.txt` — if torch is unpinned, bump the CODE - without touching torch (constraint-pin it) and compiled exts (SageAttention) survive. `docker exec` lands as uid - 1025(comfytoo), not 1000 — use `-u 1000` + the venv python `/comfy/mnt/venv/bin/python`.** See - [[reference_irv_ml1_comfyui_mmartial]]. +- `[2026-07-13]` WT #355 residual 300s hang localized to OUR LiteLLM gateway (holds 2 char-rp-reasoning requests ~21 min while the seat idles), NOT the seat — Deckard seat EXONE → `persistent-memory.d/2026-07-13-wt-355-residual-300s-hang-localized-to-our.md` -- `[2026-07-10]` **ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it).** comfy-dev - requested (Vuong-authorized) bumping the irv-ml1 `comfyui` stack (mmartial image, `/opt/docker/compose/comfyui/`, - 0.24.1) to 0.25.x for Krea-2 + LTXV 2.3. **TWO FINDINGS: (1) `DISABLE_UPGRADES=false`/USE_PIPUPGRADE bumps the - VENV (torch 2.12.1→2.13.0 + deps) but does NOT advance the ComfyUI CODE checkout** (`/comfy/mnt/ComfyUI` = - `/worktank/comfyui/run/ComfyUI` stayed 0.24.1 — pinned/detached git, comfy-dev's domain). **(2) the torch bump - broke SageAttention** (2.2.0 `_fused.so` undefined-symbol `c10::impl::cow::materialize_cow_storage` vs torch - 2.13.0) → `--use-sage-attention` (REQUIRED launch flag in COMFY_CMDLINE_EXTRA) crash-looped ComfyUI. Net: broke - the working state, zero 0.25.x payoff. **ROLLBACK WORKED**: pre-bump 16G venv snapshot - `/worktank/comfyui/venv-snapshot-comfyui-0.24.1-20260710.tar` restored (torch 2.12.1 + working SageAttention), - re-pinned DISABLE_UPGRADES=true, recreated → healthy on 0.24.1, serving :8188. Broken venv parked at - `/worktank/comfyui/run/venv.broken-torch213-20260710`. **CORRECTED PATH (sent comfy-dev, thread 01KX655V):** - comfy-dev git-advances the ComfyUI checkout to 0.25.x + reqs → I handle the torch bump + SageAttention - rebuild-against-2.13.0 + re-pin (snapshot stays as the net). **LESSON: mmartial `DISABLE_UPGRADES` gates ONLY - the venv pip-upgrades, NOT the ComfyUI git checkout; a torch bump breaks compiled exts (SageAttention) → - rebuild-after is mandatory.** Bump BLOCKED pending comfy-dev's git-advance. Stack: A6000 (NVIDIA_VISIBLE_DEVICES=1), - lkraven-owned compose+venv (uid 1000, no sudo needed), COMFY_CMDLINE_EXTRA OOM flags preserved. +- `[2026-07-13]` WT #355 turn-lifecycle fix VALIDATED on worldtree b60 — wedged turns self-terminate cancelled/stalled at the 300s stall-watchdog (turns 2064/2065 vs pre-b60 206 → `persistent-memory.d/2026-07-13-wt-355-turn-lifecycle-fix-validated-on-worldtree.md` -- `[2026-07-10]` **Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized).** Durable - systemd **--user** timer on nh3-dev (`brokkr-landscape-scan.timer`, OnCalendar `*-*-01,15 09:00:00` - America/Los_Angeles, Persistent=true; linger on) → `.service` → wrapper `~/.local/bin/brokkr-landscape-scan.sh` - runs headless `claude -p "$(cat ~/.config/brokkr-landscape-scan/prompt.txt)" --dangerously-skip-permissions` in - `~/development/brokkr-smithy` (ALTHING_HANDLE=brokkr-smithy-dev; **explicit PATH** — the --user minimal-PATH - footgun; per-run logs `~/.local/state/brokkr-landscape-scan/`). Prompt = brokkr's payload verbatim (LLM/image/TTS - new-release sweep → ranked synthesis → commit+push+notify). VALIDATED: git-push non-interactive (BatchMode - ls-remote to gitea, passphraseless key — no agent), headless claude auth (READY smoke). VALIDATED END-TO-END 2026-07-10 (manual run, exit 0): - web-sweep→synthesis→commit `2ed2f29`→PUSH of scan #2 (open-weight-releases-2026-07-24.md); triaged dwarf input + - caught baseline errors, quality strong. **HANDLE-COLLISION caught+FIXED** — the headless scan shared handle - brokkr-smithy-dev with the LIVE session + raced its inbox (eitri's dwarf-reply got stolen by the live monitor); - registered a dedicated **brokkr-scan-dev** handle (`add-handle`, driver=none) + repointed the wrapper + rewired - step-5 notify → `althing-cli post --to brokkr-smithy-dev` (NO vuong althing handle exists — confirmed). model=default - + `--max-turns 80`. First run under the new handle = 7/15. Off-cycle 07-24 doc is a validation artifact (scheduled - 1st/15th runs date to their own run-date, no collision) — operator naming-convention call pending. - **NEXT AUTO-RUN 2026-07-15 09:00 PDT.** Manual validation/first run = `systemctl --user start - brokkr-landscape-scan.service`. Open w/ brokkr (thread 01KX63G6): confirm notify-Vuong handle/mechanism + session - handle + model/turn-cap. **NEXT brokkr task (operator-sequenced after this): TTS audition env** — Higgs-TTS-3 + - ZONOS2 + Chatterbox baseline, TTFA/RTF + blind-A/B web-listen (thread 01KX6371; needs GPU-placement + HF-token - feasibility pass first; brokkr delivers the prompt set after the env's up; protocol doc in brokkr-smithy repo). +- `[2026-07-13]` Worldtree deploy bottleneck = the image build (~11 min of a ~12 min deploy), root cause the Dockerfile `uv sync → `persistent-memory.d/2026-07-13-worldtree-deploy-bottleneck-the-image-build-11-min.md` -- `[2026-07-09]` **Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker - docker log-cap SOLVED no-bounce.** (1) **Phantom**: absent from `/v1/models` + `/model/info` (config+DB - registry), zero litellm log refs — the parked "400s in /v1/models" note was STALE (already cleaned in the - 2026-07-08 gen repoint to `-heretic`); bare token survives only in 2 config COMMENTS (lines 76/80). Nothing to - remove. (2) **Log-cap**: running containers were UNCAPPED (182M json-logs, top offender 59M) because - daemon.json's `max-size 10m/max-file 3` only applies to containers CREATED AFTER a daemon restart — it never - reaches already-running ones. No-bounce fix = `/etc/logrotate.d/docker-containers` (**copytruncate** — dockerd - opens json-logs `O_APPEND` so truncate-in-place resets cleanly, no sparse-file corruption; `size 10M`, - `rotate 3`, `compress`, `su root root`), auto-picked-up by the daily `logrotate.timer`. Force-ran + gzipped the - frozen `.1` archives → **182M → ~55M** (44M active + 11M gz), every container kept its multi-week uptime - (zero bounce, verified). **LATENT FOOTGUN FLAGGED (not yet fixed, operator's call): daemon.json declares - `live-restore:true` but the RUNNING daemon has it FALSE** (daemon.json was edited after the last daemon start, - never reloaded) → the NEXT `systemctl restart docker` / crash / pkg-upgrade **bounces ALL ana-docker containers - once**. Fix WITHOUT a bounce = `systemctl reload docker` (SIGHUP loads live-restore into the running daemon; - log-opts are NOT SIGHUP-reloadable, which is why logrotate — not the daemon cap — is the enforcer for running - containers). +- `[2026-07-13]` Ledger tier-3 consumer `ledger:miranda` provisioned on personal :8081 (key b38932f5, GPG-delivered+shredded, allowlist 10.100.10.50:8770 live); → `persistent-memory.d/2026-07-13-ledger-tier-3-consumer-ledger-miranda-provisioned-on.md` -- `[2026-07-09]` **granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, - #335 Slice-4).** Changed `model_roles.yaml` memory_extractor `binds.catalog_id` `summarizer`→`gen` (overrides - intact: thinking:false/temp0/8192) on BOTH `/opt/worldtree{,-personal}/config`; `memory_distiller` left on - `summarizer` (range-scoped sed `/memory_extractor:/,/memory_distiller:/` — the naive global replace would've hit - both); backups `*.bak-preqgen-20260709`; validated via each container's OWN yaml parser. **DEMO LIVE on gen** - (b43 `d501e516732d` auto-deployed mid-edit + its restart RACED my edit by ~2min → I restarted - `worldtree-worldtree-api-1` to activate; healthy, live process resolves memory_extractor=gen). **PERSONAL - NOW LIVE on gen too** — Vuong authorized the restart (via wt-dev); restarted `worldtree-personal-worldtree-api-1` - after a PRE-FLIGHT that ran the app's OWN `load_model_roles()` (`core/llm/roles.py:121`) against the synced config - INSIDE the running `c9986cd` container: `gen` in catalog, all 9 roles resolve, no `DanglingBindingError` → proven - safe on the OLDER image BEFORE touching it (model_roles-delta-alone clean; no full-config-set sync needed). - StartedAt 20:50:55Z, healthy, resolves gen. **BOTH instances live on gen.** **LESSON: - the bind-mount `/opt/worldtree*/config` SHADOWS the baked `/app/config-defaults/` → the deploy alone never - updates the ACTIVE config; the host edit is required AND a restart activates it (role registry cached at boot) — - pre-stage BEFORE the deploy's restart or you race it.** FOLLOW-UPS (non-blocking): (a) `memory.extractor.user_pass` - parity block → self-serve from the b43 baked `defaults.yaml` (pydantic-default no-op); (b) stale `gen` provider - description (Mistral-Small-4 → qwen3.6-35b-a3b-heretic) → wt-dev owns the REPO-side `providers.yaml` fix - (operator's call — NOT purely cosmetic: the gen/dialogue + classifier entries carry Mistral-Small-4 SAMPLING - defaults that drive mask/lofn/forseti/mimir dialogue, so wt-dev re-validates for qwen), host cosmetic sync pairs - when it deploys. Gave wt-dev the VERIFIED canonical 4-alias set (backends+samplers read from the LIVE gateway - config, not the doc); corrected `docs/pfi/model-sampler-defaults.md` seat 4 (had lagged QwQ-RpR-v4 → Deckard-PKD; - live gateway was always Deckard). Operator SCOPED IN the character-RP re-point (2026-07-09): - character→char-rp / thoughtful-character→char-rp-reasoning (character-rp per wt-dev's role semantics), moving - character RP off the GENERAL qwen onto the dedicated Magidonia/Deckard seats. Relayed to wt-dev w/ the mapping - principle + a SAMPLER-OVERRIDE warning (DROP character-rp's old temp0.75/top_p0.85 overrides — carried onto the - dedicated seats they'd clobber the canonical RP tuning DOWNWARD) + ratatoskr-reach note (role call is transparent - but Magidonia/Deckard quality/latency differs from gen). DONE 2026-07-09: wt-dev committed 5d4fa4a (v1.0.0b44, - UNPUSHED — operator drives push); operator directed host-ahead-of-push, so I sourced BOTH config files directly - from that unpushed commit (local `~/development/Worldtree` checkout — capital W; `git show 5d4fa4a:config/…`) + - mirrored VERBATIM to `/opt/worldtree{,-personal}/config` on demo+personal, paired-pre-flighted via the app's - `load_model_roles()` (no DanglingBinding), restarted both → LIVE: character→char-rp, thoughtful-character + - character-rp→char-rp-reasoning, memory_extractor→gen preserved. Backups `*.bak-prerp-20260709`. context_window - VERIFIED (llama.cpp /props + char-rp-gguf `.env`): char-rp **131072**, char-rp-reasoning **262144** (gave wt-dev - to patch the repo from its interim 32768). **HOST AHEAD of repo-remote until the operator pushes 5d4fa4a** (baked - config converges with the bind-mount on push+deploy). GOTCHA: demo≠personal — PERSONAL was already partly - re-pointed (2026-07-06 AEON-era character→char-rp) so its delta was mostly stale-AEON-descriptions→Magidonia/Deckard - + character-rp + stripping personal's char-rp `default_params` temp0.7/top_p0.8 that CLOBBERED the gateway RP tuning - downward; DEMO had no char-rp catalog entries at all (b44 adds them). Diffed each instance vs b44 before applying - (both deltas = expected changeset only, nothing instance-specific clobbered). wt-dev PATCHED the context_window in **b45/3384a37** - (char-rp 131072, char-rp-reasoning 262144). BUMPED HOST-AHEAD on both instances (operator-directed 2026-07-09): - mirrored b45's providers.yaml → `/opt/worldtree{,-personal}/config`, restarted, verified LIVE (char-rp ctx - 131072, char-rp-reasoning 262144, bindings intact); backups `.bak-b44interim-20260709`. HOST now = **b45 - canonical** (providers.yaml) + b44 (model_roles unchanged b44→b45). STILL PENDING: (1) operator's batched push - of **b44+b45** (`5d4fa4a`+`3384a37`) to converge the repo-remote — host is ahead, no fork; (2) user_pass parity - block (defaults.yaml — NOT in either, separate). Threads `01KX3SGH`/`01KX48QP` (worldtree-dev), - monitor armed. See [[reference_corviduo_dev_emergency_ops]]. +- `[2026-07-10]` Heimdall grant: ratatoskr `affect.full` on PERSONAL Worldtree (operator-approved, worldtree-dev R34-v1 request) → `persistent-memory.d/2026-07-10-heimdall-grant-ratatoskr-affect-full-on-personal-worldtree.md` -- `[2026-07-09]` **mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into - gateway-chat.** Full arc (commits): gen served-name honesty rename aeon→qwen3.6-35b-a3b-heretic (99a4a17, - vLLM served-name + litellm refs, so /v1/models + spend-logs name the real model); permanent 2-container stack - (01eedd8); gateway-chat auto-voice quoted dialogue (c948013); streaming decode TTFA 4.5s→0.8s (da76829); - max_tokens 1200→2400→3500 with a context-clamp (f363fe6, 0655a37 — long lines were clipping at 14.6s, and - `repetition_penalty` 1.1 is LOAD-BEARING: at 1.0 the model never stops); AudioContext resume-on-gesture - no-sound fix (033f368); pre-chunk by QUOTED SECTION not sentence for prosody (a1f3023→f295cc1); staged clone - voices baddy/beatrice/whisper (0655a37 + runtime .wav/.txt in the voices dir); agent voicing prompt (a573514). - **Load-bearing config, all encoded in stacks/mOrpheus/: bf16 not FP8, image v0.23.0 not latest, GPU=3090 not - A6000, rep_penalty 1.1.** Serving-viability confirmed: vLLM concurrency near-linear to 8× (707 tok/s). +- `[2026-07-10]` ComfyUI v0.27.1 SUCCESS on irv-ml1 (operator-confirmed execute-now) — landed on torch 2.12.1, SageAttention preserved, crash-loop AVOIDED → `persistent-memory.d/2026-07-10-comfyui-v0-27-1-success-on-irv-ml1.md` -- `[2026-07-09]` **granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4).** - Answered their VRAM/concurrency headroom check: gen (qwen 35B-A3B heretic) has ample headroom for ~2 bursty - idle-triggered extractor calls (fixed 0.37 util; KV pool runs 0-2%; --max-num-seqs 16, near-linear batching). - Corrected their stale "gen = Mistral Small 4 / 119B-6B" belief (gen IS the qwen 35B-A3B heretic since 2026-07-08). - **This bind is INDEPENDENT of the full granite RETIRE** (reclaim ~32GB on ana-ml2 GPU1) — that stays the - operator's call, pending brokkr R33 portfolio + production-concurrency due-diligence. Tracked: althing thread - 01KX3SGH… (worldtree-dev) + brokkr's gen-absorbs-granite consult (thread 01KX2V32…) + [[reference_litellm_gateway]]. +- `[2026-07-10]` ComfyUI 0.25.x bump on irv-ml1 ATTEMPTED → FAILED → ROLLED BACK (snapshot saved it) → `persistent-memory.d/2026-07-10-comfyui-0-25-x-bump-on-irv-ml1.md` -- `[2026-07-08]` **RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD - Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (35B-A3B, util 0.37), ~4G GPU0 margin.** - Arc: (1) replaced broken Angel NVFP4 with Magidonia prose + QwQ-RpR-v4 reasoning (b268f93); (2) max-context via q8_0 - KV (f570604); (3) canonical samplers for all 4 gateway seats, dvalin-derived + char-rp A/B-tuned (aac4bcf); - (4) rebalanced gen 0.40→0.37 to fund char-rp 128K (f49c4e4); (5) RE-A/B'd the reasoning seat (operator wanted a - DRY-tolerant model): **Deckard WON** on brokkr's frozen scorer (composite 2.176, 0/30 loops, 0/30 refusals) over - RpR-v4 (3.716, 1/30 loop), Pantheon-Reasoning (1.383 but 7/30 refusals), Snowdrop+Gembrain (llama.cpp - template-incompat) — deployed (5f79b40); (6) Deckard→256K (41305bf); (7) dvalin CONFIRMED Deckard samplers = the - live A/B set is canonical (4954ca0). **GATE LESSON: a llama.cpp reasoning seat needs a STOCK template that natively - opens ``/`enable_thinking` (Qwen3.x/QwQ pass; ChatML + Gemma-4 fail) — no monkeypatching. INFRA: llama-swap - b8840 can't load Qwen3.6/Gemma-4 archs → `ghcr.io/ggml-org/llama.cpp:server-cuda` (pulled on ana-ml2).** +- `[2026-07-10]` Biweekly open-weight-releases scan cron set up for brokkr-smithy (Vuong-authorized) → `persistent-memory.d/2026-07-10-biweekly-open-weight-releases-scan-cron-set-up.md` -- `[2026-07-08]` **worldtree Mimir deploy-blocker resolved (mid-session):** synced `persona.envelopes.assistant` + - `envelope_grants:[]` verbatim from the baked canonical into BOTH corviduo-dev instances (demo+personal), - YAML-validated via each container's own parser; worldtree-dev cleared to push the Mimir-bound image. (Was my - parked R32 1C envelope-mirror come due — see [[reference_corviduo_dev_emergency_ops]] config-sync recipe.) +- `[2026-07-09]` Two parked items closed: phantom `qwen3.6-35b-a3b` alias VERIFIED already-gone; ana-docker docker log-cap SOLVED no-bounce → `persistent-memory.d/2026-07-09-two-parked-items-closed-phantom-qwen3-6-35b.md` -- `[2026-07-08]` **OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training.** - Final topology: **gen = `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-NVFP4-Experts-Only`** (LIVE, modelopt, vision, - util 0.40), **char-rp = an RP unicorn to be found on fresh context** (see Current state). Intermediate steps - ABANDONED: Pantheon-Reasoning-27B (served briefly as gen — refuses dark fiction via DeepSeek-distilled - refusal-reasoning, see Tried); Pantheon-27B-with-MTP for RP (bf16 MTP won't load on the compressed-tensors path); - Angel MS3.2-24B (my NVFP4 quant = garbage). Prefer EXISTING community NVFP4/GGUF quants over self-quanting - ("don't quant unless you have to" — operator). GGUF serving is now on the table for RP (NEVER Ollama). Gateway - sampling-defaults wiring still PENDING. +- `[2026-07-09]` granite→gen `memory_extractor` bind host-synced on demo+personal Worldtree (Vuong-directed, #335 Slice-4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-host-synced-on.md` -- `[2026-07-08]` **DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 - epochs (operator call).** Root cause: `DpoConfig` had NO `epochs` field + `_dpo_config_kwargs` didn't pass - `num_train_epochs` → DPO fell through to trl DPOConfig's default 3.0 (SFT correctly pins 1 via SftConfig.epochs - + _sft_config_kwargs). Objective SATURATED by ~epoch 0.27 (loss~0, grad~0, acc 1.0, margins~27 flat — the - off-policy qwopus rejected pairs are trivially separable), so epochs ~0.3→3 were pure over-optimization + a - ~6.5h outage. No mid-run checkpoint (save_steps 500 > total steps; save only at end) → killing lost the run. - FIX (3 edits to deployed harness, mtf-dev to canonicalize): `DpoConfig.epochs: float = 1` (mirrors SftConfig, - float for fractions); `_dpo_config_kwargs` now passes `"num_train_epochs": cfg.epochs`; recipe `dpo.epochs: 0.3`. - GPU-free verified (dpo.epochs=0.3 → num_train_epochs=0.3). Relaunched at 0.3 epoch (~30min precompute + ~12min - train = ~45min). **DONE + SANITY-CHECKED (exit 0, ~70min wall: ~30min fixed precompute + 45 steps @ ~51s/step; - train_loss 1.4e-5 @ epoch 0.301).** Fresh `data/spike/dpo_adapter/adapter_model.safetensors` (123MB) + checkpoint-45 - banked. **3-way greedy sanity (base vs SFT vs DPO, via peft load + disable_adapter/set_adapter on GPU0): - ALL THREE DISTINCT** (base≠sft≠dpo) → full SFT→DPO pipeline applies end-to-end at inference. **DPO 0.3ep is - COHERENT, fluent, NOT degenerate** (early-stop avoided over-optimization) but the quality delta on a neutral - literary prompt is SUBTLE (DPO shares SFT's structure — it continues from it — with minor stylistic drift, - arguably slightly MORE genre-clichéd). Verdict: mechanics proven, quality gain modest as predicted for 0.3ep - on off-policy pairs; the real unlock remains on-policy rejected regen + on-domain (explicit E-RP) eval + - the LitBench/holdout run. gen+rp RESTORED healthy. Next: serve fork (SGLang-finish vs merge) on the DPO - adapter — same rekey_lora_for_vllm.py (zero-z) applies unchanged (mtf-dev confirmed). +- `[2026-07-09]` mOrpheus TTS off-the-shelf voice pipeline SHIPPED end-to-end (irv-ml1) + wired into gateway-chat → `persistent-memory.d/2026-07-09-morpheus-tts-off-the-shelf-voice-pipeline-shipped.md` -- `[2026-07-08]` **T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py.** - Blocker resolved: (1) **mtf-dev's v0.0.42 stub** `_stub_missing_optional_integrations` (last-resort sys.meta_path - finder → missing mergekit/llm_blender/weave resolve to MagicMock, never called → zero numerics risk; applied - VERBATIM to deployed `src/model_training_forge/train/backend.py` after `_unsloth_available()` + call-site before - `from trl import DPOTrainer`); (2) **my cosmetic `warnings_issued` shim** (trl-0.24 DPOTrainer.__init__:405 does - `model.warnings_issued["estimate_tokens"]=True` for warning-suppression; custom Qwen3_5 class under transformers - 5.5.0 lacks the attr → `if not hasattr(model,"warnings_issued"): model.warnings_issued={}` before the - DPOTrainer(...).train() at backend.py:305 — cosmetic, zero training impact). Both edits are on the DEPLOYED - un-git'd copy only → **mtf-dev must canonicalize the warnings_issued shim into their repo** (told them). DPO - confirmed training: model loaded (851 shards), full 1196 pairs processed, in precompute_ref_log_probs (GPU0 93% - util, 54.8GB). Completion watcher armed (bg task) → restore gen+rp + verify dpo_adapter + ping mtf-dev on exit. - gen+rp STOPPED for the run (authorized window). Output → data/spike/dpo_adapter. +- `[2026-07-09]` granite→gen memory_extractor bind GREEN-lit for worldtree-dev (Worldtree #335 Slice 4) → `persistent-memory.d/2026-07-09-granite-gen-memory-extractor-bind-green-lit-for.md` -- `[2026-07-08]` **T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe.** - Operator authorized the full DPO stage (via mtf-dev) + went AFK 2h. **PROVEN LAUNCH RECIPE** (replicates the - SFT container `aeon-t1-sft` exactly, only `--stage sft`→`dpo`): `sudo docker run -d --name aeon-t1-dpo - --entrypoint python3 --gpus all -e CUDA_VISIBLE_DEVICES=0 -e MTF_FORCE_TRL=1 -e PYTHONPATH=/mtf/src - -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v /home/lkraven/model-training-forge:/mtf -v /tank:/tank - -w /mtf aeon-trainer:latest -u scripts/train.py --recipe recipes/training/qwen-3.5-122b-erp-lora/train.aeon-27b.yaml - --stage dpo`. **CRITICAL: `--entrypoint python3` is REQUIRED** — aeon-trainer's default entrypoint is - `["vllm","serve"]` (FROM vllm/vllm-openai) → without the override it runs vllm + hits a torch-ABI crash. - Dataset verified (pairs_dataset=train.flat.json=1196 pairs). **THE BLOCK:** `from trl import DPOTrainer` - (backend.py:256) eagerly pulls TRL 0.24.0's WHOLE optional-integration set — cascade: mergekit(missing)→ - immutables→**mergekit-0.1.4↔pydantic-2.13 HARD incompat** (needs pydantic==2.10.6)→llm_blender→dataclasses_json→ - **llm_blender-0.0.2↔transformers-5.5.0 HARD incompat** (TRANSFORMERS_CACHE removed, needs source patch)→weave→ - (more). NONE used by our pair-based DPO. `pip install mergekit` w/deps is UNSAFE (downgrades accelerate - 1.14→1.6). Safe partial recipe derived (core libs held: torch2.10/tf5.5.0/trl0.24.0/peft0.19.1/accel1.14.0) - but non-convergent → TRULY BLOCKING per operator's carve-out. Did NOT force-hack the proven training image. - Handed full diagnosis + recommended fix (lazy-import TRL patch, opt b) to mtf-dev (thread 01KWZG8GJX, - expects-reply, monitor armed). gen+rp RESTORED healthy. Relaunch = 1 min once mtf-dev delivers a working image. +- `[2026-07-08]` RP-SEAT CAMPAIGN CLOSED — char-rp = Magidonia-24B-v4.3 (128K), char-rp-reasoning = Deckard-PKD Qwen3.5-27B (256K); both GGUF/llama.cpp on ana-ml2 GPU0 alongside gen (3… → `persistent-memory.d/2026-07-08-rp-seat-campaign-closed-char-rp-magidonia-24b.md` + +- `[2026-07-08]` worldtree Mimir deploy-blocker resolved (mid-session): → `persistent-memory.d/2026-07-08-worldtree-mimir-deploy-blocker-resolved-mid-session.md` + +- `[2026-07-08]` OFF-THE-SHELF INFERENCE PIVOT executed — serve curated abliterated models, stop home-training → `persistent-memory.d/2026-07-08-off-the-shelf-inference-pivot-executed-serve-curated.md` + +- `[2026-07-08]` DPO was silently running 3 epochs (harness gap) → KILLED at epoch 1.2, retargeted to 0.3 epochs (operator call) → `persistent-memory.d/2026-07-08-dpo-was-silently-running-3-epochs-harness-gap.md` + +- `[2026-07-08]` T1 DPO leg is RUNNING (unblocked) — 2 fixes applied to deployed backend.py → `persistent-memory.d/2026-07-08-t1-dpo-leg-is-running-unblocked-2-fixes.md` + +- `[2026-07-08]` T1 DPO leg launch — prior BLOCK (now resolved above), kept for the launch recipe → `persistent-memory.d/2026-07-08-t1-dpo-leg-launch-prior-block-now-resolved.md` _142 older entries archived to archival-memory.md._ ## Tried and abandoned -- `[2026-07-15]` **`docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts** — `nofail` drops a mount out of remote-fs.target's blocking set, so the drop-in ordering is silently defeated (paperless still Exited(255) on reboot). Real fix = DIRECT mount->docker ordering via the fstab `x-systemd.before=docker.service` option (verify `systemctl show docker -p After` lists the mnt-*.mount units). esh-docker-vm. -- `[2026-07-15]` **The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT** — reconfirmed: `docker stop/rm -f`, `ctr -n moby task delete`, AND `systemctl restart docker` all fail to clear it; `docker exec` into a wedged container ALSO fails (`setns ... exit status 1`), so the in-place restart escape hatch is out. Worse, a daemon restart can HALF-KILL other healthy containers (knocked paperless's granian down + left it wedged). Process dead but dockerd won't reap -> phantom. NFS mounts are `_netdev,nofail` so the reboot is boot-safe. -- `[2026-07-15]` **vLLM `max-model-len` does NOT free GPU VRAM** — the KV cache POOL is sized by `gpu-memory-utilization`, not max-model-len. Lowering max-model-len only caps per-request context + drops max concurrency; the pool still fills the util budget. To actually free VRAM, lower `gpu-memory-utilization`. (Bit the char-rp-reasoning "drop KV to 150K" ask: the 150K applied but freed 0 VRAM until util dropped 0.39->0.38.) -- `[2026-07-15]` **Claude Code statusline `.cost.total_cost_usd` is per-SESSION** (Claude Code's own cache/model-aware session accounting), not a lifetime aggregate — the large value just reflects a long, multiple-times-summarized session. And the old statusline hardcoded Sonnet pricing ($3/$15) on an Opus session -> ~5x cost understatement. +- `[2026-07-15]` `docker.service After=remote-fs.target` does NOT wait for `nofail` NFS mounts → `persistent-memory.d/2026-07-15-docker-service-after-remote-fs-target-does-not.md` + +- `[2026-07-15]` The esh-docker-vm D-state/phantom-container wedge is only cleared by a host REBOOT → `persistent-memory.d/2026-07-15-the-esh-docker-vm-d-state-phantom-container.md` + +- `[2026-07-15]` vLLM `max-model-len` does NOT free GPU VRAM → `persistent-memory.d/2026-07-15-vllm-max-model-len-does-not-free-gpu.md` + +- `[2026-07-15]` Claude Code statusline `.cost.total_cost_usd` is per-SESSION → `persistent-memory.d/2026-07-15-claude-code-statusline-cost-total-cost-usd-is.md` + +- `[2026-07-14]` MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround → `persistent-memory.d/2026-07-14-mtp-on-modelopt-no-checkpoint-config-skips-the.md` + +- `[2026-07-14]` AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof → `persistent-memory.d/2026-07-14-aeon-s-working-nvfp4-mtp-rp-seat-was.md` + +- `[2026-07-14]` NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept → `persistent-memory.d/2026-07-14-nvfp4-llm-compressor-compressed-tensors-gives-no-batch.md` + +- `[2026-07-14]` NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent → `persistent-memory.d/2026-07-14-nvfp4-spike-built-the-full-mtp-serve-scaffolding.md` + +- `[2026-07-14]` MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained` → `persistent-memory.d/2026-07-14-mtp-graft-via-top-level-mtp-tensor-names.md` + +- `[2026-07-14]` gitea "test-delivery 204" is NOT proof a webhook works → `persistent-memory.d/2026-07-14-gitea-test-delivery-204-is-not-proof-a.md` + +- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data → `persistent-memory.d/2026-07-13-relaying-a-peer-s-diagnosis-as-fact-without.md` -- `[2026-07-14]` **MTP-on-modelopt: NO checkpoint config skips the spec-decode drafter's quant (vLLM 0.24 bug) — 4 config attempts failed before the runtime workaround.** All crashed the same way (`qwen3_5_mtp.py:256` `param_data.shape == loaded_weight.shape` AssertionError — bf16 mtp head loaded into a quantized drafter param): (1) mtp excludes in `config.json` (WRONG file — vLLM modelopt reads `hf_quant_config.json`); (2) specific-unfused mtp names in hf_quant_config; (3) wildcards `mtp*`/`mtp.layers.0*` (`is_layer_skipped` is EXACT-membership, NOT glob — wildcards match nothing); (4) exact fused+unfused names in both `mtp.`/`model.` prefixes. Instrumenting `is_layer_skipped` proved the drafter's exclude list holds ONLY the main model's `linear_attn` entries — the mtp excludes never reach the draft-model quant config. ONLY fix = a mounted `sitecustomize` force-skipping `mtp.*`. LESSON: don't chase checkpoint-config fixes for the mtp-drafter crash; go straight to the runtime patch. Also `nvidia-modelopt[hf]==0.43` (AEON's producer version) is a trap — it pins transformers back to 4.57 which can't load `qwen3_5` at all; use 0.45 + the FusedMoE guard in `quant_modelopt.py`. -- `[2026-07-14]` **AEON's "working NVFP4+MTP RP seat" was pantheon on compressed-tensors (0% MTP accept), not a modelopt MTP proof.** `vllm-aeon-rp`'s .env → `AEON_RP_MODEL=pantheon-27b-mtp-nvfp4`, `AEON_RP_QUANT=compressed-tensors` — it LOADED (mtp silently skipped, `exited 0`) but never accelerated. Same vLLM image (`:latest` = `sha256:4091d55` = 0.24.0) as the failed Heretic2 test, so the "AEON ran on an older vLLM" theory was wrong. Don't treat a seat that "ran" as MTP-validated without checking its `SpecDecoding` acceptance. -- `[2026-07-14]` **NVFP4 (llm-compressor / compressed-tensors) gives NO batch-1 speedup over GGUF for the Qwen3.5 GDN-hybrid, and its MTP is 0%-accept.** Measured base NVFP4 no-MTP ≈53 tok/s decode vs the GGUF NEO-CODE seat ~59.5 (llama.cpp wins single-stream; NVFP4's edge is concurrency, and this hybrid is bandwidth-bound at batch-1 with the BF16 linear_attn/GDN layers dominating). MTP spec-decode = 0% acceptance (vLLM's `Qwen3_5MTP` drafter won't load the bf16 mtp weights off a compressed-tensors main model → `Parameter … not found in params_dict`, `Avg Draft acceptance rate: 0.0%`). Pantheon is identical — its "working NVFP4+MTP" was working *structure*, never real acceleration. Working native MTP needs the **modelopt** main-model format (AEON, ~3.3/3 accept). LESSON: don't expect a faster single-stream seat from an llm-compressor NVFP4 quant of this arch; the MTP multiplier is the whole point and it requires modelopt. -- `[2026-07-14]` **NVFP4 spike: built the full MTP serve scaffolding BEFORE validating a plain NVFP4 serve was coherent.** Chased 6 sequential serve-config fixes (entrypoint doubled `serve`, arch `ForCausalLM`→`ConditionalGeneration`, `--language-model-only`, mamba-cache/`max-num-seqs`) across a **2.5hr GPU window** (quoted 30-60 min) — only to find the served model gibbers (`!!!!`). LESSON: smoke a PLAIN `/v1/completions` coherence check on the SIMPLEST config (native arch, no MTP, no splice) FIRST — validate the tracer bullet before building spec-decode scaffolding. Also cost an unnecessary re-quant (the `re:mtp.*` ignore fix that turned out moot). Diagnostic ladder in Current state. -- `[2026-07-14]` **MTP graft via top-level `mtp.*` tensor names does NOT survive `AutoModelForCausalLM.from_pretrained`** — the `Qwen3_5ForCausalLM` class doesn't expose an mtp module, so the mtp keys are DROPPED at load (quant output = 0 mtp). Fix = SPLICE the BF16 mtp tensors into the quant output post-hoc (how pantheon was built); don't rely on the graft surviving the model round-trip. -- `[2026-07-14]` **gitea "test-delivery 204" is NOT proof a webhook works** (204 = gitea *queuing*, not the listener receiving) — and a proxy test signing with the listener's OWN secret proves the listener, not gitea's real delivery. Both red herrings cost a round of the soong-lab webhook diagnosis. Diagnose from BOTH ends: sender (`docker logs gitea | grep webhook` → the `deny ''` line) AND an instrumented receiver. -- `[2026-07-13]` Relaying a peer's diagnosis as fact without confirming it against raw data. worldtree-dev diagnosed the WT #355 residual as "our llama.cpp seat wedging," which I echoed in a wrap-up; the operator challenged it and the seat logs DISPROVED it (seat completes ≤72s, idle at the wedge onset — the hang is the LiteLLM gateway). Lesson: CONFIRM peer diagnoses (esp. cross-domain ones) before acting/relaying — same discipline that caught the earlier char-rp-reasoning red-herring via a live `registry.resolve` reproduction. - `[2026-07-13]` `althing-cli reply ` (thread id, not a MESSAGE id) → "unknown message_id"; and `reply` to your OWN message self-addresses to your handle ("replying to your own message"). Reply to a PEER's message id, or use `post --to `. Bit me several times this session. -- `[2026-07-09]` **FP8 breaks mOrpheus audio-token generation.** `--quantization fp8` on the 3B → 0 valid SNAC - frames even at GREEDY (degenerate audio+text mix, no start-of-speech); bf16 is clean (28/28 frames). Quant-breaks- - TTS, same class as the Angel-NVFP4 lesson below. bf16 is REQUIRED (so the operator's "util 0.1" wish is moot — the - bf16 weights alone are 6.6GB). NB the raw-token benchmark RTF 0.50 was fp8+graphs = never real. +- `[2026-07-09]` FP8 breaks mOrpheus audio-token generation → `persistent-memory.d/2026-07-09-fp8-breaks-morpheus-audio-token-generation.md` + - `[2026-07-09]` **`vllm/vllm-openai:latest` crashes on Ampere IMPORT** — Blackwell-only kernels (oink/aiter, `has_device_capability(100)`) die during import on the 3090/A6000. Pin **v0.23.0** on irv-ml1's Ampere GPUs. (`vllm/vllm-omni:v0.18.0` has a different entrypoint — don't use it either.) + - `[2026-07-09]` **Per-frame CPU SNAC decode is too slow for streaming** — per-call overhead × ~60 frames serialized → RTF 2.2 (WORSE than whole-clip's 1.0). Fix = **windowed chunk decode** (every 6 frames decode a [2 ctx | 6 | 2 ctx] window, emit the middle 6 → seamless, O(1)/frame, RTF ~0.97, TTFA ~0.8s). -- `[2026-07-09]` **Sentence-chunking TTS loses prosody** — generating each sentence cold flattens the intonation that - spans a line. Chunk by QUOTED SECTION (whole quote = one gen call). Also: `repetition_penalty` >1.1 BREAKS cloning - (penalizes the ~1100 in-context reference audio tokens; keep ≤1.1 on the clone path). -- `[2026-07-09]` **HF whisper datasets aren't actually whispered.** Claris-Whispered-English measures voiced 0.8 - (not a whisper) + IPA transcripts; `datasets` audio decode needs torchcodec (wants CUDA-13, incompatible w/ the - cu124 venv). LPC DSP-whisperize went unstable (NaN). **kokoro `af_nicole` IS a genuine whisper** (voiced 0.24) — - that (operator's pointer) is the working whisper reference source, not TTS-voice screening or dataset-hunting. -- `[2026-07-08]` **Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE.** llm-compressor W4A4 NVFP4 - (compressed-tensors, MLP-quantized, attn/vision bf16) of the Mistral3 dense 24B produces gibberish EVEN AT GREEDY - (temp 0) → the quant itself is broken, not the tokenizer or sampler. Same recipe worked on the qwen models. - Mistral3 + W4A4 NVFP4 via llm-compressor is bad. → for the RP seat, going **GGUF (llama.cpp)** to sidestep the - whole NVFP4-quant surface. -- `[2026-07-08]` **Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24).** (a) HF `tokenizer.json` - for Mistral = **GARBAGE output** — the card's "use the official Mistral tokenizer" warning is REAL; must use the - `tekken.json`/mistral tokenizer. (b) BUT `--tokenizer-mode mistral` + vision **CRASHES** (`Failed to apply - PixtralProcessor on {'text': '[IMG]'}`; and with tekken.json present in auto mode, `CachedMistralCommonBackend has - no attribute is_fast`). So it's **mistral-tokenizer OR vision, not both** on this vLLM. Text-only + mistral - tokenizer serves clean (`--limit-mm-per-prompt '{"image": 0}'`). **GGUF/llama.cpp avoids all of this** (native - mistral tokenizer + vision). -- `[2026-07-08]` **Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base.** The base - (`llmfan46 heretic`) writes freely (thinking-off), but Gryphe distilled the reasoning traces from **DeepSeek 3.2** - (safety-aligned) onto every turn (`preserve_thinking:true`) → the model reasons ITSELF into refusals in the - `` phase (collapses to empty output). Fix: thinking-off OR an uncensor system prompt (both verified). - **Lesson: a reasoning finetune of an abliterated base can re-censor via its reasoning-trace TEACHER; the raw - abliterated base is cleaner** — this is WHY the pivot went to the llmfan46 heretic base for gen. -- `[2026-07-08]` **Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance.** MTP is a separate **bf16** head - (`mtp.*`, in `model-auxiliary.safetensors`, 15 tensors); AEON preserved it by INJECTING the bf16 head into the - quant output (NOT re-quantizing — confirmed AEON's nvfp4 mtp is bf16). Built pantheon-27b-mtp = compressed-tensors - main + injected bf16 mtp + `text_config.mtp_num_hidden_layers=1` → vLLM detected the MTP but SKIPPED the bf16 - self_attn weights → 0/192 draft tokens accepted. **The bf16 MTP head only loads on the MODELOPT main-model format - (like AEON), not compressed-tensors.** (Moot — operator dropped MTP for gen; not needed for the non-reasoning RP.) -- `[2026-07-07]` **vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639).** Adapter loads HTTP 200 - but zero deltas at inference. NOT quant (NVFP4 AND FP8 both inert). NOT adapter format (separate `zc` - adapter — correct per vLLM's `check_unexpected_modules` allowlist — loads clean but inert; the fused-key - rekey is rejected). The #47640 None-group guard-patch overlay did NOT fix it (failure is UPSTREAM of - `expand_packed_lora` — the separate→fused mapping never happens). Fix PR #47640 is OPEN (unmerged) so no - version-bump helps. Merge bakes deltas in (bypasses this) but is static. -- `[2026-07-07]` **SGLang generic image can't LOAD our NVFP4 AEON** — ModelOptModelLoader weight-shape/ - packing mismatch ([1024,5120] vs [1024,2560], 2-fp4/byte). NVFP4-on-SGLang needs the dedicated - `qwen36-27b-nvfp4` dev image or a requant to SGLang's format. bf16 loads fine (arch supported; crash was - quant-loader-specific). -- `[2026-07-07]` **SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for** - (invalid choice: 'in_proj_qkv'); `'all'` resolves to the FUSED set (qkv_proj/in_proj_qkvz). SGLang wants - its OWN packed layout (base r16 + `get_stacked_multiply=3`, NOT a pre-fused rank-48 qkv → the [48]-vs-[144] - shape assert). A THIRD adapter format; version-exact source needed (`:latest`=0.5.13, NOT `main`). -- `[2026-07-07]` **Engine invocation footguns cost several wasted serve-bounces this session** — `docker run - --rm` ate crash logs; duplicated `serve` (vLLM image entrypoint is already `["vllm","serve"]`); - `--max-lora-rank 48` invalid (choices 1/8/16/32/64… → use 64); parens in `echo` inside `ssh host -c "…"` - break the remote shell. LESSON: verify engine launch flags (`--help`, GPU-free) + never `--rm` a container - whose crash logs you need, BEFORE bouncing a production serve. -- `[2026-07-04]` **LiteLLM (this gateway version) mutates the SHARED deployment config in-place on - per-request sampler-param merge** → my deliberately-invalid `top_k=-5` forwarding-probe bled into a - param-less character-rp request (vLLM 400, ONE-OFF, self-cleared by a later valid probe). NOT - caching (none configured), NOT a config change. **Never fire invalid/distinctive sampler values at - a SHARED gateway alias with live consumers** — use a throwaway alias, or a `docker restart litellm` - flushes residual carryover. `feedback_litellm_shared_param_mutation`. -- `[2026-07-04]` **A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools - needs an explicit `Environment=PATH`** — the minimal `--user` default silently drops them. The - althing herald lost `zellij` → silent `pane-miss` for ALL config-backed TUI/pane agents; CC + FIFO - routes were unaffected, so it was invisible from a CC session. `reference_nh3_dev_althing_herald`. +- `[2026-07-09]` Sentence-chunking TTS loses prosody → `persistent-memory.d/2026-07-09-sentence-chunking-tts-loses-prosody.md` -- `[2026-07-04]` **On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS** (1-GPU + NVMe - ZeRO-Infinity offload; MoE ~10B-active cuts FLOPs but NOT the 244G base's param I/O). The only fast - on-prem path is a FULL ana-ml2 shutdown (both GPUs + the ~421G vLLM RAM freed → base fits in the - 566G CPU RAM) → CPU offload → ~1-day full-fleet outage. Cloud (no offload) = hours. `reference_t1_cloud_train_plan`. +- `[2026-07-09]` HF whisper datasets aren't actually whispered → `persistent-memory.d/2026-07-09-hf-whisper-datasets-aren-t-actually-whispered.md` -- `[2026-07-01]` **A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" - is usually the pull-only-vs-build RACE, not registry-auth.** `deploy-personal.yml` is PULL-ONLY but - fires on the `staging/vX` tag simultaneously with `deploy.yml`'s build → pulls before the push - finishes. FIX: re-run once built, or gate on `workflow_run: completed`. +- `[2026-07-08]` Angel (allura-org/MS3.2-24b-Angel) self-quanted to NVFP4 = GARBAGE → `persistent-memory.d/2026-07-08-angel-allura-org-ms3-2-24b-angel-self.md` + +- `[2026-07-08]` Mistral3 + vLLM tokenizer/vision traps (serve `MS3.2-24b`, vLLM 0.24) → `persistent-memory.d/2026-07-08-mistral3-vllm-tokenizer-vision-traps-serve-ms3-2.md` + +- `[2026-07-08]` Pantheon-Reasoning-27B refuses dark fiction DESPITE an abliterated base → `persistent-memory.d/2026-07-08-pantheon-reasoning-27b-refuses-dark-fiction-despite-an.md` + +- `[2026-07-08]` Pantheon-27B MTP on vLLM compressed-tensors = 0% acceptance → `persistent-memory.d/2026-07-08-pantheon-27b-mtp-on-vllm-compressed-tensors-0.md` + +- `[2026-07-07]` vLLM 0.24.0 qwen3_5 LoRA application = silent no-op (#47639) → `persistent-memory.d/2026-07-07-vllm-0-24-0-qwen3-5-lora-application.md` + +- `[2026-07-07]` SGLang generic image can't LOAD our NVFP4 AEON → `persistent-memory.d/2026-07-07-sglang-generic-image-can-t-load-our-nvfp4.md` + +- `[2026-07-07]` SGLang `--lora-target-modules` CLI enum REJECTS the GDN names its own resolver asks for → `persistent-memory.d/2026-07-07-sglang-lora-target-modules-cli-enum-rejects-the.md` + +- `[2026-07-07]` Engine invocation footguns cost several wasted serve-bounces this session → `persistent-memory.d/2026-07-07-engine-invocation-footguns-cost-several-wasted-serve-bounces.md` + +- `[2026-07-04]` LiteLLM (this gateway version) mutates the SHARED deployment config in-place on per-request sampler-param merge → `persistent-memory.d/2026-07-04-litellm-this-gateway-version-mutates-the-shared-deployment.md` + +- `[2026-07-04]` A systemd `--user` daemon that shells out to `~/.cargo/bin`/`~/.local/bin` tools needs an explicit `Environment=PATH` → `persistent-memory.d/2026-07-04-a-systemd-user-daemon-that-shells-out-to.md` + +- `[2026-07-04]` On-prem T1 train that keeps ANY ana-ml2 serving up = ~6-8 DAYS → `persistent-memory.d/2026-07-04-on-prem-t1-train-that-keeps-any-ana.md` + +- `[2026-07-01]` A personal-Worldtree CI deploy that fails ~85s in with "not found / unauthorized" is usually the pull-only-vs-build RACE, not registry-auth → `persistent-memory.d/2026-07-01-a-personal-worldtree-ci-deploy-that-fails-85s.md` - `[2026-07-01]` **MTP/spec-decode on a SHARED serving model helps single-stream but HURTS moderate-concurrency aggregate + silently ignores `min_p`/`logit_bias`** (qwopus `gen`: N=1 +12%,