Commit Graph

207 Commits

Author SHA1 Message Date
vh b268f93035 feat(char-rp-gguf): replace broken Angel NVFP4 with dual GGUF RP seat on ana-ml2 GPU0
char-rp        -> TheDrummer Magidonia-24B-v4.3 Q6_K (Magistral prose, ~65 tok/s,
                  zero refusal, tight POV) via llama.cpp (:8016).
char-rp-reasoning -> ArliAI QwQ-32B-RpR-v4 Q5_K_M (abliterated managed reasoning,
                  ~52 tok/s, reasoning surfaces in reasoning_content) via llama.cpp (:8018).

- New canonical stack stacks/char-rp-gguf/ (llama-server x2, GPU0-pinned, ~86/97G
  co-resident with gen). GGUF sidesteps the vLLM-NVFP4 + Mistral-tokenizer traps that
  killed the Angel serve. Never Ollama.
- Best-of-breed per seat: no single dense 24-32B is both an elite non-thinking prose
  seat AND a clean managed-reasoning seat on llama.cpp (Magidonia [THINK] boundary is
  loose; Cydonia-R1 <think> runs away; QwQ is template-managed). Pantheon-Reasoning-27B
  stays rejected (re-censors in <think>; RpR-v4 abliterated reasoning is the fix).
- Gateway rewired: char-rp->:8016, char-rp-reasoning->:8018, Mistral/QwQ samplers,
  dropped the Qwen enable_thinking kwarg. One-model Magidonia fallback documented.
- Retired the ms32-24b-angel stack.
2026-07-08 02:43:00 -07:00
vh 75851c2837 memory: snapshot — T1 SFT done (AEON-27B, adapter banked) + hot-swap BLOCKED (vLLM qwen3_5 LoRA no-op #47639), SGLang last-shot = smoke sft_adapter_zc; comfy-dev LoRA worker done (Phases 1/2/2.5) 2026-07-07 16:02:45 -07:00
vh 5c64d31094 memory: snapshot — AEON Qwen3.6-27B is now gen (qwopus displaced), litbench torn down/comfyui restored
- gen := AEON dual NVFP4 serves (vLLM 0.24 + LiteLLM v1.91.0); reasoning-trace bug
  was the LiteLLM shared-config mutation, fixed durably via distinct -thinking served-names.
- Worldtree personal character/thoughtful-character repointed to char-rp/char-rp-reasoning.
- LitBench-RM torn down, comfyui restored on irv-ml1.
2026-07-06 00:57:01 -07:00
vh 993decf3eb memory: snapshot — 2026-07-05 (T1 train venue = cloud-rec/operator-chose-ana-ml2-smoke; LitBench-RM up + comfyui displaced; character-rp shipped + #344; althing v2 herald/receiver systemd + PATH fix; glm-5.2 1M/128K; LiteLLM shared-param-mutation footgun; condensed R22 + several Recent-decisions entries) 2026-07-05 16:14:37 -07:00
vh 3c966b2631 memory: track low-pri cleanup of inert mood.decay_rate/stale_hours from deployed WT bind-mounts
R30 b15-b17 removed mood.decay_rate/mood.stale_hours from canonical (OCEAN
wall-clock OU replaced per-turn decay); deployed /opt/worldtree*/config
bind-mounts still carry them, harmless (CharacterSchema.mood is dict[str,Any]).
Tracked as an opportunistic edit-only/no-restart cleanup to restore byte-identity;
noted new optional mood.tau_base (unset->derived). Config-delta acked to worldtree-dev.
2026-07-03 22:14:58 -07:00
vh 3a627c6c26 memory: operator decided DEFER granite efficacy to the T1 run (no intermediate spike)
Records Vuong's 2026-07-02 call closing the granite-efficacy thread: no
intermediate real-efficacy granite spike (uninterpretable proxy — arch
gap + abliteration axis), efficacy validated on the real T1 run. Notes
the LitBench-less data/judge WIRING check as the correct pre-T1 de-risk
IF one is ever wanted, and that infra's remaining owed item is the queued
swappable-LoRA-on-NVFP4 load test (gated on the first T1 adapter).
2026-07-02 10:33:15 -07:00
vh 7fdda2de53 memory: granite spike = mechanical-green ONLY, efficacy not validated by design (mtf-dev confirm)
The granite-8b harness spike proved the TRL SFT->DPO->eval seam (incl. the
in-loop HoldoutEvaluator base-vs-adapter leg) runs end-to-end, but used a
12-row/12-pair synthetic writing fixture — NOT the E-RP corpus — so the
~0 anti-slop delta (-0.002) is the expected null, not an efficacy signal.
Adapter reaped; nothing to A/B. Real behaviour-shift efficacy is a T1-run
question. Sharpen both the T1 in-flight bullet and the Recent-decisions
entry so 'green' no longer reads as efficacy-validated.
2026-07-02 10:25:46 -07:00
vh 5b52673b75 memory: snapshot — 2026-07-02 (Deckard trial→revert to qwopus; MTP concurrency verdict = not-kept-on-shared-gen; worldtree #332 diagnosis + scoped-view/tunnel + CI-race lesson; mtf-dev granite harness spike; /books ESH mount) 2026-07-02 08:26:35 -07:00
vh 30c883be5d memory: snapshot — 2026-06-25 (althing v0.17 / nh3-extdev model-B mesh + zellij web pilot + Worldtree #314/#322/#317 config arc; rewrite stale althing Tools-row to v0.17; archive 10 pre-session 2026-06-20 entries) 2026-06-25 00:14:02 -07:00
vh 13bfa4a621 memory: snapshot — 2026-06-21 (compress in-flight to current; archive 21 pre-session entries to archival-memory.md) 2026-06-21 00:07:57 -07:00
vh db2953e690 memory: R22 Phase B CANCELLED (Worldtree model-agnostic → no deploy path); gateway-only 2026-06-20 22:04:37 -07:00
vh 245b217372 memory: R22 key full-open confirmed by operator (settled) 2026-06-20 21:59:40 -07:00
vh 3cb54efd59 memory: R22 key re-minted persistent (stateless consumer); old orphan revoked 2026-06-20 21:55:00 -07:00
vh b33049ce3e memory: R22 operator steer — research gated on pragmatic/deployable outcome, not art 2026-06-20 21:52:17 -07:00
vh 9d65339fb2 memory: R22 stood down to gateway, full-access key minted, Phase B parked; phantom qwen3.6 entry to clean 2026-06-20 21:49:48 -07:00
vh d9ebe8d0f4 memory: old claude-bot token (id 15) revoked — worldtree-dev fully self-serve on one token 2026-06-20 17:27:47 -07:00
vh 6430c01dad memory: claude-bot issue-scope token (id 16) minted for worldtree-dev self-serve 2026-06-20 17:25:54 -07:00
vh ec671e86c5 memory: cb2a79a readonly-admin allow-rules re-staged to demo+personal (PDP is rule-based) 2026-06-20 16:43:44 -07:00
vh b13f66aea9 memory: ratatoskr flipped to readonly — re-staged 439bebf policies.yaml to personal + safe restart reload 2026-06-20 16:29:15 -07:00
vh 2e993ac3df memory: ratatoskr resolved (operator chose admin; worldtree-dev self-served key 90db1fbd) 2026-06-20 16:24:21 -07:00
vh 5a3a75b73d docs(backups): harden + live-activate /mnt/compose automount on ana-docker 2026-06-20 16:21:05 -07:00
vh 76b317ce3e feat(backups): freshness check + daily alert timer; record rest-server-ana recovery, fstab hardening, esh-pve-nas gap, worldtree admin-key provisioning 2026-06-20 16:13:13 -07:00
vh a7b4a82dec docs(backups): add backup architecture + freshness runbook; record rest-server-ana recovery + correct ana-docker sudo path 2026-06-20 15:56:41 -07:00
vh 8f15f6bb0d memory: capture 2026-06-20 session — Worldtree v0.37.7 demo fix, gitea notifier recovery, backup diagnosis 2026-06-20 15:46:14 -07:00
vh 58ec80d58a memory: snapshot — 2026-06-20
WT capability-gateway migration (aliases + swap-transparency rule + wildcard
removal + gen-frontier + v0.37.5 staged/triggered), OmniVoice streaming /tts
(diffusion TTFA floor), arbo v0.14.9, ratatoskr :8392 + admin.events.read
brokered, claude-bot admin on vh/Worldtree. Disk incident root-caused to a
94GB unrotated langfuse-clickhouse log (+ secondary image bloat) → fixed,
image/buildx prune cron added, Langfuse retired. Backup gap: rest-server-ana
is the ACTIVE ana-side restic target down ~months on a failed ana-nas NFS
mount (NOT decommissioned — docs corrected) → recovery deferred to clean
context, along with the docker-daemon log cap. Archived the [2026-06-16]
cluster (13 Recent + 8 Tried) to archival-memory.md.
2026-06-20 13:36:54 -07:00
vh 826c2a6a64 memory: archive 15 pre-2026-06-16 entries to archival-memory.md
9 Recent decisions + 6 Tried-and-abandoned (dates [2026-06-14]/[2026-06-15]) moved
non-destructively to archival-memory.md, each stamped _Archived 2026-06-19._. Kept
the active [2026-06-14] 'migrate ALL infra access to Claude-specific credentials'
standing directive. Back-ref counts: Recent decisions 79->88, Tried-and-abandoned
70->76. persistent-memory.md 397->363 lines.
2026-06-19 21:54:45 -07:00
vh dfda60fac7 memory: snapshot — 2026-06-19 (pt2) litellm task-aliases (classifier->granite, summarizer-large->gen; gen-nt/gen-reasoning-nt added-then-removed as redundant with strip_empty_tools) + gateway-chat model-smoking web chat enhanced (auto-discover /v1/models + image upload) and stood up as a PERSISTENT nginx container on ana-docker :8091 + pi on nh3-dev wired to gen (vision, ~/.pi models.json + gen launcher, local box config) + foot-guns: litellm config-loaded models can't be hot-removed (/model/delete is DB-only; /model/new live-adds work no-bounce but dup on restart) and the * wildcard routes stale/typo'd names to decommissioned llama-swap -> misleading 'Connection error' not 'model not found' (bit a brokkr call to the renamed-away qwen-image-judge). 2026-06-19 18:51:03 -07:00
vh 378261763c memory: snapshot — 2026-06-19 gen model = Qwopus3.5-122B vision-intact NVFP4 LIVE on ana-ml2 GPU 0 (full 256K @ fp8 KV + CUDA graphs, util 0.95 + expandable_segments, 92.7 tok/s warm, 3.32x concurrency, text+image+video, tool-calling qwen3_coder; nightly+turboquant-4bit-KV proven UNNECESSARY — stable fp8 reaches 256K) replacing the bjk110 text-only qwen3.5-122b (which displaced mistral-small-4 → Worldtree character backend DARK until repointed, operator-acknowledged) + qwen-image-bench T2I judge replaced qwen3.6-35b-a3b on GPU 1 (alias image-judge) + TP=2 across both Blackwells REJECTED (PCIe-only PIX, no NVLink → all-reduce-bound, one-model-per-card is optimal; PP=2 only if a >96GB model is ever wanted) + foot-guns: MoE FusedMoE workspace is the ~3.1GB un-budgeted floor (can't fill to 0), discard cold tok/s reads (24.8 cold vs 92.7 warm). 2026-06-19 11:15:32 -07:00
vh 632124c8fb memory: nh3-extdev pi-on-GLM-5.2 wired (mark in-flight item done + residual gates) 2026-06-18 14:05:02 -07:00
vh a67d4950d0 memory: snapshot — 2026-06-18 heretic abliterated Mistral Small 4 NVFP4 BUILT + LIVE as mistral-small-4 (in-house quant device_map=cpu → native-format convert → drop-in stack under same served-name, A/B'd vs official, operator "heretic stays"; byte-equivalent to official NVFP4) + irv-ml1 VRAM consolidation (ComfyUI pinned to A6000 exclusive/48GB, audio zoo→3090, downed dia/ace-step/csm, comfy-dev torch-pin DISABLE_UPGRADES@2.12.1 + SageAttention rebuilt) + ComfyUI 9-node accel set installed for comfy-dev + ana-ml2 durable vm.overcommit_memory=1 + GLM5.2 wired + nh3-extdev sudo-less manager box + /opt/externs pi-on-GLM client workspaces.
Lessons: mmartial-comfyui root-install-leaves-root-owned-venv-files → boot-script crash-loop (chown -R 1000:1000 fix) + torch-upgrade-on-boot (DISABLE_UPGRADES); mistral HF→NVFP4 quant device_map=cpu (auto OOMs, constrained→meta-tensor) + non-mmap shard reads (safe_open mmap ENOMEMs on /tank ZFS) + NVFP4 keeps the model. prefix; HF-format Mistral4 UNSERVEABLE on vLLM (native mandatory); ComfyUI 0.24.1-not-0.19.3 version-drift kills module-level node imports + tensorrt-defaults-cu13-vs-cu12.9.

Archived the [2026-06-14] cluster (11 entries: 6 decisions + 5 foot-guns; kept the still-active credential-migration directive, infra-ops litellm key, gitea-internal-route).
2026-06-18 13:48:12 -07:00
vh 8cca365b78 memory: correct gitea action-log API note (per-job endpoint works)
Proven 2026-06-16 diagnosing arbo run #5/task 1175: the RUN-level
/runs/{id}/logs 404s, but the per-JOB endpoint
GET /api/v1/repos/{o}/{r}/actions/jobs/{job_id}/logs returns the full
plain-text log (claude-bot basic-auth, internal :3000) — no UI needed.
Also noted gitea's misleading per-step conclusions (every step shows
failure once any fails; trust the log + timestamps).
2026-06-16 14:56:12 -07:00
vh 47c30e85f9 memory: snapshot — 2026-06-16 litellm strip_empty_tools hook (d1bea13) + single-file gateway-chat.html playground (984ca3d) + claude-bot ADMIN on vh/arbo (arbo CI/CD via service account) + LitBench-RM reward judge served on irv-ml1 A6000 then taken to on-demand (held comfyui's slot) + #295 recall root-cause FLIPPED (score_breakdown-shape DISPROVEN → cold-recall agent_self scope axis vs ratatoskr conjunctive INV-005; Worldtree #297); lessons: litellm-500-Router.acompletion-missing-messages = a request missing Content-Type (NOT a gateway outage — cost 4 needless restarts), litellm-admin-UI-playground-cant-test-vLLM (#6228 empty-tools, proxy-hook-cant-reach-in-process-call), gitea-run-looks-like-never-fired-but-fired-then-skipped/failed-fast (check run list not runner). Archived the 06-09→06-13 cluster (24 entries: 17 decisions + 7 foot-guns). 2026-06-16 14:14:36 -07:00
vh f9277f5440 memory: snapshot — 2026-06-16 ratatoskr Tier-3 MEMORY plane wired (allowlist :8391, key reused, persist+dispatch GREEN; recall-injection root-caused to the score_breakdown shape seam → worldtree-dev #295) + infra-ops durable admin on corviduo (ssh alias + ssh-target) + demo/personal character model qwen→mistral (first-bind-is-default reorder, pin-safe recreate); lessons: bifrost-allowlist-is-per-port, promotion-gate=consumer-agent-memory-block-not-agent_self_enabled, WORLDTREE_IMAGE-pin-from-matrix-sibling 2026-06-16 00:13:14 -07:00
vh aeea377749 memory: snapshot — 2026-06-16 ana-ml2 dual-NVFP4 reshape (GPU0 Mistral Small 4 256K/v0.22.0-vision + GPU1 qwen36 FP8→NVFP4 + Selene FP8 judge + GPU1 grows) + NVFP4-MoE-loads-on-0.23.0 (supersedes blocked) + claude-bot service account (corviduo-org tabled) + arbo→comfy-dev ownership + gitea runner on irv-ml1 + Worldtree demo/personal capability-profile migration (pre-sync-first); lessons: vLLM-0.23-breaks-Mistral-vision (#44911), Mistral-TTFT=Triton-JIT-spikes, vh-user-not-org blocks scoped package-write, old-baseline-instances-need-full-config-set; archived the 2026-06-05/08 cluster (11 entries) 2026-06-15 22:26:11 -07:00
vh 0943d145fb memory: snapshot — 2026-06-15 (cont.) arbo v0.11.22 engine rebuild + catalog v0.11.23 (curated /workflows footer live) + althing-core v0.14.1 box-wide refresh (monitor lock fix) + comfyui VAE-decode SEGFAULT diagnosis (aimdo 0.4.8 cuda-hooks vs torch cu129/cu130 mismatch, NOT OOM); lessons: comfyui-segfault-not-OOM diagnostic, never blanket-kill peer light-monitors 2026-06-15 13:35:50 -07:00
vh 12bcd06442 memory: snapshot — 2026-06-15 ratatoskr affect smoke GREEN (Heimdall key mint+inject, allowlist, handshake+emit) + infra-ops bootstrapped on corviduo + dense Qwen3-VL-32B-NVFP4 judge A/B (lost, torn down) + MastMed cloudflared public + R18 clip+caption staged (stub smoke passed; real-voice gate pending) + LiteLLM infra-ops key; lessons: corviduo stale-:latest recreate crash, .claude.json ENOSPC repair, pkill self-match 2026-06-15 00:17:44 -07:00
vh 10f346b39e memory: snapshot — 2026-06-14 FP8 vision cutover (qwen35-vl→qwen36-vl, truthful naming, GPU-1 rebalance) + llama-swap pin drop + R16 yield probe executed + standing credential-migration directive; NVFP4-on-vLLM-blocked + sampler-warmup/profiling-race/embed-rerank-waste lessons 2026-06-14 14:47:01 -07:00
vh b45d0cd86d memory: snapshot — 2026-06-14 arbo auth-off + deploy-pipeline fix (v0.11.6, internal gitea route, catalog-only restart, scripts tracked) + storetank archive decommission (919G -> arbo 502G) + R16 inline arc closed (v1 final); archive 11 (2026-06-04 cluster) 2026-06-14 08:56:12 -07:00
vh 0fc9083d16 memory: snapshot — 2026-06-13 ana-ml2 Ada→Blackwell + NVFP4-infeasible + Qwen3.5-VL FP8 + comfyui→arbo + GPU-1 rebalance + prefix-caching; archive 8 (2026-06-03 cluster) 2026-06-13 13:30:19 -07:00
vh 005effd664 memory: snapshot — 2026-06-11 Mac Pro migration framing + GLM thinking-off + R17 v2 corpus; archive TTS-streaming arc 2026-06-11 08:27:06 -07:00
vh 5349da567c memory: snapshot — 2026-06-05 granite-FP8 cutover + Langfuse observability + worldtree #254 deferral 2026-06-05 14:14:44 -07:00
vh 1dba05d8c8 memory: snapshot — 2026-06-04 infra session (phi4/vLLM, infra-ops identity, R15/R16, brokkr svc)
Captures: phi4-mini FP8 on ana-ml2 vLLM as nevermore summarizer (granite retired,
canonical Phi-4 template after apply→revert); infra-ops NOPASSWD-sudo identity across
PFI boxes; R15/R16 brokkr-smithy stand-up + ollama 0.30.4 + A6000 UUID-pin; brokkr user
+ substrate migration + audition service; homepage incident; observability roadmap.
Foot-guns: headgun chat-template lesson, UUID GPU-pin, system-unit reaping, pyworld/
gfortran, homepage D-state. Archived 11 settled 2026-06-01 TTS-arc entries (Recent 5 +
Tried 6) to archival-memory.md; pruned 145 lines of DONE 2026-06-03 in-flight bulk.
File back under the 300-line cap (299).
2026-06-04 01:08:50 -07:00
vh 9090fb59d7 memory: archive 13 settled 2026-05-30/31 entries (Recent 7 + Tried 6) to archival-memory.md 2026-06-03 10:49:09 -07:00
vh 23f2b63884 memory: snapshot — 2026-06-03 infra session (NH3 egress proxy, R14 provisioning, yt-voice-clipper push-to-deploy, gitea allowlist, irv-ml1 disk) 2026-06-03 10:38:47 -07:00
vh fa56718ba0 memory: shelve fish-streaming (sub-realtime 0.80x); standing fp4+Blackwell revisit trigger 2026-06-02 08:59:17 -07:00
vh 02e5ea5894 memory: chatterbox-fast deployed (Phases 1-3) — :8197 A6000, fp32 5.34GB, priming reverted 2026-06-01 23:37:38 -07:00
vh a95aa75947 memory: record context-priming discard-cut abandonment (chatterbox-fast) 2026-06-01 23:27:16 -07:00
vh 7cd39001b2 feat(chatterbox-fast): Phase 1 streaming server — adaptive-chunk scheduler
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4.

- scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure
  logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by
  packing whole sentences to margin x buffered-audio; drives off measured RTF +
  sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence
  to avoid starvation (joins land on commas); a long comma-less sentence is the
  one honored-but-flagged limitation.
- test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation
  (incl. overestimated RTF) and the ratchet.
- app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le
  default, wav optional, stream/oneshot) + GET /health.
- bench.py: client — ground-truth TTFB + real 1x-consumer starvation check.

Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs
oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet
1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01.

Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token
streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan
supersedes it. Repoint persistent-memory + README at the canonical plan.
2026-06-01 22:42:28 -07:00
vh 19308ff5aa memory: snapshot — chatterbox-fast streaming build (adaptive buffer-ratchet chunking); native-turbo-streaming abandoned; fish reference_id fix + glados transcript + NFS/smithy + references-togglable; add design doc 2026-06-01 22:21:17 -07:00
vh c5bbb90980 fix(fish-s2): reference_id was a silent no-op — populate per-voice dirs + guard the regression
reference_id=<name> resolves against the DIRECTORY references/<name>/
(audio + same-basename .lab), not a flat references/<name>.wav. Voices
were staged flat with the per-name dirs left empty, so every
reference_id resolved to nothing and Fish fell back to its default
speaker — every dropdown voice produced byte-identical audio (proven:
Abigail == Imogen == no-ref, same text+seed). This was the real "no
accent" root cause, independent of the asset-engine "undefined" select
bug.

Server fix (applied to irv-ml1): populated references/<name>/<name>.wav
+ <name>.lab for all 32 voices; re-test confirms Imogen/Eleanor/
Beatrice/Abigail/no-ref now all distinct.

Durable hardening + record correction:
- playbook: normalize-layout step (flat <name>.wav -> nested dir, cp -u
  idempotent, when-gated on count mismatch) + an A/B verify gate that
  hard-fails the deploy if two reference_ids yield identical output.
- services.yaml: correct the reference_id resolution doc (dir + .lab,
  not flat wav).
- README + persistent-memory: correct the "reference_id-by-name is THE
  working path, verified" claim — it was a no-op until this fix; the
  prior ECAPA 0.79 result came through the inline base64 path.
2026-06-01 16:42:30 -07:00