memory: snapshot — 2026-06-11 Mac Pro migration framing + GLM thinking-off + R17 v2 corpus; archive TTS-streaming arc
This commit is contained in:
+94
-84
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-06-05_
|
||||
_Last updated: 2026-06-11_
|
||||
|
||||
## Repo purpose
|
||||
|
||||
@@ -24,9 +24,12 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
| `vh/mead-hall` | Bifrost tool-provider sidecar (port 5173 on dev VM 10.100.10.50) | push-to-main → CI deploys (2026-05-16) |
|
||||
| `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) |
|
||||
| `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys |
|
||||
| `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) |
|
||||
| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` |
|
||||
|
||||
(`vh/volva` + Heid were re-architected from systemd daemons to Claude Code
|
||||
session orchestrators 2026-06-08; their nh3-dev `.service` units were removed —
|
||||
no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
- **Two-layer backups** — Backrest orchestrates restic for file+DB (5
|
||||
fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for
|
||||
VM images. ana-nas is the SPOF for postgres + PBS-ANA datastore +
|
||||
@@ -88,29 +91,85 @@ Sister repos (separate gitea repos, deployed by playbooks here):
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-06-05:_
|
||||
_As of 2026-06-11:_
|
||||
|
||||
- **Granite-FP8 + observability session — all LIVE & committed (`34a43a0`, `9171e6a`).**
|
||||
- **Granite 4.1 8B FP8 is the production summarizer** (`vllm-granite` :8004, ana-ml2 GPU 1,
|
||||
50K ctx, CUDA graphs) — replaced phi4-mini, validated by brokkr (valid_format 1.0, FP8 stays).
|
||||
- **LiteLLM gateway** (:4000) routes `granite-4.1-8b`→vLLM (explicit entry shadows the `*`
|
||||
wildcard) + **Langfuse v3 wired** (ana-docker:3001, "LLM Throughput (tok/s)" dashboard built).
|
||||
- **GPU-1 retuned** (trio over-provisioned KV trimmed) → granite runs with CUDA graphs + ~10 GB
|
||||
free as a future Granite-text-LoRA hedge. Streaming through the gateway confirmed (TTFT 0.24s).
|
||||
- **ana-docker pruned** 77 GB (unused images + build cache; disk 83%→49%) to fit ClickHouse.
|
||||
- **Worldtree summarizer repoint — NO instance change now; DEFERRED to Worldtree #254** (see Recent
|
||||
decisions). worldtree-dev will ping with the providers.yaml + consumer config when #254 un-holds;
|
||||
infra-ops applies to the personal/demo/pinned bind mounts (vh@10.250.50.152, `/opt/worldtree*/config`).
|
||||
- **Commits unpushed** (`34a43a0`, `9171e6a`, nevermore `d3e19b8` in its repo) — operator's call to push.
|
||||
- **Operator flagged "new work to do"** for the next session — this snapshot is the handoff.
|
||||
- **Disclosed-keys hygiene queue** (rotate at convenience): HF token `hf_HBl…` (lkraven's), `/tmp/
|
||||
wt-personal-skaldsong-prod.key`, Worldtree `Z_AI_API_KEY`, Gitea runner reg token, `MINIFLUX_PASSWORD`
|
||||
(leaked twice). (sk-corvid + the langfuse/vastblueai-gateway keys are dev-enclosed — leakage deprioritized.)
|
||||
- **Still open from prior:** clean legacy `news-digest` on ana-docker; watch nh3-nas `/volume1`; **pin
|
||||
llama-swap to GPU 0** for clean GPU-1 separation; the `docker push 60s ceiling` mystery uninstrumented.
|
||||
- **Mac Pro migration is the big open project** — `migration-plan.md` (repo root) frames a
|
||||
WORKSTATION-ONLY migration of nh3-dev's dev env to an M2 Ultra Mac Pro Rack, racked in NH3 on
|
||||
the 10.100 subnet. nh3-dev wears two hats; only **Hat 1** (the ~40 repos, all of `~/.claude`,
|
||||
dotfiles, toolchain) moves. **Hat 2** fleet sidecars (egress SOCKS5 proxy, mead-hall, ttyd
|
||||
seats, `althing-forseti`) STAY on the Linux VM. **Phase 0 is the only time-sensitive step**:
|
||||
push the pushable repos + confirm which of ~6 no-upstream repos are local-only. Cutover =
|
||||
`rsync` the working trees, **NOT re-clone** (re-clone would lose unsaved work in ~15 repos).
|
||||
macOS gotcha pre-flagged: `/home/lkraven` → `/Users/lkraven` repath. Everything else waits on
|
||||
the hardware landing in the rack.
|
||||
|
||||
- **Two commits/branches committed-not-pushed** (operator's push call): `95b2701` (GLM thinking
|
||||
config) on main; and the R17 v2 results branch `r17-v2-characterization` on the irv-ml1
|
||||
checkout (push HELD — see decisions).
|
||||
|
||||
- **R17 v2 corpus characterization DONE on irv-ml1** (brokkr, ~8.5h): 119 clips → **18 MULTI /
|
||||
101 ELIGIBLE** (Stage-0 drop list) + 19936 candidates + ASR speech + blend verdicts. Outputs
|
||||
on the local branch; **push HELD** — no-push rule AND the `speech.json` are ASR transcripts of
|
||||
the intimate-audio batch (content-exposure call). Awaiting brokkr-collect or explicit push approval.
|
||||
|
||||
- **pi coding agent + GLM 5.1 harness live on nh3-dev** — `glm` launcher runs pi against `glm-5.1`
|
||||
(thinking-off) via the LiteLLM gateway; `glm-5.1-reasoning` for opt-in thinking.
|
||||
|
||||
- **Worldtree config-propagation lane is mature + humming** — pre-merge delta-ping → sync-on-merge.
|
||||
All of v0.33.x→v0.34.1 reconciled to the demo+personal bind-mounts; demo/personal both on v0.34.x,
|
||||
pinned stays pre-cutover (full migration at its next re-image).
|
||||
|
||||
- **Disclosed-keys hygiene queue** carries forward (rotate at convenience): HF token, the wt-personal
|
||||
keys, Gitea runner reg token, MINIFLUX_PASSWORD. `ZAI_API_KEY` (z.ai MCP web-tools key) now also
|
||||
lives in `~/.config/secrets/env.sh` (shared from Worldtree env for the Brokkr panel).
|
||||
|
||||
- **Still open from prior:** clean legacy `news-digest` on ana-docker; pin llama-swap to GPU 0; the
|
||||
`docker push 60s ceiling` mystery uninstrumented.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-06-11]` **GLM thinking inverted at the LiteLLM gateway** (operator call): `glm-5.1` defaults
|
||||
thinking-OFF; new `glm-5.1-reasoning` alias = same z.ai upstream with thinking ON (opt-in).
|
||||
Mechanism: `litellm_params.extra_body:{thinking:{type:disabled}}` — LiteLLM `drop_params` STRIPS a
|
||||
top-level `thinking`/`reasoning_effort`, but forwards `extra_body` verbatim to z.ai (the only channel
|
||||
that works; verified reasoning_tokens 0 vs >0). Shared-gateway change — affects ALL glm-5.1 callers
|
||||
(brokkr's all-models key included). (`95b2701`, auto-memory `reference_litellm_gateway`)
|
||||
|
||||
- `[2026-06-11]` **pi coding agent installed on nh3-dev as a GLM 5.1 harness** — `@earendil-works/
|
||||
pi-coding-agent` via **bun** (user-level; npm's global prefix is `/usr` → needs sudo, bun avoids it).
|
||||
Config `~/.pi/agent/models.json` (litellm provider → gateway), launcher `~/.local/bin/glm` sources
|
||||
the gateway key + selects the model. pi is OpenAI-compatible; proxy-safe compat flags for the GLM path.
|
||||
|
||||
- `[2026-06-11]` **z.ai web-tools (regin) = z.ai hosted MCP path, NOT the `/paas/v4` Tool API.** Direct
|
||||
`api.z.ai/api/paas/v4/web_search` → 429/1113 "insufficient balance" (coding-plan keys bill tools on a
|
||||
separate quota path); `open.bigmodel.cn` is the China platform (404, different account). WORKS: MCP
|
||||
streamable-HTTP at `https://api.z.ai/api/mcp/{web_search_prime,web_reader}/mcp`, `Authorization: Bearer
|
||||
$ZAI_API_KEY` (the **MCP** key — distinct from `Z_AI_API_KEY` the LLM key). Reference impl = Worldtree's
|
||||
Leif agent (`tools/web/zai_client.py` + `core/clients/mcp.py`).
|
||||
|
||||
- `[2026-06-10]` **Mac Pro migration framed: workstation-only** (M2 Ultra ARM, racked NH3 on-subnet);
|
||||
sidecars stay Linux. `migration-plan.md`. (See in-flight.)
|
||||
|
||||
- `[2026-06-10]` **Worldtree deployed-config propagation is infra-ops's OWNED lane** (operator ruling).
|
||||
worldtree-dev pings the config delta on every config-touching commit (pre-merge); infra-ops syncs
|
||||
`config/*.yaml` from MERGED canonical to the `/opt/worldtree*/config` bind-mounts on demo+personal
|
||||
(Heimdall hot-reloads policies; defaults are code-defaulted). The **v0.33.8 9-HOUR demo outage** — a
|
||||
`model_roles.yaml` hard-startup-dep that shipped in canonical 3 releases earlier but never reached the
|
||||
VM — is the failure mode this lane prevents. providers.yaml stays hand-tuned (artemis graft on
|
||||
personal). corviduo-dev emergency-ops = `ssh vh@10.250.50.152` (alias doesn't resolve, lkraven denied,
|
||||
infra-ops key excluded), docker no-sudo. (auto-memory `reference_worldtree_deploys_cicd`,
|
||||
`reference_corviduo_dev_emergency_ops`)
|
||||
|
||||
- `[2026-06-09]` **LiteLLM scoped virtual keys issued to consumers** (operator-authorized): `brokkr-
|
||||
smithy` (all-proxy-models), `arbo-prompt-enhance` (comfy-dev, granite-only). Pattern: mint via
|
||||
`/key/generate` (master `sk-corvid`), scope-restricted + rotatable, value → 600 file never the bus.
|
||||
(auto-memory `reference_litellm_gateway`)
|
||||
|
||||
- `[2026-06-08]` **volva.service + heid.service removed from nh3-dev** — vestigial systemd daemons;
|
||||
Heid/Volva re-architected from Python pollers to Claude Code session orchestrators (heid commit
|
||||
`12aa5a9`); binaries+dirs gone, volva.service was crash-looping 203/EXEC. Cleanup at heid's request
|
||||
(the bus-content-driven sudo got the harness guardrail; operator green-lit). (`6e2f80e`)
|
||||
|
||||
- `[2026-06-05]` **Granite 4.1 8B FP8 replaced phi4-mini as the production summarizer** (supersedes
|
||||
the 2026-06-04 phi4 decision below). Beat phi4 on precision in brokkr's R15 P03. **Staying FP8, not
|
||||
Q4/AWQ** — primary workload (agent memory + summarization) is high-concurrency, where FP8-on-Ada
|
||||
@@ -190,32 +249,22 @@ _As of 2026-06-05:_
|
||||
CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected
|
||||
to the same conclusion independently.
|
||||
|
||||
- `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast`
|
||||
streaming container.** Workload = single-stream interactive. **GPU placement:
|
||||
3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB
|
||||
3090-idle is expected residency, not a blocker. **Cutover: parallel catalog
|
||||
entry**, burn in beside live `chatterbox`, then flip. **Streaming approach:
|
||||
adaptive buffer-ratchet chunking** (see in-flight). Native frame-streaming
|
||||
abandoned (Tried/abandoned). Tracked: `docs/design/chatterbox-fast-plan.md`.
|
||||
|
||||
- `[2026-06-02]` **Sentence-splitting loses quality (operator-corrected).** I
|
||||
claimed naive sentence-level streaming has "zero quality loss" — WRONG. The
|
||||
T3 AR backbone conditions prosody on the WHOLE text; splitting loses
|
||||
cross-sentence prosodic context (contextual delivery, declination, affect
|
||||
continuity) even though voice timbre stays (reference-conditioned). No
|
||||
*artifacts* ≠ no *quality loss*. Hence the adaptive-chunk design (maximize
|
||||
context per chunk subject to latency budget), not fixed per-sentence splits.
|
||||
|
||||
- `[2026-06-01]` **Fish reference_id empty-dir fix shipped** (`c5bbb90`) — see
|
||||
in-flight + Tried/abandoned. Populated `references/<name>/<name>.wav`+`.lab`
|
||||
for all 32 voices; playbook gained normalize-step + A/B smoke gate. glados got
|
||||
a real transcript (ASR'd via Parakeet): the Portal "Welcome to test chamber 4"
|
||||
lines.
|
||||
|
||||
_37 older entries archived to archival-memory.md._
|
||||
_40 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-06-11]` **A completion-poll `while pgrep -f <scriptname>` SELF-MATCHES its own remote shell
|
||||
argv** — the poll's command line contains the script name, so its own `pgrep -f` always finds itself
|
||||
→ the loop never exits, the poll never fires. Use a match pattern ABSENT from the poll command (pgrep
|
||||
the python stage, or a sentinel file), not the driver's own name. (Caught only because the operator
|
||||
asked "check status"; the job had already finished cleanly.)
|
||||
|
||||
- `[2026-06-08]` **Demucs `uv pip install demucs` pulls torch 2.12/torchaudio 2.11 → `ta.save()`
|
||||
requires torchcodec → dies AFTER separating (0 stems written, rc=1).** Same class as the torch-2.12
|
||||
torchcodec foot-gun below. Fix = pin `torch==torchaudio==2.4.1` (pre-torchcodec save backend) +
|
||||
`UV_LINK_MODE=copy` for the EPERM-hardlink quirk. Lesson restated: validate the SAVE path, not just
|
||||
import + GPU inference, on a bleeding-edge torch.
|
||||
|
||||
- `[2026-06-05]` **vLLM 0.19 CUDA-graph-capture OOMs on a SHARED GPU** — it fills the KV cache to the
|
||||
`--gpu-memory-utilization` budget WITHOUT reserving graph-capture memory, so `capture_model` OOMs
|
||||
AFTER weights+KV load (model/KV log looks healthy, then crash-loops; saw 11 restarts at util 0.36
|
||||
@@ -291,43 +340,4 @@ _37 older entries archived to archival-memory.md._
|
||||
— it's analysis scripts + a poetry pyproject, no buildable package (setuptools
|
||||
package-discovery error). Install the pyproject deps directly, don't build the "package".
|
||||
|
||||
- `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), **mean 0.80x = sub-realtime**, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on `<|speaker:X|>` tags; **plain text -> batches=[whole text]** -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). **STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment** (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane).
|
||||
|
||||
|
||||
- `[2026-06-02]` **Context-priming at chunk joins (chatterbox-fast §1.6) —
|
||||
ABANDONED (discard-cut leaks the prefix).** To give a chunk backward prosodic
|
||||
context, prepend the prior sentence, generate `prefix+content` together, then
|
||||
discard the prefix audio. Built + opt-in shipped (commit d707439), live-A/B'd,
|
||||
reverted (090e70a). The kill: `generate()` returns one finished waveform with
|
||||
NO marker for where the prefix ends, and the model renders the same prefix with
|
||||
different timing solo vs followed-by-content — so locating the cut (generate
|
||||
prefix solo → measure duration → snap to nearest energy-min pause within ±0.4s)
|
||||
is a guess that left a whole clause of prefix in the output ("...without a trace
|
||||
of sarcasm," spoken twice; operator caught it). A reliable cut needs token-level
|
||||
boundaries (= the abandoned native-streaming arc) or per-chunk ASR/forced-
|
||||
alignment (heavy, imperfect, eats the latency budget). → Coherence loss at joins
|
||||
stays an ACCEPTED limitation; cold adaptive-chunk streaming judged "really good".
|
||||
Scheduler-side work that DID land + survive: affordability-gated priming math
|
||||
(a 2nd pass can't starve the buffer) — sound, but moot without a working cut.
|
||||
|
||||
- `[2026-06-02]` **Native frame-level streaming on Chatterbox-TURBO — ABANDONED
|
||||
(turbo isn't built for streaming).** Long R&D arc; record so it's not
|
||||
re-derived. (1) The model's flow is CosyVoice2-derived but `S3GenStreamer` is
|
||||
referenced-in-docstring-only (not implemented). (2) The lib's
|
||||
`flow_inference(finalize=False)` is BUGGY: the lookahead trim removes
|
||||
`pre_lookahead_len(3)*token_mel_ratio(2)=6` frames from `h` but NOT from
|
||||
`h_masks`/conds → decoder shape mismatch (e.g. 656 vs 662). A 1-line patch
|
||||
(`h_masks = h_masks[:, :, :-pre*ratio]` after the `h` trim) + sizing the
|
||||
meanflow noise to the trimmed length makes finalize=False RUN. (3) BUT the
|
||||
flow encoder uses FULL-context attention (`static_chunk_size=0`), so
|
||||
incremental/cumulative decode is **prefix-unstable** — adding tokens
|
||||
re-attends and shifts earlier mel (maxdiff ~0.30-0.39 vs one-shot,
|
||||
irrespective of fixed-noise slicing or emit-margin). (4) Forcing
|
||||
`static_chunk_size>0` on the 2 modules that carry the attr did NOT stabilize
|
||||
it (decoding_chunk_size is a forward-arg, not settable via attribute). Verdict:
|
||||
true sub-second frame-streaming on turbo needs deep model-attention surgery
|
||||
with quality risk — not worth it. Matches research ("turbo+streaming
|
||||
unsolved"; vLLM-turbo outputs noise; davidbrowne17 streaming fork is
|
||||
BASE-only). → Use adaptive-chunking instead.
|
||||
|
||||
_41 older entries archived to archival-memory.md._
|
||||
_44 older entries archived to archival-memory.md._
|
||||
|
||||
Reference in New Issue
Block a user