From 23f2b63884b36545fe351968eb84dff5a29682f0 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Wed, 3 Jun 2026 10:38:47 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=202026-06-03=20i?= =?UTF-8?q?nfra=20session=20(NH3=20egress=20proxy,=20R14=20provisioning,?= =?UTF-8?q?=20yt-voice-clipper=20push-to-deploy,=20gitea=20allowlist,=20ir?= =?UTF-8?q?v-ml1=20disk)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- persistent-memory.md | 83 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 79 insertions(+), 4 deletions(-) diff --git a/persistent-memory.md b/persistent-memory.md index 30dbd6f..e5beb2a 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-06-02_ +_Last updated: 2026-06-03_ ## Repo purpose @@ -25,6 +25,7 @@ Sister repos (separate gitea repos, deployed by playbooks here): | `vh/skaldsong` | Wizard + reader surface (port 8300, ana-docker, registry-pull pattern) | push-to-main → CI deploys (2026-05-19) | | `vh/worldtree` | Conversation API (corviduo-dev demo :8080 / personal :8081 / pinned :8082) — Heimdall auth, Bifrost integration | push-to-main → CI deploys | | `vh/volva` | Codex peer agent on althing bus (single-turn oracle, systemd daemon on nh3-dev) | manual install via `deploy/volva.service` (2026-05-18) | +| `vh/yt-voice-clipper` | YouTube → diarized voice-clip dataset builder + audition console (irv-ml1 :8000) | push-to-main → **gitea-webhook auto-deploy** to irv-ml1 (2026-06-03) — see `docs/runbooks/ytvc-autodeploy.md` | - **Two-layer backups** — Backrest orchestrates restic for file+DB (5 fleet repos, daily 01:00 PDT); PBS-ANA primary + PBS-NH3 DR mirror for @@ -87,10 +88,37 @@ Sister repos (separate gitea repos, deployed by playbooks here): ## Current state / in-flight -_As of 2026-06-02:_ +_As of 2026-06-03:_ -- **PRIMARY FOCUS — building `chatterbox-fast`, a custom streaming TTS - container; Chatterbox is becoming our MAIN TTS engine.** Operator-authorized +- **INFRA SESSION 2026-06-03 — NH3 egress proxy, R14 provisioning, yt-voice-clipper CD.** + Five threads landed (detail in the cited auto-memories + runbooks + commits): + - **NH3 egress proxy (DURABLE).** dante SOCKS5 on nh3-dev `socks5h://10.100.10.50:1080`, + internal-only (client ACL `10.100.0.0/16`), systemd. NH3 **residential** egress + (~70.230.226.88) for colo services gated on their **datacenter IP**. + `scripts/setup-nh3-egress-proxy.sh` + `docs/runbooks/nh3-egress-proxy.md` (`f27ee47`); + auto-memory `reference_nh3_egress_proxy`. + - **yt-voice-clipper on irv-ml1** (`/opt/docker/compose/yt-voice-clipper`, :8000 + console+API + bgutil `pot-provider` sidecar). yt-dlp via `YTVC_PROXY` egress beats + YouTube's datacenter bot-gate (proven end-to-end). Console + audition + trim-by-playhead + deployed; Homepage label (AI Systems) in the override. **Push-to-deploy LIVE**: gitea + webhook → HMAC listener irv-ml1:9008 → git-clone deploy dir rebuilds on push to main. + `docs/runbooks/ytvc-autodeploy.md` (`d4f180d`); auto-memory `reference_ytvc_autodeploy`. + - **R14 VC-NVV tracer (brokkr-smithy)** provisioned scope-(a) on irv-ml1: box + CUDA env + + 4 engine run-envs (knn-vc/seed-vc/knn-svc/RVC, A6000-validated; one proven on a real + VIVAE conversion) + `/mnt/smithy` nfs4 mount + VIVAE-reachability. brokkr wires arms.py + + runs. Env at `~/r14`; auto-memory `reference_irv_ml1_gpu_r14`. OPEN: irv-ml1 git-pull + deploy key for `vh/brokkr-smithy` pending brokkr authorizing the pubkey (tracked in the + R14 althing thread). + - **gitea `[webhook] ALLOWED_HOST_LIST = external, 10.100.0.0/16`** on the gitea container + (ana-docker) — the anti-SSRF default `external` was denying webhooks to the private WG net, + blocking the ytvc auto-deploy. Backup `app.ini.bak-pre-webhook`. + - **irv-ml1 disk 97%→49%** — cleared all ollama models (~194 GB) + 2023-era + `/bootstore/llm-models/Storage/` LLMs (~472 GB; 2 llmuser-owned dirs needed operator sudo). + +- **chatterbox-fast — DONE** (was the primary focus). Deployed irv-ml1 :8197, cataloged + `ready`, EXTRACTED to its own MIT repo `vh/chatterbox-fast` (`547a557`) — the eshpfi stack + is now a pointer; auto-memory `reference_chatterbox_fast_repo`. Adaptive buffer-ratchet + chunking, TTFB ~0.5s, A6000 fp32 5.34 GB. _(Historical build detail follows.)_ Operator-authorized high-effort build (incl. custom container from source). **Plan-of-attack: `docs/design/chatterbox-fast-plan.md`** (durable; carries the full executable detail). The old `docs/design/chatterbox-fast.md` was KILLED 2026-06-01 — its @@ -218,6 +246,30 @@ _As of 2026-06-02:_ ## Recent decisions +- `[2026-06-03]` **yt-voice-clipper bot-gate fix = route yt-dlp through NH3 residential + egress, NOT cookies/PO-token.** YouTube hard-flags the Irvine colo IP (LOGIN_REQUIRED on a + public video even with no cookies). Cookies + the bgutil PO-token + deno JS-runtime all + loaded fine — the gate is pure IP reputation. Operator chose proxy-via-nh3-dev → durable + dante proxy → proven. The egress proxy is a reusable fleet lever for any datacenter-IP-gated + service. + +- `[2026-06-03]` **yt-voice-clipper push-to-deploy via gitea webhook** (operator-directed, + after 6 manual rebuilds in ~40 min). Webhook (not poll) — gitea CAN reach the WG IP per the + operator. The proxy env + Homepage labels live in the **host-specific override** (untracked + → survive the auto-deploy's `git reset --hard`), NOT yt-voice-clipper-dev's image. Runbook + `d4f180d`. + +- `[2026-06-03]` **R14 scope = (a) provision-only.** infra-ops provides box + CUDA env + + engines + weights + NFS; brokkr/dev wires `arms.py` + runs — keeps infra-ops OFF the + VIVAE-processing path (VIVAE = Variably Intense Vocalizations of Affect/Emotion, CHARTER §4 + highest-liability; operator authorized R&D-eval-only, quarantined). Box = irv-ml1 (A6000 + free; ana-ml2 GPU-saturated). Per-engine venvs (divergent torch stacks); A6000 = `cuda:0` + NATIVE (≠ docker `=1`). + +- `[2026-06-03]` **Declined worldtree v0.30.4 staging deploy** — that's worldtree-team's + CI/CD lane (a developer `staging/vX.Y.Z` git-tag promote), not infra-ops. They self-corrected + to the same conclusion independently. + - `[2026-06-02]` **Chatterbox → main TTS engine; build custom `chatterbox-fast` streaming container.** Workload = single-stream interactive. **GPU placement: 3090 (device 0) if it fits else A6000 (device 1)** — shared dev stack, 20.5 GB @@ -334,6 +386,29 @@ _25 older entries archived to archival-memory.md._ ## Tried and abandoned +- `[2026-06-03]` **gitea webhook to a private IP is denied by `webhook.ALLOWED_HOST_LIST`** + (anti-SSRF; default `external` blocks private/loopback). Symptom: delivery shows + `dial tcp ...: webhook can only call allowed HTTP servers`. Fix = APPEND the target net to + ALLOWED_HOST_LIST in gitea's app.ini (keep `external`; scope tight, never `*`/`private`) + + restart gitea (act_runner job containers survive a restart). gitea runs as a container on + ana-docker (`gitea_gitea_data` volume, `/data/gitea/conf/app.ini`). + +- `[2026-06-03]` **torch-2.12 venvs need `uv pip install torchcodec`** — torchaudio 2.12 + defaults to the TorchCodec backend for `.load`; without it, real audio I/O throws "TorchCodec + is required" — and it ONLY surfaces at actual conversion, NOT at import/model-load. Lesson: + validate real I/O, not just import, when provisioning ML engine envs. (seed-vc on torch 2.4 + uses the legacy backend, exempt.) + +- `[2026-06-03]` **Backgrounding `althing-cli monitor` with an inline shell `&` (instead of + the Bash-tool `run_in_background`) orphans it** — it survives the shell exit, holds the + per-handle flock UNTRACKED (won't notify the session), and `stop-monitor` doesn't detect it. + Fix: find + kill the orphan PID (verify cwd=this repo / handle first — nh3-dev is shared, other + agents' monitors run there too), then re-arm via run_in_background. Always re-arm tracked. + +- `[2026-06-03]` **`uv pip install .` fails on SmoothKen/knn-svc** (and similar script-repos) + — it's analysis scripts + a poetry pyproject, no buildable package (setuptools + package-discovery error). Install the pyproject deps directly, don't build the "package". + - `[2026-06-02]` **Fish (fish-s2 / OpenAudio S1-mini) progressive streaming — SHELVED (sub-realtime).** Benched RTF on A6000: 0.72x (12w) / 0.82x (30w) / 0.86x (60w), **mean 0.80x = sub-realtime**, so client-side chunking would starve (same reason chatterbox-fast needs turbo's RTF>1). Root cause of the buffering (dvalin-smithy-dev deep research, verified in our code text2semantic/inference.py L600-607): Fish only chunks on `<|speaker:X|>` tags; **plain text -> batches=[whole text]** -> all semantic tokens generate before any audio (chunk_length inert). Plus a 2nd layer: kui/ASGI StreamResponse doesn't flush (header produced t=1s, delivered t=23s) -> fix = anti-buffering headers (X-Accel-Buffering:no / Transfer-Encoding:chunked) in tools/server/views.py (kept on file, not applied). A rebuild does NOT fix this (current main same logic). **STANDING REVISIT TRIGGER: when an RTX Blackwell Pro lands in the fleet -> bench fp4-quantized Fish; if RTF > ~1.5x, give it the chatterbox-fast treatment** (client-side adaptive buffer-ratchet chunker driving /v1/tts with small text pieces). Projection: fp4 (~1/4 weight bytes, memory-bound AR decode) + Blackwell (GDDR7 ~1.8TB/s vs A6000 0.77TB/s, native FP4 cores) ~ 2-3x RTF; validate fp4 voice quality (ear/ECAPA) before committing. For now Fish stays a buffered catalog entry (great for SAVED gens, not the live-audition lane).