No-sudo phase: uv, R16 world venv + pyworld (setuptools<81 pin for
pyworld's pkg_resources import). Sudo phase (run as infra-ops): R +
soundgen system deps, and pin the Ollama service to the A6000 BY UUID
(CUDA FASTEST_FIRST ordering != nvidia-smi PCI order on this box, so an
index pin is ambiguous). R15 models share the A6000's free headroom —
yt-voice-clipper + R14 keep running, card not freed.
Adds a host-agnostic elway play + fleet driver that stand up an
`infra-ops` system user (dedicated ed25519 key, NOPASSWD sudo with
log_output audit, docker group) so the infra-ops agent completes
DevOps work end-to-end instead of handing sudo steps back to the
operator. Scoped to PFI-owned Linux boxes; tiered (compute/app/
sensitive-infra) with SureFire/corviduo/esh/Synology explicitly
excluded. Validated live on irv-ml1.
worldtree-dev sent infra-ops a manual deploy request for v0.30.6 to
personal:8081; operator confirmed all three corviduo-dev Worldtree
deployments go through the Worldtree team's CI/CD, never a manual
infra-ops deploy. The old 'personal — manually managed' line was wrong
and misleading. docker-as-root pattern is emergency-ops (key restore)
only.
Auto-deploy on push to main: gitea webhook → HMAC-validated listener on irv-ml1:9008
→ git fetch/reset + docker compose up -d --build. Documents the gitea-server
ALLOWED_HOST_LIST anti-SSRF relaxation (scoped to the WG net), the irv-ml1 components
(deploy key, git-clone deploy dir preserving the proxy override/secrets, listener +
user service), and verify/debug steps.
YouTube (and a growing set of services) hard-flag datacenter IPs, bot-gating
even public content regardless of cookies/PO-tokens. Origin case: yt-voice-clipper
on irv-ml1 (Irvine colo) — every yt-dlp fetch returned LOGIN_REQUIRED. Confirmed
pure IP reputation: the same public video fetches cleanly (no cookies) once routed
through nh3-dev's residential egress (70.230.226.88).
- scripts/setup-nh3-egress-proxy.sh: idempotent dante (SOCKS5) install + config.
Internal-only ACL (10.100.0.0/16), bound to the WG interface, systemd-managed.
- docs/runbooks/nh3-egress-proxy.md: purpose, usage, security model, caveats.
Reusable fleet egress, not yt-voice-clipper-specific.
chatterbox-fast is authored software with a test suite, not a config-mirror stack —
so it moves to its own MIT-licensed, versioned, CI'd repo (gitea vh/chatterbox-fast,
v0.1.0) following the sister-repo pattern. Replace stacks/chatterbox-fast/ with a
pointer README; the moved code (scheduler/app/bench/tests/Dockerfile/compose) now
lives in the new repo. The deployed :8197 service is unaffected (still runs the
legacy devnen-based image; self-contained-image migration is an optional follow-up).
The fleet catalog entry stays in docs/asset-engine/services.yaml.
- chatterbox-fast experimental -> ready: browser audition verified end-to-end
(operator confirmed progressive playback "excellent" 2026-06-02).
- vibevoice ready -> down: no container running on irv-ml1 (connection refused);
catalog status was stale.
- voxtral: NOT a stale typo — its stack genuinely claimed :8197, the port now held
by the live chatterbox-fast. voxtral is down, so moved IT to :8201 (catalog
endpoint + source_url, stacks/voxtral/.env.example + README, host .env) rather
than disturb the live service. No live clash existed (voxtral down) but it was a
latent deploy-time collision I introduced by placing chatterbox-fast on 8197.
No catalog_version bump (status changes + endpoint correction, additive). Validates
against the schema.
Land chatterbox-fast in the asset-engine catalog as an additive service, per
asset-engine-dev's shipped streaming-audition path (asset-engine v0.1.17-19):
- streamable:true -> UI routes Generate to an ephemeral progressive-<audio>
audition (no Job/Asset); re-run on `chatterbox` to keep output.
- New service-level `streamable` bool added to services.schema.json (additive,
default false; mirrors the Pydantic model asset-engine-dev regenerates).
- Fields: text, voice (select via /voices), temperature/top_p/top_k/
repetition_penalty/seed, format (pcm default; UI forces wav). exaggeration/
cfg_weight omitted — Turbo ignores them.
- status experimental until the first real browser audition verifies progressive
playback (the one thing asset-engine-dev couldn't machine-verify).
- reproducibility + audit entries added. No catalog_version bump (additive).
Validates against the updated schema.
TTSRequest gains `seed` (0=random); seeded once per request under the lock via
torch.manual_seed + cuda.manual_seed_all. One-shot output is then byte-reproducible
for a fixed seed+params (verified: seed=42 -> identical sha256 across runs).
Streaming stays non-reproducible by design — adaptive-chunk boundaries depend on
live-measured RTF. Needed for the asset-engine catalog reproducibility contract
(parity with the chatterbox sibling, which exposes seed).
stream=false + format=wav emitted the streaming 0xFFFFFFFF-length header, so a
buffered consumer reading a complete wav got bogus RIFF/data sizes. One-shot knows
the full length, so emit correct sizes; streaming keeps the open-ended header
(length genuinely unknown up front). Verified remote: one-shot wav data size ==
bytes-44, python wave.open() reads 2.20s cleanly; streaming still 0xFFFFFFFF.
Deployed on irv-ml1 beside live chatterbox (:8196): healthy on :8197, TTFB ~0.5s,
no starvation. Measured VRAM 5.34 GB (fp32) settles the placement: the 3090's
~3.8 GB free does NOT fit, A6000 (device 1) is the only viable card.
Revert the priming feature from d707439. Live A/B caught an audible artifact: the
context-priming discard-cut left part of the throwaway prefix in the output, so a
clause ("...without a trace of sarcasm,") was spoken an extra time.
Root cause is structural: generate() returns one finished waveform with no marker
for where the prefix ends, and the model renders the same prefix with different
timing when followed by content than when generated solo — so the duration-estimate
+ energy-minimum cut is a guess and can leave a sliver (or a whole clause) of prefix
in. A reliable cut would need token-level access (the abandoned native-streaming
arc) or a per-chunk ASR/alignment pass (heavy, still imperfect, eats the latency
budget). Fails the agreed bar: "keep only if it closes the gap without a seam."
Kept from d707439: the .gitignore (build artifacts). NOT re-applied: the bundled
margin_first fix — wiring it would shrink chunk 1 (more joins = worse coherence),
against the operator's priority, and margin=0.8 there is already starvation-safe.
Coherence loss at joins stays an accepted limitation; cold streaming was judged
"really good". Phase 1 + Phase 2 parity/perf untouched. Next: Phase 3 deploy.
Prime early joins by prepending the prior sentence as backward prosodic context,
generating context+content together, then discarding the context audio. The cut
snaps to the inter-sentence pause (energy-minimum search around the context's
solo duration) with a 5ms fade-in to kill any seam click (app: _cut_at_pause /
_fade_in / Engine.generate_primed). Opt-in via request `prime` (default off).
Scheduler: priming is AFFORDABILITY-GATED so it can never starve. A primed chunk
costs ~(2·context + content)/rtf (a 2nd context-solo pass); a chunk is only primed
when buffer ≥ prime_buffer_factor (1.5) × that cost, else it falls back to a cold
generate. Consequences proven in the GPU-free sim (17 tests):
- fires on early joins for any GPU at/above rtf_prior (3.4 = 3090; A6000 ~3.8-4.0)
- self-skips (degrades to cold) on a slower-than-fleet GPU rather than starving
- never primes chunk 0 (latency-critical)
Also fixed a latent Phase-1 bug: margin_first was applied at chunk 0 (budget always
0 there) so it never did anything — now applied at chunk 1 (the first transition).
Live A/B on irv-ml1 (A6000, GLaDOS): TTFB unaffected (445 vs 467ms), no starvation;
priming fired on chunk 2 (gen 1.6s for the doubled pass). On typical text exactly
ONE early join safely primes — priming chunk 2 flattens the buffer so later/larger
chunks no longer clear the safety gate. Samples: ~/chatterbox-ab/_p2_{cold,primed}.wav.
- /voices endpoint lists predefined voice stems (excludes `_`-prefixed bench/A-B
scratch wavs); shared _predefined_wavs() also feeds default-voice discovery.
- Perf levers: TF32 matmul/cudnn + flash/mem-efficient SDPA, default ON, env-gated
(CBF_TF32 / CBF_SDPA_FLASH). Startup logs model dtype.
Measured on irv-ml1 (turbo, A6000): the model loads FLOAT32 (not the fp16 older
notes assumed). TF32+SDPA do NOT move TTFA (489->514ms, noise) — first-sentence
latency is bound by the sequential AR token decode at batch-1, not matmul
throughput. bf16 (the lever that would help) is DEFERRED: from_pretrained() has no
dtype arg and turbo's fp32 conditioning path + dtype-sensitive vocoder make a
clean cast nontrivial; not worth the quality risk at ~0.5s TTFA. torch.compile
also deferred (batch-1 regression). Findings recorded in README.
Voice management parity (predefined dir + per-request clone refs) was already in
the Phase-1 resolve path; /voices completes the surface.
Build the streaming TTS server MVP per docs/design/chatterbox-fast-plan.md §4.
- scheduler.py: adaptive buffer-ratchet chunker (the meat) — GPU-free pure
logic. First sentence emitted alone for low TTFA, then chunks ratchet ~3x by
packing whole sentences to margin x buffered-audio; drives off measured RTF +
sec/char (EMA). relieve_leader() clause-splits a too-big mid-stream sentence
to avoid starvation (joins land on commas); a long comma-less sentence is the
one honored-but-flagged limitation.
- test_scheduler.py: GPU-free simulation, 13 tests — asserts no-starvation
(incl. overestimated RTF) and the ratchet.
- app.py: FastAPI model holder + POST /tts StreamingResponse (raw PCM s16le
default, wav optional, stream/oneshot) + GET /health.
- bench.py: client — ground-truth TTFB + real 1x-consumer starvation check.
Live test on irv-ml1 (turbo, A6000, GLaDOS voice): streaming TTFB 499ms vs
oneshot 5230ms (~10x), stayed ahead of a 1x player (no starvation), ratchet
1.64->4.08->8.60->8.60s audio, measured RTF self-corrected 3.38->4.01.
Kill the superseded docs/design/chatterbox-fast.md — its §5 windowed-token
streaming was the abandoned native-frame-streaming arc; the adaptive-chunk plan
supersedes it. Repoint persistent-memory + README at the canonical plan.
Self-contained build plan for the chatterbox-fast streaming engine: the
adaptive buffer-ratchet chunking design, validated turbo API + facts, the
GPU-1 dev/test container pattern, 4 build phases, the base-fork A/B, and
watch-outs (incl. native-turbo-streaming is abandoned). Intended for a
fresh-context session to execute at full strength.
asset-engine shipped the per-field enable-toggle (v0.1.14/.15) — the
durable fix for the "form submits untouched fields" family. A field
marked togglable:true renders with an OFF-by-default switch: while off
the control is disabled (excluded from submission) AND the server skips
injecting its default, so it is genuinely not sent until the user opts
in.
Per operator direction, opt fish-s2's `references` (inline-base64
Custom-clone) field in — it already satisfies the togglable-requires-
optional validator (optional:true, no default). The advanced clone
field now renders dormant and can never silently override the Voice
dropdown again.
This is a SCHEMA change (new CatalogField property), so:
- services.schema.json: add `togglable` (boolean, default false),
mirroring the asset_engine Pydantic model that generates this schema.
- catalog_version 1 -> 2 (header: bump on schema changes).
- CATALOG-CONTRACT.md: consumer pin note -> catalog_version=2.
Scoped to `references` only. The chatterbox/dia2 clone fields are the
same family but NOT toggled: dia2 deliberately defaults voice_mode=clone
+ a clone ref as its stable out-of-box voice, and toggling that field
would change dia2's default-voice behavior (the earlier 404 fix).
Validated: jsonschema accepts togglable; additionalProperties:false
guard still rejects unknown props.
reference_id=<name> resolves against the DIRECTORY references/<name>/
(audio + same-basename .lab), not a flat references/<name>.wav. Voices
were staged flat with the per-name dirs left empty, so every
reference_id resolved to nothing and Fish fell back to its default
speaker — every dropdown voice produced byte-identical audio (proven:
Abigail == Imogen == no-ref, same text+seed). This was the real "no
accent" root cause, independent of the asset-engine "undefined" select
bug.
Server fix (applied to irv-ml1): populated references/<name>/<name>.wav
+ <name>.lab for all 32 voices; re-test confirms Imogen/Eleanor/
Beatrice/Abigail/no-ref now all distinct.
Durable hardening + record correction:
- playbook: normalize-layout step (flat <name>.wav -> nested dir, cp -u
idempotent, when-gated on count mismatch) + an A/B verify gate that
hard-fails the deploy if two reference_ids yield identical output.
- services.yaml: correct the reference_id resolution doc (dir + .lab,
not flat wav).
- README + persistent-memory: correct the "reference_id-by-name is THE
working path, verified" claim — it was a no-op until this fix; the
prior ECAPA 0.79 result came through the inline base64 path.
Pitch-shifted deepening (rubberband, formant on/off) sounded bad at every
depth (tuba / over-gravelly), so abandoned. Removed Imogen_Contralto from the
dropdown + deleted the staged variants (fish + chatterbox). Plain unmodified
Imogen remains. version 5->6.
Staged consenting VCTK Southern-England female speakers (p225/p228/p229, CC BY
4.0) as subtle-British-accent clone voices — repurposed from the on-host kyutai
tts-voices cache. Named neutrally; NOT modeled on or representing any public
figure. Added to the reference_id dropdown (32 voices total). version 3->4.
Staged 28 single-speaker dia voices + glados into /worktank/fish-s2/references/
(internal research use). Discovered the path-form references shape 500s on this
build; reference_id (by name) is the working voice path (verified live). So:
reference_id -> select 'Voice' with the 29 staged names (default Emily, female);
references demoted to advanced inline-base64 custom clones with the path->500
caveat documented. vram 10->24 (measured ~25GB in use); version 2->3.
Fish-S2 rendered as an essentially blank form — only text + a references JSON
blob — despite being the fleet's richest-control engine. Expose the real
fish-speech ServeTTSRequest levers: temperature/top_p/repetition_penalty,
latency (normal|balanced), seed, format (wav/mp3/opus), + advanced cloning
(references/reference_id) and chunking (max_new_tokens/chunk_length/normalize/
use_memory_cache). Defaults from upstream schema.py, verified live 2026-06-01
(no /openapi.json; Kui server). Sections basic/sampling/advanced.
Also corrected: seedable false->true (/tts has seed); cold_start 8s->240s
(measured compile warmup); vram 4->10GB (~9GB BF16 weights); dropped the
misleading '~150ms TTFB' for honest ~realtime throughput. version 1->2.
Chatterbox was producing poor output because the catalog pointed at the thin
OpenAI /v1/audio/speech endpoint, which exposes none of Resemble's emotion/
pacing knobs — and the devnen server's shipped default exaggeration is 1.3
(tuned for its theatrical demo presets), which over-acts.
Re-point to the wrapper's richer /tts and expose the real control surface
(exaggeration, cfg_weight, temperature, speed_factor, seed, voice_mode),
mirroring the sibling dia stack (same devnen author). Defaults sourced live
2026-06-01: exaggeration + cfg_weight = 0.5 (Resemble README 'works well for
most prompts'), temperature 0.8 / speed 1.0 / seed 0 (server generation_
defaults). The shipped 1.3 exaggeration is deliberately NOT adopted.
Voices: expose the 28 built-in predefined voices via /get_predefined_voices
(default Emily.wav, the server default_voice_id) + clone via /get_reference_
files — replacing the wrong 'OpenAI aliases only' claim. Corrected seedable:
false -> true (/tts has seed) and image_tag_mutable -> true (:latest). Bumped
service version 1 -> 2 (breaking field-shape change); status down -> ready
(live + healthy). catalog_version unchanged (no new field types).
Tear down the parked CSM stack (status: down, never successfully built).
Bring-up attempts failed at the image build: upstream
phildougherty/sesame_csm_openai pins no huggingface_hub version, which now
resolves to 1.17.0 where the `huggingface-cli` the Dockerfile relies on has
been removed (replaced by `hf`). Building would require vendoring + patching
the upstream Dockerfile.
Deep-research verdict (primary + community sourced) confirmed it isn't worth
that: the acclaimed Maya/Miles demo runs a fine-tuned, larger CSM variant
Sesame never open-sourced; the open csm-1b is the un-fine-tuned 1B base
(only the smallest of 1B/3B/8B shipped, no newer checkpoint as of mid-2026).
Ships no usable voices, can't generate text, English-only, can't stream
real-time out of the box; absent from current TTS leaderboards and dominated
by Kokoro/Dia2/Fish-S2/IndexTTS for narration.
Removes: stacks/csm/, playbooks/deploy-csm.yaml, the csm catalog service +
reproducibility_audit entries. Host state (compose dir, /worktank/csm) torn
down on irv-ml1; no container/image existed.
Capture this session's durable state: Dia2 live as dia2-2b/dia2-1b (legacy
dia retired), dia2 catalog on /tts with full controls + stable-voice default,
zonos REST adapter (built, stack down), schema regen, all pushed. New
foot-guns: dia2 image-build quirks, predefined-voices-not-in-/tts, voice_mode
clone 'undefined' 404. Archived 6 older entries (4 decisions, 2 foot-guns).
Reported: dia2-1b 404 'Reference audio file not found: undefined' when
accepting defaults — voice_mode=clone with no clone_reference_filename made
the UI submit the literal 'undefined'. Fix: voice_mode now defaults to clone
AND clone_reference_filename defaults to Abigail.wav (a staged voice), so the
out-of-box request is a valid, stable voice. Reproduced the 404 and verified
the Abigail.wav path returns 200. Folded into v2 (not yet consumed downstream).
Pre-fill the bare input textareas with format-demonstrating samples
(dia2-2b/dia2-1b: [S1]/[S2] dialogue + a nonverbal; csm: conversational;
zonos: expressive multilingual) and give csm.topk a standard default (50).
Addresses asset-engine-dev's best-practice-defaults request so the picker
forms pre-fill usefully instead of empty. No version bumps — these entries
ship fresh in this batch (no prior consumer-pinned shape).
Replace the single dia entry (legacy Dia 1.6B, retired) with two fixed-model
Dia2 entries (dia2-2b :8200, dia2-1b :8202), status ready (both exercised),
image local/dia:v2. Matching reproducibility_audit rows. catalog_version
unchanged (add/remove services = no vocab change).
Also fix a port collision I introduced earlier: the zonos-api adapter and
csm both claimed 8201 — move zonos-api to 8203 (catalog endpoint + voices
source_url, zonos .env.example, README).
NOTE FOR CONSUMERS: removing the dia id is a breaking catalog change for
asset_engine (it vendored dia in v0.1.4) — re-vendor + drop the dia tile,
add the two dia2 tiles.
The devnen wrapper is single-model and ignores the OpenAI model field, so
offering both Dia2 models to asset-engine as real per-request choices means
one fixed-model instance per model. Rework the dia stack to run two services
from a dia2-capable image:
* dia2-2b (:8200, best quality), dia2-1b (:8202, streaming) — both GPU 0
* each pins its model via a mounted /opt/docker/conf/dia2-*/config.yaml
Retire the legacy Dia 1.6B service.
New dia2-image/Dockerfile builds local/dia:v2 = upstream devnen wrapper +
the dia2 package (copied into site-packages; its pyproject build backend
yields an empty UNKNOWN wheel under the base's old setuptools) + the three
missing runtime deps (transformers/sphn/whisper-timestamped); torch 2.12 /
numpy 2.2 in the base already satisfy Dia2. Both instances verified
end-to-end (HTTP 200, Ogg/Opus 24 kHz).
zonos: new tts entry routing to the REST adapter on 8201, JSON-envelope
response with reproducibility.seed_field=seed (seedable+deterministic),
lifecycle block, section groups. Matching reproducibility_audit row.
dia: voice select had a default (S1) but no options/source_url -> empty
picker; add options [S1, S2, dialogue] (per asset-engine-dev), bump dia
to version 2. Closed select drops clone-by-filename free-text; flagged
inline for a future source_url-backed picker. catalog_version unchanged
(add-service + field-options = no vocab change).
Resolves the stale-schema gap asset-engine-dev flagged (the published
schema rejected the lifecycle field 12/14 live services already use) and
adds reproducibility.seed_field so a seedable engine can declare which
response key carries the seed used. Authoritative regen path remains
dump_schema.py against asset_engine/catalog.py; reconcile there.
Upstream Zonos ships only Gradio + Python SDK — no REST surface — so
asset-engine (which routes a clean JSON POST to /v1/audio/speech) can't
target it directly. Add a thin FastAPI adapter (stacks/zonos/adapter/):
POST /v1/audio/speech in front of the Zonos SDK, built FROM local/zonos
to reuse torch/CUDA/SDK. Returns a JSON envelope {audio, audio_format,
seed} — the seed rides back so asset-engine regenerate/fork can pin it
(Zonos is the fleet's first genuinely seedable TTS). compose gains a
zonos-api service on 8201; .env.example gains the port + voices dir.
Upstream Zyphra/Zonos ships no CMD in its Dockerfile (it launches the
app from its own compose), so our container ran the NVIDIA entrypoint,
printed the CUDA banner, exited 0, and restart-looped — nothing ever
bound 7860/8199. Add command: python3 gradio_interface.py to match
upstream, plus an explicit GRADIO_SHARE=False. Built + deployed to
irv-ml1; 8199 now serves HTTP 200 and the transformer model loads.
dia (:8200) — OpenAI-compat /v1/audio/speech, seedable (not byte-exact),
Apache-2.0 weights. Clean catalog fit; flip to ready after first exercise.
csm (:8201) — OpenAI-compat, but NO seed + temperature-sampled =
non-reproducible (contract's fix-before-adding case), catalogued by
operator direction with a reproducibility caveat + gated-license warning;
belongs at experimental once running.
Fields read from each wrapper's API docs (2026-05-31), to confirm against
live OpenAPI/Pydantic at deploy. No catalog_version bump (add-service =
no bump). NOTE: services.schema.json is stale (pre-existing — 13 errors;
live catalog uses lifecycle, schema predates it); regen via dump_schema.py.
Sesame CSM-1B via phildougherty/sesame_csm_openai — OpenAI-compat
/v1/audio/speech, context-aware conversational speech (voice-agent
layer, not a plain reader). Port 8201 on irv-ml1. Gated model:
requires CSM_HF_TOKEN (license acceptance) — placeholder in .env.example,
real token only in host .env.
dia: Nari Labs dialogue TTS (Dia 1.6B / Dia2-1B / Dia2-2B) via
devnen/Dia-TTS-Server — OpenAI-compat, fills the multi-speaker
dialogue-scene slot for skaldsong. Port 8200 on irv-ml1.
zonos: Zyphra Zonos-v0.1 (Apache-2.0, 44kHz, emotion sliders) via the
official Gradio interface. Audition surface only — no OpenAI-compat
endpoint yet (needs the FastAPI fork to become skaldsong-pluggable).
Port 8199 on irv-ml1.
Both follow the chatterbox/fish-s2 convention: local image built from a
pinned wrapper SHA via buildx git-context, .env-driven port/GPU, python
healthcheck, homepage labels.
override.conf already existed on esh-docker-vm (dockerd ExecStart), so
the creates: guard skipped the step and the After=remote-fs.target
directive was never written. Use a distinct filename
(10-after-remote-fs.conf) that systemd merges alongside override.conf,
and guard on content rather than file existence so a stale file can't
silently skip.
Adds _netdev,nofail to the 4 10.0.50.50 NFS lines in /etc/fstab and a
docker.service drop-in (After=remote-fs.target) so Docker waits for the
NFS mounts before starting NFS-bind containers. Root cause of paperless
Exited(255) on every reboot. Idempotent; backs up fstab; no reboot.
vh