diff --git a/stacks/speaches/.env.example b/stacks/speaches/.env.example new file mode 100644 index 0000000..990e21a --- /dev/null +++ b/stacks/speaches/.env.example @@ -0,0 +1,43 @@ +# speaches — irv-ml1. Copy to .env on the host and adjust. + +# ── Image ─────────────────────────────────────────────────────────────────── +# PINNED BY DIGEST, deliberately. The VAD-off setting below rides on an +# upstream variable explicitly marked unstable (`_UNSTABLE_VAD_FILTER`); an +# unpinned `:latest-cuda` could rename it on any bump, silently restoring VAD +# and shifting `no_speech_prob` semantics under Eyra's calibrated gate. +# Bumping this tag is a deliberate act that REQUIRES re-running the +# verbose_json fidelity check afterwards. Digest captured 2026-08-21. +SPEACHES_IMAGE=ghcr.io/speaches-ai/speaches@sha256:6ec12ebf890a17e0d4b242a8ba9e0eb1fb836e60e8a3c857aea9838d541579ac + +# ── Network ───────────────────────────────────────────────────────────────── +SPEACHES_PORT=8204 +SPEACHES_BIND=0.0.0.0 + +# ── GPU ───────────────────────────────────────────────────────────────────── +# 1 = RTX A6000 (26 GB free). 0 = RTX 3090, already hosting parakeet + TTS. +SPEACHES_GPU_ID=1 + +# ── Models ────────────────────────────────────────────────────────────────── +# Both tiers preloaded at startup so the first call of a meeting is not a +# cold download. Repo IDs verified against the HF API 2026-08-21. +# large-v3 (~3.1 GB fp16) — batch/accuracy tier, post-meeting job +# distil-large-v3 (~1.5 GB fp16) — low-latency draft captioning +# ⚠ distil-large-v3 is ENGLISH-ONLY. Eyra is English-only for v1 by design. +# When multilingual unparks, `Systran/faster-whisper-large-v3-turbo` is the +# multilingual replacement for the fast tier — swapping it is a .env edit. +SPEACHES_PRELOAD_MODELS=["Systran/faster-whisper-large-v3","Systran/faster-distil-whisper-large-v3"] +SPEACHES_CACHE_DIR=/worktank/speaches/cache + +# ── Inference ─────────────────────────────────────────────────────────────── +# VAD OFF — consumer requirement, do not flip without telling eyra-dev. +SPEACHES_VAD_FILTER=False +# Ampere sm_86: fp16 is correct here; no FP8/NVFP4 on this hardware. +SPEACHES_COMPUTE_TYPE=float16 +SPEACHES_NUM_WORKERS=1 +# -1 = never unload (deterministic latency). 300 = upstream default, trades +# latency for VRAM — switch if the A6000 gets tight. +SPEACHES_STT_MODEL_TTL=-1 + +# ── Misc ──────────────────────────────────────────────────────────────────── +SPEACHES_ENABLE_UI=true +SPEACHES_LOG_LEVEL=info diff --git a/stacks/speaches/README.md b/stacks/speaches/README.md new file mode 100644 index 0000000..4370d72 --- /dev/null +++ b/stacks/speaches/README.md @@ -0,0 +1,106 @@ +# speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1 + +Deployed 2026-08-21 for **Eyra** (meeting recorder) at the `eyra-dev` request. + +- **Host/GPU:** irv-ml1, **RTX A6000** (`device_ids: ["1"]` — see the GPU note below). +- **Port:** `:8204` → `http://10.100.79.3:8204`. WG/LAN-internal, **no auth** (fleet default, + agreed with the consumer for v1). +- **Image:** `ghcr.io/speaches-ai/speaches` **pinned by digest** — see "Why the digest pin". +- **Models:** `Systran/faster-whisper-large-v3` (batch tier) + + `Systran/faster-distil-whisper-large-v3` (low-latency tier). fp16, ~4.2 GB on disk, + ~5.9 GB VRAM with both resident. + +```bash +curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \ + -F "file=@clip.wav;type=audio/wav" \ + -F "model=Systran/faster-distil-whisper-large-v3" \ + -F "response_format=verbose_json" +``` + +## Why this exists next to `parakeet` (:8765) + +Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model +returning a bare `{"text": ...}`; **it has no `no_speech_prob` concept at all.** Eyra's +hallucination gate keys on per-segment `no_speech_prob`, so parakeet structurally cannot serve +it. Parakeet remains the right pick for plain text-out transcription. + +## The load-bearing requirement + +`response_format=verbose_json` must return `segments[].no_speech_prob` intact. **Verified +2026-08-21** on both tiers — every segment carries `no_speech_prob`, `avg_logprob`, +`compression_ratio`, `temperature`, `tokens`, and word timings. + +### ⚠ Measured: `no_speech_prob` alone is a WEAK hallucination gate + +Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier: + +| input | `no_speech_prob` | `avg_logprob` | `compression_ratio` | text returned | +|---|---|---|---|---| +| speech (6.6 s) | **0.018** | **-0.114** | 1.141 | correct verbatim | +| pure silence | **0.107** | **-0.650** | 0.556 | `"Thank you."` ← hallucinated | +| pink room tone | **0.077** | **-0.724** | 0.556 | `"Thank you."` ← hallucinated | + +Both non-speech inputs produced the classic Whisper `"Thank you."` hallucination, and +**`no_speech_prob` stayed under 0.11 on both** — a conventional `> 0.6` threshold would pass +them straight through. `avg_logprob` separates the same cases ~6× more decisively +(`< -0.5` catches both), and `compression_ratio` splits cleanly at 0.556 vs 1.141. + +**Recommendation to any consumer: gate on a composite, not `no_speech_prob` alone.** The field +is present and directionally correct, but its dynamic range on this stack is too compressed to +carry a threshold by itself. + +## Why VAD is pinned OFF + +`_UNSTABLE_VAD_FILTER=False`, set explicitly in `.env`, at the consumer's request. Eyra VAD-gates +upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the +audio and therefore change what `no_speech_prob` *means* per segment, underneath a gate calibrated +against this stack. + +Consequence worth stating plainly: **with VAD off, this service will happily transcribe silence +into text** (see the table above). Upstream VAD gating is what prevents that — the service is not +defending itself. + +## Why the digest pin + +`_UNSTABLE_VAD_FILTER` carries a leading underscore and the literal word "unstable" — upstream +reserves the right to rename it. On a floating `:latest-cuda`, a routine image bump could silently +drop that variable, restore VAD, and move `no_speech_prob` semantics under a calibrated consumer +gate with no error and no log line. **Bumping the pin is a deliberate act that requires re-running +the fidelity + discrimination checks above.** + +## Gotcha: `PRELOAD_MODELS` does not download + +`PRELOAD_MODELS` only *loads models already in the HF cache* — it will not fetch them. A first +boot with an empty cache starts healthy, serves `/health` 200, and exposes an **empty** +`/v1/models`. Download explicitly (URL-encode the `/` in the repo id): + +```bash +curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3" +curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3" +``` + +Related: the bind-mounted cache dir must contain a `hub/` subdirectory or **every** +`/v1/models` call 500s with `huggingface_hub.errors.CacheNotFound` while `/health` still returns +200 — a healthy container serving a broken registry. `mkdir -p /hub` owned by uid 1000. + +## GPU note — the docker/native index inversion + +`device_ids: ["1"]` is the **A6000**, verified empirically (the container reports +`NVIDIA RTX A6000`). Do not "fix" this to `0` by reading `nvidia-smi` from a native shell: +native CUDA on irv-ml1 enumerates the A6000 as `cuda:0` while the docker view has it at `1`. +`0` in compose is the RTX 3090, which already hosts parakeet and the TTS stacks. + +## Residency + +`STT_MODEL_TTL=-1` keeps both tiers resident (upstream default is 300 s). Deliberate: a +mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and +~5.9 GB against 20 GB still free is cheap. **Revisit when Eyra's diarization workload lands on +this box** — that ask is expected at their diarize milestone and will want GPU embeddings plus +gated pyannote weights. Switching to `300` trades the latency back for VRAM in one `.env` line. + +## Deploy + +```bash +scripts/deploy-stack.sh irv-ml1 speaches # diffs vs live, prompts y/N +# on host: cp .env.example .env; docker compose up -d +``` diff --git a/stacks/speaches/compose.yaml b/stacks/speaches/compose.yaml new file mode 100644 index 0000000..01a3550 --- /dev/null +++ b/stacks/speaches/compose.yaml @@ -0,0 +1,83 @@ +# speaches — OpenAI-compatible ASR (faster-whisper / CTranslate2) on irv-ml1. +# +# Consumer: Eyra (meeting recorder). Its hallucination gate keys on the +# per-segment `no_speech_prob` field returned by `response_format=verbose_json`, +# so that field surviving the API boundary is LOAD-BEARING, not cosmetic. +# +# API: +# POST /v1/audio/transcriptions — multipart; response_format=verbose_json +# GET /v1/models — registry of available STT models +# GET /health +# +# Coexists with the `parakeet` stack (:8765). Parakeet is a TDT/transducer model +# returning a bare {"text": ...} — it has no `no_speech_prob` concept at all, so +# it cannot serve this consumer. Two ASR services on one box is deliberate. +# +# All tunables live in .env — edit that, not this file. + +services: + speaches: + image: ${SPEACHES_IMAGE} + container_name: speaches + restart: unless-stopped + ports: + - "${SPEACHES_BIND:-0.0.0.0}:${SPEACHES_PORT}:8000" + environment: + - UVICORN_HOST=0.0.0.0 + - UVICORN_PORT=8000 + - LOG_LEVEL=${SPEACHES_LOG_LEVEL:-info} + # ── VAD: PINNED OFF at the consumer's explicit request ────────────── + # Eyra VAD-gates upstream on the capture edge and sends only speech + # segments. A second VAD here would re-chunk the audio and therefore + # shift what `no_speech_prob` MEANS per segment, underneath a gate + # calibrated against this stack. + # ⚠ The upstream default is True, and the variable name is explicitly + # marked unstable (leading underscore). That is exactly why + # SPEACHES_IMAGE is pinned to a digest below — an unpinned bump could + # silently rename this var, restoring VAD and moving the gate. + - _UNSTABLE_VAD_FILTER=${SPEACHES_VAD_FILTER:-False} + # ── Whisper / CTranslate2 ─────────────────────────────────────────── + # float16: irv-ml1 is Ampere (sm_86) — no native FP8/NVFP4, fp16 is the + # correct compute type here. device_index 0 is the index INSIDE the + # container, which the device_ids pin below maps to the A6000. + - WHISPER__INFERENCE_DEVICE=cuda + - WHISPER__DEVICE_INDEX=0 + - WHISPER__COMPUTE_TYPE=${SPEACHES_COMPUTE_TYPE:-float16} + - WHISPER__NUM_WORKERS=${SPEACHES_NUM_WORKERS:-1} + # ── Residency ─────────────────────────────────────────────────────── + # -1 = never unload. Deliberate: a mid-meeting model reload would be a + # multi-second stall on a latency-sensitive draft-caption tier. Both + # tiers resident is ~4.6 GB against 26 GB free on the A6000. Set to 300 + # (the upstream default) to trade that latency back for VRAM if the card + # gets tight — e.g. when Eyra's diarization workload lands here. + - STT_MODEL_TTL=${SPEACHES_STT_MODEL_TTL:--1} + - ENABLE_UI=${SPEACHES_ENABLE_UI:-true} + - PRELOAD_MODELS=${SPEACHES_PRELOAD_MODELS} + - HF_HOME=/home/ubuntu/.cache/huggingface + volumes: + # Model cache persists across recreates so a bounce is not a re-download. + - ${SPEACHES_CACHE_DIR}:/home/ubuntu/.cache/huggingface + deploy: + resources: + reservations: + devices: + - driver: nvidia + # "1" = RTX A6000 (verified empirically — nvidia-smi index 1; + # note native CUDA on this host enumerates the A6000 as 0, the + # docker view is what matters here). "0" would be the 3090, + # which already hosts parakeet + the TTS stacks. + device_ids: ["${SPEACHES_GPU_ID:-1}"] + capabilities: [gpu] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8000/health || exit 1"] + interval: 30s + timeout: 10s + retries: 3 + # First boot downloads ~4.6 GB of CT2 weights for the two preloaded tiers. + start_period: 600s + labels: + - homepage.group=AI - Audio Tools + - homepage.name=Speaches ASR + - homepage.icon=mdi-text-to-speech + - homepage.description=OpenAI-compatible faster-whisper STT, verbose_json (irv-ml1) + - homepage.href=http://10.100.79.3:${SPEACHES_PORT}