feat(speaches): OpenAI-compatible faster-whisper ASR seat on irv-ml1 A6000
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
This commit is contained in:
@@ -0,0 +1,106 @@
|
||||
# speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1
|
||||
|
||||
Deployed 2026-08-21 for **Eyra** (meeting recorder) at the `eyra-dev` request.
|
||||
|
||||
- **Host/GPU:** irv-ml1, **RTX A6000** (`device_ids: ["1"]` — see the GPU note below).
|
||||
- **Port:** `:8204` → `http://10.100.79.3:8204`. WG/LAN-internal, **no auth** (fleet default,
|
||||
agreed with the consumer for v1).
|
||||
- **Image:** `ghcr.io/speaches-ai/speaches` **pinned by digest** — see "Why the digest pin".
|
||||
- **Models:** `Systran/faster-whisper-large-v3` (batch tier) +
|
||||
`Systran/faster-distil-whisper-large-v3` (low-latency tier). fp16, ~4.2 GB on disk,
|
||||
~5.9 GB VRAM with both resident.
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \
|
||||
-F "file=@clip.wav;type=audio/wav" \
|
||||
-F "model=Systran/faster-distil-whisper-large-v3" \
|
||||
-F "response_format=verbose_json"
|
||||
```
|
||||
|
||||
## Why this exists next to `parakeet` (:8765)
|
||||
|
||||
Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model
|
||||
returning a bare `{"text": ...}`; **it has no `no_speech_prob` concept at all.** Eyra's
|
||||
hallucination gate keys on per-segment `no_speech_prob`, so parakeet structurally cannot serve
|
||||
it. Parakeet remains the right pick for plain text-out transcription.
|
||||
|
||||
## The load-bearing requirement
|
||||
|
||||
`response_format=verbose_json` must return `segments[].no_speech_prob` intact. **Verified
|
||||
2026-08-21** on both tiers — every segment carries `no_speech_prob`, `avg_logprob`,
|
||||
`compression_ratio`, `temperature`, `tokens`, and word timings.
|
||||
|
||||
### ⚠ Measured: `no_speech_prob` alone is a WEAK hallucination gate
|
||||
|
||||
Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier:
|
||||
|
||||
| input | `no_speech_prob` | `avg_logprob` | `compression_ratio` | text returned |
|
||||
|---|---|---|---|---|
|
||||
| speech (6.6 s) | **0.018** | **-0.114** | 1.141 | correct verbatim |
|
||||
| pure silence | **0.107** | **-0.650** | 0.556 | `"Thank you."` ← hallucinated |
|
||||
| pink room tone | **0.077** | **-0.724** | 0.556 | `"Thank you."` ← hallucinated |
|
||||
|
||||
Both non-speech inputs produced the classic Whisper `"Thank you."` hallucination, and
|
||||
**`no_speech_prob` stayed under 0.11 on both** — a conventional `> 0.6` threshold would pass
|
||||
them straight through. `avg_logprob` separates the same cases ~6× more decisively
|
||||
(`< -0.5` catches both), and `compression_ratio` splits cleanly at 0.556 vs 1.141.
|
||||
|
||||
**Recommendation to any consumer: gate on a composite, not `no_speech_prob` alone.** The field
|
||||
is present and directionally correct, but its dynamic range on this stack is too compressed to
|
||||
carry a threshold by itself.
|
||||
|
||||
## Why VAD is pinned OFF
|
||||
|
||||
`_UNSTABLE_VAD_FILTER=False`, set explicitly in `.env`, at the consumer's request. Eyra VAD-gates
|
||||
upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the
|
||||
audio and therefore change what `no_speech_prob` *means* per segment, underneath a gate calibrated
|
||||
against this stack.
|
||||
|
||||
Consequence worth stating plainly: **with VAD off, this service will happily transcribe silence
|
||||
into text** (see the table above). Upstream VAD gating is what prevents that — the service is not
|
||||
defending itself.
|
||||
|
||||
## Why the digest pin
|
||||
|
||||
`_UNSTABLE_VAD_FILTER` carries a leading underscore and the literal word "unstable" — upstream
|
||||
reserves the right to rename it. On a floating `:latest-cuda`, a routine image bump could silently
|
||||
drop that variable, restore VAD, and move `no_speech_prob` semantics under a calibrated consumer
|
||||
gate with no error and no log line. **Bumping the pin is a deliberate act that requires re-running
|
||||
the fidelity + discrimination checks above.**
|
||||
|
||||
## Gotcha: `PRELOAD_MODELS` does not download
|
||||
|
||||
`PRELOAD_MODELS` only *loads models already in the HF cache* — it will not fetch them. A first
|
||||
boot with an empty cache starts healthy, serves `/health` 200, and exposes an **empty**
|
||||
`/v1/models`. Download explicitly (URL-encode the `/` in the repo id):
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3"
|
||||
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3"
|
||||
```
|
||||
|
||||
Related: the bind-mounted cache dir must contain a `hub/` subdirectory or **every**
|
||||
`/v1/models` call 500s with `huggingface_hub.errors.CacheNotFound` while `/health` still returns
|
||||
200 — a healthy container serving a broken registry. `mkdir -p <cache>/hub` owned by uid 1000.
|
||||
|
||||
## GPU note — the docker/native index inversion
|
||||
|
||||
`device_ids: ["1"]` is the **A6000**, verified empirically (the container reports
|
||||
`NVIDIA RTX A6000`). Do not "fix" this to `0` by reading `nvidia-smi` from a native shell:
|
||||
native CUDA on irv-ml1 enumerates the A6000 as `cuda:0` while the docker view has it at `1`.
|
||||
`0` in compose is the RTX 3090, which already hosts parakeet and the TTS stacks.
|
||||
|
||||
## Residency
|
||||
|
||||
`STT_MODEL_TTL=-1` keeps both tiers resident (upstream default is 300 s). Deliberate: a
|
||||
mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and
|
||||
~5.9 GB against 20 GB still free is cheap. **Revisit when Eyra's diarization workload lands on
|
||||
this box** — that ask is expected at their diarize milestone and will want GPU embeddings plus
|
||||
gated pyannote weights. Switching to `300` trades the latency back for VRAM in one `.env` line.
|
||||
|
||||
## Deploy
|
||||
|
||||
```bash
|
||||
scripts/deploy-stack.sh irv-ml1 speaches # diffs vs live, prompts y/N
|
||||
# on host: cp .env.example .env; docker compose up -d
|
||||
```
|
||||
Reference in New Issue
Block a user