vh
aa5863c9a3
Deployed for Eyra (meeting recorder) per the eyra-dev request. Serves
large-v3 (batch tier) + distil-large-v3 (low-latency tier) on :8204,
fp16, both resident, ~5.9 GB VRAM against 20 GB still free.
Sits alongside the existing parakeet stack (:8765) deliberately: parakeet
is a TDT/transducer returning bare {"text": ...} and has no no_speech_prob
concept, so it structurally cannot serve this consumer.
The load-bearing requirement -- segments[].no_speech_prob surviving
response_format=verbose_json -- is VERIFIED on both tiers.
Measured finding worth more than the deployment: no_speech_prob alone is a
WEAK hallucination gate on this stack. Pure silence and pink room tone both
produced the classic Whisper 'Thank you.' hallucination while no_speech_prob
stayed under 0.11 -- a conventional >0.6 threshold passes both through.
avg_logprob separates the same cases ~6x more decisively (-0.11 speech vs
-0.65/-0.72 non-speech) and compression_ratio splits 1.141 vs 0.556.
Consumers should gate on a composite, not no_speech_prob alone. Table in
the README.
VAD pinned OFF at the consumer's request (they VAD-gate upstream on the
capture edge). Consequence stated plainly in the README: with VAD off this
service will transcribe silence into text and is not defending itself.
Image pinned BY DIGEST rather than :latest-cuda, because the VAD-off
setting rides on _UNSTABLE_VAD_FILTER -- a variable upstream explicitly
marks unstable. A floating tag could rename it on any bump, silently
restoring VAD and moving no_speech_prob semantics under a calibrated gate
with no error and no log line.
Two deployment gotchas recorded: PRELOAD_MODELS only loads models already
cached (it does not download -- use POST /v1/models/{id}), and the bind-
mounted cache needs a hub/ subdir or every /v1/models call 500s with
CacheNotFound while /health still returns 200.
107 lines
5.3 KiB
Markdown
107 lines
5.3 KiB
Markdown
# speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1
|
||
|
||
Deployed 2026-08-21 for **Eyra** (meeting recorder) at the `eyra-dev` request.
|
||
|
||
- **Host/GPU:** irv-ml1, **RTX A6000** (`device_ids: ["1"]` — see the GPU note below).
|
||
- **Port:** `:8204` → `http://10.100.79.3:8204`. WG/LAN-internal, **no auth** (fleet default,
|
||
agreed with the consumer for v1).
|
||
- **Image:** `ghcr.io/speaches-ai/speaches` **pinned by digest** — see "Why the digest pin".
|
||
- **Models:** `Systran/faster-whisper-large-v3` (batch tier) +
|
||
`Systran/faster-distil-whisper-large-v3` (low-latency tier). fp16, ~4.2 GB on disk,
|
||
~5.9 GB VRAM with both resident.
|
||
|
||
```bash
|
||
curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \
|
||
-F "file=@clip.wav;type=audio/wav" \
|
||
-F "model=Systran/faster-distil-whisper-large-v3" \
|
||
-F "response_format=verbose_json"
|
||
```
|
||
|
||
## Why this exists next to `parakeet` (:8765)
|
||
|
||
Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model
|
||
returning a bare `{"text": ...}`; **it has no `no_speech_prob` concept at all.** Eyra's
|
||
hallucination gate keys on per-segment `no_speech_prob`, so parakeet structurally cannot serve
|
||
it. Parakeet remains the right pick for plain text-out transcription.
|
||
|
||
## The load-bearing requirement
|
||
|
||
`response_format=verbose_json` must return `segments[].no_speech_prob` intact. **Verified
|
||
2026-08-21** on both tiers — every segment carries `no_speech_prob`, `avg_logprob`,
|
||
`compression_ratio`, `temperature`, `tokens`, and word timings.
|
||
|
||
### ⚠ Measured: `no_speech_prob` alone is a WEAK hallucination gate
|
||
|
||
Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier:
|
||
|
||
| input | `no_speech_prob` | `avg_logprob` | `compression_ratio` | text returned |
|
||
|---|---|---|---|---|
|
||
| speech (6.6 s) | **0.018** | **-0.114** | 1.141 | correct verbatim |
|
||
| pure silence | **0.107** | **-0.650** | 0.556 | `"Thank you."` ← hallucinated |
|
||
| pink room tone | **0.077** | **-0.724** | 0.556 | `"Thank you."` ← hallucinated |
|
||
|
||
Both non-speech inputs produced the classic Whisper `"Thank you."` hallucination, and
|
||
**`no_speech_prob` stayed under 0.11 on both** — a conventional `> 0.6` threshold would pass
|
||
them straight through. `avg_logprob` separates the same cases ~6× more decisively
|
||
(`< -0.5` catches both), and `compression_ratio` splits cleanly at 0.556 vs 1.141.
|
||
|
||
**Recommendation to any consumer: gate on a composite, not `no_speech_prob` alone.** The field
|
||
is present and directionally correct, but its dynamic range on this stack is too compressed to
|
||
carry a threshold by itself.
|
||
|
||
## Why VAD is pinned OFF
|
||
|
||
`_UNSTABLE_VAD_FILTER=False`, set explicitly in `.env`, at the consumer's request. Eyra VAD-gates
|
||
upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the
|
||
audio and therefore change what `no_speech_prob` *means* per segment, underneath a gate calibrated
|
||
against this stack.
|
||
|
||
Consequence worth stating plainly: **with VAD off, this service will happily transcribe silence
|
||
into text** (see the table above). Upstream VAD gating is what prevents that — the service is not
|
||
defending itself.
|
||
|
||
## Why the digest pin
|
||
|
||
`_UNSTABLE_VAD_FILTER` carries a leading underscore and the literal word "unstable" — upstream
|
||
reserves the right to rename it. On a floating `:latest-cuda`, a routine image bump could silently
|
||
drop that variable, restore VAD, and move `no_speech_prob` semantics under a calibrated consumer
|
||
gate with no error and no log line. **Bumping the pin is a deliberate act that requires re-running
|
||
the fidelity + discrimination checks above.**
|
||
|
||
## Gotcha: `PRELOAD_MODELS` does not download
|
||
|
||
`PRELOAD_MODELS` only *loads models already in the HF cache* — it will not fetch them. A first
|
||
boot with an empty cache starts healthy, serves `/health` 200, and exposes an **empty**
|
||
`/v1/models`. Download explicitly (URL-encode the `/` in the repo id):
|
||
|
||
```bash
|
||
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3"
|
||
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3"
|
||
```
|
||
|
||
Related: the bind-mounted cache dir must contain a `hub/` subdirectory or **every**
|
||
`/v1/models` call 500s with `huggingface_hub.errors.CacheNotFound` while `/health` still returns
|
||
200 — a healthy container serving a broken registry. `mkdir -p <cache>/hub` owned by uid 1000.
|
||
|
||
## GPU note — the docker/native index inversion
|
||
|
||
`device_ids: ["1"]` is the **A6000**, verified empirically (the container reports
|
||
`NVIDIA RTX A6000`). Do not "fix" this to `0` by reading `nvidia-smi` from a native shell:
|
||
native CUDA on irv-ml1 enumerates the A6000 as `cuda:0` while the docker view has it at `1`.
|
||
`0` in compose is the RTX 3090, which already hosts parakeet and the TTS stacks.
|
||
|
||
## Residency
|
||
|
||
`STT_MODEL_TTL=-1` keeps both tiers resident (upstream default is 300 s). Deliberate: a
|
||
mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and
|
||
~5.9 GB against 20 GB still free is cheap. **Revisit when Eyra's diarization workload lands on
|
||
this box** — that ask is expected at their diarize milestone and will want GPU embeddings plus
|
||
gated pyannote weights. Switching to `300` trades the latency back for VRAM in one `.env` line.
|
||
|
||
## Deploy
|
||
|
||
```bash
|
||
scripts/deploy-stack.sh irv-ml1 speaches # diffs vs live, prompts y/N
|
||
# on host: cp .env.example .env; docker compose up -d
|
||
```
|