# speaches — OpenAI-compatible ASR (faster-whisper) on irv-ml1

Deployed 2026-08-21 for **Eyra** (meeting recorder) at the `eyra-dev` request.

- **Host/GPU:** irv-ml1, **RTX A6000** (`device_ids: ["1"]` — see the GPU note below).
- **Port:** `:8204` → `http://10.100.79.3:8204`. WG/LAN-internal, **no auth** (fleet default,
  agreed with the consumer for v1).
- **Image:** `ghcr.io/speaches-ai/speaches` **pinned by digest** — see "Why the digest pin".
- **Models:** `Systran/faster-whisper-large-v3` (batch tier) +
  `Systran/faster-distil-whisper-large-v3` (low-latency tier). fp16, ~4.2 GB on disk,
  ~5.9 GB VRAM with both resident.

```bash
curl -X POST http://10.100.79.3:8204/v1/audio/transcriptions \
  -F "file=@clip.wav;type=audio/wav" \
  -F "model=Systran/faster-distil-whisper-large-v3" \
  -F "response_format=verbose_json"
```

## Why this exists next to `parakeet` (:8765)

Two ASR services on one box is deliberate, not drift. Parakeet is a TDT/transducer model
returning a bare `{"text": ...}`; **it has no `no_speech_prob` concept at all.** Eyra's
hallucination gate keys on per-segment `no_speech_prob`, so parakeet structurally cannot serve
it. Parakeet remains the right pick for plain text-out transcription.

## The load-bearing requirement

`response_format=verbose_json` must return `segments[].no_speech_prob` intact. **Verified
2026-08-21** on both tiers — every segment carries `no_speech_prob`, `avg_logprob`,
`compression_ratio`, `temperature`, `tokens`, and word timings.

### ⚠ Measured: `no_speech_prob` alone is a WEAK hallucination gate

Verified against synthetic speech / silence / pink-noise room tone, VAD off, distil tier:

| input | `no_speech_prob` | `avg_logprob` | `compression_ratio` | text returned |
|---|---|---|---|---|
| speech (6.6 s) | **0.018** | **-0.114** | 1.141 | correct verbatim |
| pure silence | **0.107** | **-0.650** | 0.556 | `"Thank you."` ← hallucinated |
| pink room tone | **0.077** | **-0.724** | 0.556 | `"Thank you."` ← hallucinated |

Both non-speech inputs produced the classic Whisper `"Thank you."` hallucination, and
**`no_speech_prob` stayed under 0.11 on both** — a conventional `> 0.6` threshold would pass
them straight through. `avg_logprob` separates the same cases ~6× more decisively
(`< -0.5` catches both), and `compression_ratio` splits cleanly at 0.556 vs 1.141.

**Recommendation to any consumer: gate on a composite, not `no_speech_prob` alone.** The field
is present and directionally correct, but its dynamic range on this stack is too compressed to
carry a threshold by itself.

## Why VAD is pinned OFF

`_UNSTABLE_VAD_FILTER=False`, set explicitly in `.env`, at the consumer's request. Eyra VAD-gates
upstream on the capture edge and sends only speech segments; a second VAD here would re-chunk the
audio and therefore change what `no_speech_prob` *means* per segment, underneath a gate calibrated
against this stack.

Consequence worth stating plainly: **with VAD off, this service will happily transcribe silence
into text** (see the table above). Upstream VAD gating is what prevents that — the service is not
defending itself.

## Why the digest pin

`_UNSTABLE_VAD_FILTER` carries a leading underscore and the literal word "unstable" — upstream
reserves the right to rename it. On a floating `:latest-cuda`, a routine image bump could silently
drop that variable, restore VAD, and move `no_speech_prob` semantics under a calibrated consumer
gate with no error and no log line. **Bumping the pin is a deliberate act that requires re-running
the fidelity + discrimination checks above.**

## Gotcha: `PRELOAD_MODELS` does not download

`PRELOAD_MODELS` only *loads models already in the HF cache* — it will not fetch them. A first
boot with an empty cache starts healthy, serves `/health` 200, and exposes an **empty**
`/v1/models`. Download explicitly (URL-encode the `/` in the repo id):

```bash
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-whisper-large-v3"
curl -X POST "http://localhost:8204/v1/models/Systran%2Ffaster-distil-whisper-large-v3"
```

Related: the bind-mounted cache dir must contain a `hub/` subdirectory or **every**
`/v1/models` call 500s with `huggingface_hub.errors.CacheNotFound` while `/health` still returns
200 — a healthy container serving a broken registry. `mkdir -p <cache>/hub` owned by uid 1000.

## GPU note — the docker/native index inversion

`device_ids: ["1"]` is the **A6000**, verified empirically (the container reports
`NVIDIA RTX A6000`). Do not "fix" this to `0` by reading `nvidia-smi` from a native shell:
native CUDA on irv-ml1 enumerates the A6000 as `cuda:0` while the docker view has it at `1`.
`0` in compose is the RTX 3090, which already hosts parakeet and the TTS stacks.

## Residency

`STT_MODEL_TTL=-1` keeps both tiers resident (upstream default is 300 s). Deliberate: a
mid-meeting reload would be a multi-second stall on a latency-sensitive draft-caption tier, and
~5.9 GB against 20 GB still free is cheap. **Revisit when Eyra's diarization workload lands on
this box** — that ask is expected at their diarize milestone and will want GPU embeddings plus
gated pyannote weights. Switching to `300` trades the latency back for VRAM in one `.env` line.

## Deploy

```bash
scripts/deploy-stack.sh irv-ml1 speaches     # diffs vs live, prompts y/N
# on host: cp .env.example .env; docker compose up -d
```