# parakeet-nemo — the fleet speech seat (parakeet-unified-en under NeMo)

STT seat on fv-ml1, port 8300, behind LiteLLM as `ext-stt` / `whisper-1`; the caller is `talk`.
Switched over from the sherpa-onnx int8 seat (`stacks/parakeet`) on 2026-09-30 on Prime's order,
after the A/B in `docs/pfi/parakeet-seat-ab-2026-09-30.md` (arm B-bf16w won: p50 23/27/33 ms vs
the old seat's 144/260/565 ms at 1–3/3–8/8–20 s, lower WER on every set).

## Why this runtime

The old seat was slow because of its RUNTIME: the int8 ONNX graph ran on one CPU thread. The
defects it carried — HTTP 500 above ~400 s of audio, long-form dropouts, utterance truncation
after a 1.5 s digital-silence pause — are all int8-export behaviours. This seat runs the model
under NeMo torch with bf16 weights, full-precision mel front end, and NeMo's local-attention
long-audio mode (±128), which is what removes all three defects.

## Hard-wired seat invariants (app.py — each one is load-bearing, do not "clean up")

- **bf16 cast BEFORE `.to("cuda")`** — restoring fp32 onto the GPU and casting there spikes the
  load by ~1.5 GB. GPU 0 cannot absorb that; it is shared with two vLLM seats.
- **Warm-up at the longest served length** — the CUDA-graph greedy decoder costs ~330 ms extra on
  the first call at a new maximum length. The entrypoint warm-up runs ascending silent clips
  (`WARMUP_SECONDS`, default 1,8,60).
- **`rel_pos_local_attn` ±128** — a 30-minute file transcribes in ~2.6 s in ONE request; memory
  grows linearly in length instead of quadratically.
- **`dither = 0.0`** — dither is a training-time augmentation; it makes identical files decode
  differently call to call.
- **NeMo 3.0.0 is a floor** — released 2.7.3 lacks this encoder's `att_chunk_context_size`.
- **No httptools; `--http h11` is explicit.** httptools 0.8.0 (pulled by `uvicorn[standard]`, and
  auto-selected by uvicorn when importable) writes a NUL into the response status line —
  `HTTP/1.1 200\x00OK` — that h11/httpx reject with `RemoteProtocolError: illegal status line`.
  curl tolerates it; LiteLLM reaches this seat via httpx, so every consumer would break. Proven
  A/B on the same image: `--http h11` clean, `--http httptools` dirty (2026-10-01). The image
  installs plain `uvicorn==0.53.0` for exactly this reason.

## License

`nvidia/parakeet-unified-en-0.6b` is distributed under the **NVIDIA Open Model License Agreement**
(commercial/non-commercial use permitted; Prime accepted the terms 2026-09-30). This replaces the
CC-BY-4.0 terms of the previous seat's weights for this service. Internal use: no NOTICE file
required; keep this section as the license note. Weights pinned at HF revision
`fe53cd885760c96b6a5f51a0bfd362cb4584a98b` (sha256 `ec23ed91…`), mounted read-only from
`/tank/aimodels/huggingface`, `HF_HUB_OFFLINE=1`.

## GPU 0 room

The seat rests ~2.1 GB. **Since nemo-0.1.1 it returns to rest after long requests**: the
windowed path calls `torch.cuda.empty_cache()` around each window, so a 12-min file peaks at
~3,028 MiB during the request and falls back to ~2,108 after (measured 2026-10-01, restarts=0).
Before 0.1.1 the seat PARKED at the window peak (3,582 MiB steady), and that cached peak left
gen-small no room for its runtime workspace — an EngineCore CUDA-OOM incident at 04:21 PT.
Three seat-side controls, all load-bearing:
- **`MEM_CAP_MIB=3840`** (env, default): a hard `set_per_process_memory_fraction` ceiling. An
  over-cap request answers **503** with the seat still alive (proved at cap=2000: two 503s, then
  short requests fine) — the failure lands on us, never on a neighbour's allocation.
- **`CUDA_GRAPHS=0`** (default): the CUDA-graph greedy decoder pins memory in torch's cache and
  died with an illegal-memory-access the first time `empty_cache` freed a graph-pool block.
  Graphs off costs latency (12-min file 3.0 s vs 1.2 s; short bins 35-62 ms vs 33-42 ms — still
  4-15x faster than the sherpa seat) and buys a lower, honestly-returned footprint.
- **`WINDOW_S=360`** (default): see the windowing note above; mask is T×T even under local attention.
Room came from `vllm-gen-small`: `--gpu-memory-utilization` 0.48 → 0.33 (config-only, applies at
its next restart; KV byte-pinned, boot log identical at 670,142 tokens / 2.56×). gen-small's
runtime growth was measured nvidia-smi-per-process, not computed: 3 realistic requests
(1,351 prompt / ~180 completion tokens) moved it ZERO from its 36,116 MiB — the workspace
allocation lands at engine init, right after restart (infra-ops saw the growth window at
restart+3-requests). ⚠ Before restarting ANY vLLM seat on this card, do the boot-check
arithmetic against measured `nvidia-smi` Free: required = util × total, available = Free + that
seat's own resident memory. GPU 1 is NOT an option: its free memory is intern-decision's 32k
headroom.

## Rollback

The old seat was STOPPED, not removed: `docker stop parakeet-nemo && docker start parakeet`
restores the sherpa seat on :8300 exactly as before (container and image both kept).

## Deploy

Build on fv-ml1 in a versioned dir under `/opt/docker/src/` (house convention), tag
`local/parakeet-nemo:nemo-X.Y.Z`, point `.env` at it, `docker compose up -d`. Acceptance harness
and the A/B's paired latency/WER tooling: `/tank/spikes/parakeet-ab` on fv-ml1 (do not delete).