diff --git a/persistent-memory.md b/persistent-memory.md index fdc2874..ee6ae8f 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -176,6 +176,8 @@ Every vLLM seat on a shared card (GPU 0, GPU 1) has byte-pinned KV (`--kv-cache- ### Scriberr (fv-ml1 GPU 3) +- **Diarization FIXED 2026-10-06 ~1320 (Prime: "whisper doesn't work at all, need diarized transcripts").** Three faults: no `HF_TOKEN` (pyannote is gated; now passed via compose, vault `fv-ml1/hf-token`); torchcodec 0.16 in the pyannote env against torch 2.8 (`AudioDecoder` undefined; pinned to 0.7 by hand, durable as `patches/0003`, applied at the next rebuild); and the WhisperX adapter's CPU/small/float32 defaults. Both diarization paths verified on GPU 3. → `stacks/scriberr/README.md` § Diarization + - **LIVE `scriberr:local-blackwell-a353078-dropout2`:** upstream a353078 plus patch 0001 (overlap slicer) and patch 0002 (gap retry, `PARAKEET_MODEL_PATH`), carried LOCALLY ONLY (Prime 2026-10-01: no upstream). v3 stays. `scripts/scriberr-rebuild` re-applies both. → `persistent-memory.d/2026-09-30-scriberr-slicer-gap-retry-gpu3.md` - **URL is https://scriberr.nh3.phasefinal.com since 2026-10-01** (fronted by the fleet TLS caddy on nh3-dev; Caddyfile backup `.bak-20261001-scriberr`). Prime's "can't initialize recorder" was the SecureContext rule: `getUserMedia` is refused on http://. The HTTPS name is in `ALLOWED_ORIGINS` (`playbooks/scriberr-https-origin.yaml`); http://10.251.50.54:8080 still works except for recording. ⚠ `/opt/docker/compose/scriberr` is lkraven-owned, so `deploy-stack.sh` as infra-ops FAILS there (Permission denied); upload single files with `elway --upload --sudo --owner lkraven:lkraven`. diff --git a/stacks/scriberr/README.md b/stacks/scriberr/README.md index 88d0bbb..b136e87 100644 --- a/stacks/scriberr/README.md +++ b/stacks/scriberr/README.md @@ -134,6 +134,16 @@ rebuild — a local patch to carry, which is why it was not done. - **This stack is a guest on GPU1**, which it shares with the `sec` seat. If VRAM gets tight, this is the thing that should yield. +## Diarization: what broke it and what fixed it (2026-10-06) + +Prime: "whisper doesn't work at all, and I need diarized transcripts." There were three separate faults, and all three looked like "transcription is broken": + +1. **No Hugging Face token.** pyannote's diarization models are gated. Every diarized job died at the diarize step: WhisperX with `GatedRepoError 401` on `pyannote/speaker-diarization-3.1`, but only after a full CPU transcription (12m45s for a 1 h file), and Parakeet+pyannote with "HuggingFace token is required". **Fix:** `HF_TOKEN` is passed through compose. Its value is in the host `.env` only (vault `fv-ml1/hf-token`, account lkraven, read role; access to `speaker-diarization-3.1`, `segmentation-3.0` and `speaker-diarization-community-1` was checked). +2. **The separate pyannote env had a torchcodec that cannot load.** uv resolved torchcodec 0.16.0 beside torch 2.8.0+cu128. 0.16 needs torch ≥2.11 and CUDA 13 (`libnvrtc.so.13`), so pyannote.audio 4 has no decoder and fails with `name 'AudioDecoder' is not defined`. **Fix (live, by hand):** `torchcodec==0.7.*` (its table pairs 0.7 with torch 2.8), pinned in `/tank/scriberr/whisperx-env/pyannote/pyproject.toml`, then `uv lock && uv sync` as uid 10001; the old files are kept as `*.bak-20261006`. Scriberr only rewrites that pyproject if its import test fails at startup. **Durable fix:** `patches/0003-pyannote-env-pin-torchcodec.patch` pins torch/torchaudio 2.8 and torchcodec 0.7 in the embedded pyproject, so it lands at the next rebuild. +3. **WhisperX defaults to CPU.** The adapter's defaults are `device cpu`, `model small`, `compute_type float32` (`whisperx_adapter.go`). A quick transcribe with no profile ran exactly that. **Use** device `cuda`, model `large-v3`, compute `float16`. + +**Verified on a 4-min, multi-speaker clip of a real upload, GPU 3:** WhisperX large-v3/cuda/float16 + `pyannote/speaker-diarization-3.1` exited 0 in 65 s with 2 speakers. The pyannote env script with `speaker-diarization-community-1` (the adapter's only option, used by the Parakeet profile) exited 0 in 11 s with 3 speakers. The two pipelines disagree on the speaker count, so set min/max speakers when you know it. WhisperX's own env (pyannote.audio 4.0.4) loads 3.1 fine and feeds the pipeline in-memory audio, so it never needed torchcodec. + ## Optional: summarisation via the LiteLLM gateway Scriberr speaks the OpenAI API, so point it at the fleet gateway instead of a diff --git a/stacks/scriberr/compose.yaml b/stacks/scriberr/compose.yaml index f7bd424..6178430 100644 --- a/stacks/scriberr/compose.yaml +++ b/stacks/scriberr/compose.yaml @@ -72,6 +72,15 @@ services: # and takes out the Parakeet + Sortformer backends (WhisperX survives). # `copy` trades a little disk and time for it actually working. - UV_LINK_MODE=${SCRIBERR_UV_LINK_MODE:-copy} + # Hugging Face token for the GATED pyannote models (speaker-diarization-3.1, + # segmentation-3.0). Without it every diarized job dies at the diarize step: + # WhisperX with a 401 GatedRepoError AFTER the whole transcription has run, and + # Parakeet+pyannote with "HuggingFace token is required". So it looks as if + # nothing works. Scriberr's config and both adapters read HF_TOKEN + # (internal/config/config.go), and it reaches the uv subprocesses. Value: vault + # `fv-ml1/hf-token` (account lkraven, read role; access to the gated repos was + # checked 2026-10-06). It lives only in the host .env, never in git. + - HF_TOKEN=${HF_TOKEN:-} # ── Memory settings, measured while Scriberr shared GPU 1 (2026-09-30; it moved to # GPU 3 at 1322 the same day, so the 5.5 GB budget no longer binds, but the values stand). Prime: Scriberr shared GPU 1 with # intern-decision, which holds ~9.7 GB resting / 10.3 GB peak). ────────── diff --git a/stacks/scriberr/patches/0003-pyannote-env-pin-torchcodec.patch b/stacks/scriberr/patches/0003-pyannote-env-pin-torchcodec.patch new file mode 100644 index 0000000..81326ba --- /dev/null +++ b/stacks/scriberr/patches/0003-pyannote-env-pin-torchcodec.patch @@ -0,0 +1,34 @@ +Subject: [PATCH 0003] pyannote env: pin torch/torchaudio 2.8 and torchcodec 0.7 + +The embedded pyannote env leaves torchcodec unpinned (and torch >=2.5). On fv-ml1 +(2026-10-06) uv resolved torch 2.8.0+cu128 with torchcodec 0.16.0, which needs +torch >=2.11 and CUDA 13 (libnvrtc.so.13). torchcodec's import fails, so +pyannote.audio 4 has no AudioDecoder, and every Parakeet+pyannote diarization +fails with "name 'AudioDecoder' is not defined". torchcodec's own compatibility +table pairs 0.7 with torch 2.8. Pin all three together, so a fresh env resolves +the set that was verified: community-1 diarization on GPU, 3 speakers on a 4-min +clip, 11 s. + +--- +--- a/internal/transcription/adapters/py/pyannote/pyproject.toml ++++ b/internal/transcription/adapters/py/pyannote/pyproject.toml +@@ -4,8 +4,8 @@ + description = "Audio diarization using PyAnnote" + requires-python = ">=3.10" + dependencies = [ +- "torch>=2.5.0", +- "torchaudio>=2.5.0", ++ "torch==2.8.*", ++ "torchaudio==2.8.*", + "huggingface-hub>=0.28.1", + "pyannote.audio==4.0.2" + ] +@@ -18,7 +18,7 @@ + + [tool.uv] + override-dependencies = [ +- "torchcodec ; sys_platform != 'linux' or platform_machine != 'aarch64'" ++ "torchcodec==0.7.* ; sys_platform != 'linux' or platform_machine != 'aarch64'" + ] + + [tool.uv.sources] diff --git a/stacks/scriberr/patches/README.md b/stacks/scriberr/patches/README.md index 7c217c7..71a744c 100644 --- a/stacks/scriberr/patches/README.md +++ b/stacks/scriberr/patches/README.md @@ -9,6 +9,7 @@ distinctly tagged image, and proves it before anyone deploys it. |---|---|---| | `0001-parakeet-pause-aware-slicer.patch` | upstream `a353078` (HEAD 2026-09-20) | **LIVE on fv-ml1 since 2026-09-30 1211 PT** as `scriberr:local-blackwell-a353078-slicer1`; superseded in production by 0002 on top of it; **no upstream PR (Prime, 2026-10-01)**. Both patches are carried locally only | | `0002-parakeet-model-path-and-gap-retry.patch` | 0001 | **LIVE on fv-ml1 since 2026-09-30 1602 PT** as `scriberr:local-blackwell-a353078-dropout2` (Prime: "Scriberr gets the basic fix, no surgery for the new toolkit"; v3 stays). Live check: a 20-min file peaked at 5,502 MiB on GPU 3, `retried_gaps: 1`, +29 words against the slicer1 image. Why: `docs/pfi/parakeet-dropout-investigation-2026-09-30.md` | +| `0003-pyannote-env-pin-torchcodec.patch` | 0002 | **NOT in an image yet (2026-10-06).** Pins torch/torchaudio 2.8 and torchcodec 0.7 in the embedded pyannote env pyproject. The live env was fixed by hand the same way (`../README.md` § Diarization), so this only matters when the env is rebuilt or Scriberr re-runs its setup. `git apply --check` passes on a353078+0001+0002. | Rollback for the live deploy: `SCRIBERR_IMAGE=scriberr:local-blackwell` (the