fix(scriberr): diarization works again: HF_TOKEN passthrough, pyannote env torchcodec pin (patch 0003); mirror retention; YaRN stays off
Scriberr: pyannote's diarization models are gated, and Scriberr had no Hugging Face token, so every diarized job died at the diarize step (WhisperX after a full CPU transcription). HF_TOKEN now passes through compose; the value is only in the host .env (vault fv-ml1/hf-token). The separate pyannote env had torchcodec 0.16 (needs torch >=2.11/CUDA 13) beside torch 2.8, so pyannote 4 had no AudioDecoder. Pinned to 0.7 by hand on the live env; patch 0003 makes it durable at the next rebuild. WhisperX's adapter defaults to cpu/small/float32, now documented. Both paths verified on GPU 3 with a real multi-speaker clip. pbs-nh3: the backups-mirror prune job now keeps the primary's history (last 3, daily 14, weekly 8, monthly 12, yearly 5). Dry-run: keep 738, remove 3,194 across 32 groups; first run at the job's 00:00.
This commit is contained in:
@@ -134,6 +134,16 @@ rebuild — a local patch to carry, which is why it was not done.
|
||||
- **This stack is a guest on GPU1**, which it shares with the `sec` seat. If
|
||||
VRAM gets tight, this is the thing that should yield.
|
||||
|
||||
## Diarization: what broke it and what fixed it (2026-10-06)
|
||||
|
||||
Prime: "whisper doesn't work at all, and I need diarized transcripts." There were three separate faults, and all three looked like "transcription is broken":
|
||||
|
||||
1. **No Hugging Face token.** pyannote's diarization models are gated. Every diarized job died at the diarize step: WhisperX with `GatedRepoError 401` on `pyannote/speaker-diarization-3.1`, but only after a full CPU transcription (12m45s for a 1 h file), and Parakeet+pyannote with "HuggingFace token is required". **Fix:** `HF_TOKEN` is passed through compose. Its value is in the host `.env` only (vault `fv-ml1/hf-token`, account lkraven, read role; access to `speaker-diarization-3.1`, `segmentation-3.0` and `speaker-diarization-community-1` was checked).
|
||||
2. **The separate pyannote env had a torchcodec that cannot load.** uv resolved torchcodec 0.16.0 beside torch 2.8.0+cu128. 0.16 needs torch ≥2.11 and CUDA 13 (`libnvrtc.so.13`), so pyannote.audio 4 has no decoder and fails with `name 'AudioDecoder' is not defined`. **Fix (live, by hand):** `torchcodec==0.7.*` (its table pairs 0.7 with torch 2.8), pinned in `/tank/scriberr/whisperx-env/pyannote/pyproject.toml`, then `uv lock && uv sync` as uid 10001; the old files are kept as `*.bak-20261006`. Scriberr only rewrites that pyproject if its import test fails at startup. **Durable fix:** `patches/0003-pyannote-env-pin-torchcodec.patch` pins torch/torchaudio 2.8 and torchcodec 0.7 in the embedded pyproject, so it lands at the next rebuild.
|
||||
3. **WhisperX defaults to CPU.** The adapter's defaults are `device cpu`, `model small`, `compute_type float32` (`whisperx_adapter.go`). A quick transcribe with no profile ran exactly that. **Use** device `cuda`, model `large-v3`, compute `float16`.
|
||||
|
||||
**Verified on a 4-min, multi-speaker clip of a real upload, GPU 3:** WhisperX large-v3/cuda/float16 + `pyannote/speaker-diarization-3.1` exited 0 in 65 s with 2 speakers. The pyannote env script with `speaker-diarization-community-1` (the adapter's only option, used by the Parakeet profile) exited 0 in 11 s with 3 speakers. The two pipelines disagree on the speaker count, so set min/max speakers when you know it. WhisperX's own env (pyannote.audio 4.0.4) loads 3.1 fine and feeds the pipeline in-memory audio, so it never needed torchcodec.
|
||||
|
||||
## Optional: summarisation via the LiteLLM gateway
|
||||
|
||||
Scriberr speaks the OpenAI API, so point it at the fleet gateway instead of a
|
||||
|
||||
Reference in New Issue
Block a user