29 lines
2.4 KiB
Markdown
29 lines
2.4 KiB
Markdown
# Scriberr: CUDA OOM → slicer patch → Parakeet gap retry → GPU 3 (2026-09-30)
|
||
|
||
**The OOM (0124):** Scriberr's Parakeet path hit CUDA OOM on GPU 1 beside SemIf. Prime took SemIf offline at 0135, and the 35-min job then ran clean.
|
||
- First fix, live at 0900: `PARAKEET_CHUNK_THRESHOLD_SECS=120` plus `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
|
||
- Peaks: 9,384 MiB at 300 s slices; 6,510 at 120 s; a ~5.6 GB fixed floor; 5,496 at 120 s with expandable_segments.
|
||
- Scriberr passes os.Environ() to its uv subprocess.
|
||
|
||
**Slicer patch 0001** (Prime: "build the slicer"; `docs/pfi/scriberr-slicer-bench-2026-09-30.md`):
|
||
- A 4 s overlap inside the 120 s limit, handing over at a word both chunks transcribed.
|
||
- Damaged cuts fell from 52% to 22% against a 19% background (floor ±0.08; 4 files × 3 placements).
|
||
- Pause-aware cutting measured neutral and is opt-in.
|
||
|
||
**Dropout investigation** (Prime: include a different Parakeet weight; `docs/pfi/parakeet-dropout-investigation-2026-09-30.md`):
|
||
- The v3 losses are REAL against ground truth (SCOTUS official transcript, Gutenberg #38916): 140 / 66 / 50 / 51 words per transcript.
|
||
- Cause: the v2/v3 0.6B weights collapse deep in long windows. The 1.1B models do not, but they have no punctuation.
|
||
- No decoder, context, loudness or resampling fix worked.
|
||
|
||
**Patch 0002** (gap retry plus `PARAKEET_MODEL_PATH`) re-transcribes any ≥3 s stretch of speech that produced no words, cutting the losses 80–90%.
|
||
- LIVE at 1602 as `scriberr:local-blackwell-a353078-dropout2`. Prime: "basic fix, no surgery for the new toolkit", so v3 stays.
|
||
- Rollback: `.env.bak-20260930-pre-dropout2`, or `SCRIBERR_IMAGE` set to slicer1.
|
||
|
||
**Placement:** moved to **GPU 3 at 1322** (Prime), as an on-demand tenant like Blender. It steps aside to irv-ml1's A6000 when a full-size seat claims GPU 3; the A6000 is shared with bursty ComfyUI work.
|
||
|
||
**Mechanics that survive upgrades:**
|
||
- Scriberr REWRITES its Parakeet scripts from the go:embed copy on every environment prepare, so hand edits in the env dir are wiped.
|
||
- Patches live in `stacks/scriberr/patches/` and are applied by `scripts/scriberr-rebuild`: pinned sha, `git apply --check`, a distinct tag, then the embed, unit, seam and memory stages. The budget is a 5,600 MiB regression guard.
|
||
- Upstream is quiet: 1 commit in 90 days, and the maintainer is restarting.
|
||
- The upstream PR for 0001 is prepared (`patches/upstream-pr/`) but NOT opened; that needs Prime.
|