memory: snapshot — U11a off + U11b gate; SemIf→intern-decision (Jev, 32k); Scriberr GPU 3 + slicer + gap retry; Parakeet seat switch approved for next session; 26 entries archived

2026-09-30 23:52:27 -07:00
parent 212b736836
commit 1ae324d576
27 changed files with 1608 additions and 1467 deletions
@@ -0,0 +1,28 @@
+# Scriberr: CUDA OOM → slicer patch → Parakeet gap retry → GPU 3 (2026-09-30)
+
+**The OOM (0124):** Scriberr's Parakeet path hit CUDA OOM on GPU 1 beside SemIf. Prime took SemIf offline at 0135, and the 35-min job then ran clean.
+- First fix, live at 0900: `PARAKEET_CHUNK_THRESHOLD_SECS=120` plus `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
+- Peaks: 9,384 MiB at 300 s slices; 6,510 at 120 s; a ~5.6 GB fixed floor; 5,496 at 120 s with expandable_segments.
+- Scriberr passes os.Environ() to its uv subprocess.
+
+**Slicer patch 0001** (Prime: "build the slicer"; `docs/pfi/scriberr-slicer-bench-2026-09-30.md`):
+- A 4 s overlap inside the 120 s limit, handing over at a word both chunks transcribed.
+- Damaged cuts fell from 52% to 22% against a 19% background (floor ±0.08; 4 files × 3 placements).
+- Pause-aware cutting measured neutral and is opt-in.
+
+**Dropout investigation** (Prime: include a different Parakeet weight; `docs/pfi/parakeet-dropout-investigation-2026-09-30.md`):
+- The v3 losses are REAL against ground truth (SCOTUS official transcript, Gutenberg #38916): 140 / 66 / 50 / 51 words per transcript.
+- Cause: the v2/v3 0.6B weights collapse deep in long windows. The 1.1B models do not, but they have no punctuation.
+- No decoder, context, loudness or resampling fix worked.
+
+**Patch 0002** (gap retry plus `PARAKEET_MODEL_PATH`) re-transcribes any ≥3 s stretch of speech that produced no words, cutting the losses 80–90%.
+- LIVE at 1602 as `scriberr:local-blackwell-a353078-dropout2`. Prime: "basic fix, no surgery for the new toolkit", so v3 stays.
+- Rollback: `.env.bak-20260930-pre-dropout2`, or `SCRIBERR_IMAGE` set to slicer1.
+
+**Placement:** moved to **GPU 3 at 1322** (Prime), as an on-demand tenant like Blender. It steps aside to irv-ml1's A6000 when a full-size seat claims GPU 3; the A6000 is shared with bursty ComfyUI work.
+
+**Mechanics that survive upgrades:**
+- Scriberr REWRITES its Parakeet scripts from the go:embed copy on every environment prepare, so hand edits in the env dir are wiped.
+- Patches live in `stacks/scriberr/patches/` and are applied by `scripts/scriberr-rebuild`: pinned sha, `git apply --check`, a distinct tag, then the embed, unit, seam and memory stages. The budget is a 5,600 MiB regression guard.
+- Upstream is quiet: 1 commit in 90 days, and the maintainer is restarting.
+- The upstream PR for 0001 is prepared (`patches/upstream-pr/`) but NOT opened; that needs Prime.