feat(diariz): pilot stack on fv-ml1 GPU 3 — override (GPU pin, api port), vault-rendered .env, nightly idle-gated worker restart, fleet caddy site

2026-10-19 10:50:56 -07:00
parent 2929bfc60c
commit 9aecb677a8
10 changed files with 245 additions and 0 deletions
@@ -0,0 +1,63 @@
+# Diariz (pilot) — live diarized meeting transcription on fv-ml1 GPU 3
+
+**What:** [kenhayward/Diariz](https://github.com/kenhayward/Diariz) (AGPL-3.0). It does live speaker-labelled captions in the
+browser, audio/video uploads with a full-recording re-pass, and summaries/minutes/actions through the LiteLLM gateway.
+It is a pilot **alongside Scriberr, never instead of it**. Prime uses Scriberr live in meetings, and its idle gate stands.
+Brokkr's request for Prime (2026-10-19, thread `01M5AH9QCVW7ZQ6VVXGWQGFPRS`). Prime chose GPU 3 at 1035 2026-10-19.
+
+**URL:** https://diariz.nh3.phasefinal.com (fleet TLS caddy on nh3-dev → `fv-ml1:8081`). Login `lkraven@lkraven.com`,
+password `secret get diariz/seed-password`.
+
+## Layout on fv-ml1
+
+| | |
+|---|---|
+| source | `/opt/docker/src/diariz-5a96cad`: a clean `git archive` of main `5a96cade77ba` (2026-10-16), owned by infra-ops. Upstream ships no images, so it is built locally |
+| compose | upstream `deploy/docker-compose.yml` + our `deploy/docker-compose.override.yml` (this dir). `/opt/docker/compose/diariz` is a **symlink** to that deploy dir |
+| .env | `deploy/.env` (0600). Rendered from the vault by `render-env.sh` and piped over ssh; never on disk on nh3-dev. Key list: `.env.example` |
+| project | `diariz`: postgres (pgvector pg16), redis 8, SeaweedFS S3, api (.NET), web (nginx), worker, live-worker (profile on) |
+| GPU | worker + live-worker pinned to **GPU 3** by `device_ids` (upstream reserves `count: all`). Scriberr shares the card (about 5.5 GB per job) |
+| ports | web `0.0.0.0:8081`; api `127.0.0.1:18080` (upstream's 8080 is Scriberr's); postgres `127.0.0.1:5433`; S3 `127.0.0.1:9002` |
+| data | named volumes `diariz_pgdata`, `diariz_s3data` (recordings), `diariz_workercache` (model weights), `diariz_apikeys`, `diariz_redisdata`. ⚠ Not in the restic set yet: it's a pilot |
+
+## Fleet settings (the why)
+
+- **Summaries:** `gen` (flash-next, 262,144 window) with `CHAT_CONTEXT_LENGTH=245760`, which leaves 16k for the reply. The key is LiteLLM alias **`diariz`**
+  (models `gen` + `whisper-1` only), vaulted as `diariz/litellm-key`.
+- **Dictation** (chat voice input only): `whisper-1` = the gateway's Parakeet alias. Meeting transcription runs on the worker's own
+  WhisperX large-v3 and does **not** use the Parakeet seat.
+- **Embeddings off.** Diariz's vector column is 768-d and the gateway's `qwen3-embedding` is 1024-d, so `EMBED_*` is unset and search is lexical.
+- **MCP OAuth off** (`MCP_OAUTH_ENABLED=false`). Prime can turn it on.
+- **HF token:** `fv-ml1/hf-token`, because `pyannote/speaker-diarization-3.1` is gated.
+
+## Memory behaviour and the nightly restart
+
+Read at source (`src/Diariz.Worker/pipeline.py`, `worker.py`, `gpu_memory.py`):
+- The models load **lazily on the first job** and stay loaded. Each job's working memory is freed afterwards.
+- The live worker drops its copy only when live transcription is switched off in the app.
+
+So `diariz-nightly-restart.timer` (03:00) restarts both workers to drop the models. It is **idle-gated**: GPU 3 is sampled for
+20 s, and any sample above 10% (a Diariz or a Scriberr job) means wait 15 min, retrying until 0500, then skip.
+Playbook: `playbooks/fv-ml1-diariz-nightly-restart.yaml`. Log: `journalctl -u diariz-nightly-restart`.
+
+## Operate
+
+```bash
+ssh infra-ops@10.251.50.54
+cd /opt/docker/compose/diariz        # = /opt/docker/src/diariz-5a96cad/deploy
+docker compose ps
+docker compose logs -f worker live-worker
+docker compose up -d                 # after a .env change (re-render it from nh3-dev first)
+```
+
+**Upgrade:** a new `git archive` of a new pin into `/opt/docker/src/diariz-<sha7>`. Copy the override and re-render the `.env`,
+`docker compose build`, `down` in the old dir, `up -d` in the new one (same project name, so the volumes carry over), then repoint the symlink.
+Upstream ships very often: pin a commit and read its migrations notes first.
+
+**Rollback / remove:** `docker compose down` in the deploy dir (add `-v` only to throw the data away). Disable the timer, then remove
+the `@diariz` block from `configs/fleet-tls-caddy/Caddyfile`.
+
+## Known limits (not deploy bugs)
+
+No YouTube/URL ingest (files only). Upstream issue #736: speaker-count instability. Desktop capture apps are Windows/macOS only;
+on Linux, capture runs in the browser. Quality is unmeasured, which is the point of the pilot.