From f2792183d44e044adc05c4252443bd3b5d80735a Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 26 Sep 2026 00:41:01 -0700 Subject: [PATCH] feat(nh3-ml1): LFM2.5-VL-3B (llama.cpp) + VibeVoice-ASR-Streaming-1.5B (audio.cpp) utility seats For brokkr's dataset foundry (operator-approved 2026-09-26, relayed). - stacks/lfm-vl-seat: llama.cpp server-cuda b11176 (digest-pinned), Q5_K_M + mmproj Q8_0, :8030; gateway alias lfm25-vl-3b (LiteLLM restarted, 36 s). Positive control exact; null control shows it describes a missing image. - stacks/vibevoice-asr-seat: audio.cpp v0.8.2-audio8-perf-hotfix (the GGUF's own runtime, not vibevoice.cpp) on cuda 12.8 runtime + libgomp + libsoxr, sha256-pinned; :8031 direct. LibriSpeech WER 3/69, RTF 0.07-0.14; ~31 s cold first request. --- persistent-memory.md | 1 + servers/nh3-ml1/README.md | 8 +++ stacks/lfm-vl-seat/.env.example | 14 +++++ stacks/lfm-vl-seat/README.md | 23 ++++++++ stacks/lfm-vl-seat/compose.yaml | 62 ++++++++++++++++++++++ stacks/litellm/conf/config.yaml | 14 +++++ stacks/vibevoice-asr-seat/.env.example | 2 + stacks/vibevoice-asr-seat/Dockerfile | 16 ++++++ stacks/vibevoice-asr-seat/README.md | 30 +++++++++++ stacks/vibevoice-asr-seat/compose.yaml | 50 +++++++++++++++++ stacks/vibevoice-asr-seat/conf/server.json | 15 ++++++ 11 files changed, 235 insertions(+) create mode 100644 stacks/lfm-vl-seat/.env.example create mode 100644 stacks/lfm-vl-seat/README.md create mode 100644 stacks/lfm-vl-seat/compose.yaml create mode 100644 stacks/vibevoice-asr-seat/.env.example create mode 100644 stacks/vibevoice-asr-seat/Dockerfile create mode 100644 stacks/vibevoice-asr-seat/README.md create mode 100644 stacks/vibevoice-asr-seat/compose.yaml create mode 100644 stacks/vibevoice-asr-seat/conf/server.json diff --git a/persistent-memory.md b/persistent-memory.md index 0f4f070..ebf9022 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -203,6 +203,7 @@ in `servers/pfi-gx10/README.md`. → `persistent-memory.d/2026-09-24-gx10-ac-res ## Recent decisions +- `[2026-09-26]` **Two dataset-foundry utility seats LIVE on nh3-ml1 (brokkr; operator approval relayed):** LFM2.5-VL-3B on llama.cpp `:8030` (gateway `lfm25-vl-3b`, LiteLLM restarted 36 s at 0039) and VibeVoice-ASR-Streaming-1.5B on **audio.cpp** `:8031`, not vibevoice.cpp as specced; the GGUF card names audio.cpp. Controls: VL read a synthetic image exactly, but hallucinates when no image is sent; ASR WER 3/69 on the bundled LibriSpeech clips. libsoxr added (linear resampling misheard a word). First ASR request ~31 s cold. Answered brokkr: the old 27B image seat (10.250.50.54:8015) was retired 09-14; baseline = gateway `image-judge` (Flash-Next). - `[2026-09-26]` **Coder seat STAYS on fv-ml1 (Prime).** The nh3-ml1 copy gave the same quality (teacher-forced true-code logprob diff +0.008 ± 0.019) but ran ~5× slower (64-tok FIM ~1.0 s vs 0.2 s; 63 vs 338 tok/s), and freeing 6.3 GB on fv-ml1 GPU 1 (20 GB spare) bought little. Copy removed; recipe kept in `stacks/coder-seat/`. The RTX 2000E suits embed/rerank/classify, not latency-sensitive generation. Only coder was a candidate: parakeet stays (earlier ruling), voices is a generation seat. - `[2026-09-25]` **nh3-ml1 LIVE after the NH3 visit (SB off, IGFX restored): driver + CT 109 + TEI, parity vs esh-ml1 indistinguishable (cos min 0.999993 = own floor; overlap@10 1.000 vs MRL-256 control 0.684), same speed; monitoring/DNS wired. Autonomous: lxc-pve 6.0.0-2 upgrade on nh3-pve (Docker-in-LXC fix #7006). Gateway routing + AMT follow-up are Prime's.** → `persistent-memory.d/2026-09-25-nh3-ml1-live.md` - `[2026-09-25]` **nh3-ml1 = CT 109 @ 10.100.50.80, second TEI embed/rerank backend on nh3-pve's RTX 2000E; GPU playbooks made host-generic — BUILD DEFERRED: blocked on nh3-pve Secure Boot until the NH3 visit; SB handling and gateway routing are Prime's calls.** Tracked: `7ddd116`, Current state. → `persistent-memory.d/2026-09-25-nh3-ml1-standup.md` diff --git a/servers/nh3-ml1/README.md b/servers/nh3-ml1/README.md index 331cad0..4b633d4 100644 --- a/servers/nh3-ml1/README.md +++ b/servers/nh3-ml1/README.md @@ -33,6 +33,14 @@ Prime's call (recommendation: load-share; see below). VRAM ~2.7 GB for both, so ~13 GB is free. +**Dataset-foundry utility seats (2026-09-26, brokkr; operator-approved):** +- `stacks/lfm-vl-seat`: LFM2.5-VL-3B on llama.cpp, `:8030`, gateway `lfm25-vl-3b`. +- `stacks/vibevoice-asr-seat`: VibeVoice-ASR-Streaming-1.5B on audio.cpp, + `:8031`, direct only. + +GPU total with TEI is ~7.5 of 16 GB. Both are batch workloads, which suits this +card. + **Tried and removed (2026-09-25):** a copy of the code-completion seat (Qwen2.5-Coder-1.5B). It gave the same quality but ran about 5× slower than on fv-ml1 (a 64-token FIM completion took ~1.0 s vs ~0.2 s), so it stays on fv-ml1 diff --git a/stacks/lfm-vl-seat/.env.example b/stacks/lfm-vl-seat/.env.example new file mode 100644 index 0000000..4dc5725 --- /dev/null +++ b/stacks/lfm-vl-seat/.env.example @@ -0,0 +1,14 @@ +# lfm-vl-seat tunables (nh3-ml1). Copy to `.env` on the server. + +# llama.cpp CUDA server, pinned by digest: build b11176 (commit f805c57a2), +# published 2026-09-25. Ada sm_89 is in the CUDA build's target set. +LLAMACPP_IMAGE=ghcr.io/ggml-org/llama.cpp@sha256:1f4b9cf58982dd4d7cc497aea31b1a456ca9a3a1f94f527d317d3fdee0d60ab6 + +VL_PORT=8030 +VL_ALIAS=lfm25-vl-3b +VL_MODEL_FILE=LFM2.5-VL-3B-Q5_K_M.gguf +VL_MMPROJ_FILE=mmproj-LFM2.5-VL-3B-Q8_0.gguf + +# 2 parallel slots sharing 16k context (8k each): room for a few images plus text. +VL_CTX=16384 +VL_PARALLEL=2 diff --git a/stacks/lfm-vl-seat/README.md b/stacks/lfm-vl-seat/README.md new file mode 100644 index 0000000..9f0d5b6 --- /dev/null +++ b/stacks/lfm-vl-seat/README.md @@ -0,0 +1,23 @@ +# lfm-vl-seat + +**LiquidAI LFM2.5-VL-3B**, a small vision-language model, on **nh3-ml1**, served +by llama.cpp (`ghcr.io/ggml-org/llama.cpp` server-cuda, build b11176, pinned by +digest) on `:8030`. A utility seat for brokkr's dataset foundry (image +understanding). Operator-approved 2026-09-26, relayed by brokkr-smithy-dev. + +| | | +|---|---| +| direct | `http://10.100.50.80:8030/v1/chat/completions`, model `lfm25-vl-3b` | +| gateway | `lfm25-vl-3b` (ana-docker:4000, `hosted_vllm/`, `supports_vision`) | +| files | `LFM2.5-VL-3B-Q5_K_M.gguf` + **`mmproj-LFM2.5-VL-3B-Q8_0.gguf` (required for images)** from `LiquidAI/LFM2.5-VL-3B-GGUF` @ `6f730e9a2c45` | +| VRAM | ~2.9 GB (2 slots × 8k ctx) | + +**Checks, 2026-09-26.** +- Positive control: a synthetic image (red square, blue circle, the text + "NH3 42") was described exactly at T=0, both direct and through the gateway. +- Speed: prefill ~800 tok/s, decode ~92 tok/s. +- ⚠ **Null control: with no image attached it confidently describes one anyway** + ("a solid black background"). Callers must make sure the image actually went in. + +Not wired yet: `LiquidAI/LFM2.5-VL-3B-DSpark-GGUF`, a speculative-decoding +drafter (brokkr's "optional later"). diff --git a/stacks/lfm-vl-seat/compose.yaml b/stacks/lfm-vl-seat/compose.yaml new file mode 100644 index 0000000..8cb0604 --- /dev/null +++ b/stacks/lfm-vl-seat/compose.yaml @@ -0,0 +1,62 @@ +# lfm-vl-seat — LiquidAI LFM2.5-VL-3B (vision-language) on nh3-ml1 (CT 109 on +# nh3-pve, RTX 2000E Ada 16 GB), llama.cpp server. Utility seat for brokkr's +# dataset foundry (image understanding), operator-approved 2026-09-26 (relayed +# by brokkr-smithy-dev). +# +# llama-server LFM2.5-VL-3B Q5_K_M + mmproj Q8_0 → /v1/chat/completions +# (OpenAI, with image_url content parts) :8030 +# +# ⚠ --mmproj is REQUIRED for image input; without it the model is text-only and +# says nothing about it. +# Batch work, not latency-critical: this card decodes ~5x slower than an fv-ml1 +# Blackwell (servers/nh3-ml1/README.md), which is fine for a foundry. +# +# Files (not in git): /opt/aimodels/gguf/lfm25-vl-3b/ from +# LiquidAI/LFM2.5-VL-3B-GGUF @ 6f730e9a2c45. +name: lfm-vl-seat +services: + llama-server: + image: ${LLAMACPP_IMAGE} + container_name: lfm-vl + restart: unless-stopped + ports: + - "${VL_PORT}:8080" + volumes: + - /opt/aimodels/gguf/lfm25-vl-3b:/models:ro + command: + - -m + - /models/${VL_MODEL_FILE} + - --mmproj + - /models/${VL_MMPROJ_FILE} + - --alias + - ${VL_ALIAS} + - --host + - 0.0.0.0 + - --port + - "8080" + - -ngl + - "999" + - -c + - "${VL_CTX}" + - -np + - "${VL_PARALLEL}" + - --jinja + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=VL — LFM2.5-VL-3B (llama.cpp, nh3-ml1) + - homepage.icon=mdi-image-search + - homepage.description=Image understanding for the dataset foundry (gateway lfm25-vl-3b) + - homepage.href=http://10.100.50.80:${VL_PORT} diff --git a/stacks/litellm/conf/config.yaml b/stacks/litellm/conf/config.yaml index c735a9c..698cef3 100644 --- a/stacks/litellm/conf/config.yaml +++ b/stacks/litellm/conf/config.yaml @@ -512,6 +512,20 @@ model_list: model_info: mode: rerank + # --- lfm25-vl-3b → LiquidAI LFM2.5-VL-3B (Q5_K_M + mmproj Q8_0), llama.cpp server on + # nh3-ml1 :8030 (stacks/lfm-vl-seat, 2026-09-26). Image understanding for + # brokkr's dataset foundry; send images as OpenAI image_url parts. A small VLM: + # with NO image attached it confidently describes one anyway (measured), so + # callers must check the image actually went in. Batch-grade speed (~90 tok/s). --- + - model_name: lfm25-vl-3b + litellm_params: + model: hosted_vllm/lfm25-vl-3b + api_base: http://10.100.50.80:8030/v1 + api_key: os.environ/VLLM_API_KEY + model_info: + mode: chat + supports_vision: true + # --- coder-fast → Qwen2.5-Coder-1.5B (BASE), FIM code-completion seat (ana-ml2 # GPU1 :8020, vLLM; deep-research pick 2026-07-27). For Zed editor inline # edit-predictions via the LEGACY /v1/completions endpoint with Qwen FIM diff --git a/stacks/vibevoice-asr-seat/.env.example b/stacks/vibevoice-asr-seat/.env.example new file mode 100644 index 0000000..c43f844 --- /dev/null +++ b/stacks/vibevoice-asr-seat/.env.example @@ -0,0 +1,2 @@ +# vibevoice-asr-seat tunables (nh3-ml1). Copy to `.env` on the server. +ASR_PORT=8031 diff --git a/stacks/vibevoice-asr-seat/Dockerfile b/stacks/vibevoice-asr-seat/Dockerfile new file mode 100644 index 0000000..312f81b --- /dev/null +++ b/stacks/vibevoice-asr-seat/Dockerfile @@ -0,0 +1,16 @@ +# audio.cpp server (0xShug0/audio.cpp) for VibeVoice-ASR-Streaming on nh3-ml1. +# The release's Linux CUDA build ("cuda12.8-colab") is a bare binary that needs the +# CUDA 12 runtime libs (cudart, cublas, cufft, nccl) plus libgomp. This image +# supplies them, plus libsoxr (without it audio.cpp falls back to linear +# resampling for the 16k→24k step). Built locally on the host; not pushed. +FROM nvidia/cuda:12.8.1-runtime-ubuntu22.04 +ARG AUDIOCPP_TAG=v0.8.2-audio8-perf-hotfix +ARG AUDIOCPP_SHA256=ccfb35869d67e520801981392189a1ab8b6c829151093e0816ecd6a2d828d8c8 +RUN apt-get update && apt-get install -y --no-install-recommends libgomp1 libsoxr0 curl ca-certificates \ + && rm -rf /var/lib/apt/lists/* +RUN curl -fsSL -o /tmp/a.tgz "https://github.com/0xShug0/audio.cpp/releases/download/${AUDIOCPP_TAG}/audio-${AUDIOCPP_TAG}-bin-ubuntu-x64-cuda12.8-colab.tar.gz" \ + && echo "${AUDIOCPP_SHA256} /tmp/a.tgz" | sha256sum -c - \ + && mkdir -p /app && tar xzf /tmp/a.tgz -C /app --no-same-owner && rm /tmp/a.tgz \ + && chmod +x /app/audiocpp_server +WORKDIR /app +ENTRYPOINT ["/app/audiocpp_server"] diff --git a/stacks/vibevoice-asr-seat/README.md b/stacks/vibevoice-asr-seat/README.md new file mode 100644 index 0000000..07782f1 --- /dev/null +++ b/stacks/vibevoice-asr-seat/README.md @@ -0,0 +1,30 @@ +# vibevoice-asr-seat + +**Microsoft VibeVoice-ASR-Streaming-1.5B** (Q4_K) on **nh3-ml1**, served by +**audio.cpp** (`0xShug0/audio.cpp` v0.8.2-audio8-perf-hotfix, Linux CUDA 12.8 +build) on `:8031`, with direct access only. A utility seat for brokkr's dataset +foundry (speech → text with inline speaker labels). Operator-approved 2026-09-26, +relayed by brokkr-smithy-dev. + +- `POST /v1/audio/transcriptions`: multipart `file=@x.wav`, `model=vibevoice-asr-streaming-1.5b`. + Returns `{"text": " \n Speaker 0:...", "timing": {"rtf": ...}}`. +- `POST /v1/audio/transcriptions/live?model=…&sample_rate=16000&channels=1&sample_format=s16le` + takes chunked raw PCM. +- `GET /v1/models`, `GET /health`. + +**Why audio.cpp and not vibevoice.cpp or llama.cpp.** The +`christopherthompson81/VibeVoice-ASR-Streaming-1.5B-GGUF` files are audio.cpp +packages with their sidecars embedded, per the model card, so no separate +tokenizer is needed. The release binary expects the CUDA 12 runtime libraries, +so `Dockerfile` puts it on `nvidia/cuda:12.8.1-runtime` with a SHA-256-pinned +download, plus `libgomp1` and **`libsoxr0`**. Without libsoxr it falls back to +linear resampling (16k→24k), and on one clip that turned "cutter" into "country". + +**Checks, 2026-09-26.** On the 4 LibriSpeech clips shipped with audio.cpp, WER +is 3/69 = 4.35%, identical across 3 reps. Two of the three errors are "I'm" vs +"I am" normalization. RTF is 0.07–0.14. ⚠ **The first request after a start +takes ~31 s** (CUDA graph warmup); later ones take 0.3–1 s. VRAM ~2.0 GB. + +The model card recommends Q8_0 (3.3 GB; its CUDA WER is 5.80% vs Q4_K's 7.25%, +on 69 words). To switch, change `path` in `conf/server.json`, download the file, +and recreate the container. diff --git a/stacks/vibevoice-asr-seat/compose.yaml b/stacks/vibevoice-asr-seat/compose.yaml new file mode 100644 index 0000000..a3da884 --- /dev/null +++ b/stacks/vibevoice-asr-seat/compose.yaml @@ -0,0 +1,50 @@ +# vibevoice-asr-seat — Microsoft VibeVoice-ASR-Streaming-1.5B on nh3-ml1 (CT 109, +# RTX 2000E Ada 16 GB), served by audio.cpp. Utility seat for brokkr's dataset +# foundry (speech → text, speaker-attributed), operator-approved 2026-09-26 +# (relayed by brokkr-smithy-dev). +# +# audiocpp_server vibevoice-asr-streaming-1.5b (Q4_K) :8031 +# POST /v1/audio/transcriptions (OpenAI-style, whole file) +# POST /v1/audio/transcriptions/live?... (chunked 16 kHz s16le PCM) +# GET /v1/models, /health +# +# Runtime is audio.cpp, per the GGUF's own model card (the GGUF is an audio.cpp +# package with its sidecars embedded, so no separate tokenizer file). It is NOT +# vanilla llama.cpp and NOT mudler's vibevoice.cpp. +# Direct access only; no gateway entry. +# +# Files (not in git): /opt/aimodels/gguf/vibevoice-asr-streaming-1.5b/ from +# christopherthompson81/VibeVoice-ASR-Streaming-1.5B-GGUF @ ff9615110299. +name: vibevoice-asr-seat +services: + audiocpp: + build: + context: . + image: local/audiocpp:v0.8.2-audio8-perf-hotfix + container_name: vibevoice-asr + restart: unless-stopped + command: ["--config", "/config/server.json", "--no-ui", "--log"] + ports: + - "${ASR_PORT}:8080" + volumes: + - /opt/aimodels/gguf/vibevoice-asr-streaming-1.5b:/models:ro + - /opt/docker/conf/vibevoice-asr-seat/server.json:/config/server.json:ro + deploy: + resources: + reservations: + devices: + - driver: nvidia + device_ids: ["0"] + capabilities: [gpu] + healthcheck: + test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 120s + labels: + - homepage.group=AI - Eval & Retrieval + - homepage.name=ASR — VibeVoice Streaming 1.5B (audio.cpp, nh3-ml1) + - homepage.icon=mdi-microphone-message + - homepage.description=Streaming speech-to-text for the dataset foundry (direct, no gateway) + - homepage.href=http://10.100.50.80:${ASR_PORT}/v1/models diff --git a/stacks/vibevoice-asr-seat/conf/server.json b/stacks/vibevoice-asr-seat/conf/server.json new file mode 100644 index 0000000..bcc27ba --- /dev/null +++ b/stacks/vibevoice-asr-seat/conf/server.json @@ -0,0 +1,15 @@ +{ + "host": "0.0.0.0", + "port": 8080, + "backend": "cuda", + "threads": 6, + "models": [ + { + "id": "vibevoice-asr-streaming-1.5b", + "family": "vibevoice_asr_streaming", + "path": "/models/vibevoice-asr-streaming-1.5b-q4_k.gguf", + "task": "asr", + "mode": "streaming" + } + ] +}