0dc8e9096e2a52bd8f72292c05240e69cdb3e75d
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 vh
|
300ecc1276 |
voices-seat: ship lv-hemingway (ckpt850), and replace the memorisation control that passed it
Live on vllm-voices (fv-ml1 GPU0 :8027) beside voices-base, lv-yarros and lv-bronte.
Healthy 190 s after recreate, four models served, GPU0 96,092 -> 96,090 MiB. The adapter
was verified byte-identical to checkpoint-850 by sha256 across both transfer hops, and the
seat was verified by generating, not by reading its config: base emits 170 words of <think>
planning and never writes the passage, lv-hemingway writes the scene.
Gate design was pre-registered before any generation existed (
|
||
|
vh
|
caa04801f3 |
fix(parakeet): move the seat from the empty GPU 3 to GPU 0
Placed on GPU 3 first because it was the empty card. That was the wrong read:
the seat is ~800 MiB, under 1% of a 96 GB card, so the question was never "where
does it fit" but "whose headroom is cheapest to spend".
vLLM sizes its KV cache as a fraction of TOTAL VRAM, not free VRAM. A resident
tenant on an otherwise-clean card therefore does not cost its own megabytes — it
costs the profiling margin of whatever full-size seat lands there later, and
flash-next needs 93 GiB of 96. A 96 GB card at 2 MiB can still take that; the
same card at 922 MiB is one where the next big seat needs its utilization
hand-trimmed, which this repo's flash-next history shows is both thin and silent
when it goes wrong.
Committed utilization per card is the number that governs, not free bytes:
GPU 0 0.40 + 0.48 = 0.88 ~13 GB spare <- moved here
GPU 1 0.52+0.24+0.10+0.055+0.03+0.03 = 0.975 ~4.3 GB
GPU 2 0.96 ~1.8 GB
GPU 3 - kept empty as reserve
GPU 3 is back to 2 MiB / 97,247 MiB free and is now documented as a deliberate
reserve rather than a spare.
Post-move n=5 on the same clip: 0.68 / 0.54 / 0.54 / 0.52 / 0.53 s, median 0.54 s
against 0.50 s on GPU 3. The spreads overlap at this sample size and no difference
is claimed; the GPU 3 figure was taken on an idle card and is now noted as a best
case, since the seat shares GPU 0 with the hot serving path. Silence control and
the gateway round-trip both re-verified after the move.
Also records both Parakeet endpoints (FV v3 on Blackwell, IRV v2 on a 3090) and
the four confounds that make them not an A/B pair, sent to tts-dev for the bench.
|
||
|
vh
|
b9b14b5baf |
feat(parakeet): stand up Parakeet STT on fv-ml1 GPU 3 + LiteLLM ext-stt/whisper-1
Retargets the existing sherpa-onnx stack from irv-ml1 to fv-ml1's utility card and puts it behind the gateway. GPU 3 was the only card with room: 0/1/2 carry the vLLM seats at 84-95.5 GB of 96. Changes: - compose: pin GPU via `device_ids: ["3"]` (the dead on-host stub used `count: all`, which would have handed a 0.6B ASR seat all four cards); join traefik-net; port 8300; homepage href to the live FV address. - .env.example: default to the v3 int8 model (25 European languages, 464 MiB) rather than English-only v2; models to /tank/parakeet/models. - app.py: warm the recognizer at startup before uvicorn accepts traffic. The warmup is not an optimisation. ONNX Runtime's CUDA EP compiles and autotunes lazily on the FIRST DECODE, and on sm_120 that measured 45.7s cold (reproduced at 45.1s on a second container) against ~0.50s warm. A 45s first request is indistinguishable from a hang and LiteLLM's default timeout abandons it long before it returns. Decoding 1s of silence at load moves the cost inside the healthcheck's 300s start_period; first real request after restart is now 0.65s. Verification, because "provider=cuda" in the log is only an echo of the env var: ORT falls back to CPU silently and still returns correct text, so the service being up and the transcript being right establishes nothing. The discriminator is a process on GPU 3 (922 MiB), confirmed. Controls both directions — a known TTS sentence transcribes near-exactly (positive), 3s of digital silence returns empty (null). Warm throughput 0.50s median on an 8.52s clip, n=5, spread 0.47-0.65s, single-stream, one clip: a smoke measurement with its harness stated, not a benchmark. Gateway aliases `ext-stt` (engine-neutral, mirrors ext-tts) and `whisper-1` (OpenAI-compatible drop-in) registered via POST /model/new, i.e. LiteLLM's Postgres store where the ext-tts family already lives — no gateway restart, and config.yaml is consequently not a complete picture of what the gateway serves. Both verified end to end. The aliases use a raw IP deliberately: ana-docker resolves no .internal names at all (resolv.conf points at 1.1.1.1), and LiteLLM only reaches irv-ml1 through a hand-pinned extra_hosts entry. A second hosts entry would mean recreating the container and bouncing the gateway for every consumer. Also records the svos_miranda plugin validation pass and its structural findings, and notes that the irv-ml1 parakeet is still running — there are two now, and retiring the old one is the operator's call. |
||
|
vh
|
959a743256 |
feat(fv): invert the watchdog to break-glass; fv-ml1 off the mesh
Operator's design, and a better one. Once the FV SNAT rules landed, fv-ml1's mesh membership was redundant for routing and its only remaining value was as a second way in. Keeping it enrolled bought a standing second door; joining on demand buys the same recovery path without one. normal tailscaled stopped + disabled; fleet reached via the gateway SNAT fault nh3-dev / nh3-docker unreachable while the WAN is up action start tailscaled + tailscale up -> reachable at its 100.64.x address fv-ml1 is now off the mesh and its node record deleted. Verified it still reaches NH3, ESH, Anaheim, Irvine and the internet on the SNAT path alone, then the break-glass fired on cue (counted 1..4, joined at 5 as 100.64.0.10), answered ping and ssh from nh3-dev, and was closed again cleanly. No auto-leave, deliberately: once open the door stays open until a human runs systemctl disable --now tailscaled. A watchdog that re-closes on recovery flaps, and a flapping recovery path is down exactly when someone finally looks. It also skips entirely when already on the mesh, which is what makes it idempotent after firing. The question exposed a hole worth more than the redesign. The stored rejoin key was one of the 2026-09-12 FV cutover keys, expiring 2026-09-19 -- a break-glass credential that dies in four days and fails silently at the only moment it matters. Replaced with a dedicated 1-year reusable key (headscale ID 8, expires 2027-09-15), vaulted as fv-ml1/headscale-breakglass-key, root:600 on the host. That also closes the standing self-join risk rather than trading it: the two stale reusable keys (IDs 5, 6) are expired, so the mesh now has exactly one live reusable key -- purpose-built, on a host we control -- instead of two orphans nobody owned. Rejoin uses --accept-routes=false and the reason is in the script: on 2026-09-14 tailscale up --accept-routes on this box accepted its OWN subnet from the gateway and black-holed it. That happened with a human watching; here it runs unattended, during an incident, on a box already in trouble. |
||
|
vh
|
8c8559b8ec |
feat(fv): mesh dead-man's switch on fv-ml1; partial progress on FV cross-site routing
WATCHDOG (done, proven). fv-mesh-watchdog probes two independent anchors every
minute and, after 5 consecutive failures, puts Tailscale back to known-good:
accept-routes off, re-up against headscale with a stored key. It touches
nothing else — a watchdog with a wide remit is a second way to lose the box.
Two anchors that cannot share a failure mode: a plain-internet one and a
mesh-only one. If BOTH fail the site uplink is down, Tailscale cannot fix that,
and it deliberately does nothing — thrashing tailscaled during an ISP outage
turns a wait into an incident. Disable file at /etc/fv-watchdog.disable for
planned work.
Proven by positive control, not assumed: counter incremented 1..4 without
acting, fired the restore at 5 (tailscale up ran, tailscaled restarted), and
reset to 0 once the real anchor returned. fv-ml1 stayed reachable throughout.
This exists because a on fv-ml1 black-holed it
from its own LAN earlier the same day: it accepted 10.251.0.0/16 from the
gateway — its OWN subnet — and routed the local network through the tunnel.
FV CROSS-SITE ROUTING (partial). Two changes landed, the path is still broken:
1. acceptSubnetRoutes 0 -> 1 on the FV gateway's tailscale plugin, via
settings/set + service/reconfigure (the documented apply, not a reboot).
The GATEWAY now has 10.0/16, 10.100/16 and 10.250/16 in its routing table
and reaches NH3 and ESH itself. It could not before.
2. Remote-site MASQUERADE rules on nh3-scale. The existing jump matched only
-s 100.64.0.0/10, so traffic from another site's LAN never entered
MESH-EXIT and kept its original source; an NH3 host then replied via its
own LAN router instead of back through nh3-scale, making the path
asymmetric. The rule is confirmed firing (counter increments on FV
traffic) but does not complete the path.
Still failing: fv-ml1 -> NH3/ESH LAN addresses. Mesh addresses work perfectly
from fv-ml1 (100.64.0.1, 100.64.0.4), Anaheim works over the metro link, and
the FV firewall log shows the outbound passing on tailscale0 with
src=10.251.50.54 and no reply ever returning. The remaining gap is forwarded
FV-LAN traffic specifically, not the gateway's own.
Full regression sweep clean: nh3-dev, ana-docker and esh-docker-vm all reach
all four sites plus the internet.
|
||
|
vh
|
91bda3c480 |
fv-ml1: complete the cutover — rename, renumber, DNS, and the LiteLLM repoint
The box is physically at Fountain Valley, renamed, renumbered onto 10.251/16, and serving inference again. This lands the repo half of that. Host: hostname ana-ml2 -> fv-ml1, pinned to 10.251.50.54 by a dnsmasq reservation so the address the runbook, DNS and LiteLLM all assume is the address it actually has. Its headscale node is renamed too. The sweep ran from scripts/fv-ml1-rename-sweep.sh, whose allowlist is the reason this diff touches current-state files and not the record. Dated persistent-memory entries, archival-memory and incident notes still say ana-ml2 in 31 and 62 places respectively, because that is what the box was when those things happened. Rewriting them would make the history lie. LiteLLM was the load-bearing piece and needed more than the api_base sed the runbook describes. Twenty api_base entries repointed, but a grep-and-verify pass also caught a LIVE pass_through_endpoints target for the scalar-judge reward route still on the old address -- an api_base-only substitution would have left it dead. Four prose references describing current state were repointed as well; one historical note recording where a hand-test was run is deliberately left pointing at 10.250.50.54. Two facts in the server tables were wrong and are corrected here. The site is Fountain Valley, not Anaheim. And the box has FOUR RTX PRO 6000 Blackwell Max-Q, not two -- verified by nvidia-smi -L and independently by PCI enumeration of four GB202GL devices. That is 391 GB of VRAM rather than 196, which changes what fits on it. DNS: fv-ml1, fv-ml1-bmc and fv-gw added under the fv site via the piggyback approach, scriberr re-homed, and the ana-ml2 records removed. Applied to all three resolvers. The BMC record carries a warning that its 802.1q VLAN tag must stay disabled -- it shipped tagging VLAN 250 into an untagged port, which made it invisible to every network-side diagnostic and is the reason it appeared dead through several cable changes. Verified end to end: summarizer and sec both answer through the Anaheim gateway across the mesh to FV seats on different ports. |