diff --git a/archival-memory.md b/archival-memory.md index c74d3e2..d5252d4 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -4,6 +4,31 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re ## Recent decisions (archived) +- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` + _Archived 2026-09-12._ + +- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md` + _Archived 2026-09-12._ + +- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` + _Archived 2026-09-12._ + +- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md` + _Archived 2026-09-12._ + +- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry. + _Archived 2026-09-12._ + +- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`. + _Archived 2026-09-12._ + +- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md` + _Archived 2026-09-12._ + +- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md` + _Archived 2026-09-12._ + + - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] _Archived 2026-09-11._ diff --git a/persistent-memory.d/2026-09-12-fv-cutover-executed.md b/persistent-memory.d/2026-09-12-fv-cutover-executed.md new file mode 100644 index 0000000..32887ab --- /dev/null +++ b/persistent-memory.d/2026-09-12-fv-cutover-executed.md @@ -0,0 +1,92 @@ +# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving + +Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the +ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading. + +## Final state + + fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare) + vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8 + fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation + fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge + seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr + +`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned. +Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim +LiteLLM gateway, across the mesh, to FV seats on different ports. + +## ⭐⭐ FOUR GPUs, not two — every record was wrong + +`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4), +independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device +is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and +the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and +seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate. +⚠ Also means the FV circuit was likely specced against half the real draw; operator says +it is on its own circuit and fine for now. + +## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side + +The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was +**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a). +`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was +tagging every frame into an untagged access port, so the switch silently discarded them. + +**No cable position could ever have fixed this**, and no network-side diagnostic can see +it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an +in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`, +which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static. +Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200. + +## Routing: what actually had to change (not what we assumed) + +The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned +`10.251.0.0/16` automatically the moment headscale approved the route. What was missing: + +1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first + UDM static route appeared to do nothing — asymmetric drop, not a missing forward route. +2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and + the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first). + +All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact +afterwards. + +## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs + +fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine. +But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly +(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at +the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS +(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite +anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED. + +## ⚠ Removed a stale if-up hook + +`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes +via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is +down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`. + +## Credentials, all vaulted and read-back verified + +`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`, +`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus +`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19). +⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` +(fv-ml1) — redundant now, should be deleted. + +## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours + +Every write API (reboot, service restart, tunables) returns **403** with only a session +cookie. The CSRF token lives in a `