memory: fleet IPv6 state + verified VPN topology; ana-wg key material locked down
Durable capture ahead of the ESH fiber install (2026-08-18) that puts the house behind CGNAT and breaks Site Magic on IPv4 -- IPv6 becomes the escape hatch and the likely first consumer of fleet v6. Topology verified rather than assumed: Site Magic between UniFi units, IPsec IKEv2 colo<->UniFi, and WireGuard as a remote-access convention only, host-based on ana-wg behind a FortiGate UDP VIP. The FortiGate port-forwards and never terminates WireGuard, so FortiOS 7.2's lack of native WG is a non-issue. IPv6 today: NH3 WAN live, colo and ESH none. AT&T delegates exactly one /64 at NH3 -- established by forcing the prefix ID from auto to 0 and observing the subnet not move, since the c110/c11f pattern otherwise reads as a /60. PD enabled on nh3-iot to measure, then reverted; all five NH3 LANs are back to ipv6_interface_type=none. Also fixed on ana-wg: wg0.conf, keys/*_priv, keys/*_psk and the client configs were mode 644 with private key material in them. Now 600, with keys/ and configs/ at 700. wg-quick@wg0 stayed active, three peers intact. Corrects two stale in-flight rows: the DS regeneration is retired, not queued, and SPEC-ds-regeneration.md is deleted rather than untracked.
This commit is contained in:
@@ -113,7 +113,7 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res
|
||||
|
||||
- **🟢 GEN SEAT — RESOLVED 2026-08-17 (the whole multi-day degeneration saga).** Primary gen = the in-house **JonathanColetti/Heretic mixed NVFP4+FP8 build** (`/tank/aimodels/qwen38-27b-uncensored-nvfp4-mixed`, FP8 attention) on **vLLM nightly PINNED** `vllm/vllm-openai:nightly-311b3513…` (`v0.27.2rc1.dev150`, carries #51113 mamba fix), **MTP ON, prefix-caching ON**. Operator-confirmed **coherent through 60k tokens** real multi-turn. Root cause = TWO compounding real causes: (1) genuine vLLM `qwen3_5_mtp`×GDN partial-accept bug (#51113, architectural across vLLM/SGLang/llama.cpp, fixed by nightly), and (2) **AEON's full W4A4** being lowest-fidelity on the known activation gradient (W4A4 < W4+FP8 < W4+bf16) → ~15-20% stochastic degeneration on top of (1). **AEON PURGED** (re-pullable `sakamakismile/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4`). Full lesson `docs/pfi/model-quantization-playbook.md` §3.8. Primary **until the DavidAU Qwen3.8 lands.** ⚠ **pinned nightly is bleeding-edge — move to a stable release once #51113 ships in one (the standing follow-up).** 7 aliases (gen/gen-reasoning/summarizer/-large/classifier/image-judge/qwen-image-bench) all route here. Seat carries `--default-chat-template-kwargs '{"reasoning_effort":"medium"}'` (per-request overridable, affects gen-reasoning only). Commits `d28a371`,`2f2bbce`,`2185964`.
|
||||
|
||||
- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr QUEUED** (8 dropped operational+meta axes, spec at `services/refusal-probe/darkscarlett-regen-spec.md`) — gated on the GPU1 window, no deadline.
|
||||
- **🔵 RP SEAT — FABLE-FUSION serving `char-rp-reasoning` (evaluation window, unchanged this session).** `fablefusion-charrp-probe` ana-ml2 GPU1 `:8019` serving `char-rp-probe` (`kkuspa/Qwen3.6-27B-Fable-Fusion-711-…-MTP-NVFP4A16`). LiteLLM `char-rp-reasoning` + `char-rp-fable` both route to it (deliberate repoint, documented in `stacks/litellm/conf/config.yaml`). `darkscarlett-charrp-reasoning` is `compose down`, weights intact at `/tank/aimodels/darkscarlett-nvfp4-work/`. **⏳ STILL AWAITING operator's hands-on read of FF prose** (refusal question settled: FF 15.8% vs DS 92.5% cold-framing; DS v1.0 never abliterated). ⚠ FF reasons 2.1–4.6k chars → use `max_tokens` ≥3072. `ReadyArt/Dark-Scarlett-27B-v2.0` (Qwen3.8) is GATED (`403 awaiting review`) — operator ruled not-interesting, do NOT re-propose. **DS regeneration for brokkr RETIRED 2026-08-17 — unqueued, do NOT run** (`9c1405b`): brokkr withdrew on the operator's call because (a) our `ictrl-pair-unwrapped`/`-wrapped` control isolates the classifier over-fire cleanly where DS's cross-class delta only bounded it, and (b) **DS v2 releases soon**, so a k=5 v1 baseline baselines a superseded version. Spec at `services/refusal-probe/darkscarlett-regen-spec.md` stays banked as the record of the run that will not happen (axes + per-class grading asymmetry still correct, checklist struck through). No GPU1 window was ever spent. A DS-**v2** characterization would be a fresh purpose-scoped ask.
|
||||
|
||||
- **🟢 LOBE CHAT — LIVE on esh-docker-vm `:3210` (2026-08-17).** Replaces the hand-rolled `gateway-chat` HTML surface. `stacks/lobe-chat/`, image `lobehub/lobe-chat` (143 MB compressed vs Open WebUI's 1.8 GB — the weight call). Scoped LiteLLM key `lobe-chat-esh` (free-local models only; paid GLM/Kimi BLOCKED, verified). Secrets vaulted `esh-docker-vm/lobe-chat-*`. TTS = a SPLIT: endpoint env-driven (inherits `OPENAI_PROXY_URL`→`ext-tts`), but voice/model/format UI-only. System-agent repointed off its `gpt-5-mini` default onto fleet models via `SYSTEM_AGENT` env. **⏳ REMAINING: one-time human UI pass** to enable TTS + set `response_format:mp3`. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`.
|
||||
|
||||
@@ -125,14 +125,18 @@ _As of 2026-08-17 — **quiet; gen-seat degeneration saga CLOSED.** Gen seat res
|
||||
|
||||
- **EVIDENCE HOLD (partial):** WT #394 FILE half STILL STANDS — do NOT delete on-disk gen dirs (`fiction/rex390-dcc`, `rex392-dcc`, `b59c147c5ce0`); rex393-fiction-* + r42-gate-* KEEP.
|
||||
|
||||
- **🟡 FLEET IPv6 — mapped, nothing enabled; WAITING ON ADDRESSES.** Driver is **ESH fiber installing 2026-08-18 landing the house behind CGNAT**, which breaks **Site Magic** (NH3↔ESH) on IPv4 → v6 is the escape hatch and the likely first consumer. State: **NH3 WAN live** (`2600:1700:b25:c110::48`, AT&T delegates exactly **one /64**), **colo none** (FortiGate has zero v6), **ESH none** (both WANs `wan_type_v6=disabled`). NH3 LANs all reverted to `ipv6_interface_type=none` per operator. Work when addresses land: v6 on `ana-wg` eth0 + a **v6 port-forward for UDP 31337** on the FortiGate (its WG socket is already dual-stack — no WG reconfig), flip the UDM WG server off `v4`-pinned binding, and **AAAA records** so the dynamic prefixes at all three sites don't break endpoints. Full detail + access recipes → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
|
||||
|
||||
- **OPEN FOLLOW-UPS (parked):** move gen seat off pinned-nightly to stable once #51113 ships; Lobe one-time TTS UI pass; delete the 1.8GB litellm dump; `harden-esh-docker-vm` (park id 28, PROMOTED — Tier-1 done, `/mnt/books` stays hard w/ watchdog); chatterbox-fast build-context divergence; #363 research-wing ingest (no deadline); optionally attach our MTP reproducer to vllm#47087 (needs a GitHub identity — operator's call).
|
||||
|
||||
- **althing monitor** ARMED (handle `infra-ops`). ⚠️ Re-arm ONLY after a real FIRE (rc0), never after a plain operator turn (bounces rc3); spawn `althing-wake-listener` as its OWN `run_in_background` task, never chained with `&` (orphans it — hit this twice 2026-08-17, `stop-monitor` reclaims).
|
||||
|
||||
- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` + `services/refusal-probe/SPEC-ds-regeneration.md` UNTRACKED.
|
||||
- **eshpfi push state:** operator pushes **manually** (this session's commits from `766c658`→`2185964` are the operator's to push). ⚠ **Push over the INTERNAL gitea route** — `git push ssh://git@10.250.50.70:222/vh/esh-pfi-infrastructure.git main:main`; `origin` resolves the public edge (`38.120.12.44`) which fail2bans fleet-host egress. `graphify-out/GRAPH_REPORT.md` churns every commit (ignore); `stacks/heretic2-charrp-reasoning/` UNTRACKED. (`services/refusal-probe/SPEC-ds-regeneration.md` was a truncated duplicate draft — **deleted 2026-08-17** on operator instruction; `darkscarlett-regen-spec.md` is the single canonical spec.)
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-08-17]` **Fleet IPv6 mapped + the real VPN topology verified; the driver is CGNAT at ESH, not the WireGuard mesh.** New ESH fiber (installing 2026-08-18) lands the house behind **CGNAT**, which breaks **Site Magic** (NH3↔ESH `sdwan-mesh-tunnel`) on IPv4 — so IPv6 becomes load-bearing as the escape hatch, and that is its most likely first consumer. Topology as VERIFIED (a prior turn assumed wrong and was corrected): UniFi↔UniFi = **Site Magic**; colo↔UniFi = **IPsec IKEv2** (`pfi-ana-nh3` 158M/165M pkt = the workhorse, `ana-to-eshudm`); **WireGuard is an RA convention only, host-based on `ana-wg`** UDP 31337 behind a FortiGate VIP — the FortiGate never terminates WG (FortiOS 7.2 has none; 7.4 added it) so "upgrade the edge for WireGuard" is a **non-problem, do not re-derive**. IPv6 today: **NH3 WAN live** `2600:1700:b25:c110::48`, **colo none**, **ESH none**. **AT&T delegates exactly ONE /64** (`2600:1700:b25:c11f::/64`) — proven by forcing prefix-ID auto→`0` and watching the subnet NOT move, because the `c110`/`c11f` pattern otherwise reads convincingly as a /60. A mesh needs a routable **WAN** address, **not** PD. `ana-wg`'s WG socket is **already dual-stack** (`[::]:31337`) → v6 RA needs an address + a v6 port-forward, no WG reconfig. ⚠ UDM legacy `rest/firewallrule` returns **0 rules** (zone-based firewall) — use `v2/…/firewall-policies`; inbound v6 is default-deny and held. All three endpoints will be **dynamic** → extend the existing hostname pattern (`ana-fw`/`nh3.phasefinal.com`) to **AAAA**. Enabled PD on `nh3-iot` to measure, **reverted on operator instruction** (all 5 LANs back to `none`, verified). Also fixed: **`ana-wg` WireGuard key material was world-readable** (`wg0.conf` + `keys/*_priv` + `*_psk` + client `configs/*.conf` at 644) → now 600, dirs 700, service untouched. Detail → `persistent-memory.d/2026-08-17-fleet-ipv6-mesh.md`.
|
||||
|
||||
- `[2026-08-17]` **Gen-seat multi-day degeneration RESOLVED — two compounding real causes, not one; the meta-lesson is "a mitigation that HELPS but doesn't FIX means a second cause, not a wrong one."** vLLM `qwen3_5_mtp`×GDN bug (#51113, real, fixed by nightly) + AEON full-W4A4 being lowest-fidelity (W4A4<W4+FP8<W4+bf16) → ~15-20% stochastic degeneration. Fixed by mixed FP8-attn build on pinned nightly. AEON purged. Also banked: **stochastic (~15-20%) degeneration is invisible to a small synthetic probe — n=1 "clean" validated THREE non-fixes (MTP-off, APC-off, nightly-alone) that all failed in real use; get the operator's real transcript, do not trust your own probe.** Full → `docs/pfi/model-quantization-playbook.md` §3.8 (+ §3.7 MTP-multi-turn). Commits `d28a371`,`2f2bbce`,`2185964`.
|
||||
|
||||
- `[2026-08-17]` **Lobe Chat chosen over Open WebUI (weight: 143 MB vs 1.8 GB) + stood up on esh-docker-vm; scoped LiteLLM key blocks paid models; System-Agent `gpt-5-mini` default repointed via env.** TTS env-vs-UI resolved as a split (endpoint env-driven, voice/model UI-only). tts-dev onboarding closed both directions; ballad/verse aliased so no voice can 404 the router. Commits `e9362de`,`163a725`,`cac75cb`,`933253d`,`25fa18e`.
|
||||
|
||||
Reference in New Issue
Block a user