vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

memory: snapshot — FV cutover executed, BMC online, four GPUs not two

Browse Source 
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1
live on 10.251/16 and serving, BMC recovered after finding it was tagging
802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB
count that invalidates prior sizing assumptions.

Also records two things that cost real time and would cost it again: OPNsense
write APIs need an X-CSRFToken scraped from a script block, not a hidden form
input, and a 403 that coincides with the box going down looks exactly like a
successful reboot.

Auto-archived 8 settled entries (Recent decisions: 8). The index stays over
the 300-line cap at 385 because the guards hold -- nearly every remaining
dated entry is under 14 days old or carries open deferred work, and an
over-cap file that keeps live decisions beats a scannable one that lost any.
This commit is contained in:
vh
2026-09-12 22:10:14 -07:00
parent 48e34e1849
commit 3e727dbeb5
3 changed files with 173 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
persistent-memory.md
+56 -49
View File
@@ -1,6 +1,6 @@
# Persistent memory — eshpfi-management
_Last updated: 2026-09-11 ~17:45 PT (fv-ml1 relocation cutover PREPPED for tomorrow; Anaheim recovered except ana-ml2 which relocates; BabyYarros COMPLETE + evaluated; sentinel-r3 quant done, cyber-preview to re-run at FV)_
_Last updated: 2026-09-12 ~19:30 PT (FV cutover EXECUTED — fv-ml1 live at Fountain Valley, renamed/renumbered/serving; BMC online after a VLAN-250 find; four GPUs not two)_
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
> `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -110,46 +110,69 @@ no longer deployed sidecars here. See Recent decisions.)
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
## Current state / in-flight
_As of 2026-09-11 ~17:45 PT._
_As of 2026-09-12 ~19:30 PT._
### fv-ml1 relocation — cutover PREPPED, executes TOMORROW
- **ana-ml2 → fv-ml1**, moving to a NEW Fountain Valley colo (`10.251.0.0/16`) tomorrow;
its power draw is the root cause of the repeated Anaheim breaker trips. Fully staged,
nothing deployed: runbook `docs/runbooks/fv-ml1-cutover.md`, rename sweep
`scripts/fv-ml1-rename-sweep.sh` (dry-run default, history-safe), exact DNS + LiteLLM
commands inside the runbook. See Recent decisions `[2026-09-11]` fv-ml1 for the full plan.
- Load-bearing at cutover: **LiteLLM `api_base` 10.250.50.54→10.251.50.54** (darkens every
inference alias if missed), DNS piggyback records, OPNsense as tailscale subnet-router.
Box already down (clean cutover); `/tank` is local ZFS so data travels with the chassis.
- **Anaheim rack left DARK until the move** (operator) — nothing to bring up, it relocates.
### FV colo — LIVE. Cutover done, one gap open.
- **fv-ml1** (ex ana-ml2) is racked at Fountain Valley, renamed, on `10.251.50.54`, mesh
node `100.64.0.7`; **vb-gateway** OPNsense on `10.251.50.1` / `100.64.0.8`; **BMC** on
`10.251.250.50`. Public `fv.phasefinal.com` → `172.83.89.66`. `tank` intact, all vLLM
seats healthy, inference verified through the Anaheim gateway. All three sites reach FV
by real IP. → `persistent-memory.d/2026-09-12-fv-cutover-executed.md`
- ⚠ **OPEN GAP: fv-ml1 cannot initiate to fleet LAN IPs** (10.100/10.250/10.0 all fail;
mesh IPs and internet fine, inbound fine). Return-path issue at the far gateways. Not
biting yet — DNS is MagicDNS, inference is inbound — but blocks fv-ml1 pulling from any
fleet LAN host. **Next concrete task.**
- ⚠ Plaintext creds to delete: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` (fv-ml1).
All four are vaulted under `fv-gateway/` and read-back verified.
- ⚠ FV WAN rule `InfraOps` is scoped to alias `fleet_egress` (NH3 70.230.226.88 / ANA
38.120.12.42 / ESH 128.177.138.182). Operator wants it up a few days, then close.
- ⚠ headscale preauth keys `headscale/preauth-fv-{router,client}-7d-20260912` expire
**2026-09-19** — revoke after the build settles.
### Anaheim colo — recovered except ana-ml2
- Full-site power/breaker outage ~15:0x PT; recovered ~16:39 EXCEPT ana-ml2 (no power,
relocating). The gitea-wide 403 (crowdsec crash → traefik bouncer fail-closed) was fixed
by restarting crowdsec then traefik; LiteLLM + everything else healthy. ⚠ recurring
post-power-loss step, now in the recovery runbook memory.
### Next up — the VRAM the fleet didn't know it had
- fv-ml1 has **4× RTX PRO 6000 = 391 GB**, not the 196 GB every doc claimed. Operator:
"we have some fun things to do with the vram we now have." Revisit seat placement and
whether seats split across irv-ml1/gx10 can consolidate. Nothing decided yet.
### BabyYarros — COMPLETE + evaluated
- Both arms trained (**Base 2.5263 @ ckpt-125**, overfits within epoch; **Instruct 2.6114 @
178**) and evaluated: voice moved toward Yarros **above the 0.046 measured noise floor**
(Base +0.157, Instruct +0.076), Instruct renders **beats 9/10**. Booth `babyyarros-voice`.
⛔ Full frozen adjudication (romantasy control panel + 2nd seed + gen seat for beat-incumbent)
DEFERRED — needs the gen seat back. See Recent decisions `[2026-09-11]`.
### BabyYarros — complete; pair-corpus rebuild is the next step
- Both arms trained + evaluated; Instruct renders beats 9/10 by a **lexical** metric that
overstates it (echo-contaminated). Measured on the operator's own Janis beat: the beat's
grammar controls tense (0/4 → 4/4 present), anti-echo prompting fails 11/12, and prompt
tuning alone gets ~1-2/4 usable. → `persistent-memory.d/2026-09-11-babyyarros-*.md`
- **Agreed direction:** rebuild the corpus as instruction PAIRS (option C), with only the
INSTRUCTION side LLM-generated (backtranslated beats over real Yarros paragraphs) —
generating responses would distil away the voice. Measured supply: only 318 single
paragraphs are 90-140w, so the pair unit must be a ~4-paragraph scene window
(6,445 non-overlapping, 88% corpus coverage). Critical path is response-only loss
masking in `train_voice_lora.py` (currently `labels = ids.clone()`), ~1 day.
- ⛔ Frozen adjudication still deferred (needs the gen seat, now back at FV).
### Quants — sentinel-r3 done, cyber-preview to re-run
- **sentinel-r3** NVFP4 (grafted base MTP head) COMPLETE at `/tank/aimodels/sentinel-r3-nvfp4-mixed`
(survives — ZFS). Acceptance/A-B deferred (needs a serving slot). **cyber-preview** NVFP4
died mid-quant with the ana-ml2 outage — re-run when fv-ml1 is up; both bf16 sources safe on /tank.
### Quants — cyber-preview to re-run
- **sentinel-r3** NVFP4 complete at `/tank/aimodels/sentinel-r3-nvfp4-mixed`; acceptance
A-B still needs a serving slot. **cyber-preview** died mid-quant in the Anaheim outage —
re-runnable now that FV is up.
### gx10 on althing; Jetson planning
- postbox installed on gx10 (handle `gx10`, **send-only** — no reader on its inbox, it's a
headless notifier/watcher-host; reply-expecting watchers post as `infra-ops`).
- Jetson AGX Orin — discussed as an **ESH House Computer** (cameras via Frigate + local ASR/TTS);
its native fit is vision/perception. Discussion only, not committed. Jetson Nano generation TBD.
### Jetson AGX Orin — 64 GB, in hand, unassigned
- Operator has the 64 GB dev kit plus ~6 undeployed cameras, a depth camera and lidar for
an unbuilt robot. House runs **Home Assistant + UniFi Protect, not Frigate**; Protect
exposes RTSPS per-channel (verified, currently off). Recommendation on the table: the
Orin is the **robot brain** (Isaac ROS), not the house computer — ESH already owns four
idle hardware decoders and the house voice stack is CPU-work that belongs beside HA on
esh-docker-vm. Blocked on which depth camera and lidar. Not decided.
### eBay listing pipeline — scoped, unstarted
- Opto-mechanical resale. Verified eBay's **Sell Inventory API** is the right surface
(bulk item/offer/publish, 25/call; requires Business Policies opt-in) and that
`getItemAspectsForCategory` turns listing generation into schema population. Plan:
EXIF-gap clustering → VLM identification → Booth ask for approval → bulk publish.
Nothing built.
## Recent decisions
- `[2026-09-12]` ⭐⭐⭐ **FV CUTOVER EXECUTED — fv-ml1 live at Fountain Valley, renamed, renumbered to 10.251/16, serving inference; BMC online after finding it was tagging 802.1q VLAN 250 into an untagged port; and the box has FOUR RTX PRO 6000 (391 GB VRAM), not the two every doc claimed.** Also: OPNsense write APIs need an `X-CSRFToken` header scraped from a `<script>` block — session cookies alone 403, which is why reboots appeared to work while the operator was power-cycling by hand. → `persistent-memory.d/2026-09-12-fv-cutover-executed.md`
- `[2026-09-12]` **esh-vm-db Restic fixed:** stale April PG dump caused by TCP auth + masked errors; peer-auth/fail-closed hook and bounded retries installed. Snapshot bc5eeaff and repository check verified. → `persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md`
- `[2026-09-11]` **Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips `memory.reader.enabled` or `memory.writer.enabled` on any deployment without infra-ops first confirming the memory root is writable by the container's uid.** The reader **REFUSES AT BOOT** if it cannot append+read back `<memory root>/reader/canary.jsonl` (deliberate, the #335 typo'd-reranker precedent: refuse loudly, never silently disable); per-euid subdirs are created lazily and only warn, so the **root canary is the only boot-blocking check**. The writer degrades rather than refuses. Both ship DARK (`enabled: false`, parity-only `config/defaults.yaml`) until the operator schedules the tracer skeleton. ⭐ **Measured 2026-09-11 on corviduo-dev — all three deployments PASS**: demo :8080 uid **0** and personal :8081 uid **0** both have `/data/state/memory` at 1000:1000 755 writable; pinned :8082 uid **1000** lacks `memory/` but its parent `/data/state` is 1000:1000 755 so it can create it. ⚠ I had predicted personal was uid 1000 and warned it would fail — **wrong, retracted**; only pinned runs as 1000, and it passes anyway. ⚠ Re-probe immediately before any flip: a permissions reading is a claim about its own date, not about boot time. Heimdall side is clear too — demo and personal grant 7x `tool.*`, pinned uses image defaults, and the lone `tool.evidence.*` is additive, so `tool.memory_read` needs no policy change. Thread `01M2A05WED5W`.
- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@<tag>` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md`
@@ -334,22 +357,6 @@ _As of 2026-09-11 ~17:45 PT._
- `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`.
- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md`
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md`
- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry.
- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`.
- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
- `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
@@ -369,7 +376,7 @@ _As of 2026-09-11 ~17:45 PT._
_20 older entries archived to archival-memory.md._
_28 older entries archived to archival-memory.md._
## Tried and abandoned