memory: snapshot — FV cutover executed, BMC online, four GPUs not two
Captures the Fountain Valley cutover as executed rather than planned: fv-ml1 live on 10.251/16 and serving, BMC recovered after finding it was tagging 802.1q VLAN 250 into an untagged port, and the corrected four-GPU / 391 GB count that invalidates prior sizing assumptions. Also records two things that cost real time and would cost it again: OPNsense write APIs need an X-CSRFToken scraped from a script block, not a hidden form input, and a 403 that coincides with the box going down looks exactly like a successful reboot. Auto-archived 8 settled entries (Recent decisions: 8). The index stays over the 300-line cap at 385 because the guards hold -- nearly every remaining dated entry is under 14 days old or carries open deferred work, and an over-cap file that keeps live decisions beats a scannable one that lost any.
This commit is contained in:
@@ -4,6 +4,31 @@ _Entries moved out of persistent-memory.md to keep the active file scannable. Re
|
||||
|
||||
## Recent decisions (archived)
|
||||
|
||||
- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry.
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`.
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
|
||||
_Archived 2026-09-12._
|
||||
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
_Archived 2026-09-11._
|
||||
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# `[2026-09-12]` FV cutover EXECUTED — the box is at Fountain Valley, renamed, renumbered, serving
|
||||
|
||||
Operator drove to the new Fountain Valley colo and racked the OPNsense gateway + the
|
||||
ex-ana-ml2 GPU box. Everything below is verified by observation, not by config reading.
|
||||
|
||||
## Final state
|
||||
|
||||
fv.phasefinal.com 172.83.89.66 public A record, DNS-only (Cloudflare)
|
||||
vb-gateway 10.251.50.1 OPNsense 26.7.3_11, mesh node 100.64.0.8
|
||||
fv-ml1 10.251.50.54 mesh node 100.64.0.7, DHCP reservation
|
||||
fv-ml1-bmc 10.251.250.50 own interface bge1, outside the bridge
|
||||
seats vllm-gen/:8015, mog-sec/:8019, erp-seat, gemma4-charrp, scriberr
|
||||
|
||||
`tank` 11.6T ONLINE with 6.14T allocated — the data travelled with the chassis as planned.
|
||||
Inference verified end-to-end: `summarizer` and `sec` both answer through the Anaheim
|
||||
LiteLLM gateway, across the mesh, to FV seats on different ports.
|
||||
|
||||
## ⭐⭐ FOUR GPUs, not two — every record was wrong
|
||||
|
||||
`nvidia-smi -L` counts **4× RTX PRO 6000 Blackwell Max-Q @ 97,887 MiB** (C3/C4/E3/E4),
|
||||
independently confirmed by PCI enumeration of four GB202GL devices (the fifth VGA device
|
||||
is the ASPEED BMC framebuffer). That is **391 GB VRAM, not 196**. CLAUDE.md, README and
|
||||
the cutover runbook all said "dual". Corrected. ⚠ This materially changes model sizing and
|
||||
seat placement — worth revisiting whether seats split across irv-ml1/gx10 can consolidate.
|
||||
⚠ Also means the FV circuit was likely specced against half the real draw; operator says
|
||||
it is on its own circuit and fine for now.
|
||||
|
||||
## ⭐⭐ The BMC was tagging 802.1q VLAN 250 — invisible from the network side
|
||||
|
||||
The BMC never ARPed on any port through an hour of cable-swapping. Its addressing was
|
||||
**correct the whole time** (static 10.251.250.50/24, gw .1, MAC 7c:c2:55:60:fe:8a).
|
||||
`ipmitool lan print 1` from inside the chassis showed `802.1q VLAN ID : 250` — it was
|
||||
tagging every frame into an untagged access port, so the switch silently discarded them.
|
||||
|
||||
**No cable position could ever have fixed this**, and no network-side diagnostic can see
|
||||
it — ARP, DHCP leases, port counters and firewall logs all just show silence. Only an
|
||||
in-chassis `ipmitool lan print` reveals it. Fixed with `ipmitool lan set 1 vlan id off`,
|
||||
which resets the IP to 0.0.0.0 (normal Supermicro behaviour), then re-set the static.
|
||||
Now: 0.37 ms, ports 443/80/623/5900 open, ATEN web UI and Redfish both HTTP 200.
|
||||
|
||||
## Routing: what actually had to change (not what we assumed)
|
||||
|
||||
The scale nodes needed **nothing** — all three already had `RouteAll: true` and learned
|
||||
`10.251.0.0/16` automatically the moment headscale approved the route. What was missing:
|
||||
|
||||
1. **`--accept-routes` on vb-gateway.** It had no return path to the fleet, so the first
|
||||
UDM static route appeared to do nothing — asymmetric drop, not a missing forward route.
|
||||
2. **Three gateway statics**, `10.251.0.0/16` → local scale node: NH3 UDM, ESH UDM, and
|
||||
the Anaheim FortiGate (config backed up to flash as `pre-fv-route-20260912` first).
|
||||
|
||||
All three sites now reach FV by real IP; Anaheim's pre-existing routing verified intact
|
||||
afterwards.
|
||||
|
||||
## ⚠ Known gap — fv-ml1 cannot initiate to fleet LAN IPs
|
||||
|
||||
fv-ml1 reaches mesh IPs (100.64.x) and the internet, and the fleet reaches fv-ml1 fine.
|
||||
But fv-ml1 → 10.100.x / 10.250.x / 10.0.x all FAIL. Packets are forwarded correctly
|
||||
(`pass on tailscale0 10.251.50.54 -> 10.100.50.40`), so the gap is on the return side at
|
||||
the far gateways. **Not currently biting**: DNS goes via tailscale MagicDNS
|
||||
(100.100.100.100) and inference is inbound, so nothing depends on it yet. Will bite
|
||||
anything that needs fv-ml1 to pull from a fleet LAN host (NAS, registries). UNRESOLVED.
|
||||
|
||||
## ⚠ Removed a stale if-up hook
|
||||
|
||||
`/etc/network/if-up.d/mesh-routes` on the box still injected Anaheim mesh return routes
|
||||
via ana-scale 10.250.50.45 on `enp97s0f0np0.50`. Inert at FV only because that NIC is
|
||||
down — a landmine if it ever comes up. Removed; backup at `/root/mesh-routes.retired-20260912`.
|
||||
|
||||
## Credentials, all vaulted and read-back verified
|
||||
|
||||
`fv-gateway/opnsense-api-key`, `fv-gateway/opnsense-api-secret`,
|
||||
`fv-gateway/infra-ops-password`, `fv-gateway/root-password`, plus
|
||||
`headscale/preauth-fv-{router,client}-7d-20260912` (expire 2026-09-19).
|
||||
⚠ Plaintext copies still on disk: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io`
|
||||
(fv-ml1) — redundant now, should be deleted.
|
||||
|
||||
## ⚠⚠ OPNsense API: session cookies are NOT enough, and this cost hours
|
||||
|
||||
Every write API (reboot, service restart, tunables) returns **403** with only a session
|
||||
cookie. The CSRF token lives in a `<script>` block as
|
||||
`xhr.setRequestHeader("X-CSRFToken", "…")` — **not** a hidden form input, so scraping for
|
||||
`<input type=hidden>` finds nothing. With the header supplied, writes return 200.
|
||||
Config *restores* via the legacy `diag_backup.php` form always worked (it carries a token).
|
||||
|
||||
⚠⚠ **Because the reboot API 403'd silently, I repeatedly reported reboots as mine when
|
||||
the operator was power-cycling the box by hand.** A 403 that coincides with the box going
|
||||
down looks exactly like success. Verify a reboot by observing the box go down AND the
|
||||
change take effect — never by the API's response.
|
||||
|
||||
Runbook: `docs/runbooks/fv-ml1-cutover.md`, `docs/runbooks/fv-onsite-playbook.md`.
|
||||
Commits `91bda3c` (rename/renumber/DNS/LiteLLM), `17586ab` (proxy retired),
|
||||
`87d7555` + `48e34e1` (playbook bodies + remaining live targets).
|
||||
+56
-49
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-11 ~17:45 PT (fv-ml1 relocation cutover PREPPED for tomorrow; Anaheim recovered except ana-ml2 which relocates; BabyYarros COMPLETE + evaluated; sentinel-r3 quant done, cyber-preview to re-run at FV)_
|
||||
_Last updated: 2026-09-12 ~19:30 PT (FV cutover EXECUTED — fv-ml1 live at Fountain Valley, renamed/renumbered/serving; BMC online after a VLAN-250 find; four GPUs not two)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -110,46 +110,69 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
|
||||
## Current state / in-flight
|
||||
_As of 2026-09-11 ~17:45 PT._
|
||||
_As of 2026-09-12 ~19:30 PT._
|
||||
|
||||
### fv-ml1 relocation — cutover PREPPED, executes TOMORROW
|
||||
- **ana-ml2 → fv-ml1**, moving to a NEW Fountain Valley colo (`10.251.0.0/16`) tomorrow;
|
||||
its power draw is the root cause of the repeated Anaheim breaker trips. Fully staged,
|
||||
nothing deployed: runbook `docs/runbooks/fv-ml1-cutover.md`, rename sweep
|
||||
`scripts/fv-ml1-rename-sweep.sh` (dry-run default, history-safe), exact DNS + LiteLLM
|
||||
commands inside the runbook. See Recent decisions `[2026-09-11]` fv-ml1 for the full plan.
|
||||
- Load-bearing at cutover: **LiteLLM `api_base` 10.250.50.54→10.251.50.54** (darkens every
|
||||
inference alias if missed), DNS piggyback records, OPNsense as tailscale subnet-router.
|
||||
Box already down (clean cutover); `/tank` is local ZFS so data travels with the chassis.
|
||||
- **Anaheim rack left DARK until the move** (operator) — nothing to bring up, it relocates.
|
||||
### FV colo — LIVE. Cutover done, one gap open.
|
||||
- **fv-ml1** (ex ana-ml2) is racked at Fountain Valley, renamed, on `10.251.50.54`, mesh
|
||||
node `100.64.0.7`; **vb-gateway** OPNsense on `10.251.50.1` / `100.64.0.8`; **BMC** on
|
||||
`10.251.250.50`. Public `fv.phasefinal.com` → `172.83.89.66`. `tank` intact, all vLLM
|
||||
seats healthy, inference verified through the Anaheim gateway. All three sites reach FV
|
||||
by real IP. → `persistent-memory.d/2026-09-12-fv-cutover-executed.md`
|
||||
- ⚠ **OPEN GAP: fv-ml1 cannot initiate to fleet LAN IPs** (10.100/10.250/10.0 all fail;
|
||||
mesh IPs and internet fine, inbound fine). Return-path issue at the far gateways. Not
|
||||
biting yet — DNS is MagicDNS, inference is inbound — but blocks fv-ml1 pulling from any
|
||||
fleet LAN host. **Next concrete task.**
|
||||
- ⚠ Plaintext creds to delete: `/tmp/opn.pw` (nh3-dev), `/tmp/io.pw` + `/tmp/key.io` (fv-ml1).
|
||||
All four are vaulted under `fv-gateway/` and read-back verified.
|
||||
- ⚠ FV WAN rule `InfraOps` is scoped to alias `fleet_egress` (NH3 70.230.226.88 / ANA
|
||||
38.120.12.42 / ESH 128.177.138.182). Operator wants it up a few days, then close.
|
||||
- ⚠ headscale preauth keys `headscale/preauth-fv-{router,client}-7d-20260912` expire
|
||||
**2026-09-19** — revoke after the build settles.
|
||||
|
||||
### Anaheim colo — recovered except ana-ml2
|
||||
- Full-site power/breaker outage ~15:0x PT; recovered ~16:39 EXCEPT ana-ml2 (no power,
|
||||
relocating). The gitea-wide 403 (crowdsec crash → traefik bouncer fail-closed) was fixed
|
||||
by restarting crowdsec then traefik; LiteLLM + everything else healthy. ⚠ recurring
|
||||
post-power-loss step, now in the recovery runbook memory.
|
||||
### Next up — the VRAM the fleet didn't know it had
|
||||
- fv-ml1 has **4× RTX PRO 6000 = 391 GB**, not the 196 GB every doc claimed. Operator:
|
||||
"we have some fun things to do with the vram we now have." Revisit seat placement and
|
||||
whether seats split across irv-ml1/gx10 can consolidate. Nothing decided yet.
|
||||
|
||||
### BabyYarros — COMPLETE + evaluated
|
||||
- Both arms trained (**Base 2.5263 @ ckpt-125**, overfits within epoch; **Instruct 2.6114 @
|
||||
178**) and evaluated: voice moved toward Yarros **above the 0.046 measured noise floor**
|
||||
(Base +0.157, Instruct +0.076), Instruct renders **beats 9/10**. Booth `babyyarros-voice`.
|
||||
⛔ Full frozen adjudication (romantasy control panel + 2nd seed + gen seat for beat-incumbent)
|
||||
DEFERRED — needs the gen seat back. See Recent decisions `[2026-09-11]`.
|
||||
### BabyYarros — complete; pair-corpus rebuild is the next step
|
||||
- Both arms trained + evaluated; Instruct renders beats 9/10 by a **lexical** metric that
|
||||
overstates it (echo-contaminated). Measured on the operator's own Janis beat: the beat's
|
||||
grammar controls tense (0/4 → 4/4 present), anti-echo prompting fails 11/12, and prompt
|
||||
tuning alone gets ~1-2/4 usable. → `persistent-memory.d/2026-09-11-babyyarros-*.md`
|
||||
- **Agreed direction:** rebuild the corpus as instruction PAIRS (option C), with only the
|
||||
INSTRUCTION side LLM-generated (backtranslated beats over real Yarros paragraphs) —
|
||||
generating responses would distil away the voice. Measured supply: only 318 single
|
||||
paragraphs are 90-140w, so the pair unit must be a ~4-paragraph scene window
|
||||
(6,445 non-overlapping, 88% corpus coverage). Critical path is response-only loss
|
||||
masking in `train_voice_lora.py` (currently `labels = ids.clone()`), ~1 day.
|
||||
- ⛔ Frozen adjudication still deferred (needs the gen seat, now back at FV).
|
||||
|
||||
### Quants — sentinel-r3 done, cyber-preview to re-run
|
||||
- **sentinel-r3** NVFP4 (grafted base MTP head) COMPLETE at `/tank/aimodels/sentinel-r3-nvfp4-mixed`
|
||||
(survives — ZFS). Acceptance/A-B deferred (needs a serving slot). **cyber-preview** NVFP4
|
||||
died mid-quant with the ana-ml2 outage — re-run when fv-ml1 is up; both bf16 sources safe on /tank.
|
||||
### Quants — cyber-preview to re-run
|
||||
- **sentinel-r3** NVFP4 complete at `/tank/aimodels/sentinel-r3-nvfp4-mixed`; acceptance
|
||||
A-B still needs a serving slot. **cyber-preview** died mid-quant in the Anaheim outage —
|
||||
re-runnable now that FV is up.
|
||||
|
||||
### gx10 on althing; Jetson planning
|
||||
- postbox installed on gx10 (handle `gx10`, **send-only** — no reader on its inbox, it's a
|
||||
headless notifier/watcher-host; reply-expecting watchers post as `infra-ops`).
|
||||
- Jetson AGX Orin — discussed as an **ESH House Computer** (cameras via Frigate + local ASR/TTS);
|
||||
its native fit is vision/perception. Discussion only, not committed. Jetson Nano generation TBD.
|
||||
### Jetson AGX Orin — 64 GB, in hand, unassigned
|
||||
- Operator has the 64 GB dev kit plus ~6 undeployed cameras, a depth camera and lidar for
|
||||
an unbuilt robot. House runs **Home Assistant + UniFi Protect, not Frigate**; Protect
|
||||
exposes RTSPS per-channel (verified, currently off). Recommendation on the table: the
|
||||
Orin is the **robot brain** (Isaac ROS), not the house computer — ESH already owns four
|
||||
idle hardware decoders and the house voice stack is CPU-work that belongs beside HA on
|
||||
esh-docker-vm. Blocked on which depth camera and lidar. Not decided.
|
||||
|
||||
### eBay listing pipeline — scoped, unstarted
|
||||
- Opto-mechanical resale. Verified eBay's **Sell Inventory API** is the right surface
|
||||
(bulk item/offer/publish, 25/call; requires Business Policies opt-in) and that
|
||||
`getItemAspectsForCategory` turns listing generation into schema population. Plan:
|
||||
EXIF-gap clustering → VLM identification → Booth ask for approval → bulk publish.
|
||||
Nothing built.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-12]` ⭐⭐⭐ **FV CUTOVER EXECUTED — fv-ml1 live at Fountain Valley, renamed, renumbered to 10.251/16, serving inference; BMC online after finding it was tagging 802.1q VLAN 250 into an untagged port; and the box has FOUR RTX PRO 6000 (391 GB VRAM), not the two every doc claimed.** Also: OPNsense write APIs need an `X-CSRFToken` header scraped from a `<script>` block — session cookies alone 403, which is why reboots appeared to work while the operator was power-cycling by hand. → `persistent-memory.d/2026-09-12-fv-cutover-executed.md`
|
||||
|
||||
- `[2026-09-12]` **esh-vm-db Restic fixed:** stale April PG dump caused by TCP auth + masked errors; peer-auth/fail-closed hook and bounded retries installed. Snapshot bc5eeaff and repository check verified. → `persistent-memory.d/2026-09-12-esh-vm-db-restic-repair.md`
|
||||
|
||||
- `[2026-09-11]` **Worldtree memory-split (U6) — PROTOCOL AGREED with worldtree-dev: nobody flips `memory.reader.enabled` or `memory.writer.enabled` on any deployment without infra-ops first confirming the memory root is writable by the container's uid.** The reader **REFUSES AT BOOT** if it cannot append+read back `<memory root>/reader/canary.jsonl` (deliberate, the #335 typo'd-reranker precedent: refuse loudly, never silently disable); per-euid subdirs are created lazily and only warn, so the **root canary is the only boot-blocking check**. The writer degrades rather than refuses. Both ship DARK (`enabled: false`, parity-only `config/defaults.yaml`) until the operator schedules the tracer skeleton. ⭐ **Measured 2026-09-11 on corviduo-dev — all three deployments PASS**: demo :8080 uid **0** and personal :8081 uid **0** both have `/data/state/memory` at 1000:1000 755 writable; pinned :8082 uid **1000** lacks `memory/` but its parent `/data/state` is 1000:1000 755 so it can create it. ⚠ I had predicted personal was uid 1000 and warned it would fail — **wrong, retracted**; only pinned runs as 1000, and it passes anyway. ⚠ Re-probe immediately before any flip: a permissions reading is a claim about its own date, not about boot time. Heimdall side is clear too — demo and personal grant 7x `tool.*`, pinned uses image defaults, and the lone `tool.evidence.*` is additive, so `tool.memory_read` needs no policy change. Thread `01M2A05WED5W`.
|
||||
|
||||
- `[2026-09-11]` ⭐ **Plex hardware transcoding on the Arc A580 FIXED (esh-pve-nas LXC 105) — every setting was already correct and the fault was one layer below them.** `intel-media-va-driver` **22.3.1** (Apr 2023, stock jammy) predates Arc/DG2 support and exports only `__vaDriverInit_1_14`, against the libva **2.22 Plex BUNDLES** and loads via RPATH. Passthrough, cgroups, `plex` in video+render, HuC authenticated, Plex Pass, `HardwareAcceleratedCodecs=1` and the Arc already selected as `HardwareDevicePath` — all good the whole time. Fixed with Intel's client-GPU repo (rolling `jammy client`) → iHD **24.3.4** (`__vaDriverInit_1_22`) + a consistent libva **2.22.0.2-87** set, now **pinned + `apt-mark hold`** (verified: a simulated upgrade moves 152 packages, touches none of the six). Also repaired a **half-finished prior attempt** — libva/libva-drm hand-installed at 2.22 with `libva-x11` left at 2.14, killing every X11 VA-API app on `va_fool_postp`. ⚠⚠ **`pct snapshot` REFUSES on a bind-mounted guest AND STILL EXITS 0** (LXC 105 has `mp0: /tank/media`) — use `zfs snapshot nvme/subvol-105-disk-0@<tag>` and read it back. ⚠⚠ **A synthetic `Plex Transcoder` run is NOT a valid test** (Plex bundles its own libc among 61 libs; my harness failed identically before and after a fix that worked — no positive control, so its negatives were worthless). Only a **forced transcode** settles it: PASS *names the device* (`testing API vaapi for device '/dev/dri/renderD129' (Intel DG2 [Arc A580])`). ⚠ The original empty `final decoder: , final encoder:` was an **absence of evidence**, not failure — `TranscodeSession` was 0. Jellyfin LXC 107 left alone (operator: not actively used). → `persistent-memory.d/2026-09-11-plex-arc-vaapi.md`, runbook `docs/runbooks/plex-arc-vaapi-jammy.md`
|
||||
@@ -334,22 +357,6 @@ _As of 2026-09-11 ~17:45 PT._
|
||||
|
||||
- `[2026-09-01]` **Idle VRAM on this fleet is a RESERVED scratch pool, not waste.** Operator declined raising `vllm-mog-sec` from `gpu-memory-utilization 0.52`: single-user dev fleet, KV headroom nobody will consume is worth less than room for ephemeral models and small training runs. vLLM's "fully utilize gpu memory" startup hint does NOT apply here. Tracked in auto-memory `feedback_idle_vram_is_reserved_not_waste`.
|
||||
|
||||
- `[2026-08-28]` **althing v3 flag day (U9b) executed, then six releases to 3.1.1 in one afternoon — and the post office MOVED to nh3-docker.** Every v2 command deleted; 73 handles seeded and verified by set difference; 5,043 orphaned wake FIFOs deleted (v2 named them per-session+PID, v3 per-handle). Image now registry-pulled, digest-pinned, under the `claude-bot` namespace. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
|
||||
|
||||
- `[2026-08-28]` **A stale `ALTHING_HANDLE` silently reads another agent's inbox and reports it empty — a SECOND route into the failure v3 exists to prevent.** Outbound mis-signing sometimes gets caught; inbound never does. Shipped as a 3.1.1 warning. ⚠ My `session_handles.json` grounding was wrong (v2 artifact, v3 never opens it) and the same stale source had survived inside my statusline rewrite. → `persistent-memory.d/2026-08-28-handle-resolution-wrong-inbox.md`
|
||||
|
||||
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE, and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured 5.4-18.4 GB, so 27 GB is 3-4 long-lived sessions. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
|
||||
|
||||
- `[2026-08-28]` **`sec` moved to ana-ml2 GPU0 and is serving** (operator-directed) — GPU1 had ~28 GB free against the ~51 GB it reserves, so it could not start there. Re-arms the two-GPU load condition on a circuit that tripped 36h earlier; accepted with the constraint stated. → `persistent-memory.d/2026-08-28-sec-seat-gpu0.md`
|
||||
|
||||
- `[2026-08-28]` **BELAYED by the operator, both explicitly: (a) a cgroup memory cap on CC sessions, (b) putting ana-gw + ana-wg + one BMC on separate power.** Both were my recommendations; neither is open work. Do not re-raise as new — the atop ramps that would inform (a) are now being collected, so revisit only with a week of data. Tracking surface: this entry.
|
||||
|
||||
- `[2026-08-28]` **The deployed CC plugin copies are a release step nobody owns.** `sync_skill.sh` covers the SKILL, not the plugin; both copies must be rsync'd from the repo's `plugin/` on every althing release or they carry the previous release's bugs into the live surface. Raised with forseti for their release notes. Tracking surface: althing thread `01M14QHZNDKDK8KH9DN92VF6VE`.
|
||||
|
||||
- `[2026-08-28]` **althing v3.0.0 flag day (U9b) executed — the post office replaced the P2P bus on both boxes, one-way.** 73 handles seeded and verified by set difference; 5,043 orphaned v2 wake FIFOs deleted (v2 named them per-session+PID and never reaped; v3 names them per-handle, so the leak is bounded by construction); v2 db left inert. → `persistent-memory.d/2026-08-28-althing-v3-cutover.md`
|
||||
|
||||
- `[2026-08-28]` **nh3-dev's three OOM events attribute to CLAUDE CODE — and the "no kernel evidence" was a permissions artifact.** journald was persistent all along; `journalctl` silently shows only your own messages outside `adm`. Single CC sessions measured at 5.4-18.4 GB, so 27 GB is 3-4 mature sessions, not the ~66 a 408 MB estimate implies. sysstat + atop now instrument the ramp. → `persistent-memory.d/2026-08-28-nh3-dev-oom-attribution.md`
|
||||
|
||||
- `[2026-08-25]` **Run 2's base is an OPEN OPERATOR DECISION, deliberately not staged** — four options with materially different safety postures, detailed in Current state. Tracked at althing thread `01M0WQ8W5574KMEVCHCEKEXNS5`. ⚠ Do not let it get filed as a config knob; it is a reversal of the trainee-selection decision.
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** Operator: "note the fused MoE kernel for round two… if we nail it soon, the math has us wanting to restart the run anyway." Training MFU is **8.6%** (27.1 of a benchmarked 313.8 TFLOPS) because `transformers` runs the Gemma-4 experts in a Python loop — 128 experts × 30 layers, ~11,500 iterations per step under gradient checkpointing. ⚠ **The same fused 3-D expert layout that made bitsandbytes skip 88.5% of the model is exactly what a grouped GEMM wants** — the format is good for storage and for fused kernels, and hostile only to naive iteration. Two fixes: `group_by_length` (−29.9% compute, free, but breaks the seeded order manifest and re-opens a batch-composition call brokkr already made) and a grouped-GEMM/compiled MoE forward (the remaining ~10×). **Not applied to the live run** — restarting mid-flight to change batch ordering was judged a bad trade at step ~50 of 1,312.
|
||||
@@ -369,7 +376,7 @@ _As of 2026-09-11 ~17:45 PT._
|
||||
|
||||
|
||||
|
||||
_20 older entries archived to archival-memory.md._
|
||||
_28 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
|
||||
Reference in New Issue
Block a user