memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning

persistent-memory.md

@@ -1,6 +1,6 @@
 # Persistent memory — eshpfi-management
 
-_Last updated: 2026-09-02_
+_Last updated: 2026-09-03_
 
 > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
 > `Written:` stamp is under an hour old, read it (it carries the in-flight
@@ -108,49 +108,46 @@ no longer deployed sidecars here. See Recent decisions.)
   (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
 ## Current state / in-flight
 
-_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is
-baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has
-not arrived._
+_As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every
+item below is a live commitment or a known-open risk, not work in progress._
 
-- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
-  Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
-  factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
-  (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it
-  receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout
-  can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both
-  are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose
-  nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are
-  **512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count
-  from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking.
-  → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
-- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
-  `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
-  **TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills
-  all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB
-  at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim
-  irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a
-  from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered
-  and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
-- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps.
-  ⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive
-  each way** with **13 hosts dark including three SureFire CLIENT machines**, which no
-  `save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not
-  executed** — operator stood it down.
-  → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
-- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
-  by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
-  "trust the HTTP status, not `error_code`" still applies on running instances.
-- **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD`
-  from the vault rather than an inline value.
-- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart
-  hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether
-  their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because
-  they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render
-  agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that
-  evaporates the moment ComfyUI reloads.
+- **Nothing is running.** No training, no deploys pending, no background jobs.
+  35 commits sit unpushed on `main` — all docs, runbooks, memory and two
+  scripts; push is the operator's call.
+
+- **`web_search` needs a session restart to appear.** The SearXNG MCP server is
+  registered at user scope and `claude mcp list` reports it Connected, but MCP
+  servers load at session start — this session does not have the tool.
+
+- ⚠ **SearXNG general search is effectively single-engine.** `google cse`
+  carries it; `brave`/`duckduckgo`/`startpage` CAPTCHA even from NH3's
+  residential egress. If google cse breaks it goes quiet the same way it just
+  did. `scripts/searxng-health.sh` is the detector.
+
+- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** Fleecing means a
+  slow `pbs-ana` can no longer wedge the guest, but a job that once ran at
+  941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
+  Nightly at 21:00, `all 1`. Worth its own investigation.
+
+- ⚠ **pfi-gx10 is now single-path.** Wired only on a reserved 10.100.50.60; the
+  Wi-Fi escape hatch is deliberately gone. A switch-port or reservation failure
+  is a rack visit — on the box run 3c was moved to.
+
+- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their
+  first client-premises release cut (U10, unscheduled). They wrote the gate down
+  specifically so it would not quietly become never; ping expected when U10 is
+  scheduled.
+
+- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** —
+  deliberate, they are the operator's personal machines and registering them
+  implies PFI-managed. They now carry real config, so the omission is a choice
+  to revisit, not an oversight.
 
 ## Recent decisions
 
+- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md`
+- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I <wired>` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md`
+- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md`
 - `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
 - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
 - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
@@ -233,7 +230,6 @@ not arrived._
 
 - `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
 
-- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
 
 - `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
 
@@ -243,7 +239,6 @@ not arrived._
 
 - `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md`
 
-- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
 
 - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
 
@@ -274,7 +269,6 @@ not arrived._
 
 
 
-- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
 
 - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
 
@@ -282,7 +276,7 @@ not arrived._
 
 _Older entries archived to archival-memory.md._
 
-_239 older entries archived to archival-memory.md._
+_242 older entries archived to archival-memory.md._
 
 ## Tried and abandoned