From 43135a2768a065f36dcc95ab10ba04d1ce0e62f9 Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Thu, 3 Sep 2026 22:22:11 -0700 Subject: [PATCH] =?UTF-8?q?memory:=20snapshot=20=E2=80=94=20searxng=20move?= =?UTF-8?q?,=20gx10=20rack=20networking,=20mac=20fleet=20provisioning?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- archival-memory.md | 159 ++++++++++++++++++ .../2026-08-12-secrets-broker.md | 45 ----- .../2026-08-19-esh-pve-freeze-dns-spof.md | 108 ------------ .../2026-09-03-gx10-rack-network.md | 46 +++++ .../2026-09-03-mac-fleet-dsh.md | 51 ++++++ .../2026-09-03-searxng-nh3-move.md | 67 ++++++++ persistent-memory.md | 82 +++++---- 7 files changed, 361 insertions(+), 197 deletions(-) delete mode 100644 persistent-memory.d/2026-08-12-secrets-broker.md delete mode 100644 persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md create mode 100644 persistent-memory.d/2026-09-03-gx10-rack-network.md create mode 100644 persistent-memory.d/2026-09-03-mac-fleet-dsh.md create mode 100644 persistent-memory.d/2026-09-03-searxng-nh3-move.md diff --git a/archival-memory.md b/archival-memory.md index 9d3b655..43028d0 100644 --- a/archival-memory.md +++ b/archival-memory.md @@ -3251,3 +3251,162 @@ Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/l - `[2026-08-02]` **`mv complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/ /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t `** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop). _Archived 2026-08-20._ +`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.** + +**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`, +on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi +`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions): +`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org +shared to the operator's primary account, so he sees items too), folder = hostname, item name = +`/`, title-derived slug. Small text → item note; small binary → base64 hidden field; +**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source +metadata fields; idempotent upsert keyed by name. + +**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login +(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation +session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from +backfill. + +**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an +undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant + +both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI +(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments). + +**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's +`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip +verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives / +cargo noise. + +**Post-launch (jackdaw-dev feedback).** Added **`secret rm `** (bw soft-delete to trash, +recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning** +on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose +warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix. +Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing +stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear. + +**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of +truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it** +rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create +`vh/stonehenge-park`. + +**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack +(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`, +`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment + +resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note). +Consumer-confirmed end-to-end by jackdaw-dev. + +Auto-memory: `reference_secrets_broker_cli`. +_Archived 2026-09-03._ + +- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`. +_Archived 2026-09-03._ + +# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it + +Reported by the operator as "routing or DNS issues on the PVC wifi." It was +neither: the internet was healthy the entire time (gateway reporting 3 ms and +209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero +loss). **The house had no name resolution because one VM was down.** + +## The SPOF: one resolver, cross-VLAN, no fallback + +`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired +userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on +`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every +client on the VLAN loses DNS, and it presents as "the wifi is broken." + +It was the only network in the house exposed this way. `Default`, `esh-mgmt`, +`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out); +`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8. + +**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45` +primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver, +verified answering. Applied via the Classic API, +`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73` +with the full object (GET → modify one field → PUT), `rc: ok`. **This was also +the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key +was write-tested. See [[reference_unifi_udm_integration_api_keys]]. + +⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in +parallel, so once AdGuard is back a real share of lookups go to the gateway and +**skip ad-blocking**. This converts a total outage into degraded-but-working. +The actual fix for blocking integrity is a second AdGuard instance NOT on +esh-pve. + +## Root cause: hard freeze, no diagnostics, two suspects + +`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at +**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no +thermal event**. Powered on with its 10G link up, but not answering ARP. + +Two changes landed the day before, and they are not exclusive: + +1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed + `proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the + box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First + boot on the new kernel lasted **20 hours**. +2. **GPU passthrough.** The last kernel messages of the dead boot are + `vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102 + `esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`, + **39 minutes before the freeze**. + +A vfio/i915 regression in the newer kernel would produce exactly this +signature. `6.8.12-16` is still installed and is the held-in-reserve rollback. + +**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the +operator — it is on-demand and there has been no demand. That removes the +suspect without a kernel rollback. + +## Why nobody could recover it remotely — and the fix + +Nothing on the box could reboot it: + +- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a + hard kernel freeze: the frozen kernel is the thing that would have to fire + its timer. This is the trap — the machine *looked* watchdog-protected. +- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only + pets the device while an HA client is connected, and this cluster has no HA + resources. +- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+** + (Intel X710, port 27 on the Garage switch) and presents exactly one MAC. + **AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated + Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged. + Cabling one and provisioning AMT in MEBx remains the open item for *control*; + the watchdog below is the fix for *recovery*. + +**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the +PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is +blacklisted and unloaded, `watchdog-mux` is masked. Verified live: +`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1, +journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves +idempotency (6 skipped / 6 verify OK). + +Firmware does **not** block the TCO timer here — checked for the +`unable to reset NO_REBOOT flag` line before committing to the approach; the +board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`. + +⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever +configured on esh-pve this must be reverted. Not a near-term concern: +`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss +already costs quorum and the survivor would fence itself — HA here would reduce +availability, not raise it. + +⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.** +Proving it means deliberately wedging the host. Untested-but-armed is still +strictly better than softdog; treat a real firing as unconfirmed until tested. + +## Diagnostic corrections worth keeping + +- **"No route to host" was the dead host, not a routing gap.** Two claims made + mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed + over the NH3↔ESH tunnel, and that a firewall isolates it from the server + VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve` + SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from + `esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the + target subnet before concluding "the subnet is unreachable."** +- **UDM `uptime` on a client record is association time, not host uptime.** It + read 2.2 days while the host had been up 20 hours. Use + `journalctl --list-boots` on the host for real boot history. +- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts + that are demonstrably online). `stat/sta` is the live view. +_Archived 2026-09-03._ diff --git a/persistent-memory.d/2026-08-12-secrets-broker.md b/persistent-memory.d/2026-08-12-secrets-broker.md deleted file mode 100644 index 796cc25..0000000 --- a/persistent-memory.d/2026-08-12-secrets-broker.md +++ /dev/null @@ -1,45 +0,0 @@ -`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.** - -**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`, -on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi -`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions): -`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org -shared to the operator's primary account, so he sees items too), folder = hostname, item name = -`/`, title-derived slug. Small text → item note; small binary → base64 hidden field; -**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source -metadata fields; idempotent upsert keyed by name. - -**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login -(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation -session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from -backfill. - -**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an -undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant + -both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI -(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments). - -**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's -`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip -verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives / -cargo noise. - -**Post-launch (jackdaw-dev feedback).** Added **`secret rm `** (bw soft-delete to trash, -recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning** -on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose -warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix. -Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing -stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear. - -**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of -truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it** -rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create -`vh/stonehenge-park`. - -**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack -(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`, -`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment + -resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note). -Consumer-confirmed end-to-end by jackdaw-dev. - -Auto-memory: `reference_secrets_broker_cli`. diff --git a/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md b/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md deleted file mode 100644 index 40b74f8..0000000 --- a/persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md +++ /dev/null @@ -1,108 +0,0 @@ -# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it - -Reported by the operator as "routing or DNS issues on the PVC wifi." It was -neither: the internet was healthy the entire time (gateway reporting 3 ms and -209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero -loss). **The house had no name resolution because one VM was down.** - -## The SPOF: one resolver, cross-VLAN, no fallback - -`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired -userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on -`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every -client on the VLAN loses DNS, and it presents as "the wifi is broken." - -It was the only network in the house exposed this way. `Default`, `esh-mgmt`, -`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out); -`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8. - -**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45` -primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver, -verified answering. Applied via the Classic API, -`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73` -with the full object (GET → modify one field → PUT), `rc: ok`. **This was also -the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key -was write-tested. See [[reference_unifi_udm_integration_api_keys]]. - -⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in -parallel, so once AdGuard is back a real share of lookups go to the gateway and -**skip ad-blocking**. This converts a total outage into degraded-but-working. -The actual fix for blocking integrity is a second AdGuard instance NOT on -esh-pve. - -## Root cause: hard freeze, no diagnostics, two suspects - -`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at -**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no -thermal event**. Powered on with its 10G link up, but not answering ARP. - -Two changes landed the day before, and they are not exclusive: - -1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed - `proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the - box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First - boot on the new kernel lasted **20 hours**. -2. **GPU passthrough.** The last kernel messages of the dead boot are - `vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102 - `esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`, - **39 minutes before the freeze**. - -A vfio/i915 regression in the newer kernel would produce exactly this -signature. `6.8.12-16` is still installed and is the held-in-reserve rollback. - -**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the -operator — it is on-demand and there has been no demand. That removes the -suspect without a kernel rollback. - -## Why nobody could recover it remotely — and the fix - -Nothing on the box could reboot it: - -- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a - hard kernel freeze: the frozen kernel is the thing that would have to fire - its timer. This is the trap — the machine *looked* watchdog-protected. -- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only - pets the device while an HA client is connected, and this cluster has no HA - resources. -- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+** - (Intel X710, port 27 on the Garage switch) and presents exactly one MAC. - **AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated - Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged. - Cabling one and provisioning AMT in MEBx remains the open item for *control*; - the watchdog below is the fix for *recovery*. - -**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the -PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is -blacklisted and unloaded, `watchdog-mux` is masked. Verified live: -`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1, -journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves -idempotency (6 skipped / 6 verify OK). - -Firmware does **not** block the TCO timer here — checked for the -`unable to reset NO_REBOOT flag` line before committing to the approach; the -board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`. - -⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever -configured on esh-pve this must be reverted. Not a near-term concern: -`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss -already costs quorum and the survivor would fence itself — HA here would reduce -availability, not raise it. - -⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.** -Proving it means deliberately wedging the host. Untested-but-armed is still -strictly better than softdog; treat a real firing as unconfirmed until tested. - -## Diagnostic corrections worth keeping - -- **"No route to host" was the dead host, not a routing gap.** Two claims made - mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed - over the NH3↔ESH tunnel, and that a firewall isolates it from the server - VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve` - SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from - `esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the - target subnet before concluding "the subnet is unreachable."** -- **UDM `uptime` on a client record is association time, not host uptime.** It - read 2.2 days while the host had been up 20 hours. Use - `journalctl --list-boots` on the host for real boot history. -- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts - that are demonstrably online). `stat/sta` is the live view. diff --git a/persistent-memory.d/2026-09-03-gx10-rack-network.md b/persistent-memory.d/2026-09-03-gx10-rack-network.md new file mode 100644 index 0000000..006deed --- /dev/null +++ b/persistent-memory.d/2026-09-03-gx10-rack-network.md @@ -0,0 +1,46 @@ +- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.** + +`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only. + +**Operator ruling, and the better design:** put the address on the +**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A +host-side static works until the box moves, and then it is a stale netplan file +on a machine whose address you no longer know. A reservation moves with the MAC. + + UniFi switch port 22 native network -> nh3-servers (VLAN 50) + UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60 + host unchanged, still DHCP + +`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and +was NOT used — annotated as retired at its top. Its safety *ordering* was +followed and is still right. + +⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd +`10.100.0.111` from `nh3-default`. The switch port had to be repointed before +anything else could work. Do not assume a racked port is on the VLAN you asked +for. + +⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two +unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and +written back. + +⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still +up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits +directly on the userland subnet — so "I can reach it on the new address" proved +NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on +inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what +settled it: + + ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10 + ping -c2 -I enP7s7 1.1.1.1 0% loss egress + +Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to +`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot — +verified by reading `/var/lib/NetworkManager/NetworkManager.state` back. + +⚠ **The box now has exactly ONE path.** If that switch port or the reservation +breaks it is a rack visit; the escape hatch is deliberately gone. Correct end +state for a racked server, but a posture change from the desk setup — and this +is the box run 3c moved to. + +Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`. diff --git a/persistent-memory.d/2026-09-03-mac-fleet-dsh.md b/persistent-memory.d/2026-09-03-mac-fleet-dsh.md new file mode 100644 index 0000000..dd0600a --- /dev/null +++ b/persistent-memory.d/2026-09-03-mac-fleet-dsh.md @@ -0,0 +1,51 @@ +- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.** + + vuongs-mac-mini 10.100.79.2 infra-ops + lkraven + esh-macbook-air 10.0.10.83 infra-ops + lkraven + esh-mac-studio 10.0.10.10 infra-ops + vhpfi + +Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32 +random chars and vaulted at `/infra-ops-password`. `dsh` runs in the +operator's own account on each, on a **device-scoped** LiteLLM key +(`-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than +trusted from the mint) — a laptop travels, and losing one should be one +revocation, not a fleet key rotation. + +`scripts/provision-mac-dsh.sh [name]` carries every trap; the +operator-run half is `docs/runbooks/mac-provisioning.md`. + +⚠ **`sudo -u ` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit +`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at +a **working install in another account**. Only filesystem permissions stopped +it. The script now refuses to run unless `$HOME` matches the target. + +⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl` +leaves `/Users/` root-owned; the account authenticates, gets a shell, +reports the right `$HOME`, and cannot write to it — surfacing as a bare +"Permission denied" hours later. + +⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers +`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a +bad password, AND a user outside `com.apple.access_ssh`. This produced a false +diagnosis twice in one session — once where the password was a typo +(`no-password` vs `nopassword`) and I blamed the access group, once where the +Studio's operator account is **`vhpfi`, not `lkraven`**. Check +`dscl . -list /Users` FIRST. + +⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and +no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with +"Operation is not permitted without secure token unlock". `dscl` works precisely +because there is no token to desync. True on all three Macs. + +⚠ **FileVault kills remote access across reboots** — the machine sits at the +pre-boot unlock screen with no network. Nothing unattended should depend on a +Mac being reachable after a restart. + +⚠ macOS has no `adduser`, `useradd`, or `timeout`. + +Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook; +`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a +two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the +provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of +`agent-default-model` — without it the picker offers three models the gateway +does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`. diff --git a/persistent-memory.d/2026-09-03-searxng-nh3-move.md b/persistent-memory.d/2026-09-03-searxng-nh3-move.md new file mode 100644 index 0000000..0892f0a --- /dev/null +++ b/persistent-memory.d/2026-09-03-searxng-nh3-move.md @@ -0,0 +1,67 @@ +- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.** + +**The failure.** The ana-docker instance answered `/healthz` every 30s, showed +`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while +returning **0 results for every query tested**. It was running **2026.4.17 +against a current 2026.9.3**: 4.5 months of engine scrapers rotting against +sites that had changed their markup. SearXNG ships near-daily releases for +exactly this reason. + +⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A +container created in April on `:latest` is pinned to April forever. + +⚠ **`/healthz` proves the web app answers and says NOTHING about whether search +works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that +was a stalled backup, the runner audit that trusted liveness for identity, and +the statusline bell that measured a mechanism. + +**Proven before acting**: the new image, same settings file, same host, same +query, in a throwaway container → **20 results where the running one returned +0**. Network was ruled out first — from inside the container DNS resolved and +mojeek/wikipedia were reachable, so engines were reachable and the parsers were +the broken part. + +**Why NH3 and not an in-place update** (operator's call, and the measurement +backs it): + + ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA + nh3-docker egress 70.230.226.88 residential -> not gated the same way + +Search engines gate datacenter ranges. Same reason the fleet keeps a residential +SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied +around. `outgoing.proxies` has the fallback commented in place if NH3's egress +ever changes. + +⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from +NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`, +`github`, `stackoverflow`, `marginalia` work. **General search is effectively +single-engine** — if google cse breaks, it goes quiet again. + +**Two config defects, both silent:** `base_url` still named +`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins, +so nothing broke and the file lied to every reader); and the +`karmasearch.videos` removal key never matched because the engine's real name +has a space in it. + +**`scripts/searxng-health.sh` asserts results > 0** across three unrelated +queries. That is the only check that could have caught this — the mechanism was +healthy throughout. + +**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered +`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets +`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an +empty list is indistinguishable from a broken aggregator, which is precisely how +this hid. Same principle as althing's "unreachable post office is an OUTAGE, +never an empty inbox". + +⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1 +`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install +--force` served a CACHED build** and silently reinstalled the old code — the +installed file still had the v1 API after the source no longer did. +`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what +caught it. + +Old instance stopped and removed; DNS alias repointed to +`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at +`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]] +for the other UniFi-side change the same day. diff --git a/persistent-memory.md b/persistent-memory.md index 75ac0bf..db3ac18 100644 --- a/persistent-memory.md +++ b/persistent-memory.md @@ -1,6 +1,6 @@ # Persistent memory — eshpfi-management -_Last updated: 2026-09-02_ +_Last updated: 2026-09-03_ > **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its > `Written:` stamp is under an hour old, read it (it carries the in-flight @@ -108,49 +108,46 @@ no longer deployed sidecars here. See Recent decisions.) (no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`. ## Current state / in-flight -_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is -baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has -not arrived._ +_As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every +item below is a live commitment or a known-open risk, not work in progress._ -- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000 - Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores - factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD** - (the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it - receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout - can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both - are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose - nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are - **512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count - from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking. - → `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md` -- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a) - `zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving. - **TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills - all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB - at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim - irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a - from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered - and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md` -- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps. - ⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive - each way** with **13 hosts dark including three SureFire CLIENT machines**, which no - `save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not - executed** — operator stood it down. - → `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md` -- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed - by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships, - "trust the HTTP status, not `error_code`" still applies on running instances. -- **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD` - from the vault rather than an inline value. -- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart - hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether - their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because - they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render - agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that - evaporates the moment ComfyUI reloads. +- **Nothing is running.** No training, no deploys pending, no background jobs. + 35 commits sit unpushed on `main` — all docs, runbooks, memory and two + scripts; push is the operator's call. + +- **`web_search` needs a session restart to appear.** The SearXNG MCP server is + registered at user scope and `claude mcp list` reports it Connected, but MCP + servers load at session start — this session does not have the tool. + +- ⚠ **SearXNG general search is effectively single-engine.** `google cse` + carries it; `brave`/`duckduckgo`/`startpage` CAPTCHA even from NH3's + residential egress. If google cse breaks it goes quiet the same way it just + did. `scripts/searxng-health.sh` is the detector. + +- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** Fleecing means a + slow `pbs-ana` can no longer wedge the guest, but a job that once ran at + 941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms. + Nightly at 21:00, `all 1`. Worth its own investigation. + +- ⚠ **pfi-gx10 is now single-path.** Wired only on a reserved 10.100.50.60; the + Wi-Fi escape hatch is deliberately gone. A switch-port or reservation failure + is a rack visit — on the box run 3c was moved to. + +- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their + first client-premises release cut (U10, unscheduled). They wrote the gate down + specifically so it would not quietly become never; ping expected when U10 is + scheduled. + +- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** — + deliberate, they are the operator's personal machines and registering them + implies PFI-managed. They now carry real config, so the omission is a choice + to revisit, not an oversight. ## Recent decisions +- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md` +- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I ` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md` +- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md` - `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md` - `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md` - `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md` @@ -233,7 +230,6 @@ not arrived._ - `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md` -- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md` - `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md` @@ -243,7 +239,6 @@ not arrived._ - `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md` -- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`. - `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now"). @@ -274,7 +269,6 @@ not arrived._ -- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md` - `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/.wav` + `transcripts/.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]] @@ -282,7 +276,7 @@ not arrived._ _Older entries archived to archival-memory.md._ -_239 older entries archived to archival-memory.md._ +_242 older entries archived to archival-memory.md._ ## Tried and abandoned