memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning
This commit is contained in:
@@ -3251,3 +3251,162 @@ Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/l
|
||||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||||
_Archived 2026-08-20._
|
||||
|
||||
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
|
||||
|
||||
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
|
||||
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
|
||||
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
|
||||
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
|
||||
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
|
||||
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
|
||||
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
|
||||
metadata fields; idempotent upsert keyed by name.
|
||||
|
||||
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
|
||||
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
|
||||
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
|
||||
backfill.
|
||||
|
||||
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
|
||||
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
|
||||
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
|
||||
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
|
||||
|
||||
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
|
||||
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
|
||||
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
|
||||
cargo noise.
|
||||
|
||||
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
|
||||
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
|
||||
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
|
||||
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
|
||||
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
|
||||
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
|
||||
|
||||
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
|
||||
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
|
||||
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
|
||||
`vh/stonehenge-park`.
|
||||
|
||||
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
|
||||
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
|
||||
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
|
||||
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
|
||||
Consumer-confirmed end-to-end by jackdaw-dev.
|
||||
|
||||
Auto-memory: `reference_secrets_broker_cli`.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
|
||||
|
||||
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
|
||||
neither: the internet was healthy the entire time (gateway reporting 3 ms and
|
||||
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
|
||||
loss). **The house had no name resolution because one VM was down.**
|
||||
|
||||
## The SPOF: one resolver, cross-VLAN, no fallback
|
||||
|
||||
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
|
||||
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
|
||||
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
|
||||
client on the VLAN loses DNS, and it presents as "the wifi is broken."
|
||||
|
||||
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
|
||||
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
|
||||
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
|
||||
|
||||
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
|
||||
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
|
||||
verified answering. Applied via the Classic API,
|
||||
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
|
||||
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
|
||||
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
|
||||
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
|
||||
|
||||
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
|
||||
parallel, so once AdGuard is back a real share of lookups go to the gateway and
|
||||
**skip ad-blocking**. This converts a total outage into degraded-but-working.
|
||||
The actual fix for blocking integrity is a second AdGuard instance NOT on
|
||||
esh-pve.
|
||||
|
||||
## Root cause: hard freeze, no diagnostics, two suspects
|
||||
|
||||
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
|
||||
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
|
||||
thermal event**. Powered on with its 10G link up, but not answering ARP.
|
||||
|
||||
Two changes landed the day before, and they are not exclusive:
|
||||
|
||||
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
|
||||
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
|
||||
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
|
||||
boot on the new kernel lasted **20 hours**.
|
||||
2. **GPU passthrough.** The last kernel messages of the dead boot are
|
||||
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
|
||||
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
|
||||
**39 minutes before the freeze**.
|
||||
|
||||
A vfio/i915 regression in the newer kernel would produce exactly this
|
||||
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
|
||||
|
||||
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
|
||||
operator — it is on-demand and there has been no demand. That removes the
|
||||
suspect without a kernel rollback.
|
||||
|
||||
## Why nobody could recover it remotely — and the fix
|
||||
|
||||
Nothing on the box could reboot it:
|
||||
|
||||
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
|
||||
hard kernel freeze: the frozen kernel is the thing that would have to fire
|
||||
its timer. This is the trap — the machine *looked* watchdog-protected.
|
||||
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
|
||||
pets the device while an HA client is connected, and this cluster has no HA
|
||||
resources.
|
||||
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
|
||||
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
|
||||
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
|
||||
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
|
||||
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
|
||||
the watchdog below is the fix for *recovery*.
|
||||
|
||||
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
|
||||
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
|
||||
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
|
||||
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
|
||||
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
|
||||
idempotency (6 skipped / 6 verify OK).
|
||||
|
||||
Firmware does **not** block the TCO timer here — checked for the
|
||||
`unable to reset NO_REBOOT flag` line before committing to the approach; the
|
||||
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
|
||||
|
||||
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
|
||||
configured on esh-pve this must be reverted. Not a near-term concern:
|
||||
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
|
||||
already costs quorum and the survivor would fence itself — HA here would reduce
|
||||
availability, not raise it.
|
||||
|
||||
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
|
||||
Proving it means deliberately wedging the host. Untested-but-armed is still
|
||||
strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||||
|
||||
## Diagnostic corrections worth keeping
|
||||
|
||||
- **"No route to host" was the dead host, not a routing gap.** Two claims made
|
||||
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
|
||||
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
|
||||
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
|
||||
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
|
||||
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
|
||||
target subnet before concluding "the subnet is unreachable."**
|
||||
- **UDM `uptime` on a client record is association time, not host uptime.** It
|
||||
read 2.2 days while the host had been up 20 hours. Use
|
||||
`journalctl --list-boots` on the host for real boot history.
|
||||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||||
that are demonstrably online). `stat/sta` is the live view.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
|
||||
|
||||
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
|
||||
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
|
||||
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
|
||||
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
|
||||
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
|
||||
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
|
||||
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
|
||||
metadata fields; idempotent upsert keyed by name.
|
||||
|
||||
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
|
||||
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
|
||||
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
|
||||
backfill.
|
||||
|
||||
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
|
||||
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
|
||||
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
|
||||
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
|
||||
|
||||
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
|
||||
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
|
||||
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
|
||||
cargo noise.
|
||||
|
||||
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
|
||||
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
|
||||
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
|
||||
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
|
||||
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
|
||||
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
|
||||
|
||||
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
|
||||
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
|
||||
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
|
||||
`vh/stonehenge-park`.
|
||||
|
||||
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
|
||||
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
|
||||
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
|
||||
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
|
||||
Consumer-confirmed end-to-end by jackdaw-dev.
|
||||
|
||||
Auto-memory: `reference_secrets_broker_cli`.
|
||||
@@ -1,108 +0,0 @@
|
||||
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
|
||||
|
||||
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
|
||||
neither: the internet was healthy the entire time (gateway reporting 3 ms and
|
||||
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
|
||||
loss). **The house had no name resolution because one VM was down.**
|
||||
|
||||
## The SPOF: one resolver, cross-VLAN, no fallback
|
||||
|
||||
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
|
||||
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
|
||||
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
|
||||
client on the VLAN loses DNS, and it presents as "the wifi is broken."
|
||||
|
||||
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
|
||||
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
|
||||
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
|
||||
|
||||
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
|
||||
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
|
||||
verified answering. Applied via the Classic API,
|
||||
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
|
||||
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
|
||||
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
|
||||
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
|
||||
|
||||
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
|
||||
parallel, so once AdGuard is back a real share of lookups go to the gateway and
|
||||
**skip ad-blocking**. This converts a total outage into degraded-but-working.
|
||||
The actual fix for blocking integrity is a second AdGuard instance NOT on
|
||||
esh-pve.
|
||||
|
||||
## Root cause: hard freeze, no diagnostics, two suspects
|
||||
|
||||
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
|
||||
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
|
||||
thermal event**. Powered on with its 10G link up, but not answering ARP.
|
||||
|
||||
Two changes landed the day before, and they are not exclusive:
|
||||
|
||||
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
|
||||
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
|
||||
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
|
||||
boot on the new kernel lasted **20 hours**.
|
||||
2. **GPU passthrough.** The last kernel messages of the dead boot are
|
||||
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
|
||||
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
|
||||
**39 minutes before the freeze**.
|
||||
|
||||
A vfio/i915 regression in the newer kernel would produce exactly this
|
||||
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
|
||||
|
||||
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
|
||||
operator — it is on-demand and there has been no demand. That removes the
|
||||
suspect without a kernel rollback.
|
||||
|
||||
## Why nobody could recover it remotely — and the fix
|
||||
|
||||
Nothing on the box could reboot it:
|
||||
|
||||
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
|
||||
hard kernel freeze: the frozen kernel is the thing that would have to fire
|
||||
its timer. This is the trap — the machine *looked* watchdog-protected.
|
||||
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
|
||||
pets the device while an HA client is connected, and this cluster has no HA
|
||||
resources.
|
||||
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
|
||||
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
|
||||
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
|
||||
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
|
||||
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
|
||||
the watchdog below is the fix for *recovery*.
|
||||
|
||||
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
|
||||
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
|
||||
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
|
||||
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
|
||||
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
|
||||
idempotency (6 skipped / 6 verify OK).
|
||||
|
||||
Firmware does **not** block the TCO timer here — checked for the
|
||||
`unable to reset NO_REBOOT flag` line before committing to the approach; the
|
||||
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
|
||||
|
||||
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
|
||||
configured on esh-pve this must be reverted. Not a near-term concern:
|
||||
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
|
||||
already costs quorum and the survivor would fence itself — HA here would reduce
|
||||
availability, not raise it.
|
||||
|
||||
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
|
||||
Proving it means deliberately wedging the host. Untested-but-armed is still
|
||||
strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||||
|
||||
## Diagnostic corrections worth keeping
|
||||
|
||||
- **"No route to host" was the dead host, not a routing gap.** Two claims made
|
||||
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
|
||||
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
|
||||
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
|
||||
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
|
||||
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
|
||||
target subnet before concluding "the subnet is unreachable."**
|
||||
- **UDM `uptime` on a client record is association time, not host uptime.** It
|
||||
read 2.2 days while the host had been up 20 hours. Use
|
||||
`journalctl --list-boots` on the host for real boot history.
|
||||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||||
that are demonstrably online). `stat/sta` is the live view.
|
||||
@@ -0,0 +1,46 @@
|
||||
- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.**
|
||||
|
||||
`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
|
||||
|
||||
**Operator ruling, and the better design:** put the address on the
|
||||
**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A
|
||||
host-side static works until the box moves, and then it is a stale netplan file
|
||||
on a machine whose address you no longer know. A reservation moves with the MAC.
|
||||
|
||||
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
|
||||
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
|
||||
host unchanged, still DHCP
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and
|
||||
was NOT used — annotated as retired at its top. Its safety *ordering* was
|
||||
followed and is still right.
|
||||
|
||||
⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd
|
||||
`10.100.0.111` from `nh3-default`. The switch port had to be repointed before
|
||||
anything else could work. Do not assume a racked port is on the VLAN you asked
|
||||
for.
|
||||
|
||||
⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two
|
||||
unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and
|
||||
written back.
|
||||
|
||||
⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still
|
||||
up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits
|
||||
directly on the userland subnet — so "I can reach it on the new address" proved
|
||||
NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on
|
||||
inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what
|
||||
settled it:
|
||||
|
||||
ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10
|
||||
ping -c2 -I enP7s7 1.1.1.1 0% loss egress
|
||||
|
||||
Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to
|
||||
`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot —
|
||||
verified by reading `/var/lib/NetworkManager/NetworkManager.state` back.
|
||||
|
||||
⚠ **The box now has exactly ONE path.** If that switch port or the reservation
|
||||
breaks it is a rack visit; the escape hatch is deliberately gone. Correct end
|
||||
state for a racked server, but a posture change from the desk setup — and this
|
||||
is the box run 3c moved to.
|
||||
|
||||
Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`.
|
||||
@@ -0,0 +1,51 @@
|
||||
- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.**
|
||||
|
||||
vuongs-mac-mini 10.100.79.2 infra-ops + lkraven
|
||||
esh-macbook-air 10.0.10.83 infra-ops + lkraven
|
||||
esh-mac-studio 10.0.10.10 infra-ops + vhpfi
|
||||
|
||||
Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32
|
||||
random chars and vaulted at `<name>/infra-ops-password`. `dsh` runs in the
|
||||
operator's own account on each, on a **device-scoped** LiteLLM key
|
||||
(`<name>-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than
|
||||
trusted from the mint) — a laptop travels, and losing one should be one
|
||||
revocation, not a fleet key rotation.
|
||||
|
||||
`scripts/provision-mac-dsh.sh <host> <account> [name]` carries every trap; the
|
||||
operator-run half is `docs/runbooks/mac-provisioning.md`.
|
||||
|
||||
⚠ **`sudo -u <user>` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit
|
||||
`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at
|
||||
a **working install in another account**. Only filesystem permissions stopped
|
||||
it. The script now refuses to run unless `$HOME` matches the target.
|
||||
|
||||
⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl`
|
||||
leaves `/Users/<account>` root-owned; the account authenticates, gets a shell,
|
||||
reports the right `$HOME`, and cannot write to it — surfacing as a bare
|
||||
"Permission denied" hours later.
|
||||
|
||||
⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers
|
||||
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
|
||||
bad password, AND a user outside `com.apple.access_ssh`. This produced a false
|
||||
diagnosis twice in one session — once where the password was a typo
|
||||
(`no-password` vs `nopassword`) and I blamed the access group, once where the
|
||||
Studio's operator account is **`vhpfi`, not `lkraven`**. Check
|
||||
`dscl . -list /Users` FIRST.
|
||||
|
||||
⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and
|
||||
no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
|
||||
"Operation is not permitted without secure token unlock". `dscl` works precisely
|
||||
because there is no token to desync. True on all three Macs.
|
||||
|
||||
⚠ **FileVault kills remote access across reboots** — the machine sits at the
|
||||
pre-boot unlock screen with no network. Nothing unattended should depend on a
|
||||
Mac being reachable after a restart.
|
||||
|
||||
⚠ macOS has no `adduser`, `useradd`, or `timeout`.
|
||||
|
||||
Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook;
|
||||
`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a
|
||||
two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the
|
||||
provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of
|
||||
`agent-default-model` — without it the picker offers three models the gateway
|
||||
does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`.
|
||||
@@ -0,0 +1,67 @@
|
||||
- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
|
||||
|
||||
**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
|
||||
`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
|
||||
returning **0 results for every query tested**. It was running **2026.4.17
|
||||
against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
|
||||
sites that had changed their markup. SearXNG ships near-daily releases for
|
||||
exactly this reason.
|
||||
|
||||
⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
|
||||
container created in April on `:latest` is pinned to April forever.
|
||||
|
||||
⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
|
||||
works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
|
||||
was a stalled backup, the runner audit that trusted liveness for identity, and
|
||||
the statusline bell that measured a mechanism.
|
||||
|
||||
**Proven before acting**: the new image, same settings file, same host, same
|
||||
query, in a throwaway container → **20 results where the running one returned
|
||||
0**. Network was ruled out first — from inside the container DNS resolved and
|
||||
mojeek/wikipedia were reachable, so engines were reachable and the parsers were
|
||||
the broken part.
|
||||
|
||||
**Why NH3 and not an in-place update** (operator's call, and the measurement
|
||||
backs it):
|
||||
|
||||
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
||||
nh3-docker egress 70.230.226.88 residential -> not gated the same way
|
||||
|
||||
Search engines gate datacenter ranges. Same reason the fleet keeps a residential
|
||||
SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
|
||||
around. `outgoing.proxies` has the fallback commented in place if NH3's egress
|
||||
ever changes.
|
||||
|
||||
⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
|
||||
NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
|
||||
`github`, `stackoverflow`, `marginalia` work. **General search is effectively
|
||||
single-engine** — if google cse breaks, it goes quiet again.
|
||||
|
||||
**Two config defects, both silent:** `base_url` still named
|
||||
`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
|
||||
so nothing broke and the file lied to every reader); and the
|
||||
`karmasearch.videos` removal key never matched because the engine's real name
|
||||
has a space in it.
|
||||
|
||||
**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
|
||||
queries. That is the only check that could have caught this — the mechanism was
|
||||
healthy throughout.
|
||||
|
||||
**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
|
||||
`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
|
||||
`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
|
||||
empty list is indistinguishable from a broken aggregator, which is precisely how
|
||||
this hid. Same principle as althing's "unreachable post office is an OUTAGE,
|
||||
never an empty inbox".
|
||||
|
||||
⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
|
||||
`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
|
||||
--force` served a CACHED build** and silently reinstalled the old code — the
|
||||
installed file still had the v1 API after the source no longer did.
|
||||
`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
|
||||
caught it.
|
||||
|
||||
Old instance stopped and removed; DNS alias repointed to
|
||||
`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
|
||||
`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
|
||||
for the other UniFi-side change the same day.
|
||||
+38
-44
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-09-02_
|
||||
_Last updated: 2026-09-03_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under an hour old, read it (it carries the in-flight
|
||||
@@ -108,49 +108,46 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
(no NOPASSWD)** — stage model pulls to `/home`, not root-owned `/worktank`.
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-09-02 — **nothing is running. Operator stood the session down.** The GX10 is
|
||||
baselined and outfitted but still on the desk on Wi-Fi; run 3c is unlaunched; the Ada box has
|
||||
not arrived._
|
||||
_As of 2026-09-03 — **nothing is mid-action; the session closed clean.** Every
|
||||
item below is a live commitment or a known-open risk, not work in progress._
|
||||
|
||||
- **▶ ADA INFERENCE SERVER (R750xa `JPJ1ZP3`) — awaiting an iDRAC inventory.** 2× RTX 6000
|
||||
Ada to be fitted; ComfyUI's new home at NH3. **Order regardless: 8× `M04W6` RDIMM (restores
|
||||
factory 256 GB), 2× NVIDIA `930-00030-1546-000` 12VHPWR adapters, and 2× 2 TB SATA SSD**
|
||||
(the as-bought 1.92 TB pair mirrored is ~1.75 TiB — SMALLER than the 1.82 TiB pool it
|
||||
receives from, landing the migration at ~84% full; pruning cannot fix it, only vdev layout
|
||||
can). ⚠ **The RAM line may be FREE** — operator has 2× Dell R640 spares at 768 GB each; both
|
||||
are DDR4 RDIMM platforms and the Silver 4314s cap this box at 2666 anyway, so R640 DIMMs lose
|
||||
nothing. **768 GB is not a shape the R750xa takes** (16 slots, 8 ch/socket): targets are
|
||||
**512 GB** (16× 32 GB) or **1 TB** (16× 64 GB). ⚠ Confirm DIMM part numbers AND slot count
|
||||
from iDRAC before ordering. ~1 kW loaded — pick the NH3 circuit before racking.
|
||||
→ `persistent-memory.d/2026-09-01-ada-inference-server-r750xa.md`
|
||||
- **⏸ ADA MIGRATION — strategy settled, cutover window is the operator's.** Branch (a)
|
||||
`zfs send`, ~3.9 h for ~1.5 TB at a measured 99 MB/s, incremental so irv-ml1 keeps serving.
|
||||
**TWO decisions with the operator: (1) drives for the R750xa — he is leaning 6, which fills
|
||||
all 8 bays and makes drive SIZE the permanent ceiling; recommend 8 uniform in raidz2 (~10.5 TB
|
||||
at 14%) over 6 mixed, since a raidz vdev caps at its smallest member; (2) retain vs reclaim
|
||||
irv-ml1's `/storetank` after the move — RETAIN recommended** (two paywalled pins mean a
|
||||
from-source rebuild cannot reproduce the stack). comfy-dev's per-file prune audit is offered
|
||||
and **explicitly NOT gating**. → `persistent-memory.d/2026-09-01-ada-migration-branch-a.md`
|
||||
- **⏸ RUN 3c UNLAUNCHED.** Config `/tank/erp-tune/run-03c.json` validated; 604 steps.
|
||||
⚠ **The GX10 is the right home if it moves** — an Anaheim breaker trip is a **40-minute drive
|
||||
each way** with **13 hosts dark including three SureFire CLIENT machines**, which no
|
||||
`save_steps` change mitigates. Port scoped (~2.5 GB + an aarch64 venv rebuild) but **not
|
||||
executed** — operator stood it down.
|
||||
→ `persistent-memory.d/2026-09-01-pfi-gx10-onboarding.md`
|
||||
- **⏸ Worldtree `route_not_found` awaiting the operator's DEPLOY PUSH.** Approved and landed
|
||||
by worldtree-dev as `ece0250c` (wire 2.5.0→2.6.0). main auto-deploys demo. Until it ships,
|
||||
"trust the HTTP status, not `error_code`" still applies on running instances.
|
||||
- **⏸ Deferred, no blocker:** convert the live synapse compose to read `POSTGRES_PASSWORD`
|
||||
from the vault rather than an inline value.
|
||||
- **🔵 Awaiting others:** a **Claude Code restart** to load althing plugin 0.1.1's SessionStart
|
||||
hook (never once run on this box — see the deploy entry); **regin-smithy-dev** on whether
|
||||
their live waiter was meant to give them push (live waiter + `mode: pull`, ambiguous because
|
||||
they intentionally released their pane route); **tts-dev** to ask comfy-dev for a no-render
|
||||
agreement during their FireRedAudio bench, since the A6000's ~44 GB free is a snapshot that
|
||||
evaporates the moment ComfyUI reloads.
|
||||
- **Nothing is running.** No training, no deploys pending, no background jobs.
|
||||
35 commits sit unpushed on `main` — all docs, runbooks, memory and two
|
||||
scripts; push is the operator's call.
|
||||
|
||||
- **`web_search` needs a session restart to appear.** The SearXNG MCP server is
|
||||
registered at user scope and `claude mcp list` reports it Connected, but MCP
|
||||
servers load at session start — this session does not have the tool.
|
||||
|
||||
- ⚠ **SearXNG general search is effectively single-engine.** `google cse`
|
||||
carries it; `brave`/`duckduckgo`/`startpage` CAPTCHA even from NH3's
|
||||
residential egress. If google cse breaks it goes quiet the same way it just
|
||||
did. `scripts/searxng-health.sh` is the detector.
|
||||
|
||||
- ⚠ **The nh3-dev backup throughput cause is UNEXPLAINED.** Fleecing means a
|
||||
slow `pbs-ana` can no longer wedge the guest, but a job that once ran at
|
||||
941 MiB/s ran at 1.4 MiB/s with the link up and pbs-ana answering in 11 ms.
|
||||
Nightly at 21:00, `all 1`. Worth its own investigation.
|
||||
|
||||
- ⚠ **pfi-gx10 is now single-path.** Wired only on a reserved 10.100.50.60; the
|
||||
Wi-Fi escape hatch is deliberately gone. A switch-port or reservation failure
|
||||
is a rack visit — on the box run 3c was moved to.
|
||||
|
||||
- **Open commitment to vastblue-dev:** a dedicated CI runner, gated on their
|
||||
first client-premises release cut (U10, unscheduled). They wrote the gate down
|
||||
specifically so it would not quietly become never; ping expected when U10 is
|
||||
scheduled.
|
||||
|
||||
- **Neither Mac nor the Studio is in `servers/` or `dns/internal.yaml`** —
|
||||
deliberate, they are the operator's personal machines and registering them
|
||||
implies PFI-managed. They now carry real config, so the omission is a choice
|
||||
to revisit, not an oversight.
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-09-03]` **SearXNG returned ZERO results for every query while reporting `healthy` for 7 days — 4.5 months stale.** Moved to nh3-docker (residential egress beats the colo's CAPTCHA-gated 38.120.12.42), updated, and exposed to every CC session as the user-scope `web_search` MCP tool. ⚠ `/healthz` cannot tell you whether search works. → `persistent-memory.d/2026-09-03-searxng-nh3-move.md`
|
||||
- `[2026-09-03]` **pfi-gx10 racked: VLAN 50 via a DHCP RESERVATION on the UDM, not a host static — operator ruling, so the box stays portable.** ⚠ The racked port arrived on the NATIVE VLAN; ⚠ `port_overrides` is a whole-array PUT; ⚠ prove inter-VLAN routing with `ping -I <wired>` BEFORE downing the Wi-Fi escape hatch. Now single-path. → `persistent-memory.d/2026-09-03-gx10-rack-network.md`
|
||||
- `[2026-09-03]` **Three Macs onboarded (mini / Air / Studio) with infra-ops, NOPASSWD sudo, rotated+vaulted passwords and `dsh` on device-scoped keys — and the fourth is `scripts/provision-mac-dsh.sh`, not a fourth hand-run.** ⚠ `sudo -u` keeps the CALLER's `$HOME` and nearly wiped a working install; ⚠ a wrong USERNAME is indistinguishable from a wrong password. → `persistent-memory.d/2026-09-03-mac-fleet-dsh.md`
|
||||
- `[2026-09-03]` **nh3-dev wedged for ~40 min and it was the BACKUP, not the disk — a stalled cross-site vzdump holding every guest write via `copy-before-write`.** Symptoms screamed dying disk: 45 writes in flight completing zero, jbd2 + flush kworkers in D state 33 min, io pressure full 96%, load 26, `virtio_ring` in the stack. ⚠ **The discriminator was the ABSENCE of errors** — no SCSI/ATA/IO errors, rpool ONLINE 21%, guest fs 79%, memory fine, and **Dirty only 3.8 MB** (so nothing backed up in page cache; it was stuck BELOW the block layer). ⚠ **The hypervisor was IDLE** — load 0.63, io pressure 0.00, zpool ~0 writes: nothing was reaching the disk because the filter held it. Cause: `vzdump` of VM 102 → **pbs-ana** did 1% at 64 MiB/s then collapsed to **1.4 MiB/s for 35 min**; Proxmox backups interpose a `copy-before-write` filter, so every guest write queues behind the backup's copy-out. FIX = cancel the task (`pvesh delete /nodes/localhost/tasks/<UPID>`); filter detached, inflight 45→0, D-states gone, 191 MB/s dsync restored. ⚠ **`fleecing 0` on the job is why a slow TARGET can stall a GUEST** — fleecing routes copy-before-write to a fast local image instead. Job = `backup-5d8f1221-8f71`, **daily 21:00, `all 1`, storage pbs-ana** → recurs nightly until changed. A prior run of this VM managed 941 MiB/s read, so 1.4 MiB/s is degradation, not normal. → `docs/runbooks/nh3-dev-io-stall.md`
|
||||
- `[2026-09-02]` **althing deploy is SIX surfaces, and #6 is outside the althing repo: `~/.claude/settings.json` `crossSessionInbound: "accept"`.** Without it Claude Code HOLDS every cc poke — it auto-delivers only when the sender's permission-mode class matches, and the herald is a daemon that asserts none, so the notice goes to a human watching the pane instead of to the session. ⚠ **The seat reports declared, reachable and green throughout** — same failure shape as the SessionStart hook that was never deployed. Set on nh3-dev by forseti 09:28 with operator authorization (diff verified: one key, backup at `/tmp/settings.json.bak-20260902T092829`). Operator's reasoning: the herald reaches only local seats and a **pane poke already types+Enters into a session**, so the socket channel is strictly NARROWER than what it replaces — stating the existing trust boundary, not widening it. Cost without it is first-contact-only (in-memory correspondent record), not per-message. ⚠ **No attestation exists for the herald to send** — CC identifies a sender by verified pid against the session registry and reads that session's LIVE runtime mode; a daemon is not in it, and `from_mode` on a `type:"user"` frame is never consulted. **`deploy-althing.sh` reports surface 6 and deliberately never SETS it** — a deploy script that edits its own trust settings grants itself trust. → `docs/runbooks/althing-deploy.md`
|
||||
- `[2026-09-02]` **`vastblue` gitea org created (id 8, private, owner `vh`) with empty repo `vastblue/platform`** — third entity namespace alongside `corviduo` and `pfi`; most repos still live under `vh/`. Home of VastBlueDocumentAI + the anchor healthcare-billing SPA (signed 3-yr client contract). **Org scope was the decision**: org-level runner registration and secrets are inherited free by the DocumentAI repo when it splits out, and that is the only binding expensive to retrofit. Deliberately NOT set: org runner (instance-wide `ana-docker-runner` already serves it; org scope is for the DEDICATED runner, deferred to U10) and org secrets (none exist yet; a guessed secret looks bound). ⚠ **Dedicated runner is gated on the first client-premises release cut**, not on the first green pipeline — the risk is another repo's CI sharing a root-level daemon with a build that ships to a healthcare client, see the runner entry above. Push needs no credential: vastblue-dev is on nh3-dev and git-SSH there auths as `vh`. → `stacks/gitea-runner/README.md`
|
||||
@@ -233,7 +230,6 @@ not arrived._
|
||||
|
||||
- `[2026-08-20]` **Cold-Fusion abliteration — Robinson recipe captured; the fight was the environment, not the recipe.** Stock Cold-Fusion measured ~33% creative refusal → worth abliterating ourselves (supersedes waiting for DavidAU's heretic build). Recipe maps 1:1 (131 tensors); capture succeeded only in **fp32** — transformers' Qwen3.5 DeltaNet linear-attn NaNs nondeterministically in bf16 without the unbuildable `causal-conv1d` kernel (precision cancellation, not overflow). Direction finite at layer 22 but agreement 0.59 (vs Robinson's 0.99) → **calibration-set expansion is next.** → `persistent-memory.d/2026-08-20-coldfusion-abliteration-capture.md`
|
||||
|
||||
- `[2026-08-19]` **A *software* watchdog is not watchdog protection — esh-pve froze for 4.5h holding one.** softdog cannot fire when the kernel it runs in is wedged, and Proxmox's `watchdog-mux` never arms without HA resources, so the box *looked* protected and wasn't. Moved to the PCH `iTCO_wdt` under systemd. Also: a single cross-VLAN DNS entry with no secondary turns any VM outage into a whole-site outage. → `persistent-memory.d/2026-08-19-esh-pve-freeze-dns-spof.md`
|
||||
|
||||
- `[2026-08-19]` **Fleet `.internal` DNS built and live — git-sourced, agent-managed, three resolvers.** Zone-scoped authority (ESH's hand-made `esteban.net` rewrites survive); the colo had no resolver at all; v6 column empty on purpose because SLAAC addresses rotate. → `persistent-memory.d/2026-08-19-fleet-internal-dns.md`
|
||||
|
||||
@@ -243,7 +239,6 @@ not arrived._
|
||||
|
||||
- `[2026-08-19]` **Four unmanaged stacks found on live hosts — two quietly broken.** A dashboard card is a cheap census of what is actually running; check whether the stack is even in `stacks/` before debugging the symptom. → `persistent-memory.d/2026-08-19-unmanaged-stacks-searxng-seafile.md`
|
||||
|
||||
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
|
||||
|
||||
- `[2026-08-19]` **AI-tab Dormant regrouping BELAYED by the operator** — six seats (char-rp Magidonia, char-rp-reasoning Heretic2, Granite summarizer, Qwen-Image-Bench, Skaldsong, Chatterbox Fast) show amber EXITED inside live groups rather than `AI - Dormant`. Fix is a label change + recreate per stack; needs the operator's read on which are retired vs temporarily down. `untracked by operator choice` (his words: "belay the ai dormant regrouping for now").
|
||||
|
||||
@@ -274,7 +269,6 @@ not arrived._
|
||||
|
||||
|
||||
|
||||
- `[2026-08-10→12]` **secrets-broker: per-box Vaultwarden credential store SHIPPED + consumer-confirmed.** `secret` CLI (`put/get/list/rm/backfill`, bw-backed) on `~/.local/bin`; 25 nh3-dev secrets backfilled + round-trip-verified; `rm` + new-namespace warning added post-launch; standing "vault is the credential source of truth" directive now global. → `persistent-memory.d/2026-08-12-secrets-broker.md`
|
||||
|
||||
- `[2026-08-09→10]` **dots.tts (rednote-hilab) TTS burn-in on irv-ml1 + canonical voice corpus built (`voices/`).** Operator-directed eval to potentially replace chatterbox-fast. **dots.tts VERIFIED real** (canonical HF ns `dots-studio/`, `rednote-hilab/dots.tts-*` redirects there; Apache-2.0; PyPI `dots.tts` 0.2.1; 2B continuous-AR = semantic enc + Qwen2.5-1.5B LLM + flow-matching acoustic head over 48kHz AudioVAE; zero-shot clone from wav+transcript). **Runs on Ampere 3090** (sm_86, bf16, no fp8 dep); **optimized RTF 0.22** at num_steps=10 (`from_pretrained(..., optimize=True)` CUDA graphs — raw unoptimized was 1.21), **~6GB VRAM**, 48kHz, streams (`generate_stream`). Venv+cache at `irv-ml1:/home/lkraven/dots-tts` (~10GB). **Operator design calls:** SGLang Omni serving (OpenAI `/v1/audio/speech`), transcribe-refs-first, `soar` variant. ⚠ Omni serves soar but its continuous-batching + streaming opts are **mf-only** (soar = single-request) — non-issue for ratatoskr's single-consumer RP surface. **KEY FINDING — dots is highly sensitive to an accurate AND sentence-bounded reference transcript:** mismatched transcript → 0.16s collapse; over-long/messy transcript → reference-audio BLEEDS as an output prefix; mid-clause trim → dangling-word leak (glados "we'll", emmie "And,"). RECIPE (baked into `voices/derive.py`): trim ref to a clean ~6–10s clip ending on a sentence boundary + accurate transcript of exactly that clip. **CANONICAL VOICE CORPUS** stood up in eshpfi `voices/` (operator idea): engine-agnostic `canonical/<v>.wav` + `transcripts/<v>.txt` → per-engine ref sets DERIVED by `derive.py` reading `engines.yaml` profiles (dots/chatterbox/zonos); canonical wavs git-tracked (small/curated), `derived/` gitignored. **4 voices optimized + verified CLEAN for dots: donut, glados, emmie, miranda** (glados canonical is low-SR 16kHz — flagged upgrade candidate). ⚠ GPU GOTCHA: irv-ml1 native CUDA orders **A6000=device0** (ComfyUI-full) — pin the 3090 with `CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0`; and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments` CONFLICTS with `optimize=True` CUDA graphs (curr_block error). Booths: `dots-vs-chatterbox`, `dots-voices-optimized`. **SHIPPED 2026-08-10:** operator A/B verdict "dots is very good" → containerized as a **thin FastAPI wrapper over DotsTtsRuntime** (chosen over SGLang Omni — Omni's batching is mf-only, unneeded for ratatoskr's single consumer; wrapper is SERIALIZED one-gen-at-a-time via a threading.Lock, Omni+mf = parked API-compatible escalation if multi-consumer ever lands). **LIVE on irv-ml1:8198** (`local/dots-tts:v1`, OpenAI `/v1/audio/speech` + `/health` + `/v1/voices`, container healthy, both stream + non-stream verified CLEAN, 4 voices donut/glados/emmie/miranda) alongside chatterbox :8197 (nothing repointed). Stack = `stacks/dots-tts/` (Dockerfile/app.py/compose/.env.example/README). ⚠ CONTAINER GOTCHA: `optimize=True` (torch.compile/inductor/triton) needs a **C compiler at RUNTIME** — slim image must `apt install build-essential` or model-load dies "Failed to find C compiler" (host venv had gcc ambient, masking it); persist `TORCHINDUCTOR_CACHE_DIR` to a mounted dir or every restart re-JITs ~5min. Corpus home = eshpfi `voices/` (operator ruled keep-here). **REMAINING: ratatoskr client cutover** to :8198 `/v1/audio/speech` (Phase-2 tail, peer-coupled — draft the ask). [[reference_chatterbox_fast_repo]] [[reference_zonos_tts_stack]] [[reference_verify_hf_repo_ids_before_pull]]
|
||||
|
||||
@@ -282,7 +276,7 @@ not arrived._
|
||||
|
||||
_Older entries archived to archival-memory.md._
|
||||
|
||||
_239 older entries archived to archival-memory.md._
|
||||
_242 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
|
||||
Reference in New Issue
Block a user