memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning
This commit is contained in:
@@ -1,45 +0,0 @@
|
||||
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
|
||||
|
||||
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
|
||||
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
|
||||
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
|
||||
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
|
||||
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
|
||||
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
|
||||
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
|
||||
metadata fields; idempotent upsert keyed by name.
|
||||
|
||||
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
|
||||
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
|
||||
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
|
||||
backfill.
|
||||
|
||||
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
|
||||
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
|
||||
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
|
||||
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
|
||||
|
||||
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
|
||||
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
|
||||
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
|
||||
cargo noise.
|
||||
|
||||
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
|
||||
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
|
||||
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
|
||||
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
|
||||
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
|
||||
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
|
||||
|
||||
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
|
||||
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
|
||||
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
|
||||
`vh/stonehenge-park`.
|
||||
|
||||
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
|
||||
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
|
||||
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
|
||||
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
|
||||
Consumer-confirmed end-to-end by jackdaw-dev.
|
||||
|
||||
Auto-memory: `reference_secrets_broker_cli`.
|
||||
@@ -1,108 +0,0 @@
|
||||
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
|
||||
|
||||
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
|
||||
neither: the internet was healthy the entire time (gateway reporting 3 ms and
|
||||
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
|
||||
loss). **The house had no name resolution because one VM was down.**
|
||||
|
||||
## The SPOF: one resolver, cross-VLAN, no fallback
|
||||
|
||||
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
|
||||
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
|
||||
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
|
||||
client on the VLAN loses DNS, and it presents as "the wifi is broken."
|
||||
|
||||
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
|
||||
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
|
||||
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
|
||||
|
||||
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
|
||||
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
|
||||
verified answering. Applied via the Classic API,
|
||||
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
|
||||
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
|
||||
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
|
||||
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
|
||||
|
||||
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
|
||||
parallel, so once AdGuard is back a real share of lookups go to the gateway and
|
||||
**skip ad-blocking**. This converts a total outage into degraded-but-working.
|
||||
The actual fix for blocking integrity is a second AdGuard instance NOT on
|
||||
esh-pve.
|
||||
|
||||
## Root cause: hard freeze, no diagnostics, two suspects
|
||||
|
||||
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
|
||||
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
|
||||
thermal event**. Powered on with its 10G link up, but not answering ARP.
|
||||
|
||||
Two changes landed the day before, and they are not exclusive:
|
||||
|
||||
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
|
||||
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
|
||||
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
|
||||
boot on the new kernel lasted **20 hours**.
|
||||
2. **GPU passthrough.** The last kernel messages of the dead boot are
|
||||
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
|
||||
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
|
||||
**39 minutes before the freeze**.
|
||||
|
||||
A vfio/i915 regression in the newer kernel would produce exactly this
|
||||
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
|
||||
|
||||
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
|
||||
operator — it is on-demand and there has been no demand. That removes the
|
||||
suspect without a kernel rollback.
|
||||
|
||||
## Why nobody could recover it remotely — and the fix
|
||||
|
||||
Nothing on the box could reboot it:
|
||||
|
||||
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
|
||||
hard kernel freeze: the frozen kernel is the thing that would have to fire
|
||||
its timer. This is the trap — the machine *looked* watchdog-protected.
|
||||
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
|
||||
pets the device while an HA client is connected, and this cluster has no HA
|
||||
resources.
|
||||
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
|
||||
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
|
||||
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
|
||||
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
|
||||
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
|
||||
the watchdog below is the fix for *recovery*.
|
||||
|
||||
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
|
||||
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
|
||||
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
|
||||
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
|
||||
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
|
||||
idempotency (6 skipped / 6 verify OK).
|
||||
|
||||
Firmware does **not** block the TCO timer here — checked for the
|
||||
`unable to reset NO_REBOOT flag` line before committing to the approach; the
|
||||
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
|
||||
|
||||
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
|
||||
configured on esh-pve this must be reverted. Not a near-term concern:
|
||||
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
|
||||
already costs quorum and the survivor would fence itself — HA here would reduce
|
||||
availability, not raise it.
|
||||
|
||||
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
|
||||
Proving it means deliberately wedging the host. Untested-but-armed is still
|
||||
strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||||
|
||||
## Diagnostic corrections worth keeping
|
||||
|
||||
- **"No route to host" was the dead host, not a routing gap.** Two claims made
|
||||
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
|
||||
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
|
||||
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
|
||||
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
|
||||
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
|
||||
target subnet before concluding "the subnet is unreachable."**
|
||||
- **UDM `uptime` on a client record is association time, not host uptime.** It
|
||||
read 2.2 days while the host had been up 20 hours. Use
|
||||
`journalctl --list-boots` on the host for real boot history.
|
||||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||||
that are demonstrably online). `stat/sta` is the live view.
|
||||
@@ -0,0 +1,46 @@
|
||||
- `[2026-09-03]` **pfi-gx10 racked and networked: VLAN 50 via a DHCP RESERVATION, not a host static; Wi-Fi down.**
|
||||
|
||||
`pfi-gx10.nh3.internal` → **10.100.50.60**, wired only.
|
||||
|
||||
**Operator ruling, and the better design:** put the address on the
|
||||
**switch/firewall side** as a DHCP reservation and leave the host on DHCP. A
|
||||
host-side static works until the box moves, and then it is a stale netplan file
|
||||
on a machine whose address you no longer know. A reservation moves with the MAC.
|
||||
|
||||
UniFi switch port 22 native network -> nh3-servers (VLAN 50)
|
||||
UniFi client reservation -> 30:c5:99:3d:a7:45 = 10.100.50.60
|
||||
host unchanged, still DHCP
|
||||
|
||||
`playbooks/gx10-rack-network.yaml` was pre-written to apply a **host static** and
|
||||
was NOT used — annotated as retired at its top. Its safety *ordering* was
|
||||
followed and is still right.
|
||||
|
||||
⚠ **The port arrived on the native VLAN**, not the server VLAN — it DHCP'd
|
||||
`10.100.0.111` from `nh3-default`. The switch port had to be repointed before
|
||||
anything else could work. Do not assume a racked port is on the VLAN you asked
|
||||
for.
|
||||
|
||||
⚠ **`port_overrides` is a WHOLE-ARRAY PUT.** Anything omitted is deleted. Two
|
||||
unrelated overrides (ports 21, 23) were read, backed up to a file, preserved and
|
||||
written back.
|
||||
|
||||
⚠ **The step that is easy to skip and expensive to miss:** while Wi-Fi was still
|
||||
up, traffic from the box to nh3-dev **preferred `wlP9s9`** — that interface sits
|
||||
directly on the userland subnet — so "I can reach it on the new address" proved
|
||||
NOTHING about the wired path. Downing Wi-Fi on that evidence is a coin flip on
|
||||
inter-VLAN routing, and losing it is a rack visit. Forcing the interface is what
|
||||
settled it:
|
||||
|
||||
ping -c3 -I enP7s7 10.100.10.50 0% loss VLAN 50 -> VLAN 10
|
||||
ping -c2 -I enP7s7 1.1.1.1 0% loss egress
|
||||
|
||||
Only then did Wi-Fi come down, as its own step, `/etc/netplan` backed up to
|
||||
`/etc/netplan.bak-preWifiDown`. `nmcli radio wifi off` persists across reboot —
|
||||
verified by reading `/var/lib/NetworkManager/NetworkManager.state` back.
|
||||
|
||||
⚠ **The box now has exactly ONE path.** If that switch port or the reservation
|
||||
breaks it is a rack visit; the escape hatch is deliberately gone. Correct end
|
||||
state for a racked server, but a posture change from the desk setup — and this
|
||||
is the box run 3c moved to.
|
||||
|
||||
Runbook `docs/runbooks/gx10-rack-network.md`; commit `a95717e`.
|
||||
@@ -0,0 +1,51 @@
|
||||
- `[2026-09-03]` **Three Macs onboarded with infra-ops + NOPASSWD sudo + the DeepSeek Harness, and the fourth is a script instead of a fourth hand-run.**
|
||||
|
||||
vuongs-mac-mini 10.100.79.2 infra-ops + lkraven
|
||||
esh-macbook-air 10.0.10.83 infra-ops + lkraven
|
||||
esh-mac-studio 10.0.10.10 infra-ops + vhpfi
|
||||
|
||||
Each: key auth, `visudo`-validated NOPASSWD drop-in, password rotated to 32
|
||||
random chars and vaulted at `<name>/infra-ops-password`. `dsh` runs in the
|
||||
operator's own account on each, on a **device-scoped** LiteLLM key
|
||||
(`<name>-dsh`, scoped to `gen-reasoning`, scope verified 200/403 rather than
|
||||
trusted from the mint) — a laptop travels, and losing one should be one
|
||||
revocation, not a fleet key rotation.
|
||||
|
||||
`scripts/provision-mac-dsh.sh <host> <account> [name]` carries every trap; the
|
||||
operator-run half is `docs/runbooks/mac-provisioning.md`.
|
||||
|
||||
⚠ **`sudo -u <user>` KEEPS THE CALLER'S `$HOME`.** Without `-H` and an explicit
|
||||
`HOME=`, `"$HOME/.local"` resolved to the caller's home and an `rm -rf` aimed at
|
||||
a **working install in another account**. Only filesystem permissions stopped
|
||||
it. The script now refuses to run unless `$HOME` matches the target.
|
||||
|
||||
⚠ **An account may not own its own home.** A `sudo mkdir` before `sysadminctl`
|
||||
leaves `/Users/<account>` root-owned; the account authenticates, gets a shell,
|
||||
reports the right `$HOME`, and cannot write to it — surfacing as a bare
|
||||
"Permission denied" hours later.
|
||||
|
||||
⚠ **A wrong USERNAME looks exactly like a wrong password.** sshd answers
|
||||
`Permission denied (publickey,password,keyboard-interactive)` for a bad user, a
|
||||
bad password, AND a user outside `com.apple.access_ssh`. This produced a false
|
||||
diagnosis twice in one session — once where the password was a typo
|
||||
(`no-password` vs `nopassword`) and I blamed the access group, once where the
|
||||
Studio's operator account is **`vhpfi`, not `lkraven`**. Check
|
||||
`dscl . -list /Users` FIRST.
|
||||
|
||||
⚠ **Rotation: use `dscl . -passwd`, not `sysadminctl`.** With FileVault on and
|
||||
no Secure Token on the account, `sysadminctl -resetPasswordFor` refuses with
|
||||
"Operation is not permitted without secure token unlock". `dscl` works precisely
|
||||
because there is no token to desync. True on all three Macs.
|
||||
|
||||
⚠ **FileVault kills remote access across reboots** — the machine sits at the
|
||||
pre-boot unlock screen with no network. Nothing unattended should depend on a
|
||||
Mac being reachable after a restart.
|
||||
|
||||
⚠ macOS has no `adduser`, `useradd`, or `timeout`.
|
||||
|
||||
Harness config (all machines): `high` → the seat's `xhigh` via the gateway hook;
|
||||
`maxTokens 32768` (the 256000 default left 6144 for input and overflowed on a
|
||||
two-word prompt); `defaultContextWindow 262144`; and `models:` **replacing** the
|
||||
provider's hard-coded DeepSeek catalog, which the web GUI reads INDEPENDENTLY of
|
||||
`agent-default-model` — without it the picker offers three models the gateway
|
||||
does not serve while headless runs work fine. Commits `6ca455a`, `926fc2f`.
|
||||
@@ -0,0 +1,67 @@
|
||||
- `[2026-09-03]` **SearXNG was returning ZERO results for every query while reporting `healthy` — moved to nh3-docker, updated, and exposed to every CC session as an MCP tool.**
|
||||
|
||||
**The failure.** The ana-docker instance answered `/healthz` every 30s, showed
|
||||
`Up 7 days (healthy)` with 0 restarts, and had a green Homepage card — while
|
||||
returning **0 results for every query tested**. It was running **2026.4.17
|
||||
against a current 2026.9.3**: 4.5 months of engine scrapers rotting against
|
||||
sites that had changed their markup. SearXNG ships near-daily releases for
|
||||
exactly this reason.
|
||||
|
||||
⚠ **`:latest` means "latest AT PULL TIME".** Nothing re-pulls on its own. A
|
||||
container created in April on `:latest` is pinned to April forever.
|
||||
|
||||
⚠ **`/healthz` proves the web app answers and says NOTHING about whether search
|
||||
works.** That is the whole lesson. Same shape as the nh3-dev "failing disk" that
|
||||
was a stalled backup, the runner audit that trusted liveness for identity, and
|
||||
the statusline bell that measured a mechanism.
|
||||
|
||||
**Proven before acting**: the new image, same settings file, same host, same
|
||||
query, in a throwaway container → **20 results where the running one returned
|
||||
0**. Network was ruled out first — from inside the container DNS resolved and
|
||||
mojeek/wikipedia were reachable, so engines were reachable and the parsers were
|
||||
the broken part.
|
||||
|
||||
**Why NH3 and not an in-place update** (operator's call, and the measurement
|
||||
backs it):
|
||||
|
||||
ana-docker egress 38.120.12.42 datacenter -> DuckDuckGo/Startpage CAPTCHA
|
||||
nh3-docker egress 70.230.226.88 residential -> not gated the same way
|
||||
|
||||
Search engines gate datacenter ranges. Same reason the fleet keeps a residential
|
||||
SOCKS5 proxy on nh3-dev for yt-dlp — applied at the source instead of proxied
|
||||
around. `outgoing.proxies` has the fallback commented in place if NH3's egress
|
||||
ever changes.
|
||||
|
||||
⚠ **Not a complete fix.** `brave`, `duckduckgo`, `startpage` still CAPTCHA from
|
||||
NH3. `google cse` carries general search at ~20 results/query; `yandex`, `wiby`,
|
||||
`github`, `stackoverflow`, `marginalia` work. **General search is effectively
|
||||
single-engine** — if google cse breaks, it goes quiet again.
|
||||
|
||||
**Two config defects, both silent:** `base_url` still named
|
||||
`searxng.pfi.local`, retired 2026-08-19, while the env said otherwise (env wins,
|
||||
so nothing broke and the file lied to every reader); and the
|
||||
`karmasearch.videos` removal key never matched because the engine's real name
|
||||
has a space in it.
|
||||
|
||||
**`scripts/searxng-health.sh` asserts results > 0** across three unrelated
|
||||
queries. That is the only check that could have caught this — the mechanism was
|
||||
healthy throughout.
|
||||
|
||||
**The MCP tool** — `services/searxng-mcp`, `uv tool install`, registered
|
||||
`claude mcp add --scope user searxng searxng-mcp`, so every CC session gets
|
||||
`web_search`. ⚠ Zero results **raise** rather than returning an empty list: an
|
||||
empty list is indistinguishable from a broken aggregator, which is precisely how
|
||||
this hid. Same principle as althing's "unreachable post office is an OUTAGE,
|
||||
never an empty inbox".
|
||||
|
||||
⚠ Written against **mcp 2.x** (`FastMCP` → `MCPServer`; the v1
|
||||
`@app.list_tools()` decorator is gone and fails at import). ⚠ **`uv tool install
|
||||
--force` served a CACHED build** and silently reinstalled the old code — the
|
||||
installed file still had the v1 API after the source no longer did.
|
||||
`--reinstall --no-cache` fixed it; `md5sum` of source vs installed is what
|
||||
caught it.
|
||||
|
||||
Old instance stopped and removed; DNS alias repointed to
|
||||
`searxng.nh3.internal` → 10.100.50.40. Secret vaulted at
|
||||
`nh3-docker/searxng-secret`. Commit `0f748ea`. See [[2026-09-03-gx10-rack-network]]
|
||||
for the other UniFi-side change the same day.
|
||||
Reference in New Issue
Block a user