memory: snapshot — searxng move, gx10 rack networking, mac fleet provisioning
This commit is contained in:
@@ -3251,3 +3251,162 @@ Gateway = LiteLLM on ana-docker `10.250.50.70:4000`; kimi-k3 config in `stacks/l
|
||||
- `[2026-08-02]` **`mv <job> complete/ → failed/` RENAMED the job to `failed` because failed/ didn't exist.** worldtree-dev's round-2 unblock command (`mv /data/state/ingestion/complete/<job> /data/state/ingestion/failed/`) assumed `failed/` existed; on PERSONAL muninn it did NOT (fresh instance — root was `active/ complete/ pending/ sources/`, no `failed/`). `mv src nonexistent/` **renames** src→nonexistent, so job1 became the `failed` dir and job2 nested inside it. Caught on post-move `ls` (failed/ held job *contents*, not two subdirs), reconstructed via complete/ as watcher-safe scratch + rebuilt `failed/` (worldtree:worldtree 755) — NO data loss. **Lessons:** (1) before `mv X into-dir/`, verify the dir EXISTS (`[ -d dir ]`) — an empty `ls dir/ 2>/dev/null` is AMBIGUOUS (missing vs empty), which was the preflight miss that let it through; (2) the correct guard is **`mv -t <targetdir> <src>`** (`--target-directory`): it refuses a MISSING target loudly (rc=1, "No such file or directory", nothing moved) — this is the house convention for queue/state moves now. TESTED by muninn-dev on coreutils 9.1: a **trailing slash does NOT protect** — `mv src failed/` with `failed/` missing STILL silently renames to `failed` (rc=0); "just add the slash" is a false guard. (`mkdir -p failed/` first also works, but `mv -t` inverts the failure from silent-wrong to loud-safe in one flag.) Container `sh` is dash — no `(` in echo strings. **SILENT failure mode (muninn-dev carry-forward):** a misplaced ingestion-state move doesn't crash anything — `list_jobs()` stays OK, loose files are inert; the ONLY symptom is the job quietly absent from the board (`job_row`→None, requeue→not_found/404, looks IDENTICAL to the original block). So after ANY state move, verify the job is actually ON THE BOARD (`job_row` found + guards pass), don't trust mv exit codes — and confirm `job.dispatch.json` survived (requeue refuses a dispatch-less job with the same not_requeueable symptom). Cross-checked + all-clear'd by muninn-dev, who correctly refused to mutate ingestion_root (INV-MG-1) and flagged instead. **DON'T TIDY (round-2 pending):** both DCC + P&P jobs currently REST in personal `failed/` with manifests reading `state: complete` until round-2 requeue runs — deliberate + load-bearing (`requeue` keys on DIRECTORY PLACEMENT, not manifest state); looks wrong to anyone cold, leave it exactly as-is. **Round-2 sequencing:** the requeue is **mimir-dev's** browser flow (pending their operator's board-vs-API ruling); **muninn-dev** is the gate confirmer (runs the post-move board-check inside its custody — the right split, don't reach across INV-MG-1); **infra-ops** = the #381 restart after both jobs go terminal, then later the supervised main-collection sweep. Guard-verified HOLD LIFTED by muninn-dev 02:36Z. **ARC COMPLETE (2026-08-03 ~05:49):** both books terminal — DCC `mimir-6351554e8e8f` 705 concepts + P&P `mimir-f3887c9b97b7` 667, extracted AND indexed, 5/5 phases, 0 failures/truncations (validates the #385 budget fix vs April's 785 control); **#381 restart-after-ingest FIRED** (personal api, healthz/readyz 200 ~25s), retrieval-visibility confirmed (search_library returns DCC+P&P from fiction post-restart); handed ratatoskr-verify go to worldtree-dev. **Delete-sweep precondition NOW MET** — the stale DCC rows in `main` are genuine duplicates of live `fiction` rows, so worldtree-dev's supervised sweep of the ~785 April orphans is unblocked (still comes to me supervised: snapshot + operator-in-loop).
|
||||
_Archived 2026-08-20._
|
||||
|
||||
`[2026-08-10→12]` **secrets-broker — per-box Vaultwarden credential store, SHIPPED + consumer-confirmed.**
|
||||
|
||||
**What.** A per-dev-box credential store over the fleet Vaultwarden (`vaultwarden.phasefinal.com`,
|
||||
on ana-docker, DB on pfi-postgres, in the pg_dump backup set). The `secret` CLI at eshpfi
|
||||
`services/secrets-broker/secret` (also installed to `~/.local/bin/secret`, on PATH for all sessions):
|
||||
`put / get / list / rm / backfill`. Stores into the **`infra-ops` org's Default collection** (org
|
||||
shared to the operator's primary account, so he sees items too), folder = hostname, item name =
|
||||
`<host>/<path>`, title-derived slug. Small text → item note; small binary → base64 hidden field;
|
||||
**>6000 B → a bw attachment** (Vaultwarden caps notes at ~10000 encrypted chars); sha256 + source
|
||||
metadata fields; idempotent upsert keyed by name.
|
||||
|
||||
**Auth.** Bootstraps from `~/.config/secrets-broker/bootstrap.env` (0600): apikey login
|
||||
(`BW_CLIENTID`/`BW_CLIENTSECRET`) + master-password unlock (`--passwordenv`) → per-invocation
|
||||
session. That file is **secrets-zero** (it unlocks the vault, can't live in it) and is excluded from
|
||||
backfill.
|
||||
|
||||
**Client = `bw`, NOT `rbw`.** rbw was the operator's first choice but its `register` returned an
|
||||
undebuggable 400 against this Vaultwarden despite valid creds (a direct `client_credentials` grant +
|
||||
both prelogin paths return 200; rbw emits no HTTP logs). Switched to the official `bw` CLI
|
||||
(user-prefix npm install) — clean unattended flow, full write support (org collections + attachments).
|
||||
|
||||
**Backfill.** Local-only (each box backs up itself; NOT a fleet daemon). Scanned nh3-dev's
|
||||
`~/development/*/{env.sh,.env}` + `~/.config` credential files, **25 items stored + round-trip
|
||||
verified** (2 large via attachment). Excludes bootstrap.env / `.example` / `~/AIPA-Data` archives /
|
||||
cargo noise.
|
||||
|
||||
**Post-launch (jackdaw-dev feedback).** Added **`secret rm <name>`** (bw soft-delete to trash,
|
||||
recoverable) — closes the "no delete path, append-only" gap; and a **new-top-level-namespace warning**
|
||||
on `put` (stderr, non-blocking) — catches a typo'd/missing host prefix at store time. Chose
|
||||
warn-not-auto-prefix because domain-scoped names (`gitea/…`, `certs/…`) would misfire on auto-prefix.
|
||||
Deferred edge recorded in the contract: the warning is non-blocking, so a scripted put suppressing
|
||||
stderr can still mis-namespace — add an opt-in `--strict` only if scripted callers appear.
|
||||
|
||||
**Standing directive (now GLOBAL in `~/.claude/CLAUDE.md`):** the vault is the credential source of
|
||||
truth — **`secret put` durable secrets into it AND `secret get` the creds a task needs FROM it**
|
||||
rather than reading on-disk copies. Dogfooded by pulling the gitea `vh` token from the vault to create
|
||||
`vh/stonehenge-park`.
|
||||
|
||||
**Deploy shape.** Not a service / no daemon — per-box; a new dev box duplicates the stack
|
||||
(`services/secrets-broker/README.md`): npm-install `bw` to `~/.local`, drop a per-box `bootstrap.env`,
|
||||
`secret backfill`. Commits: `41359ea` (CLI + contract), `850a197` (backfill 25/25 + attachment +
|
||||
resilient run), `a249073` (rm + namespace warning), `a1304b7` (deferred-edge contract note).
|
||||
Consumer-confirmed end-to-end by jackdaw-dev.
|
||||
|
||||
Auto-memory: `reference_secrets_broker_cli`.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
- `[2026-08-19]` **`claude-bot` granted read on `vh/waterland`** (operator-empowered, verified `admin:false push:false pull:true`) so irv-ml1 can self-update without the operator's site-admin token living on a GPU box. Precedent for the standing migrate-off-operator-creds directive: grant the service account, wire a repo-scoped 0600 credential helper, keep the remote URL clean. Commit `8189076`.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
# `[2026-08-19]` esh-pve hard-froze for 4.5h — and took the whole house's DNS with it
|
||||
|
||||
Reported by the operator as "routing or DNS issues on the PVC wifi." It was
|
||||
neither: the internet was healthy the entire time (gateway reporting 3 ms and
|
||||
209/26 Mbps; 1.1.1.1 and 8.8.8.8 answering at ~3 ms from inside ESH with zero
|
||||
loss). **The house had no name resolution because one VM was down.**
|
||||
|
||||
## The SPOF: one resolver, cross-VLAN, no fallback
|
||||
|
||||
`esh-userland` (VLAN 10, `10.0.10.0/24` — the `PVC` SSID *and* the wired
|
||||
userland LAN) handed out **exactly one DNS server, `10.0.50.45`** — AdGuard, on
|
||||
`esh-docker-vm`, on the **server** VLAN. No secondary. That VM dies, every
|
||||
client on the VLAN loses DNS, and it presents as "the wifi is broken."
|
||||
|
||||
It was the only network in the house exposed this way. `Default`, `esh-mgmt`,
|
||||
`esh-server` and `esh-cameras` run DNS on auto (the gateway hands itself out);
|
||||
`esh-iot` and `ESH-WG` point at 1.1.1.1 + 8.8.8.8.
|
||||
|
||||
**Fixed** (operator-approved): `esh-userland` now hands out `10.0.50.45`
|
||||
primary, **`10.0.10.1` (the gateway) secondary** — the UDM's own resolver,
|
||||
verified answering. Applied via the Classic API,
|
||||
`PUT /proxy/network/api/s/default/rest/networkconf/687985eae5d15b673cef1a73`
|
||||
with the full object (GET → modify one field → PUT), `rc: ok`. **This was also
|
||||
the first confirmed WRITE on the ESH UDM key** — previously only the NH3 key
|
||||
was write-tested. See [[reference_unifi_udm_integration_api_keys]].
|
||||
|
||||
⚠️ **A secondary is not clean failover.** macOS/iOS query resolvers in
|
||||
parallel, so once AdGuard is back a real share of lookups go to the gateway and
|
||||
**skip ad-blocking**. This converts a total outage into degraded-but-working.
|
||||
The actual fix for blocking integrity is a second AdGuard instance NOT on
|
||||
esh-pve.
|
||||
|
||||
## Root cause: hard freeze, no diagnostics, two suspects
|
||||
|
||||
`esh-pve` (Minisforum MS-01, i9-13900H, `productname: YajuuSenpai`) froze at
|
||||
**03:34:39**. The journal stops mid-operation — **no panic, no OOM, no MCE, no
|
||||
thermal event**. Powered on with its 10G link up, but not answering ARP.
|
||||
|
||||
Two changes landed the day before, and they are not exclusive:
|
||||
|
||||
1. **New kernel.** A large `apt` batch on **2026-08-18 07:00:21** installed
|
||||
`proxmox-kernel-6.8.12-42-pve`; clean reboot at 07:08:44. Before that the
|
||||
box had **4.5 months of uptime** (Mar 30 → Aug 18) on `6.8.12-16`. First
|
||||
boot on the new kernel lasted **20 hours**.
|
||||
2. **GPU passthrough.** The last kernel messages of the dead boot are
|
||||
`vfio-pci 0000:01:00.0/.1: enabling device` at **02:55:17** — VM 102
|
||||
`esh-vm-workstation` starting with `hostpci0: 0000:01:00,pcie=1,x-vga=1`,
|
||||
**39 minutes before the freeze**.
|
||||
|
||||
A vfio/i915 regression in the newer kernel would produce exactly this
|
||||
signature. `6.8.12-16` is still installed and is the held-in-reserve rollback.
|
||||
|
||||
**VM 102 is now pinned off** (`qm set 102 --onboot 0`, stopped) per the
|
||||
operator — it is on-demand and there has been no demand. That removes the
|
||||
suspect without a kernel rollback.
|
||||
|
||||
## Why nobody could recover it remotely — and the fix
|
||||
|
||||
Nothing on the box could reboot it:
|
||||
|
||||
- **`softdog` was the loaded watchdog.** A *software* watchdog cannot rescue a
|
||||
hard kernel freeze: the frozen kernel is the thing that would have to fire
|
||||
its timer. This is the trap — the machine *looked* watchdog-protected.
|
||||
- **Proxmox's `watchdog-mux` held `/dev/watchdog` but never armed it.** It only
|
||||
pets the device while an HA client is connected, and this cluster has no HA
|
||||
resources.
|
||||
- **vPro/AMT was unusable.** The MS-01 reaches the network only via **SFP+**
|
||||
(Intel X710, port 27 on the Garage switch) and presents exactly one MAC.
|
||||
**AMT cannot ride a discrete/SFP+ NIC** — it needs the chipset-integrated
|
||||
Intel PHY, i.e. one of the two i226 RJ45 ports, and both are unplugged.
|
||||
Cabling one and provisioning AMT in MEBx remains the open item for *control*;
|
||||
the watchdog below is the fix for *recovery*.
|
||||
|
||||
**Fixed:** `playbooks/esh-pve-hardware-watchdog.yaml` — systemd now owns the
|
||||
PCH hardware watchdog (`iTCO_wdt`, `RuntimeWatchdogSec=60`), `softdog` is
|
||||
blacklisted and unloaded, `watchdog-mux` is masked. Verified live:
|
||||
`watchdog0: identity=iTCO_wdt state=active timeout=60s`, held by PID 1,
|
||||
journal `Using hardware watchdog 'iTCO_wdt', version 6`. Playbook re-run proves
|
||||
idempotency (6 skipped / 6 verify OK).
|
||||
|
||||
Firmware does **not** block the TCO timer here — checked for the
|
||||
`unable to reset NO_REBOOT flag` line before committing to the approach; the
|
||||
board reports `Found a Intel PCH TCO device (Version=6, TCOBASE=0x0400)`.
|
||||
|
||||
⚠️ **Masking `watchdog-mux` trades away HA fencing.** If Proxmox HA is ever
|
||||
configured on esh-pve this must be reverted. Not a near-term concern:
|
||||
`esh-pve-cluster` is **two nodes with no qdevice**, so a single node loss
|
||||
already costs quorum and the survivor would fence itself — HA here would reduce
|
||||
availability, not raise it.
|
||||
|
||||
⚠️ **The watchdog is configured and armed, but has NOT been proven to fire.**
|
||||
Proving it means deliberately wedging the host. Untested-but-armed is still
|
||||
strictly better than softdog; treat a real firing as unconfirmed until tested.
|
||||
|
||||
## Diagnostic corrections worth keeping
|
||||
|
||||
- **"No route to host" was the dead host, not a routing gap.** Two claims made
|
||||
mid-incident were wrong: that the mgmt VLAN (`10.0.250.0/24`) is not routed
|
||||
over the NH3↔ESH tunnel, and that a firewall isolates it from the server
|
||||
VLAN. Both were artifacts of esh-pve being dead. With it up, `root@esh-pve`
|
||||
SSHes fine from nh3-dev at 7.5 ms, and `10.0.250.1` answers from
|
||||
`esh-pve-nas` in 0.078 ms. **Control-test against a *different* host on the
|
||||
target subnet before concluding "the subnet is unreachable."**
|
||||
- **UDM `uptime` on a client record is association time, not host uptime.** It
|
||||
read 2.2 days while the host had been up 20 hours. Use
|
||||
`journalctl --list-boots` on the host for real boot history.
|
||||
- **`rest/user` `last_seen` is not maintained** (it read ~203 days for hosts
|
||||
that are demonstrably online). `stat/sta` is the live view.
|
||||
_Archived 2026-09-03._
|
||||
|
||||
Reference in New Issue
Block a user