memory: snapshot — 2026-10-04 ~0546 (albok 0.1.6; 8 settled 09-19 entries archived)
This commit is contained in:
@@ -13559,3 +13559,45 @@ Commits `156e126` (applied), `1a35181` (reverted).
|
||||
|
||||
**`api_key: !ENV SEARXNG_BRAVE_API_KEY` in searxng settings** — this build has NO `!ENV` YAML constructor, so the file was unparseable and the container crash-looped ten times with fleet search down. There is no env-var path into that file at all: the loader reads only `SEARXNG_SETTINGS_PATH` and the entrypoint substitutes only `ultrasecretkey`. Literal or nothing.
|
||||
_Archived 2026-10-03._
|
||||
|
||||
## Recent decisions (archived 2026-10-04 batch)
|
||||
|
||||
# `[2026-09-19]` FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).
|
||||
|
||||
⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.
|
||||
|
||||
⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` Three agents commit as one git author, and closing that gap took three instruments to get right.
|
||||
|
||||
⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.
|
||||
|
||||
⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.
|
||||
|
||||
⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.
|
||||
|
||||
**ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with `timeout waiting on systemd`, manufacturing a real `job errors` every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ **Policy, not a fix:** the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit `vmid` INCLUDE list, so exclusion = removal; reason recorded in the job's own `comment`. e50e57b, `playbooks/esh-exclude-vm102-from-backup.yaml`.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.
|
||||
|
||||
⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-19]` `infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.
|
||||
|
||||
⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.
|
||||
|
||||
⭐⭐ **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** Post office on nh3-docker (built from `althing@6db955f`, manifest `sha256:df0709b3`, **10.9 s** recreate, volume preserved) and the nh3-extdev system wheel + herald. forseti independently verified both. ⚠ Everything CONTENT-verified, never tag-verified: `postbox --version` inside the image before the push and inside the running container after; data continuity proven by reading forseti's own message back out of the running 3.7.0 store. ⚠ **`deploy-stack.sh` rsyncs with `--delete`, so the host-side `.bak-<version>` compose convention is GONE** — it converges rather than accretes; rollback is git history + the retained 3.6.3 registry digest. ⚠ **The claim bug:** a 45-minute operation claim was refreshed *and then released* by deploy-stack.sh's exit trap, silently dropping the protection mid-rollout. Fixed 3e7d3a3 — `ops-log claim` exits **10** when the claim is already the caller's and leaves the holder file UNTOUCHED (a refresh would overwrite the reason and TTL the original claimant chose).
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.
|
||||
|
||||
⭐⭐ **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** A root-requiring `when:` (`pvesh get …` exits 255 unprivileged) returns non-zero → elway reports **SKIPPED** → a playbook that never performed its change reports overall **OK**, because "skipped" is indistinguishable from working idempotency. Three consecutive runs of the VM-102 exclusion reported success while changing nothing; **only the unconditional verify phase caught it.** `creates:` failed the other way — a `/root/…` path is unreadable to the login user, so `test -e` said absent and the step re-ran forever. Fixed (754db4b); sudo-less steps unchanged. ⚠ Audit any existing playbook with a root-requiring condition — a green run proves less than you thought. Two neighbouring traps: never `grep -A<n>` a PVE job in `jobs.cfg` (`vmid` sorts LAST, so adding a `comment` pushed it out of the window and every condition inverted); and `pvesh` JSON carries both a top-level `"enabled":1` and a nested `"fleecing":{"enabled":"0"}`, so a greedy `.*"enabled":` sed matches the wrong one.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.
|
||||
|
||||
**ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed nightly since ~09-06 with `timeout waiting on systemd`, manufacturing a real `job errors` every night. infra-hermes argued correctly that a stopped guest still holds state and the job defect is the real fault; the operator overrode on what the guest actually IS — the one input neither agent had. ⚠ **Policy, not a fix:** the systemd-scope failure on stopped guests is UNDIAGNOSED, merely no longer reached on this node. A stopped guest that DOES matter would hit the same wall. Job uses an explicit `vmid` INCLUDE list, so exclusion = removal; reason recorded in the job's own `comment`. e50e57b, `playbooks/esh-exclude-vm102-from-backup.yaml`.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).
|
||||
|
||||
⭐⭐ **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** The ceiling is **1920 W**, not 2400 — a GPU host running for hours is a continuous load, so NEC's 80% rule governs. Measured: BMC `dcmi power reading` 390 W instantaneous / 461 W max over 2423 s **at idle GPUs**, so the non-GPU baseline is ~313 W (EPYC 9254 24C/96T, 5+ drives, 4 PSUs). GPUs are capped **275 W** each against a **300 W** stock TGP (`power.default_limit`) — a 100 W saving across four cards, NOT the 200 W you get by measuring against the 325 W firmware ceiling; the operator corrected me on exactly that. Derived worst case **~1625 W capped (85%)** vs **~1725 W at stock (90%)**. ⚠ **Keep the caps** — 90% leaves nothing for a heavier-than-estimated R420, PSU efficiency, or a warm day. ⚠⚠ **The coupling is worse than the trip:** OPNsense IS the FV edge and shares the breaker with the thing most likely to trip it, so an overload takes the router with it and removes the remote path needed to recover. Four PSUs do not help — they are all downstream of one breaker. ⚠ **NOT measured:** fv-ml1 under real 4-GPU load (the 461 W max is idle-ish), whether the BMC reports AC or DC (±10% ≈ 150 W at load), and the R420's actual draw. Recorded in `servers/fv-ml1/README.md` § Power. ⚠ Also fixed there: the README claimed **2x** GPUs; `nvidia-smi` reports **four**.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` `infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.
|
||||
|
||||
⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.
|
||||
|
||||
⭐⭐⭐ **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** (1) `backup-freshness-alert.sh` called `althing-cli`, DELETED by the 2026-08-28 v3 cutover — the check detected stale backups every morning and told nobody. Swapped to `postbox`, added the missing `ALTHING_POST_OFFICE` to the unit **and its installer** (the binary swap alone would have failed differently), recipient `infra-ops`→`infra-hermes` (it was mailing itself), exit 2 now means the ALARM is broken vs exit 1 the backups, and `--test-alert` is a positive control because nothing had ever exercised the healthy path. (2) The checker is now **coverage-aware** — a guest is "not backed up by policy" only when NO enabled vzdump job covers it, a UNION across jobs: reading one `exclude` list would have silently stopped alarming on ana CT 109, which is excluded from the 03:00 job AND has its own 22:00 job. (3) It now reads **vzdump TASK STATUS**, because snapshot age is structurally blind to a job that runs and errors nightly — that cost 6 days on both CT 107 and VM 102, and it immediately found a third case on esh-nas-pve whose guests all read 0-1h FRESH. ⚠ **`ops-log` earned its keep on day one**: it captured all three failed playbook attempts automatically. Commits e979ccb, 5be25be, ba26852.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.
|
||||
|
||||
⭐⭐⭐ **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** Central on nh3-dev because both agents are the same unix user there (journald was the tempting alternative and would have split the log silently along the `infra-ops`/`lkraven` axis). Claim is `mkdir`-atomic, held across the whole diff→prompt→apply window, auto-breaks stale AND logs the break. `deploy-stack.sh` refuses a foreign claim (exit 3); ops-log being broken never blocks a deploy. 136-stack baseline laid so the detector starts from today. ⚠ Caught my own instrument saying "clean" for an unreachable host — now INCOMPLETE/exit 5. See in-flight § BUILT for the four settled design questions and the known hole (actors outside nh3-dev).
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-19]` Three agents commit as one git author, and closing that gap took three instruments to get right.
|
||||
|
||||
⭐⭐⭐ **Three agents commit as one git author, and closing that gap took three instruments to get right.** An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git (all commits are Vuong Hoang by convention), from the ops log (host changes only) or from the bus. Turned out to be infra-hermes; forseti disclaimed it and the files' own `~/.hermes/hermes-agent` imports gave it away. Three fixes, each exposing the next: **(1)** `0dc8e90` a post-commit hook recording the althing handle; **(2)** `4e778ae` its fallback said `lkraven@nh3-dev`, which *reads like an answer while being the absence of one* — now `unattributed(login)`, and handles never contain `@` or parens; **(3)** `9141a41` the hook was **appended** behind graphify's **eight `exit 0` paths**, so a `graphify-out/`-only or empty commit could NEVER be recorded. ⚠ **Root cause was control flow, not the lock race proposed** — see `feedback_control_flow_before_concurrency`. Attribution must run FIRST, never behind another hook's interestingness filter. ⚠ Also: `ALTHING_HANDLE` lived only in `althing-infra-hermes-seat-run.sh`, not in `hermes-gateway.service` / the pump unit — hermes fixed it with drop-ins. ⚠ **`systemctl show -p Environment` reports the CONFIGURED env, not the running process's** — it read correct while `/proc/<pid>/environ` did not have the variable at all. Check the process, never the config.
|
||||
+5
-13
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-03 ~2045 PT (esh-nas-pve on PVE 9; nh3-pve on PVE 9; albok-service 0.1.5 + vh/albok pushed on Prime's go; QNAP LCD menu fixed; RustDesk 1.1.16; NH3 Mac mini asleep, fixed on site tomorrow.)_
|
||||
_Last updated: 2026-10-04 ~0546 PT (albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,9 +115,9 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
_As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
|
||||
### 2026-10-03: live now (as of ~2045 PT)
|
||||
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
|
||||
|
||||
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
|
||||
- **nh3-pve-2** (MS-03, NH3) is live at 10.100.250.62 on nh3-mgmt:
|
||||
@@ -132,7 +132,7 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
|
||||
- **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. Lead for later: .166 is Prime's Wi-Fi MacBook (`Mac`, 36:c7:30:13:aa:28, nh3-userland). So it is Wi-Fi → wired VM on the same /24; check the SSID's client-isolation / proxy-ARP settings first.
|
||||
- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **`pve` (esh-pve) is still on 8.4.20; separate window** (DNS VM, CT 108 mesh, esh-ml1 NVIDIA → `proxmox-default-headers` + a /boot check first). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
|
||||
- **NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, Wi-Fi, 10.100.10.121) asleep since ~1407; MeshCentral + RustDesk can't reach it.** Wake-on-LAN from nh3-dev (same subnet) did NOT wake it (Wi-Fi had disassociated). Prime fixes it on site 2026-10-04: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally plus Ethernet.
|
||||
- **albok-service 0.1.5 LIVE (2039) after Prime's "push it"; 0.1.4 went out at 1611 the same way.** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
|
||||
- **albok-service 0.1.6 LIVE (2026-10-04 0545; review fold: a locked keyword-index build); 0.1.5 at 2039 and 0.1.4 at 1611, each after Prime's "push it".** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
|
||||
- **QNAP front-panel LCD on esh-nas-pve fixed (1614), AWAITING Prime's visual check.** Upstream DigitalFunk/QnapLCD-Proxmox opened on a TrueNAS-only screen (`cli`) and died on Proxmox; it also hid vmbr0's IP. Our menu (host/IP, PVE version, uptime, IPs, pools) + a restarting unit: `services/qnap-lcd/`, `playbooks/qnap-lcd.yaml`; the originals are kept beside.
|
||||
- **Uptime Kuma watches the Booth (#2) but NOTHING on Miranda's chain** (svos :8770, hermes-gateway 127.0.0.1:8765); its alerts route through nh3-dev's :8096 bridge, the same host. Prime has NOT decided; recommended SVOS check + gateway heartbeat + a 2nd alert path off nh3-dev (mailrise on ana-docker).
|
||||
- **Done today, nothing open:**
|
||||
@@ -429,21 +429,13 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
- `[2026-09-20]` **`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** — infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a… → `persistent-memory.d/2026-09-20-ravenpen-com-registered-by-the-operator-the-09-18-hold-is.md`
|
||||
|
||||
- `[2026-09-19]` **FV is a dedicated 20 A circuit carrying fv-ml1 AND the R420 running OPNsense, and nothing else (operator-confirmed).** → `persistent-memory.d/2026-09-19-fv-is-a-dedicated-20-a-circuit-carrying-fv-ml1-and-the-r420.md`
|
||||
|
||||
- `[2026-09-19]` **althing 3.7.0 rolled out to both infra-ops surfaces — and the rollout broke the claim tooling built that morning.** → `persistent-memory.d/2026-09-19-althing-3-7-0-rolled-out-to-both-infra-ops-surfaces-and-the.md`
|
||||
|
||||
- `[2026-09-19]` **Three agents commit as one git author, and closing that gap took three instruments to get right.** — An unattributable commit (`e43e262`) appeared in the push set between two of mine — unidentifiable from git… → `persistent-memory.d/2026-09-19-three-agents-commit-as-one-git-author-and-closing-that-gap.md`
|
||||
|
||||
- `[2026-09-19]` **The backup alarm had no wire for three weeks, and fixing it exposed two more instruments that pass without looking.** → `persistent-memory.d/2026-09-19-the-backup-alarm-had-no-wire-for-three-weeks-and-fixing-it.md`
|
||||
|
||||
- `[2026-09-19]` **elway evaluated `when:` / `creates:` / `removes:` / `changed_when:` WITHOUT the step's sudo, and it fails silently in the dangerous direction.** → `persistent-memory.d/2026-09-19-elway-evaluated-when-creates-removes-changedwhen-without.md`
|
||||
|
||||
- `[2026-09-19]` **ESH VM 102 (`esh-vm-workstation`) excluded from the nightly backup job — operator ruling.** — It is a Windows 11 Parsec/RDP **sandbox** (no password, no state to recover), and its vzdump had failed… → `persistent-memory.d/2026-09-19-esh-vm-102-esh-vm-workstation-excluded-from-the-nightly.md`
|
||||
|
||||
- `[2026-09-19]` **The ops log is BUILT — `scripts/ops-log`, automatic writers, and a detector for the path they cannot cover.** → `persistent-memory.d/2026-09-19-the-ops-log-is-built-scripts-ops-log-automatic-writers-and.md`
|
||||
|
||||
- `[2026-09-19]` **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** — infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day… → `persistent-memory.d/2026-09-19-infra-hermes-is-this-session-s-assistant-and-the-division.md`
|
||||
|
||||
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
|
||||
- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
|
||||
@@ -469,7 +461,7 @@ _As of 2026-10-03 ~2045 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_172 older entries archived to archival-memory.md._
|
||||
_180 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
|
||||
Reference in New Issue
Block a user