memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived)
This commit is contained in:
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-15]` Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)
|
||||
|
||||
Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`). ⚠ When the `svos_miranda` plugin arrives it will reference the **dispatch key, not the bearer** (expected), and its `tools` array is legitimately **seven or eight** entries; any other number is a real fault. Commit `e641931`.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.
|
||||
|
||||
⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.
|
||||
|
||||
**`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
@@ -1,80 +0,0 @@
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
@@ -1,81 +0,0 @@
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-20]` `ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.
|
||||
|
||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||
@@ -0,0 +1,36 @@
|
||||
# `[2026-10-04]` The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — what each host taught
|
||||
|
||||
**Every hypervisor we run is on PVE 9.2.21 / kernel 7.0.14-20, and both backup servers are on PBS 4.2.7 / Debian 13.7.**
|
||||
nh3-pve + esh-nas-pve 2026-10-03; pfi-pve (0656), esh-pve (0742), pbs-nh3 + pbs-ana (1131 / 1140) 2026-10-04; nh3-pve-2
|
||||
and esh-pve-2 were installed on 9. sfsrv-ana (SureFire, PVE 8) is the client's and was not touched. Plans + logs:
|
||||
`docs/runbooks/pfi-pve-pve9-upgrade-plan.md`, `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`,
|
||||
`docs/runbooks/pbs-pbs4-upgrade-plan.md`. Commits 075ba09, b67ab48 (pfi-pve), 9a6b235, 73129a0 (esh-pve), d0c1bc6 (PBS).
|
||||
|
||||
## What each one taught
|
||||
|
||||
- **nh3-pve (10-03): no `proxmox-headers-7.0` was installed, so the nvidia DKMS module never built for 7.0 and nh3-ml1
|
||||
(CT 109) came up with no GPU.** Fixed with `proxmox-default-headers` + headers + `dkms autoinstall`
|
||||
(`playbooks/nh3-pve-pve9-postfix.yaml`). On any DKMS host, confirm `proxmox-default-headers` BEFORE the upgrade and
|
||||
`dkms status` for the new kernel BEFORE the reboot.
|
||||
- **esh-nas-pve (10-03): `/boot` is a separate 488 MB LV**; with four kernels the second 7.0 initramfs hit "No space left"
|
||||
mid-dist-upgrade. Trim old kernels and check `df /boot` first.
|
||||
- **pfi-pve booted UEFI from a 2023 GRUB because `grub-pc` was the installed package** (the PVE 7.4 install was never
|
||||
switched to `grub-efi-amd64`). That is the wiki's "UEFI + root on LVM may fail to boot" trap; prep swapped it and a
|
||||
PVE 8 reboot test proved it before trixie. Check `dpkg -l grub-pc grub-efi-amd64` + `efibootmgr -v` on any PVE host
|
||||
before its next major upgrade.
|
||||
- **A pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale CT 114 is the only site-to-site path) and ~20 min of the LiteLLM
|
||||
gateway** (14-guest boot storm on HDD-backed ospool; LiteLLM's prisma start is slow under it), which also silences
|
||||
Miranda (her `base_url` is the ANA gateway; recorded in CLAUDE.md, fallback parked as park 98). Break-glass verified:
|
||||
FortiGate WAN SSH `infra-ops@38.120.12.42` is tunnel-independent and reaches the iDRAC (Enterprise licence).
|
||||
- **esh-pve's vmbr0 still listed `enp87s0`**, dead since the 2026-09-24 GPU renumbered the igc ports; every boot logged
|
||||
"bridge configuration failed". Fixed to the ports actually forwarding before the upgrade. One reboot (Prime on site,
|
||||
his call over a PVE 8 test): mesh ~1 min, DNS/HA/Zigbee ~2 min; nvidia DKMS built for 7.0 because
|
||||
`proxmox-default-headers` was already in (the nh3-pve trap).
|
||||
- **ifupdown2 3.3 (PVE 9) makes `ifreload -a -s` exit 1 on PVE's own `bridge-fd 0`.** Cosmetic: nh3-pve and esh-nas-pve
|
||||
show it and run fine. Pre-reboot checks now fail only on OTHER ifupdown2 output.
|
||||
- **PBS were "PBS on plain Debian"** (no `proxmox-backup` meta package), so `pbs3to4` FAILs one check by design; the
|
||||
real requirement was asserted directly. Driven as root through `qm guest exec` (no infra-ops login then; added after,
|
||||
Prime) — and the first in-band `apt` died mid-dpkg on BOTH VMs when qemu-guest-agent restarted itself (auto-memory
|
||||
`feedback_no_inband_apt_through_guest_agent`). Proven after: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync.
|
||||
- **The PVE 9 vzdump warns on Windows VMs without the Microsoft 2023 Secure Boot certs** (`ms-cert=2023k`). Moot for
|
||||
now: Prime retired VM 101 (DC) and VM 104 (nh3-laser) the same day; enroll with `qm enroll-efi-keys` before any start.
|
||||
@@ -0,0 +1,23 @@
|
||||
# `[2026-10-04]` pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; two new Cloudflare tokens
|
||||
|
||||
**pfi-ana-webhost (VM 110, Debian 11 + Virtualmin) is destroyed** (Prime: "retire the webhost — we may want one back in
|
||||
the future, and if so we'll build it fresh"). It served nothing live: its one vhost `dragonfirepro.com` was drop-caught
|
||||
by a third party in 2025-09, and `dragonfireacoustics.com` moved on 2026-09-24 to **Cloudflare Pages in the client's own
|
||||
account** (project `dragonfireacoustics`, source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, now pushed to
|
||||
private gitea `vh/dragonfireacoustics-site`; domain on Cloudflare Registrar to 2035-10-30, transfer-locked). Found via an
|
||||
agent-feedback memory in albok, not the fleet; the 2026-09-17 "expires 10-30 / webhost is dead" entries are superseded.
|
||||
Cleanup: FortiGate policy 18, VIPs `web-to-webhost` + `ssh-to-webhost` (SSH was open to the internet), address, service,
|
||||
DHCP reservation; public A `pfi-ana-webhost.phasefinal.com`; `pfi-ana-webhost.ana.internal`; inventory moved to
|
||||
`docs/retired/pfi-ana-webhost/`. **38.120.12.45 is unmapped.** Last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`.
|
||||
|
||||
**phasefinal.com moved off ana-docker to Cloudflare Pages** (project `phasefinal`, our account `vh@phasefinal`):
|
||||
www + apex CNAME → `phasefinal.pages.dev`, apex → www as a zone Redirect Rule (Pages `_redirects` cannot cross hosts),
|
||||
`_headers` carries the old nginx headers, `404.html` added (without it Pages serves the home page for every unknown
|
||||
path). Privacy page's hosting line rewritten with Prime-approved wording. Deploy: `stacks/phasefinal-web/deploy-pages.sh`.
|
||||
Nobody had recorded why it went to ana-docker on 2026-08-29; it was the default placement, not a weighed choice.
|
||||
|
||||
**Cloudflare tokens:** Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`,
|
||||
Cloudflare name `infra-ops-api`); infra-ops minted **`infra-ops-pages`** with it (vault
|
||||
`nh3-dev/.config/cloudflare/infra-ops-pages-token`: Pages Write + all zones Zone Read, DNS Write, Dynamic URL Redirects
|
||||
Write, Cache Settings Write, Cache Purge). Account tokens verify at `/accounts/<id>/tokens/verify`, not `/user/…`.
|
||||
Details: auto-memory `reference_infra_ops_cloudflare_dns_token`. Commits 55e4d01, 513f6ef, 905862a.
|
||||
Reference in New Issue
Block a user