memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived)
This commit is contained in:
@@ -13601,3 +13601,191 @@ Commits `156e126` (applied), `1a35181` (reverted).
|
||||
|
||||
⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
_Archived 2026-10-04._ (Its pointer target, `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`, stays live: the parked breeze-tts entry still uses it.)
|
||||
|
||||
# `[2026-09-15]` Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)
|
||||
|
||||
Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`). ⚠ When the `svos_miranda` plugin arrives it will reference the **dispatch key, not the bearer** (expected), and its `tools` array is legitimately **seven or eight** entries; any other number is a real fault. Commit `e641931`.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.
|
||||
|
||||
⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.
|
||||
|
||||
**`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-20]` `ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.
|
||||
|
||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
Reference in New Issue
Block a user