memory: snapshot — 2026-10-04 2300 (PVE 9 + PBS 4 fleet-wide, webhost destroyed, phasefinal on Pages, sudo-io archiver; 7 settled entries archived)
This commit is contained in:
@@ -13601,3 +13601,191 @@ Commits `156e126` (applied), `1a35181` (reverted).
|
||||
|
||||
⭐⭐⭐ **`infra-hermes` is this session's ASSISTANT, and the division of labour is now standing policy.** infra-ops keeps **improving infrastructure tooling** plus the hard calls; infra-hermes does **day-to-day checks, triage and routine operations**; either may perform infra ops; **infra-ops may task him downward** and he escalates upward as needed. Three operator answers, verbatim in intent: (1) **build the ops log** — see in-flight; (2) **he is NOT Miranda**, so the global CLAUDE.md's sole-trusted-relay exception does not cover him and a directive he relays is information rather than authorization (reversible relayed work executes, irreversible or fleet-affecting goes to the operator); (3) **yes, task him**. Structural facts recorded in `CLAUDE.md` § "infra-hermes IS a real peer" rather than here, because a fresh session must have them without reading this file. ⚠ He is a Hermes bus seat on nh3-dev (`althing-pump-infra-hermes.service`, enabled, route declared) — round trip proven both directions 2026-09-19 04:46.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
_Archived 2026-10-04._ (Its pointer target, `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`, stays live: the parked breeze-tts entry still uses it.)
|
||||
|
||||
# `[2026-09-15]` Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)
|
||||
|
||||
Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`). ⚠ When the `svos_miranda` plugin arrives it will reference the **dispatch key, not the bearer** (expected), and its `tools` array is legitimately **seven or eight** entries; any other number is a real fault. Commit `e641931`.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.
|
||||
|
||||
⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.
|
||||
|
||||
**`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
# `[2026-09-20]` `ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.
|
||||
|
||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||
_Archived 2026-10-04._
|
||||
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-15]` Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)
|
||||
|
||||
Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`). ⚠ When the `svos_miranda` plugin arrives it will reference the **dispatch key, not the bearer** (expected), and its `tools` array is legitimately **seven or eight** entries; any other number is a real fault. Commit `e641931`.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.
|
||||
|
||||
⚠⚠ **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** RDAP: registrar **eNom, LLC** (IANA 48, Tucows; abuse@enom.com), created 2008-10-30, status `active` **only** — no `clientTransferProhibited`. Registrant redacted (Verisign RDAP is thin, eNom's own endpoint 404s). Meanwhile `dragonfirepro.com` was re-registered 2025-09-27 by a Hungarian registrar and is now **for sale on expireddomains.com** — the customer lapsed it and a drop-catcher took it. ⭐ **This also makes the cert fix mandatory rather than tidy: two of the three SANs name a domain a THIRD PARTY now owns, so that request can never validate, and our Virtualmin has been retrying it often enough to get the Let's Encrypt account PAUSED.** Nobody is minding this domain — 13-month-dead cert, 403 homepage, sibling already gone — so the six-week clock is a real risk to an 18-year-old .com with a live site. Customer-facing; nothing touched.
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-17]` `dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.
|
||||
|
||||
**`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** It is a ServerAlias on the `dragonfirepro.com` Virtualmin vhost (DocumentRoot `/home/dragonfirepro/public_html`, suexec 1001), which is the **only enabled site on the box**. State: `www.` → 38.120.12.45 → DNAT to 10.250.50.52 (**proven, not inferred — identical cert SHA-256 inside and out**), Apache answers **403**, and the Let's Encrypt cert (`CN=dragonfirepro.com`, SAN `www.dragonfireacoustics.com`) **expired 2025-08-17, 13 months ago**. The apex points to 199.250.192.76, not ours and not answering at all. ⚠ **CORRECTED — the customer did not migrate `dragonfirepro.com`, they LOST it.** RDAP: re-registered **2025-09-27** through Domain Science Kft (Hungary, IANA 3882) on `dns-redirect.com` nameservers, and it now redirects to **expireddomains.com listed FOR SALE** as an "established .com (6y)". It lapsed, dropped and was sniped. (The earlier "moved to a SaaS platform" read of its 6 foreign A records was wrong — that is parking infrastructure.) `/home/dragonfirepro` mtime 2025-04-12. ⚠ The VM is **Debian 11**, whose LTS window closed end of August 2026 — an unsupported OS exposed on public 80/443 for a site serving nothing. Retire / fix / tell-the-customer-to-repoint is an OPERATOR call: it is a customer relationship, not a technical one. Nothing touched.
|
||||
@@ -1,80 +0,0 @@
|
||||
# [2026-09-18] althing search returned ZERO for every hyphenated query — silently, for everyone
|
||||
|
||||
Routed in by heid on behalf of peedlar-dev as "a message reached storage but never
|
||||
entered the inbox unread-set or search index". Two symptoms, and **neither was what
|
||||
was reported**.
|
||||
|
||||
## Symptom 1 — "never delivered": NOT A DEFECT
|
||||
|
||||
Message 2571 was fully intact: recipients row present, in the FTS index, `read_at`
|
||||
set. **Zero messages fleet-wide have no recipient row — 0 of 2583.**
|
||||
|
||||
Grouping the reader's `read_at` values gives one group per `postbox read`
|
||||
transaction, and the second consumed **2570 AND 2571 together**. `postbox read`
|
||||
advances over exactly what it printed, so that read printed both.
|
||||
|
||||
⚠ **The evidence was truncated.** Three of four tee'd files sat at exact 4 KiB
|
||||
multiples ending mid-sentence; `read-2-of-3.md` was 8,192 bytes while message 2570's
|
||||
body ALONE is 64,975. It held under 13% of 2570 and never reached 2571. That read
|
||||
emitted ≥127 KB and 8 KB survived. The reasoning — "a read that included 2571 would
|
||||
have printed it into one of these files" — was sound on a premise that failed.
|
||||
|
||||
## Symptom 2 — A REAL DEFECT, and far wider than one message
|
||||
|
||||
**Any query containing a hyphen returned zero hits, silently, for every handle.**
|
||||
|
||||
'nh3-docker' -> OperationalError: no such column: docker
|
||||
'"nh3-docker"' -> 20 rows
|
||||
'tts-dev' -> OperationalError: no such column: dev
|
||||
'"tts-dev"' -> 47 rows
|
||||
|
||||
FTS5 reads the hyphen as a column filter and tries to resolve the trailing token as
|
||||
a column name. `store.py` passed the caller's query to `MATCH` unescaped; `search()`
|
||||
caught the `OperationalError`, ran its probe (which succeeded — the store was
|
||||
healthy), and returned `[]`, indistinguishable from "no results".
|
||||
|
||||
On this fleet that is most proper nouns we own: `nh3-docker`, `irv-ml1`,
|
||||
`esh-docker-vm`, `fv-ml1`, `tts-dev`, `peedlar-dev`.
|
||||
|
||||
⚠ **The discriminator was working as designed.** It correctly separated
|
||||
store-broken from caller-typo. The defect was that it classified an ordinary
|
||||
hyphenated query as a typo.
|
||||
|
||||
## The fix, and the boundary that held
|
||||
|
||||
Routed to forseti (althing's owner) rather than patched — infra-ops owns the
|
||||
**rollout**, not the codebase. forseti implemented on operator instruction and
|
||||
shipped **v3.6.3** (`5de41b7`): valid FTS syntax intact, `OperationalError` retries
|
||||
once as an escaped literal phrase before the health probe. Plus a read-summary line
|
||||
(`consumed N messages: 2570, 2571`) that makes the truncation case detectable.
|
||||
|
||||
⚠ **I twice declined to act on relayed authorization** — "Vuong requested deployment"
|
||||
arriving through a peer is not the operator instructing me, and a post-office bounce
|
||||
takes the bus down for every handle. Got it directly, then executed.
|
||||
|
||||
## Rollout mechanics worth keeping
|
||||
|
||||
- **No CI builds the post-office image.** `deploy.yaml` ships althing-*chamber* to
|
||||
ana-docker; `ci.yaml` runs tests and the frontend. The root `Dockerfile` IS the
|
||||
post office and nothing publishes it — a build-and-push step, not a pull.
|
||||
- Built from `git archive v3.6.3`, **not a working tree**; archive hash verified
|
||||
identical after transfer. My wheel came out byte-identical to forseti's.
|
||||
- Bus outage **19:13:40 → 19:13:57 UTC, 17 seconds**.
|
||||
- Deployed digest `sha256:978f85533674ee248d6c6f29c54ffab0bc2cb16332c18c9fb8bfda1d566e2de4`;
|
||||
rollback pin kept as `compose.yaml.bak-3.6.2`.
|
||||
- ⚠ **Canonical `stacks/althing-post-office/compose.yaml` had to be re-pinned too.**
|
||||
forseti caught it: hand-deploying leaves canonical intent stale, so the next routine
|
||||
`deploy-stack.sh` would have silently ROLLED THE BUS BACK. Commit `670ac9e`.
|
||||
|
||||
## Still open — two postbox fixes proposed to forseti
|
||||
|
||||
1. **`--body -` is taken as a literal hyphen, not stdin.** Four real messages lost to
|
||||
it today (tts-dev ×3, ledger-dev earlier). Store audit: 5 of 2,637 messages have a
|
||||
one-character body and **every accidental one is a literal `-`**; the fifth is a
|
||||
deliberate `'x'` probe. forseti's scope: reserve `--body -` for stdin in both verbs.
|
||||
**Awaiting operator.** ⚠ Empty bodies are already rejected (PRE-306, verified) — no
|
||||
minimum-length rule is needed.
|
||||
2. A bounded consuming read — **withdrawn**, and rightly. A message-count limit does
|
||||
not bound one very large body, so it would not have saved peedlar-dev.
|
||||
|
||||
Until it ships, the mitigation is entirely sender-side: pipe the body, never `--body`.
|
||||
@@ -1,81 +0,0 @@
|
||||
# [2026-09-18] SearXNG had ONE working general web engine — now seven, and the health script never noticed
|
||||
|
||||
The fleet `web_search` MCP tool every nh3-dev session uses was answering from
|
||||
`google cse` alone, and had been for at least a day, with every instrument green.
|
||||
|
||||
## The root cause, which is subtler than "engines were disabled"
|
||||
|
||||
Of **55 general-category engines, 7 were enabled by default** — and six of those
|
||||
seven are dictionary, translation, currency and encyclopedia engines that return
|
||||
nothing for an ordinary web query. `google cse` was the only general WEB engine on
|
||||
the instance. When its quota exhausted, a query returned **HTTP 200 with an empty
|
||||
`results` array**.
|
||||
|
||||
⚠ **The distinction that hid it: `inactive: false` only makes an engine
|
||||
SELECTABLE; `disabled: false` is what puts it in the DEFAULT set.** The other 48
|
||||
were selectable-but-off, which an API client has no way to change.
|
||||
|
||||
## Three wrong causes published before the right one
|
||||
|
||||
Recorded because the pattern repeated all day and each was reasoned, not careless.
|
||||
|
||||
1. **"The ESH egress move broke it."** Search egress had been moved on 09-17 to a
|
||||
SOCKS5 proxy on esh-scale, and a 2026-09-03 comment claimed NH3's residential
|
||||
address avoided CAPTCHAs. Reverting to direct NH3 egress produced a
|
||||
**byte-identical** failure. Falsified.
|
||||
2. **"Stale suspension timers."** A live `!ddg` bang probe on a freshly restarted
|
||||
container also returned CAPTCHA. Falsified.
|
||||
3. **"The version bump fixed it."** Updating `2026.9.3` → `2026.9.18` appeared to
|
||||
restore all four engines — but **another session had removed the captcha'd
|
||||
scrapers from the engine set four minutes before that measurement**, and I did
|
||||
not check what else had moved. Two changes, one measurement, attributed to mine.
|
||||
|
||||
Every one asserted causation from correlation with **no baseline measurement of the
|
||||
working state**.
|
||||
|
||||
## The fix
|
||||
|
||||
Five keyless engines added to the default set, each bang-probed on this egress
|
||||
first rather than trusted from a list:
|
||||
|
||||
duckduckgo web 10 mojeek Suspended excluded
|
||||
bing 10 qwant CAPTCHA excluded
|
||||
yep 20 startpage Suspended excluded
|
||||
yahoo 7 brave Suspended excluded
|
||||
wiby 12 seznam timeout excluded
|
||||
|
||||
Plus `braveapi` with a real key. **General web engines: 1 → 7.**
|
||||
|
||||
Verified under the live failure condition rather than a simulated one — `google cse`
|
||||
was STILL quota-suspended and three consecutive queries returned 38–41 results from
|
||||
4–5 engines each. Later with braveapi: 55–63 results from six.
|
||||
|
||||
## Two durable rules this settles
|
||||
|
||||
⭐ **This stack tracks `:latest` ON PURPOSE — do not "improve" it by pinning.**
|
||||
Upstream ships engine-handler fixes continuously because providers change bot
|
||||
gating continuously; being current IS the mitigation, and a digest pin would have
|
||||
frozen the breakage in place. (The althing post office is pinned for the opposite
|
||||
reason — it is the message bus and must not move underfoot.)
|
||||
|
||||
⭐ **The Brave API key is committed in plaintext by explicit operator decision**
|
||||
(2026-09-18): *"if the service is useless, so is the key"* — free tier, rate-limited
|
||||
service, not worth the machinery of keeping it out of git. It is also vaulted at
|
||||
`nh3-docker/searxng-brave-api-key`. **This is scoped to one low-value credential and
|
||||
is not a change to the no-secrets-in-git rule.** It cannot be un-committed; rotation
|
||||
means a new key at Brave, never a history rewrite, because the repo is shared.
|
||||
|
||||
⚠ There is **no env-var path into `searxng-settings.yml`** in this build. The loader
|
||||
reads only `SEARXNG_SETTINGS_PATH`; the entrypoint substitutes only `ultrasecretkey`.
|
||||
`SEARXNG_BRAVE_API_KEY` reaches the container and is never read — it is kept in the
|
||||
compose, unused, with a comment saying so.
|
||||
|
||||
## ⚠ Still open: the health script cannot see any of this
|
||||
|
||||
`scripts/searxng-health.sh` printed `✓ searching` with six engines answering AND
|
||||
with one. It gates on "did any results come back" and prints failed engines as
|
||||
informational. Search ran on a single engine for a day with every instrument green.
|
||||
**Fix it to fail on blocked engines before trusting it again.**
|
||||
|
||||
Commits `9a428fd` `274d3e2` `9219942` `5b20b02`. See
|
||||
[[2026-09-18-searxng-esh-egress-reverted]].
|
||||
-3
@@ -1,3 +0,0 @@
|
||||
# `[2026-09-20]` `ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.
|
||||
|
||||
**`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a peer relay with no registrar credential on our side; the standing instruction was to post registrar + expiry back on thread `01M2SERDB3DR3RV7JS1J0GMF0H` only once the operator bought it himself. Done. Facts from **RDAP (Verisign, authoritative)** rather than a dashboard: registrar **Cloudflare, Inc.** (IANA 1910), registered 2026-09-20T20:12:30Z, **expires 2028-09-20** (two-year), `clientTransferProhibited`. Zone `2df4c5eb4ea4b9410423bdebcb6c5192` active on `vh@phasefinal`, activated 0.4 s after creation — registered THROUGH Cloudflare Registrar, which is why the zone's `original_registrar` is null. ⚠ **The zone is BARE — zero DNS records**, so the name resolves to nothing and mail to it bounces; correct for bought-not-built, but say so before anyone points at it. ⚠ **Scope boundary measured, not assumed:** the fleet `infra-ops` Cloudflare token is Zone·DNS·Edit and **403s on the Registrar API** — I can build records in the zone, and I can NOT read auto-renew state, renew, or transfer. **Auto-renew is therefore UNCONFIRMED**; do not let anyone assume it. ⚠ The token is **vaulted, not on disk** — `secret get 'nh3-dev/.config/cloudflare/infra-ops-dns-token'`; the memory's "vaulted at nh3-dev/..." names a VAULT KEY, and reading it as a filesystem path wastes a step.
|
||||
@@ -0,0 +1,36 @@
|
||||
# `[2026-10-04]` The whole fleet is on PVE 9.2.21 and PBS 4.2.7 — what each host taught
|
||||
|
||||
**Every hypervisor we run is on PVE 9.2.21 / kernel 7.0.14-20, and both backup servers are on PBS 4.2.7 / Debian 13.7.**
|
||||
nh3-pve + esh-nas-pve 2026-10-03; pfi-pve (0656), esh-pve (0742), pbs-nh3 + pbs-ana (1131 / 1140) 2026-10-04; nh3-pve-2
|
||||
and esh-pve-2 were installed on 9. sfsrv-ana (SureFire, PVE 8) is the client's and was not touched. Plans + logs:
|
||||
`docs/runbooks/pfi-pve-pve9-upgrade-plan.md`, `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`,
|
||||
`docs/runbooks/pbs-pbs4-upgrade-plan.md`. Commits 075ba09, b67ab48 (pfi-pve), 9a6b235, 73129a0 (esh-pve), d0c1bc6 (PBS).
|
||||
|
||||
## What each one taught
|
||||
|
||||
- **nh3-pve (10-03): no `proxmox-headers-7.0` was installed, so the nvidia DKMS module never built for 7.0 and nh3-ml1
|
||||
(CT 109) came up with no GPU.** Fixed with `proxmox-default-headers` + headers + `dkms autoinstall`
|
||||
(`playbooks/nh3-pve-pve9-postfix.yaml`). On any DKMS host, confirm `proxmox-default-headers` BEFORE the upgrade and
|
||||
`dkms status` for the new kernel BEFORE the reboot.
|
||||
- **esh-nas-pve (10-03): `/boot` is a separate 488 MB LV**; with four kernels the second 7.0 initramfs hit "No space left"
|
||||
mid-dist-upgrade. Trim old kernels and check `df /boot` first.
|
||||
- **pfi-pve booted UEFI from a 2023 GRUB because `grub-pc` was the installed package** (the PVE 7.4 install was never
|
||||
switched to `grub-efi-amd64`). That is the wiki's "UEFI + root on LVM may fail to boot" trap; prep swapped it and a
|
||||
PVE 8 reboot test proved it before trixie. Check `dpkg -l grub-pc grub-efi-amd64` + `efibootmgr -v` on any PVE host
|
||||
before its next major upgrade.
|
||||
- **A pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale CT 114 is the only site-to-site path) and ~20 min of the LiteLLM
|
||||
gateway** (14-guest boot storm on HDD-backed ospool; LiteLLM's prisma start is slow under it), which also silences
|
||||
Miranda (her `base_url` is the ANA gateway; recorded in CLAUDE.md, fallback parked as park 98). Break-glass verified:
|
||||
FortiGate WAN SSH `infra-ops@38.120.12.42` is tunnel-independent and reaches the iDRAC (Enterprise licence).
|
||||
- **esh-pve's vmbr0 still listed `enp87s0`**, dead since the 2026-09-24 GPU renumbered the igc ports; every boot logged
|
||||
"bridge configuration failed". Fixed to the ports actually forwarding before the upgrade. One reboot (Prime on site,
|
||||
his call over a PVE 8 test): mesh ~1 min, DNS/HA/Zigbee ~2 min; nvidia DKMS built for 7.0 because
|
||||
`proxmox-default-headers` was already in (the nh3-pve trap).
|
||||
- **ifupdown2 3.3 (PVE 9) makes `ifreload -a -s` exit 1 on PVE's own `bridge-fd 0`.** Cosmetic: nh3-pve and esh-nas-pve
|
||||
show it and run fine. Pre-reboot checks now fail only on OTHER ifupdown2 output.
|
||||
- **PBS were "PBS on plain Debian"** (no `proxmox-backup` meta package), so `pbs3to4` FAILs one check by design; the
|
||||
real requirement was asserted directly. Driven as root through `qm guest exec` (no infra-ops login then; added after,
|
||||
Prime) — and the first in-band `apt` died mid-dpkg on BOTH VMs when qemu-guest-agent restarted itself (auto-memory
|
||||
`feedback_no_inband_apt_through_guest_agent`). Proven after: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync.
|
||||
- **The PVE 9 vzdump warns on Windows VMs without the Microsoft 2023 Secure Boot certs** (`ms-cert=2023k`). Moot for
|
||||
now: Prime retired VM 101 (DC) and VM 104 (nh3-laser) the same day; enroll with `qm enroll-efi-keys` before any start.
|
||||
@@ -0,0 +1,23 @@
|
||||
# `[2026-10-04]` pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; two new Cloudflare tokens
|
||||
|
||||
**pfi-ana-webhost (VM 110, Debian 11 + Virtualmin) is destroyed** (Prime: "retire the webhost — we may want one back in
|
||||
the future, and if so we'll build it fresh"). It served nothing live: its one vhost `dragonfirepro.com` was drop-caught
|
||||
by a third party in 2025-09, and `dragonfireacoustics.com` moved on 2026-09-24 to **Cloudflare Pages in the client's own
|
||||
account** (project `dragonfireacoustics`, source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, now pushed to
|
||||
private gitea `vh/dragonfireacoustics-site`; domain on Cloudflare Registrar to 2035-10-30, transfer-locked). Found via an
|
||||
agent-feedback memory in albok, not the fleet; the 2026-09-17 "expires 10-30 / webhost is dead" entries are superseded.
|
||||
Cleanup: FortiGate policy 18, VIPs `web-to-webhost` + `ssh-to-webhost` (SSH was open to the internet), address, service,
|
||||
DHCP reservation; public A `pfi-ana-webhost.phasefinal.com`; `pfi-ana-webhost.ana.internal`; inventory moved to
|
||||
`docs/retired/pfi-ana-webhost/`. **38.120.12.45 is unmapped.** Last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`.
|
||||
|
||||
**phasefinal.com moved off ana-docker to Cloudflare Pages** (project `phasefinal`, our account `vh@phasefinal`):
|
||||
www + apex CNAME → `phasefinal.pages.dev`, apex → www as a zone Redirect Rule (Pages `_redirects` cannot cross hosts),
|
||||
`_headers` carries the old nginx headers, `404.html` added (without it Pages serves the home page for every unknown
|
||||
path). Privacy page's hosting line rewritten with Prime-approved wording. Deploy: `stacks/phasefinal-web/deploy-pages.sh`.
|
||||
Nobody had recorded why it went to ana-docker on 2026-08-29; it was the default placement, not a weighed choice.
|
||||
|
||||
**Cloudflare tokens:** Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`,
|
||||
Cloudflare name `infra-ops-api`); infra-ops minted **`infra-ops-pages`** with it (vault
|
||||
`nh3-dev/.config/cloudflare/infra-ops-pages-token`: Pages Write + all zones Zone Read, DNS Write, Dynamic URL Redirects
|
||||
Write, Cache Settings Write, Cache Purge). Account tokens verify at `/accounts/<id>/tokens/verify`, not `/user/…`.
|
||||
Details: auto-memory `reference_infra_ops_cloudflare_dns_token`. Commits 55e4d01, 513f6ef, 905862a.
|
||||
+22
-41
@@ -1,6 +1,6 @@
|
||||
# Persistent memory — eshpfi-management
|
||||
|
||||
_Last updated: 2026-10-04 ~1145 PT (PBS 4 on both; pfi-pve + esh-pve on PVE 9; albok-service 0.1.6 live after Prime's push go; esh-nas-pve + nh3-pve on PVE 9; QNAP LCD fixed, awaiting Prime's look; NH3 Mac mini asleep, fixed on site today.)_
|
||||
_Last updated: 2026-10-04 ~2300 PT (every hypervisor on PVE 9.2.21 + both PBS on 4.2.7; pfi-ana-webhost destroyed, phasefinal.com on Cloudflare Pages; sudo-io archiver + bulk-data exemption; fv-ml1 root pool cleaned; fv-ml1 Docker move + guest patch campaign parked.)_
|
||||
|
||||
> **Always check for `/tmp/infra-ops-handoff.md`** — if it exists and its
|
||||
> `Written:` stamp is under **8 hours** old, read it (it carries the in-flight
|
||||
@@ -115,36 +115,18 @@ no longer deployed sidecars here. See Recent decisions.)
|
||||
|
||||
## Current state / in-flight
|
||||
|
||||
_As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
_As of 2026-10-04 ~2300 PT. The newest subsection is first; older subsections carry their own dates._
|
||||
|
||||
### 2026-10-03/04: live now (as of 2026-10-04 ~0546 PT)
|
||||
### 2026-10-04: live now (as of 2026-10-04 ~2300 PT)
|
||||
|
||||
- **sudo session recordings: archived weekly since 2026-10-04 (Prime: "archived and backed up, then cleared … a regular thing")**: `services/sudo-io-archive/` (nh3-dev user timer, Sun 05:40; sessions >30 d → `~/archive/sudo-io/<host>/`, sha256-verified, then removed on the host by `tar --remove-files` over the archived list; nh3-dev's vzdump to PBS backs them up). First pass took 4.8 G from 11 hosts. ⚠ The volume is DATA piped through sudo, not commands: a 14.45 G recording of `sudo tar` of a model dir on fv-ml1, and a ~380 M `sudo rsync` of the HA volume nightly on esh-docker-vm. **Bulk data now kept out (Prime same day): sudoers `BULK_DATA` (rsync/tar/cat/dd/compressors/docker save|export|cp) `!log_output` on all 19 recording hosts; pack drops any stream >64 MiB; ~31 G of existing bulk streams pruned on hosts and the archives rebuilt (store 4.8 G → 83 M).** Lessons in the scripts: pull the outbox WITHOUT sudo (sudo ≥1.9.14 truncated piped output and re-records it), read the host list on fd 3 (ssh ate stdin).
|
||||
- **fv-ml1 root pool cleaned (2026-10-04)**: unused vLLM images/containers + build cache (~93 G), user HF/uv/pip caches merged to tank and cleared (zroot 124 G → 194 G free); `HF_HOME=/tank/aimodels/huggingface` system-wide. Docker data-root still on zroot (move = outage window, Prime). → `servers/fv-ml1/README.md`
|
||||
- **pfi-ana-webhost DESTROYED 2026-10-04 ~1726 (Prime: retire, build fresh if ever needed)**: `qm destroy 110 --purge`; last image `pbs-ana:backup/vm/110/2026-10-04T10:28:18Z`. All FortiGate objects for it deleted (policy 18, VIPs, address, service, DHCP reservation; **38.120.12.45 now unmapped**), public + internal DNS records removed, inventory moved to `docs/retired/pfi-ana-webhost/`. ⭐ dragonfireacoustics.com = Cloudflare Pages in the CLIENT's account (source on nh3-dev `~/development/adhoc/dragonfireacoustics-site`, pushed 2026-10-04 to private gitea `vh/dragonfireacoustics-site`). **Supersedes the two 2026-09-17 dragonfire entries.**
|
||||
- **phasefinal.com on Cloudflare Pages since 2026-10-04 ~1720** (project `phasefinal`, our account `vh@phasefinal`): www + apex CNAME → `phasefinal.pages.dev`; nginx container on ana-docker retired. Privacy page hosting line rewritten (Prime-approved wording). `_headers` + `404.html` added. Apex → www 301 restored as a zone Redirect Rule. Tokens: Prime vaulted an account **token-creator** (`nh3-dev/.config/cloudflare/infra-ops-token-creator`); infra-ops minted **`infra-ops-pages`** with it (vault `…/infra-ops-pages-token`: Pages Write + all-zones DNS/redirect/cache). Deploy: `stacks/phasefinal-web/deploy-pages.sh` (proven with the new token).
|
||||
- **VM 101 PFI-ANA-DC (pfi-pve) and VM 104 nh3-laser (nh3-pve) RETIRED from running (Prime 2026-10-04 1634: "we aren't running the DC or the remote laser")**: 101 shut down cleanly, both `onboot 0`, both excluded from their nightly vzdump (last backups kept). That also ends the PVE 9 Secure Boot/VirtIO backup warnings. ⚠ If either is ever started again: enroll the Microsoft 2023 certs first (`qm enroll-efi-keys`, BitLocker check).
|
||||
- **infra-ops login on pbs-ana + pbs-nh3** (Prime 2026-10-04): uid 850, fleet key, NOPASSWD + `log_output`, bootstrapped via guest agent; `servers/pbs-*/ssh-target` now infra-ops.
|
||||
- **PBS 3 → 4: DONE 2026-10-04 1144** (Prime via Miranda): pbs-nh3 then pbs-ana, both PBS 4.2.7 / Debian 13.7. VM backups first (pbs-ana had none anywhere), disk snapshots `pre-pbs4`, read-only datastores, detached dist-upgrades. Proven: v3→v4 sync, PVE 9→PBS 4 write (ct/113), v4→v4 sync. ⚠ My first prep killed apt mid-dpkg on both (qemu-guest-agent restarted itself under the in-band apt); repaired in ~5 min, PBS never stopped serving. **Open:** sfsrv-ana (PVE 8) 04:00 run on 10-05 is the one untested pair; drop `pre-pbs4` snapshots after it; vzdumps until ~10-11. → `docs/runbooks/pbs-pbs4-upgrade-plan.md`
|
||||
- **pfi-pve (Anaheim): DONE, PVE 9.2.21 / kernel 7.0.14-20 since 0656 2026-10-04** (Prime: "drive it", "proceed"). Prep fixed a real boot trap: the box boots UEFI but had `grub-pc`, so the ESP GRUB was the 2023 install's; now `grub-efi-amd64`. NIC pins, microcode, a PVE 8 reboot test (passed), then trixie. All 14 guests + services verified 0715. ⚠ Each pfi-pve reboot = ~4 min of NH3/ESH↔ANA (ana-scale) and **~20 min of LiteLLM, which also silences Miranda**. Break-glass = FortiGate WAN SSH `infra-ops@38.120.12.42` (verified). ⚠ ifupdown2 3.3 `ifreload -a -s` exits 1 on PVE's own `bridge-fd 0` — cosmetic (same on nh3-pve/esh-nas-pve). **Open: `lvremove -y pve/root-pre-pve9` on/after 2026-10-05** (asked infra-hermes); no `zpool upgrade`. → `docs/runbooks/pfi-pve-pve9-upgrade-plan.md`
|
||||
- **nh3-pve: DONE. ON PVE 9.2.21 / kernel 7.0.14-20 (booted 1402).** Post-check: every guest is back (the stopped 104/108 are as before), ZFS healthy, DNS, post office, albok, svos/hermes, mesh and AMT all OK. ⚠ **nh3-ml1 (CT 109) is DOWN:** no `proxmox-headers-7.0` was installed, so the nvidia 580.178.04 DKMS module was never built for 7.0, `nvidia-persistenced` failed, and CT 109 has no /dev/nvidia*. **FIXED 1409 (Prime go, `playbooks/nh3-pve-pve9-postfix.yaml`):** installed `proxmox-default-headers` + headers-7.0.14-20, nvidia DKMS built for 7.0 (580.178.04 compiles fine), nh3-ml1 back (TEI :8001/:8013 healthy, embed dim 1024); intel-microcode installed (non-free-firmware added; takes effect next boot); `pve-enterprise.sources` off; pve8to9 now 0 FAIL / 1 WARN (dkms, benign). VM 102 snapshot `pre-rootgrow-20261002` DELETED (Prime go). No `pre-pve9` snapshot had been taken. **POST-CHECK DONE: nh3-pve clean.** ⚠ Lesson: on a PVE host with DKMS, check that `proxmox-default-headers` is installed BEFORE a major upgrade, or the new kernel boots with no module.
|
||||
- **nh3-pve-2** (MS-03, NH3, 10.100.250.62): **follow-ups DONE 2026-10-04** (Prime rebooted 0724; .62, nic0/AMT, CIRA connected, enterprise repos off, up to date). No guests; purpose TBD. → `servers/nh3-pve-2/README.md`
|
||||
- **esh-pve-2** (MS-03, ESH) is unplugged by Prime on purpose. vmstore + AMT phone-home are done. The permanent address is deferred by Prime ("address can wait"; `servers/esh-pve-2/README.md`). Run `playbooks/pve-nag-patch.yaml` on it when it is back.
|
||||
- **cira-tunnel-watchdog** on pfi-tacticalrmm has been live since 1232 and drops :4433 tunnels silent >180 s. Watch `journalctl -t cira-tunnel-watchdog`.
|
||||
- **RustDesk server (ana-docker) updated 1.1.14 → 1.1.16 at 1424** (same key; ports 21115-21119 reachable via rustdesk.phasefinal.com). Private key vaulted first as `ana-docker/rustdesk/id_ed25519`. "Change ID" is refused by the OSS server BY DESIGN (Pro feature; `rendezvous_server.rs` answers TCP RegisterPk with NOT_SUPPORT). Workaround, read at client source and untested: stop RustDesk, delete `enc_id` from RustDesk.toml, set a plain `id`, restart. → `servers/ana-docker/README.md`
|
||||
- **Booth vs 10.100.10.166 (Prime, ~1420): that client cannot reach the Booth.** The Booth was up and answered from 4 hosts at 1417; .166 is on nh3-dev's own subnet. Prime interrupted my check, so it is OPEN and resumes only on his word. Lead for later: .166 is Prime's Wi-Fi MacBook (`Mac`, 36:c7:30:13:aa:28, nh3-userland). So it is Wi-Fi → wired VM on the same /24; check the SSID's client-isolation / proxy-ARP settings first.
|
||||
- **esh-nas-pve: DONE, PVE 9.2.21 / kernel 7.0.14-20 since 1604** (Prime on site; ESH DNS never dropped; the /boot-LV-full trap was hit and fixed). Keep `nvme/ROOT@pre-pve9` about a week (until ~2026-10-10), then destroy it. `openipmi.service` fails at boot (no IPMI on the QNAP); harmless, maskable. **esh-pve: DONE 2026-10-04 0742, PVE 9.2.21 / 7.0.14-20** (Prime on site, one reboot; ESH mesh ~1 min, DNS/HA/Zigbee ~2 min). Prep found + fixed a stale vmbr0 `bridge-ports enp87s0` (dead since the 09-24 GPU renumbered the igc ports). nvidia DKMS built for 7.0. **The ESH cluster is uniform 9.2.21.** Open: `lvremove pve/root-pre-pve9` on/after 10-05 (infra-hermes). → `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`
|
||||
- **NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, Wi-Fi, 10.100.10.121) asleep since ~1407; MeshCentral + RustDesk can't reach it.** Wake-on-LAN from nh3-dev (same subnet) did NOT wake it (Wi-Fi had disassociated). Prime fixes it on site 2026-10-04: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally plus Ethernet.
|
||||
- **albok-service 0.1.6 LIVE (2026-10-04 0545; review fold: a locked keyword-index build); 0.1.5 at 2039 and 0.1.4 at 1611, each after Prime's "push it".** Each vh/albok push is a per-release go from Prime, not standing authorization. 0.1.5 = search 4.5 s → ~0.25 s. Deploy BETWEEN Nemi walks: `flock -n ~/.local/state/albok-nemi/run.lock true` (the 0.1.4 redeploy cut the 1601 walk; a catch-up walk 1612-1625 was clean). → `stacks/albok-service/README.md`
|
||||
- **QNAP front-panel LCD on esh-nas-pve fixed (1614), AWAITING Prime's visual check.** Upstream DigitalFunk/QnapLCD-Proxmox opened on a TrueNAS-only screen (`cli`) and died on Proxmox; it also hid vmbr0's IP. Our menu (host/IP, PVE version, uptime, IPs, pools) + a restarting unit: `services/qnap-lcd/`, `playbooks/qnap-lcd.yaml`; the originals are kept beside.
|
||||
- **Uptime Kuma watches the Booth (#2) but NOTHING on Miranda's chain** (svos :8770, hermes-gateway 127.0.0.1:8765); its alerts route through nh3-dev's :8096 bridge, the same host. Prime has NOT decided; recommended SVOS check + gateway heartbeat + a 2nd alert path off nh3-dev (mailrise on ana-docker).
|
||||
- **Done today, nothing open:**
|
||||
- tank verified healthy 0629, disk kept, Miranda told;
|
||||
- albok-service 0.1.2 + the Nemi hourly timer;
|
||||
- the post-deletion Worldtree gate (infra-hermes runs it as a regression check);
|
||||
- the PVE subscription popup patched on nh3-pve, nh3-pve-2, pfi-pve and esh-pve;
|
||||
- nh3-pve PVE 9 + its post-fixes; nh3-dev's VM 102 snapshot deleted.
|
||||
- **ESH PVE 8→9: DONE** (esh-nas-pve 10-03, esh-pve 10-04). `docs/runbooks/esh-pve-cluster-pve9-upgrade-plan.md`.
|
||||
- **Nothing is mid-action.** Every hypervisor is on PVE 9.2.21 / 7.0.14-20 (pfi-pve 0656, esh-pve 0742, nh3-pve + esh-nas-pve 10-03, nh3-pve-2 fresh) and both PBS on 4.2.7 (1144). → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
- **Scheduled with infra-hermes (do not redo):** 10-05 0815 PT check that sfsrv-ana's (PVE 8) 04:00 backup reached pbs-ana, then `qm delsnapshot` 100/105 `pre-pbs4` (if stale/errored he sends infra-ops URGENT and keeps them); on/after 10-05 0900 drop `pve/root-pre-pve9` on pfi-pve + esh-pve (job `pve9-drop-pre-snapshot`); 10-11 0900 delete the two one-off PBS VM vzdumps. esh-nas-pve `nvme/ROOT@pre-pve9`: destroy ~10-10 (ours). Never `zpool upgrade` any pool yet.
|
||||
- **Weekly sudo-io archiver** first scheduled run Sun 2026-10-11 05:40 (nh3-dev user timer); bulk data no longer recorded fleet-wide. → `services/sudo-io-archive/README.md`
|
||||
- **phasefinal.com on Cloudflare Pages; pfi-ana-webhost destroyed; dragonfireacoustics.com is the client's Pages.** → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||
- **Worldtree memory gate 20261004T143011Z FAIL** (one run filed the forbidden `u-negative-ephemeral` fact; extractor is `gen` on fv-ml1). With worldtree-dev, with the infra timeline (an esh-pve reboot overlapped run 1). The 2026-10-05 0730 run shows whether it repeats.
|
||||
- **fv-ml1:** root pool 194 G free; lkraven's `~/.cache/huggingface/hub/models--Cloudflare--clef-flash` (18 G, duplicate of tank) was left because bench-jev-2026-10-04 was using it; delete once that bench is done.
|
||||
- **Open, Prime's calls (unchanged from 10-03):** QNAP LCD visual check (`services/qnap-lcd/`); NH3 Mac mini (`Mac-mini`, 00:c5:85:13:e0:92, 10.100.10.121, asleep since 10-03 ~1407) wake fix on site: `sudo pmset -a sleep 0 disksleep 0 displaysleep 10 womp 1 autorestart 1 powernap 0`, ideally Ethernet; Booth unreachable from 10.100.10.166 (Prime's Wi-Fi MacBook; resume only on his word; lead: SSID client isolation / proxy-ARP); Uptime Kuma has nothing on Miranda's chain (svos :8770, hermes-gateway); mask `openipmi.service` on esh-nas-pve; run `playbooks/pve-nag-patch.yaml` on esh-pve-2 when it is plugged back in.
|
||||
- **Live, nothing open:** albok-service 0.1.8 (all three wings reindexed, albok-dev verified); cira-tunnel-watchdog on pfi-tacticalrmm; RustDesk 1.1.16 on ana-docker (OSS refuses "Change ID" by design; see `servers/ana-docker/README.md`); nh3-pve-2 follow-ups done; VM 101 (DC) + VM 104 (nh3-laser) retired.
|
||||
|
||||
### Parakeet speech seat: unified-en under NeMo, LIVE (2026-10-01)
|
||||
|
||||
@@ -316,6 +298,14 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
## Recent decisions
|
||||
|
||||
- `[2026-10-04]` **The whole fleet is on PVE 9.2.21 and PBS 4.2.7** — per-host traps (pfi-pve's grub-pc on UEFI, esh-pve's dead bridge port, nh3-pve's missing 7.0 headers, PBS via guest agent) and what a pfi-pve reboot costs. → `persistent-memory.d/2026-10-04-fleet-on-pve-9-and-pbs-4.md`
|
||||
- `[2026-10-04]` **pfi-ana-webhost destroyed; phasefinal.com on Cloudflare Pages; token-creator + `infra-ops-pages` tokens** — the dragonfire sites were never ours to serve; 38.120.12.45 now unmapped. → `persistent-memory.d/2026-10-04-webhost-destroyed-phasefinal-on-pages.md`
|
||||
- `[2026-10-04]` **sudo session recordings: archived weekly, bulk data never kept (Prime)** — `BULK_DATA !log_output` on all 19 recording hosts, archiver drops streams >64 MiB, ~31 G pruned. → `services/sudo-io-archive/README.md`
|
||||
- `[2026-10-04]` **VM 101 PFI-ANA-DC + VM 104 nh3-laser retired (Prime: not in use)** — off, `onboot 0`, out of vzdump, backups kept; enroll MS 2023 Secure Boot certs before any restart.
|
||||
- `[2026-10-04]` **PARKED (Prime): move fv-ml1's Docker data-root to tank** — needs a model-seat outage window; tracked at park `move-fv-ml1-s-docker-storage-off-the-root-ssd` (id 99).
|
||||
- `[2026-10-04]` **PARKED (Prime): guest OS patch campaign** — ~20 guests with 70–350 pending updates incl. security + kernels; plan in the item; tracked at park `guest-os-patch-campaign-across-the-fleet-debian` (id 100).
|
||||
- `[2026-10-04]` **PARKED (Prime): a model path for SVOS/Miranda that does not depend on the ANA LiteLLM gateway** — tracked at park `give-svos-and-miranda-a-model-path-that-doesn-t` (id 98).
|
||||
|
||||
- `[2026-10-03]` **PVE subscription popup patched out on nh3-pve, nh3-pve-2, pfi-pve, esh-pve (Prime: "every host")**: an anchored one-line edit plus an apt hook. esh-nas-pve was already de-nagged; esh-pve-2, sfsrv-ana and the PBS hosts are not done. → `services/pve-nag-patch/README.md`
|
||||
- `[2026-10-03]` **nh3-pve-2 finished (Prime on site): host 10.100.250.62 on nh3-mgmt via X710 `nic3`, AMT on `nic0` with `auto` kept up, 1 TB wiped into `vmstore` (913 GiB), stale boot entry gone** → `persistent-memory.d/2026-10-03-nh3-pve-2-finished-prime-on-site-host-10-100-250.md`
|
||||
- `[2026-10-03]` **CIRA tunnel watchdog LIVE on pfi-tacticalrmm (Prime go, 1232)** → `persistent-memory.d/2026-10-03-cira-tunnel-watchdog-live-on-pfi-tacticalrmm-prime-go-1232.md`
|
||||
@@ -430,7 +420,6 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
- `[2026-09-21]` **`vh/forgefirm` mirrored** — from `github.com/openglow-org/forgefirm`, following the house convention read off the existing 17: `vh/`… → `persistent-memory.d/2026-09-21-vh-forgefirm-mirrored.md`
|
||||
|
||||
- `[2026-09-20]` **`ravenpen.com` REGISTERED by the operator — the 09-18 hold is discharged and hamr-dev is answered.** — infra-ops deliberately did NOT execute this twice, because it was a non-refundable purchase reaching us as a… → `persistent-memory.d/2026-09-20-ravenpen-com-registered-by-the-operator-the-09-18-hold-is.md`
|
||||
|
||||
|
||||
|
||||
@@ -440,34 +429,26 @@ _As of 2026-10-04 ~0546 PT. The newest subsection is first; older subsections ca
|
||||
|
||||
|
||||
|
||||
- `[2026-09-18]` ⭐⭐ **SearXNG had ONE working general web engine and every health check said fine.** 7 of 55 were enabled-by-default and six of those are dictionary/translation engines — `inactive: false` only makes an engine SELECTABLE, `disabled: false` puts it in the DEFAULT set. Now seven. ⭐ This stack tracks `:latest` ON PURPOSE (upstream ships engine-handler fixes continuously; a pin freezes breakage). ⭐ The Brave key is committed in plaintext by explicit operator decision — scoped to one low-value credential, NOT a change to the no-secrets rule. → `persistent-memory.d/2026-09-18-searxng-one-engine-to-seven.md`
|
||||
- `[2026-09-18]` ⭐⭐ **althing search returned ZERO for every hyphenated query — silently, for every handle, and on this fleet that is most of our hostnames.** FTS5 read the hyphen as a column filter; `search()` caught the error, its probe passed, and it returned `[]`. Routed to forseti (they own the code, I own rollout) → **v3.6.3** deployed, 17 s bus outage. The reported symptom, a message that never arrived, was NOT a defect: the reporter's evidence file was truncated at 8 KiB. ⚠ No CI builds the post-office image. → `persistent-memory.d/2026-09-18-althing-363-hyphen-search.md`
|
||||
|
||||
- `[2026-09-17]` ⚠ SUPERSEDED 2026-10-04 (renewed to 2035 at Cloudflare Registrar, locked) — **`dragonfireacoustics.com` expires 2026-10-30 — six weeks — at eNom with NO transfer lock, and its sibling domain was already lost exactly this way.** → `persistent-memory.d/2026-09-17-dragonfireacoustics-com-expires-2026-10-30-six-weeks-at.md`
|
||||
⚠ **If it is ever transferred, DNS does NOT come with the registration** — the nameservers are eNom's `name-services.com` and the zone must be recreated first or mail dies. The whole zone is two facts plus a landmine: `*` (WILDCARD) → 199.250.192.76 which is **dead** (no HTTP at all, and it is what the apex/mail/webmail/admin/ftp all answer with), `www` → 38.120.12.45 (us), and **7 Google Workspace MX records that must not be lost**. No DNSSEC (`delegationSigned: false`), so no transfer complication. ⚠ Also found: **no SPF and no DMARC** at the apex on a Google Workspace domain — a live deliverability problem independent of everything else. Transfer gate is the **TAC/EPP code from the eNom account**, not the lock; the missing lock is not authorization. 60-day rule is satisfied (last changed 2025-10-24).
|
||||
- `[2026-09-17]` ⚠ SUPERSEDED 2026-10-04 (site moved to Cloudflare Pages 09-24; webhost retired) — **`dragonfireacoustics.com` IS configured on `pfi-ana-webhost`, and the whole thing is dead — a forgotten public-facing VM.** → `persistent-memory.d/2026-09-17-dragonfireacoustics-com-is-configured-on-pfi-ana-webhost.md`
|
||||
|
||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion.md`
|
||||
- `[2026-09-17]` **The althing route-declaring SessionStart hook is documented but NOT installed on nh3-dev** — `dev_launch.py` has zero occurrences of "route", no hook declares one, and every live route was hand-declared… → `persistent-memory.d/2026-09-17-the-althing-route-declaring-sessionstart-hook-is-documented.md`
|
||||
|
||||
- `[2026-09-17]` **PARKED lv-krakauer, and the reason is a selection criterion the line was missing: ask whether the author HAS a voice before investigating whether the…** → `persistent-memory.d/2026-09-17-parked-lv-krakauer-and-the-reason-is-a-selection-criterion-2.md`
|
||||
|
||||
- `[2026-09-15]` ⚠⚠ **`--gpu-memory-utilization` DOES NOT PREDICT RESIDENT VRAM — measure it, never compute it.** Wrong in **both** directions on fv-ml1: `vllm-cyberprev` util 0.40 (expect ~39,155 MiB) holds **47,124** (+8 GB over); `vllm-gen-small` util 0.48 (expect ~46,986) holds **36,942** (−10 GB under). Planning a placement off the fractions would have been 8 GB wrong. Read `nvidia-smi --query-compute-apps`. Full per-seat residency table + the breeze shuffle arithmetic → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
- `[2026-09-15]` **breeze-tts stays on irv-ml1; the TTS-stack move to fv-ml1 is PARKED (park id 75, `move-the-tts-stack-breeze-tts-bragi-tts-gateway`), triggered on evacuating embed/rerank/reward.** ⚠ Trigger as stated says "gpu0" but those three are on **GPU 1** (~0.16 util, ~15.7 GB; GPU 1 is the tight card at 0.975 / 4,336 MiB free) — confirm which he meant before executing. All three services move together because only `breeze-tts` is GPU-resident (~10.3 GiB, **growing**) while `bragi` and `tts-gateway` are CPU proxies, and co-location is what avoids a cross-site hop per TTS call. **breeze-tts sizing — original recommendation NOT to move it.** ~**10.3 GiB** measured under load at 53 min uptime, **up from 9.2 GiB** shortly after warm-up (it grows; n=2, plateau unmeasured) — so GPU 0's 11,982 MiB free is a **1.7 GB margin and shrinking**, on the live chat serving path. ⚠ Two measurement traps: it reports **nothing at idle on the wrong card** (`BREEZE_GPU_DEVICES=0` = the **3090**, not the A6000), and an early reading understates it. ⭐ The real objection is **topology**: `tts-gateway` is on irv-ml1 and reaches it same-box, so moving breeze alone adds a cross-site hop to every TTS call against a 478 ms first-sample budget. GPU 3 would fit it but spends the reserve. → `persistent-memory.d/2026-09-15-breeze-placement-sizing.md`
|
||||
|
||||
- `[2026-09-15]` **Hermes bearer rotation hold released — svos-dev split their HS256 signing key off the shared value (svos `7165272`)** → `persistent-memory.d/2026-09-15-hermes-bearer-rotation-hold-released-svos-dev-split-their.md`
|
||||
|
||||
- `[2026-09-13]` **STANDING POLICY (operator): cap GPU power limits at BUILD time, not after discovering the constraint.** → `persistent-memory.d/2026-09-13-standing-policy-operator-cap-gpu-power.md`
|
||||
|
||||
- `[2026-08-25]` **Fused MoE kernel path — DEFERRED, tracked at park `fused-moe-kernel-path-for-gemma-4-moe-training` (id 47).** → `persistent-memory.d/2026-08-25-fused-moe-kernel-path-deferred-tracked-at-park-fused-moe.md`
|
||||
|
||||
- `[2026-08-24]` **`nconnect=8` on `/mnt/smithy` — approved but DEFERRED at operator instruction** → `persistent-memory.d/2026-08-24-nconnect-8-on-mnt-smithy-approved-but-deferred-at.md`
|
||||
|
||||
_180 older entries archived to archival-memory.md._
|
||||
_187 older entries archived to archival-memory.md._
|
||||
|
||||
## Tried and abandoned
|
||||
|
||||
- `[2026-10-04]` **`apt-get` run in-band through `qm guest exec`** — qemu-guest-agent was in the upgrade set, restarted itself, and systemd killed apt mid-dpkg on BOTH PBS VMs. Run apt in a `systemd-run` unit. (auto-memory `feedback_no_inband_apt_through_guest_agent`)
|
||||
- `[2026-10-04]` **Pulling files with `ssh host 'sudo cat …'`** — sudo ≥ 1.9.14 truncated the piped stream at exit (Debian 13 hosts), and `log_output` re-recorded the whole file as a new session. Read without sudo (group-readable file).
|
||||
- `[2026-10-01]` **Size + mtime scans to pick reclaimable "staging" on a model store** → `persistent-memory.d/2026-10-01-size-mtime-scans-to-pick-reclaimable-staging-on-a.md`
|
||||
- `[2026-09-30]` **Shorter Parakeet slices as Scriberr's memory fix** → `persistent-memory.d/2026-09-30-shorter-parakeet-slices-as-scriberr-s-memory-fix.md`
|
||||
- `[2026-09-30]` **Whole-file local-attention Parakeet in Scriberr (context 255/255).** OOM past 16 GB on a 35-min file. Local attention inside chunks is also non-deterministic run to run.
|
||||
|
||||
Reference in New Issue
Block a user