memory: snapshot — gx10 unracked and next up as an inference+training box

Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W
appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified
holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the
first move is a throughput probe rather than a harness port.

Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1
keeping its eight services; the Synapse 39-release upgrade with its one-way schema
migration, the appservice namespace opening and the admin API lockdown; the ratified room
alias convention; and a named failure class — a correct check aimed at the wrong object —
with six instances from one day across three sessions.

persistent-memory.d/2026-09-01-matrix-upgrade-and-hardening.md

@@ -0,0 +1,102 @@
+# `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
+
+## The upgrade
+
+**Synapse v1.120.0 → v1.159.0** (21 months, 39 releases) and **Element-web v1.11.80 →
+v1.12.27**. Schema migrations applied cleanly through schema 94. Postgres deliberately left
+at 16 — changing two stateful things at once destroys failure attribution.
+
+⚠ **Schema migrations are ONE-WAY.** v1.120 cannot start against a v1.159 database. Rollback
+is restore-from-dump, not revert-the-tag. Verified pre-upgrade dump (739 TOC entries from a
+33 MB database) plus all four config files at
+`/opt/docker/backups/synapse-preupgrade-20260901T174625Z/`.
+
+Reviewed every upgrade note in the range; nothing applicable bit us (PG 11/12/13 drops — we
+are on 16; MSC3861/MAS items — no MAS; s3-storage-provider and worker media quarantine — not
+in use).
+
+## The appservice namespace — `exclusive: true` → `false`
+
+The `aipa-bridge` registration claimed `@[a-z][a-z0-9_-]*:matrix.phasefinal.com` **exclusively**
+— every localpart on the server. 14 of 14 accounts fell inside it; 13 were appservice-owned.
+
+**`exclusive` governs who ELSE may act, not what the appservice may do.** On a homeserver
+with registration disabled, one admin and no competing actor, it bought anti-squatting
+protection against a threat that cannot occur, while locking out every other means of
+account creation — admin registration returned `M_EXCLUSIVE` with no explanation.
+
+⚠ **Do NOT narrow the users regex to a prefix** — all 13 accounts fall inside it and would be
+orphaned. ⚠ **Do NOT rename the `id`** — Synapse keys account ownership on `aipa-bridge` in
+the `users` table. The FILE may be renamed; the id may not.
+
+The narrow **aliases** namespace (`#aipa-debug-*`) was left exclusive — specific, costs nothing.
+
+⚠ The registration is named `aipa`, but the service behind it is **`wt-matrix-bridge`**, the
+Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. AIPA is a dead project name
+on a live service, and it is why infra-ops mis-routed a provisioning request to worldtree-dev.
+**Operator ruling: worldtree-dev writes the bridge code; infra-ops OPERATES this instance and
+has full authority over it.**
+
+## `/_synapse/admin` closed to the internet
+
+Synapse mounts its admin API on the same vhost as the client API, so publishing
+`matrix.phasefinal.com` published the admin surface — it **answered 200 from the open
+internet**. A higher-priority router (explicit priority 100) now scopes it behind an
+`ipallowlist`.
+
+Verified from a **genuinely external vantage** — the NH3 residential egress proxy, because
+testing from a fleet host sits inside the allow-list and proves nothing: admin **403**,
+client API **200**, Element unaffected.
+
+⚠ **The `10.0.0.0/8` entry matches NOTHING and that is expected.** The hostname resolves
+publicly, so fleet hosts hairpin out their own WAN — a request from nh3-dev measured as
+`70.230.226.88`. The rule is effectively **deny-all through Traefik**, which is intended:
+admin work goes via `docker exec synapse` against `localhost:8008` and never touches Traefik.
+Allow-listing the sites' WAN addresses was **rejected** — dynamic, and a stale entry either
+locks us out or hands admin to whoever inherits the address.
+
+## Conventions ratified (operator, 2026-09-01)
+
+    #<agent>-<purpose>:matrix.phasefinal.com
+
+Mirrors the existing `@<agent>:` user-ID convention. Proposed by ledger-dev. Rationale is
+**"so the room IDENTITY carries the tier"** — deliberately NOT "so the push payload carries
+the room name", which is true only for clients without a notification service extension.
+Pre-existing rooms are not renamed ("The High Seat", `!NiVVoMsyoHCBRPrrrn`).
+
+## Push reality — measured, and it inverts the obvious reading
+
+The registered pusher (`@vhoang`, Element X iOS) uses **`"format": "event_id_only"`** via
+matrix.org's sygnal. That payload carries event_id, room_id and counts — **no room name, no
+sender, no content**. It still produces a useful notification because `mutable-content: 1`
+means Element X runs a **Notification Service Extension**: iOS wakes it with the near-empty
+payload and it **fetches the event and renders the notification on the device**.
+
+1. The tier-in-room-identity scheme works — but **via the client fetch**, not the payload.
+   `m.room.name` must be set at creation; the ALIAS is not what reaches the phone. Synapse
+   sends `ctx["name"]` (the `m.room.name` state event) and omits the key entirely if unset.
+2. **`push: include_content: false` is irrelevant for clients with an NSE.** It bites clients
+   without one.
+3. ⚠ **Server-invisible failure mode:** if the phone cannot reach the homeserver at wake time
+   the fetch fails and iOS shows the bare word "Notification". **Synapse records
+   `last_success` and sees a delivered push.**
+
+Self-hosted sygnal **considered and rejected** — sygnal is a relay to FCM/APNs, not a
+replacement, so it removes matrix.org and nothing else; and with `event_id_only` the path
+already carries nothing worth protecting.
+
+## QR sign-in — requires MAS, deferred
+
+MSC4108 hard-requires `matrix_authentication_service`; Synapse refuses to start otherwise.
+MSC4388 enables independently but is only the rendezvous **channel**, not a login flow.
+Deferred: MAS is a service, a database and a migration of every account off built-in auth,
+and the v1.139.0 note warns `/register` from **old appservice implementations may break under
+MAS** — precisely the bridge owning 13 of 15 accounts.
+
+## Shared-secret registration gotcha
+
+`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**, it does
+not **terminate**. A trailing `\x00` yields `HMAC incorrect`. Run inside the container against
+`localhost:8008`; port 8008 is not published to the host.
+
+Full doc: `docs/pfi/vm-102-matrix-synapse.md` (`931bac8`, `73866f6`).