vh
7142657749
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions.
103 lines
5.8 KiB
Markdown
103 lines
5.8 KiB
Markdown
# `[2026-09-01]` Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
|
|
|
|
## The upgrade
|
|
|
|
**Synapse v1.120.0 → v1.159.0** (21 months, 39 releases) and **Element-web v1.11.80 →
|
|
v1.12.27**. Schema migrations applied cleanly through schema 94. Postgres deliberately left
|
|
at 16 — changing two stateful things at once destroys failure attribution.
|
|
|
|
⚠ **Schema migrations are ONE-WAY.** v1.120 cannot start against a v1.159 database. Rollback
|
|
is restore-from-dump, not revert-the-tag. Verified pre-upgrade dump (739 TOC entries from a
|
|
33 MB database) plus all four config files at
|
|
`/opt/docker/backups/synapse-preupgrade-20260901T174625Z/`.
|
|
|
|
Reviewed every upgrade note in the range; nothing applicable bit us (PG 11/12/13 drops — we
|
|
are on 16; MSC3861/MAS items — no MAS; s3-storage-provider and worker media quarantine — not
|
|
in use).
|
|
|
|
## The appservice namespace — `exclusive: true` → `false`
|
|
|
|
The `aipa-bridge` registration claimed `@[a-z][a-z0-9_-]*:matrix.phasefinal.com` **exclusively**
|
|
— every localpart on the server. 14 of 14 accounts fell inside it; 13 were appservice-owned.
|
|
|
|
**`exclusive` governs who ELSE may act, not what the appservice may do.** On a homeserver
|
|
with registration disabled, one admin and no competing actor, it bought anti-squatting
|
|
protection against a threat that cannot occur, while locking out every other means of
|
|
account creation — admin registration returned `M_EXCLUSIVE` with no explanation.
|
|
|
|
⚠ **Do NOT narrow the users regex to a prefix** — all 13 accounts fall inside it and would be
|
|
orphaned. ⚠ **Do NOT rename the `id`** — Synapse keys account ownership on `aipa-bridge` in
|
|
the `users` table. The FILE may be renamed; the id may not.
|
|
|
|
The narrow **aliases** namespace (`#aipa-debug-*`) was left exclusive — specific, costs nothing.
|
|
|
|
⚠ The registration is named `aipa`, but the service behind it is **`wt-matrix-bridge`**, the
|
|
Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. AIPA is a dead project name
|
|
on a live service, and it is why infra-ops mis-routed a provisioning request to worldtree-dev.
|
|
**Operator ruling: worldtree-dev writes the bridge code; infra-ops OPERATES this instance and
|
|
has full authority over it.**
|
|
|
|
## `/_synapse/admin` closed to the internet
|
|
|
|
Synapse mounts its admin API on the same vhost as the client API, so publishing
|
|
`matrix.phasefinal.com` published the admin surface — it **answered 200 from the open
|
|
internet**. A higher-priority router (explicit priority 100) now scopes it behind an
|
|
`ipallowlist`.
|
|
|
|
Verified from a **genuinely external vantage** — the NH3 residential egress proxy, because
|
|
testing from a fleet host sits inside the allow-list and proves nothing: admin **403**,
|
|
client API **200**, Element unaffected.
|
|
|
|
⚠ **The `10.0.0.0/8` entry matches NOTHING and that is expected.** The hostname resolves
|
|
publicly, so fleet hosts hairpin out their own WAN — a request from nh3-dev measured as
|
|
`70.230.226.88`. The rule is effectively **deny-all through Traefik**, which is intended:
|
|
admin work goes via `docker exec synapse` against `localhost:8008` and never touches Traefik.
|
|
Allow-listing the sites' WAN addresses was **rejected** — dynamic, and a stale entry either
|
|
locks us out or hands admin to whoever inherits the address.
|
|
|
|
## Conventions ratified (operator, 2026-09-01)
|
|
|
|
#<agent>-<purpose>:matrix.phasefinal.com
|
|
|
|
Mirrors the existing `@<agent>:` user-ID convention. Proposed by ledger-dev. Rationale is
|
|
**"so the room IDENTITY carries the tier"** — deliberately NOT "so the push payload carries
|
|
the room name", which is true only for clients without a notification service extension.
|
|
Pre-existing rooms are not renamed ("The High Seat", `!NiVVoMsyoHCBRPrrrn`).
|
|
|
|
## Push reality — measured, and it inverts the obvious reading
|
|
|
|
The registered pusher (`@vhoang`, Element X iOS) uses **`"format": "event_id_only"`** via
|
|
matrix.org's sygnal. That payload carries event_id, room_id and counts — **no room name, no
|
|
sender, no content**. It still produces a useful notification because `mutable-content: 1`
|
|
means Element X runs a **Notification Service Extension**: iOS wakes it with the near-empty
|
|
payload and it **fetches the event and renders the notification on the device**.
|
|
|
|
1. The tier-in-room-identity scheme works — but **via the client fetch**, not the payload.
|
|
`m.room.name` must be set at creation; the ALIAS is not what reaches the phone. Synapse
|
|
sends `ctx["name"]` (the `m.room.name` state event) and omits the key entirely if unset.
|
|
2. **`push: include_content: false` is irrelevant for clients with an NSE.** It bites clients
|
|
without one.
|
|
3. ⚠ **Server-invisible failure mode:** if the phone cannot reach the homeserver at wake time
|
|
the fetch fails and iOS shows the bare word "Notification". **Synapse records
|
|
`last_success` and sees a delivered push.**
|
|
|
|
Self-hosted sygnal **considered and rejected** — sygnal is a relay to FCM/APNs, not a
|
|
replacement, so it removes matrix.org and nothing else; and with `event_id_only` the path
|
|
already carries nothing worth protecting.
|
|
|
|
## QR sign-in — requires MAS, deferred
|
|
|
|
MSC4108 hard-requires `matrix_authentication_service`; Synapse refuses to start otherwise.
|
|
MSC4388 enables independently but is only the rendezvous **channel**, not a login flow.
|
|
Deferred: MAS is a service, a database and a migration of every account off built-in auth,
|
|
and the v1.139.0 note warns `/register` from **old appservice implementations may break under
|
|
MAS** — precisely the bridge owning 13 of 15 accounts.
|
|
|
|
## Shared-secret registration gotcha
|
|
|
|
`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**, it does
|
|
not **terminate**. A trailing `\x00` yields `HMAC incorrect`. Run inside the container against
|
|
`localhost:8008`; port 8008 is not published to the host.
|
|
|
|
Full doc: `docs/pfi/vm-102-matrix-synapse.md` (`931bac8`, `73866f6`).
|