vh
7142657749
Run 3c's new intended home is the GX10 rather than a power triage on ana-ml2: a ~240 W appliance instead of the kilowatt-class box that tripped the breaker, and 121 GB unified holds the 49 GB bf16 base comfortably where ana-ml2 was tight. The box is bare, so the first move is a throughput probe rather than a harness port. Also banked: the Ada migration settling on zfs send with branch (b) ruled out by irv-ml1 keeping its eight services; the Synapse 39-release upgrade with its one-way schema migration, the appservice namespace opening and the admin API lockdown; the ratified room alias convention; and a named failure class — a correct check aimed at the wrong object — with six instances from one day across three sessions.
5.8 KiB
[2026-09-01] Matrix: 39-release Synapse upgrade, appservice namespace opened, admin API closed
The upgrade
Synapse v1.120.0 → v1.159.0 (21 months, 39 releases) and Element-web v1.11.80 → v1.12.27. Schema migrations applied cleanly through schema 94. Postgres deliberately left at 16 — changing two stateful things at once destroys failure attribution.
⚠ Schema migrations are ONE-WAY. v1.120 cannot start against a v1.159 database. Rollback
is restore-from-dump, not revert-the-tag. Verified pre-upgrade dump (739 TOC entries from a
33 MB database) plus all four config files at
/opt/docker/backups/synapse-preupgrade-20260901T174625Z/.
Reviewed every upgrade note in the range; nothing applicable bit us (PG 11/12/13 drops — we are on 16; MSC3861/MAS items — no MAS; s3-storage-provider and worker media quarantine — not in use).
The appservice namespace — exclusive: true → false
The aipa-bridge registration claimed @[a-z][a-z0-9_-]*:matrix.phasefinal.com exclusively
— every localpart on the server. 14 of 14 accounts fell inside it; 13 were appservice-owned.
exclusive governs who ELSE may act, not what the appservice may do. On a homeserver
with registration disabled, one admin and no competing actor, it bought anti-squatting
protection against a threat that cannot occur, while locking out every other means of
account creation — admin registration returned M_EXCLUSIVE with no explanation.
⚠ Do NOT narrow the users regex to a prefix — all 13 accounts fall inside it and would be
orphaned. ⚠ Do NOT rename the id — Synapse keys account ownership on aipa-bridge in
the users table. The FILE may be renamed; the id may not.
The narrow aliases namespace (#aipa-debug-*) was left exclusive — specific, costs nothing.
⚠ The registration is named aipa, but the service behind it is wt-matrix-bridge, the
Worldtree PERSONAL instance on corviduo-dev 10.250.50.152:8010. AIPA is a dead project name
on a live service, and it is why infra-ops mis-routed a provisioning request to worldtree-dev.
Operator ruling: worldtree-dev writes the bridge code; infra-ops OPERATES this instance and
has full authority over it.
/_synapse/admin closed to the internet
Synapse mounts its admin API on the same vhost as the client API, so publishing
matrix.phasefinal.com published the admin surface — it answered 200 from the open
internet. A higher-priority router (explicit priority 100) now scopes it behind an
ipallowlist.
Verified from a genuinely external vantage — the NH3 residential egress proxy, because testing from a fleet host sits inside the allow-list and proves nothing: admin 403, client API 200, Element unaffected.
⚠ The 10.0.0.0/8 entry matches NOTHING and that is expected. The hostname resolves
publicly, so fleet hosts hairpin out their own WAN — a request from nh3-dev measured as
70.230.226.88. The rule is effectively deny-all through Traefik, which is intended:
admin work goes via docker exec synapse against localhost:8008 and never touches Traefik.
Allow-listing the sites' WAN addresses was rejected — dynamic, and a stale entry either
locks us out or hands admin to whoever inherits the address.
Conventions ratified (operator, 2026-09-01)
#<agent>-<purpose>:matrix.phasefinal.com
Mirrors the existing @<agent>: user-ID convention. Proposed by ledger-dev. Rationale is
"so the room IDENTITY carries the tier" — deliberately NOT "so the push payload carries
the room name", which is true only for clients without a notification service extension.
Pre-existing rooms are not renamed ("The High Seat", !NiVVoMsyoHCBRPrrrn).
Push reality — measured, and it inverts the obvious reading
The registered pusher (@vhoang, Element X iOS) uses "format": "event_id_only" via
matrix.org's sygnal. That payload carries event_id, room_id and counts — no room name, no
sender, no content. It still produces a useful notification because mutable-content: 1
means Element X runs a Notification Service Extension: iOS wakes it with the near-empty
payload and it fetches the event and renders the notification on the device.
- The tier-in-room-identity scheme works — but via the client fetch, not the payload.
m.room.namemust be set at creation; the ALIAS is not what reaches the phone. Synapse sendsctx["name"](them.room.namestate event) and omits the key entirely if unset. push: include_content: falseis irrelevant for clients with an NSE. It bites clients without one.- ⚠ Server-invisible failure mode: if the phone cannot reach the homeserver at wake time
the fetch fails and iOS shows the bare word "Notification". Synapse records
last_successand sees a delivered push.
Self-hosted sygnal considered and rejected — sygnal is a relay to FCM/APNs, not a
replacement, so it removes matrix.org and nothing else; and with event_id_only the path
already carries nothing worth protecting.
QR sign-in — requires MAS, deferred
MSC4108 hard-requires matrix_authentication_service; Synapse refuses to start otherwise.
MSC4388 enables independently but is only the rendezvous channel, not a login flow.
Deferred: MAS is a service, a database and a migration of every account off built-in auth,
and the v1.139.0 note warns /register from old appservice implementations may break under
MAS — precisely the bridge owning 13 of 15 accounts.
Shared-secret registration gotcha
HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin") — the null separates, it does
not terminate. A trailing \x00 yields HMAC incorrect. Run inside the container against
localhost:8008; port 8008 is not published to the host.
Full doc: docs/pfi/vm-102-matrix-synapse.md (931bac8, 73866f6).