esh-pfi-infrastructure/docs/pfi/vm-102-matrix-synapse.md

VM 102 — Matrix Synapse Deployment

Overview

Matrix (Synapse) is deployed on VM 102 (PFI-ANA-Docker, 10.250.50.70) as the primary human-to-agent communication channel for AIPA. The stack consists of:

• Synapse — Matrix homeserver (event routing, auth, persistence)
• PostgreSQL 16 — Synapse database backend
• Element Web — Web client for users

Federation is disabled (internal-only deployment). Registration is disabled (admin-created accounts only).

⚠ The build steps below date from the AIPA era and describe how this was originally stood up. They are kept for provenance. For what is actually running, read Current state immediately below — the versions, the appservice owner, and the bridge have all moved since.

---

Current state — 2026-09-01

Synapse	v1.159.0 (upgraded from v1.120.0, 2026-09-01 — 39 releases, 21 months)
Element Web	v1.12.27 (from v1.11.80)
PostgreSQL	16-alpine — deliberately NOT upgraded alongside Synapse
Appservice	registration id: aipa-bridge, but the service behind it is wt-matrix-bridge, the Worldtree PERSONAL instance on corviduo-dev 10.250.50.152:8010. The aipa name is legacy.
Accounts	15 — 13 appservice-owned, plus @vhoang (admin) and @miranda
Encryption	zero encrypted rooms, server-wide
Federation	federation_domain_whitelist: [] — present but EMPTY, so federation is effectively OFF. The federation endpoint still answers, which is misleading.

Ops ownership: worldtree-dev writes the bridge code; infra-ops operates this instance and has full authority to change it (operator, 2026-09-01). Do not route provisioning or config requests to worldtree-dev.

Upgrade procedure (learned 2026-09-01)

Synapse schema migrations are one-way. v1.120 cannot start against a v1.159 database. Rollback is restore-from-dump, not revert-the-tag. Therefore:

D=/opt/docker/backups/synapse-preupgrade-$(date -u +%Y%m%dT%H%M%SZ)
docker exec synapse-db pg_dump -U synapse -d synapse --format=custom --compress=6 > $D/synapse.dump
docker exec -i synapse-db pg_restore --list < $D/synapse.dump | wc -l   # verify it reads back

Snapshot compose.yaml, homeserver.yaml, aipa_appservice.yaml and element-config.json alongside it. Then bump tags and up -d; watch for Schema now up to date in the logs.

A benign startup warning — Failed to listen on 0.0.0.0 ... Address already in use — is normal Linux dual-stack behaviour after it binds [::]:8008. Not a fault.

Appservice namespace — why exclusive is false

The aipa-bridge registration claims a user namespace of @[a-z][a-z0-9_-]*:matrix.phasefinal.com — effectively every localpart on the server. It was exclusive: true until 2026-09-01.

exclusive governs who ELSE may act, not what the appservice may do. On a homeserver with registration disabled, one admin, and no competing actor, it bought anti-squatting protection against a threat that cannot occur, while locking out every other means of account creation — admin shared-secret registration returned M_EXCLUSIVE for any conventional name, with no explanation of why.

Now exclusive: false, regex unchanged. The bridge is unaffected: it creates and puppets exactly as before. The narrow aliases namespace (#aipa-debug-*) is left exclusive — it is specific and costs nothing.

⚠ Do not narrow the users regex to a prefix. All 13 appservice accounts fall inside it; narrowing would orphan them. ⚠ Do not rename the id — Synapse keys account ownership on aipa-bridge in the users table. The FILE may be renamed.

Shared-secret registration — the HMAC has no trailing null

POST /_synapse/admin/v1/register. The MAC is HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin") — the null separates, it does not terminate. A trailing \x00 after notadmin yields HMAC incorrect. Run it inside the container against localhost:8008; port 8008 is not published to the host.

---

Conventions

User IDs

@<agent>:matrix.phasefinal.com — one account per agent or persona.

Room aliases — ratified by the operator 2026-09-01

#<agent>-<purpose>:matrix.phasefinal.com

Proposed by ledger-dev, ratified by the operator. Mirrors the user-ID convention rather than introducing a second, independent axis.

• Ownership is legible from the alias alone — #miranda-summons is unambiguously Miranda's without opening the room.
• Namespaced on the AGENT, not a product, so it survives a consuming project's rename.
• #aipa-debug-* is avoided by construction — no agent is named aipa-debug, and that range is the bridge's exclusive alias namespace.
• Load-bearing rationale, not tidiness: purpose-as-tier is what makes a summons triageable so the room IDENTITY carries the tier. (Stated carefully: it is not that the push payload carries the room name — see the pusher note below.)

Known cost, accepted deliberately: a genuinely shared room with no single owner has no natural slot. Preferred over pre-building a namespace for a room that may never exist; name that case deliberately if it arrives.

Pre-existing rooms are not renamed. "The High Seat" (!NiVVoMsyoHCBRPrrrn) predates the convention, has no canonical alias, and works. Renaming a live proven channel to satisfy a scheme it did not exist under is the churn the convention exists to prevent. New rooms follow it.

---

Push notifications — what actually reaches a phone

Established 2026-09-01 while proving Ledger's summons channel.

The registered pusher (@vhoang, Element X iOS) uses "format": "event_id_only" via https://matrix.org/_matrix/push/v1/notify. That payload carries event_id, room_id and counts — no room name, no sender, no content.

It still produces a useful notification because mutable-content: 1 means Element X runs a Notification Service Extension: iOS wakes it with the near-empty payload, and the extension then fetches the event from the homeserver and renders the notification on the device.

Three consequences:

1. The tier-in-room-identity scheme works, but via the client fetch — not because the push carried the name. m.room.name must still be set at creation.
2. push: include_content: false is irrelevant for clients with an NSE. The extension reads the event directly. It does bite clients without one.
3. Server-invisible failure mode: if the phone cannot reach matrix.phasefinal.com at wake time, the fetch fails and iOS shows the fallback — the bare word "Notification". Synapse records last_success and sees a delivered push. The degradation happens entirely on the device.

Because the payload carries nothing meaningful, self-hosted sygnal was considered and rejected — it would remove matrix.org from a path that already carries no content, room name or sender.

⚠ Sygnal is a relay to FCM/APNs, not a replacement for them. Self-hosting it removes matrix.org and nothing else. "Notifications must not leave the fleet" is not satisfiable with mobile push at any price.

---

Sign in with QR code — requires Matrix Authentication Service

Asked 2026-09-01; not enabled. From Synapse 1.159.0's own config validator:

if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) \
        and not auth_delegated:
    raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled")

QR sign-in is MSC4108, an OIDC flow that hard-requires MAS. Synapse refuses to start with the flag set and no MAS.

MSC4388 (msc4388_mode: off|open|authenticated) enables independently, but it is only the rendezvous channel — the transport, not a login flow.

Deferred because MAS is a service, a database, and a migration of every account off built-in auth — and the v1.139.0 upgrade note warns that /register from old application service implementations may break under MAS, which is precisely the bridge owning 13 of the 15 accounts. Revisit when MAS becomes mandatory rather than optional, in its own change window, with the bridge regression-tested either side.

---

Architecture

User (Element client)
       │  m.room.message events
       ▼
┌──────────────────────┐
│  Synapse             │  Matrix homeserver — event routing, auth, persistence
│  (homeserver)        │  Port 8008 (Client-Server API)
└──────────┬───────────┘
           │  Appservice push  PUT /_matrix/app/v1/transactions/{txnId}
           ▼
┌──────────────────────────────────────────┐
│  AIPA Matrix Bridge  (core/matrix_bridge.py)  │
│  Port 8009                                │
└──────────────────────────────────────────┘

Full appservice bridge details: VM 102 — Matrix Appservice Configuration

---

Components

Component	Image	Port	Purpose
synapse	matrixdotorg/synapse:latest	8008	Matrix homeserver (Client-Server API)
synapse-db	postgres:16	—	Synapse database
element-web	vectorim/element-web:latest	8080	Web client for users

---

Storage Paths

Host Path	Purpose
/opt/docker/compose/synapse/	Compose file (managed by Dockge)
/opt/docker/conf/synapse/	Configuration files
/opt/docker/data/synapse/	Synapse data (homeserver.yaml, signing keys, appservice reg)
/opt/docker/conf/synapse/element-config.json	Element Web client config

Follows the VM 102 convention: compose in /opt/docker/compose/<service>/, config in /opt/docker/conf/<service>/.

Volumes

Volume / Path	Container Path	Purpose
/opt/docker/data/synapse	/data	Synapse config, signing keys, media store
synapse-db-data (named vol)	/var/lib/postgresql/data	PostgreSQL persistent data

---

Step 1 — Create Directory Structure

mkdir -p /opt/docker/data/synapse
mkdir -p /opt/docker/conf/synapse
mkdir -p /opt/docker/compose/synapse

---

Step 2 — Generate Synapse Config

docker run --rm \
  -v /opt/docker/data/synapse:/data \
  -e SYNAPSE_SERVER_NAME=matrix.pfi.local \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate

This writes /opt/docker/data/synapse/homeserver.yaml and /opt/docker/data/synapse/matrix.pfi.local.signing.key.

Do not modify server_name after generation — it is permanent.

---

Step 3 — Edit homeserver.yaml

Open /opt/docker/data/synapse/homeserver.yaml and apply:

# Use PostgreSQL instead of SQLite (required for production)
database:
  name: psycopg2
  args:
    user: synapse
    password: synapse_db_password   # match POSTGRES_PASSWORD in compose
    database: synapse
    host: synapse-db
    cp_min: 5
    cp_max: 10

# Disable open registration — accounts are created by admin only
enable_registration: false

# Disable federation (internal deployment only)
federation_domain_whitelist: []

# Allow the application service to be registered (add AFTER generating the AS file)
app_service_config_files:
  - /data/aipa_appservice.yaml

---

Step 4 — Docker Compose

File: /opt/docker/compose/synapse/docker-compose.yml

---
# =============================================================================
# Synapse Matrix Homeserver — AIPA internal deployment on VM 102
# =============================================================================
#
# Conventions:
#   - Config:  /opt/docker/conf/synapse/
#   - Data:    /opt/docker/data/synapse/  (bind mount) + synapse-db-data (named vol)
#   - Compose: /opt/docker/compose/synapse/
#   - Network: synapse-net (dedicated, not on traefik-net)
#
# Notes:
#   - Federation disabled (internal only)
#   - Port 8448 (federation) commented out
#   - Resource limits set for VM 102 (8 vCPU, 16 GB RAM)

services:
  synapse-db:
    image: postgres:16
    container_name: synapse-db
    restart: unless-stopped
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: synapse_db_password
      POSTGRES_DB: synapse
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C"
    volumes:
      - synapse-db-data:/var/lib/postgresql/data
    networks:
      - synapse-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse"]
      interval: 10s
      timeout: 5s
      retries: 5

  synapse:
    image: matrixdotorg/synapse:latest
    container_name: synapse
    restart: unless-stopped
    depends_on:
      synapse-db:
        condition: service_healthy
    ports:
      - "8008:8008"   # Client-Server API (HTTP)
      # - "8448:8448" # Server-Server API (federation) — disabled for internal use
    volumes:
      - /opt/docker/data/synapse:/data
    networks:
      - synapse-net
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: "2.0"
        reservations:
          memory: 256M
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8008/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

  element-web:
    image: vectorim/element-web:latest
    container_name: element-web
    restart: unless-stopped
    volumes:
      - /opt/docker/conf/synapse/element-config.json:/app/config.json:ro
    ports:
      - "8080:80"
    networks:
      - synapse-net

volumes:
  synapse-db-data:

networks:
  synapse-net:
    name: synapse-net

Start

cd /opt/docker/compose/synapse
docker compose up -d

Verify

docker compose ps
curl http://localhost:8008/health

---

Step 5 — Element Web Configuration

File: /opt/docker/conf/synapse/element-config.json

{
    "default_server_config": {
        "m.homeserver": {
            "base_url": "http://10.250.50.70:8008",
            "server_name": "matrix.pfi.local"
        }
    },
    "brand": "AIPA",
    "default_theme": "dark",
    "disable_guests": true,
    "disable_login_language_selector": true
}

---

Step 6 — Create Admin User

docker exec -it synapse register_new_matrix_user \
  -u admin \
  -p 'yourpassword' \
  -a \
  http://localhost:8008

The -a flag makes the user an admin. Omit for regular users.

Agent users (@atlas, @linus, @hermione) are virtual — managed by the appservice. Do not register them as real accounts.

---

Port Allocation

Port	Service	Purpose	Protocol
8008	Synapse	Client-Server API	HTTP
8080	Element Web	Web client	HTTP
8009	AIPA Bridge	Appservice endpoint	HTTP

See docker-stack.md for the full VM 102 port allocation table.

---

Network

This deployment uses a dedicated synapse-net network (not traefik-net), because:

• Synapse is accessed directly by IP (no public domain routing needed)
• The AIPA bridge connects to Synapse at http://localhost:8008 from the host
• Element Web connects at the VM IP:8008 from the browser

If TLS/reverse proxy is added later, join traefik-net and add Traefik labels.

---

Security Notes

• Synapse is exposed on port 8008 (HTTP). For any externally accessible deployment, put it behind a TLS-terminating reverse proxy (Traefik/nginx) and restrict 8009 to internal access only.
• Registration is disabled (enable_registration: false) — accounts created by admin only.
• Federation is disabled (federation_domain_whitelist: []) — internal use only.
• The as_token and hs_token in the appservice registration are secrets equivalent to admin credentials. Store in env.sh (gitignored), never commit.

---

Troubleshooting

Symptom	Likely Cause	Fix
Bridge starts but Synapse doesn't push events	Appservice URL wrong in registration YAML	Verify url: is reachable from the Synapse container; check docker inspect synapse network
401 errors from Synapse	as_token mismatch	Verify token in aipa_appservice.yaml matches MATRIX_AS_TOKEN env var
Element can't connect to homeserver	Wrong base_url in element-config.json	Must be the IP/hostname Element's browser can reach, not the Docker container name
Synapse won't start	Database connection failure	Verify synapse-db is healthy first; check password matches in homeserver.yaml and compose

---

Next Steps

After Synapse is running and healthy, configure the AIPA appservice bridge: → VM 102 — Matrix Appservice Configuration

---

Sources

• Source deployment guide: projects/matrix/matrix-deployment.md (2026-04-11)
• VM 102 Proxmox config: configs/pfi-ana/proxmox/vm-102.conf
• Docker Stack conventions: docs/pfi-ana/docker-stack.md
• Matrix Protocol Reference: infrastructure/matrix-docker-deployment.md (KB)