esh-pfi-infrastructure/docs/pfi/vm-102-matrix-synapse.md

# VM 102 — Matrix Synapse Deployment

## Overview

Matrix (Synapse) is deployed on VM 102 (PFI-ANA-Docker, `10.250.50.70`) as the primary
human-to-agent communication channel for AIPA. The stack consists of:

- **Synapse** — Matrix homeserver (event routing, auth, persistence)
- **PostgreSQL 16** — Synapse database backend
- **Element Web** — Web client for users

Federation is disabled (internal-only deployment). Registration is disabled (admin-created accounts only).

> ⚠ **The build steps below date from the AIPA era and describe how this was
> originally stood up.** They are kept for provenance. For what is actually
> running, read *Current state* immediately below — the versions, the appservice
> owner, and the bridge have all moved since.

---

## Current state — 2026-09-01

| | |
|---|---|
| Synapse | **v1.159.0** (upgraded from v1.120.0, 2026-09-01 — 39 releases, 21 months) |
| Element Web | **v1.12.27** (from v1.11.80) |
| PostgreSQL | **16-alpine** — deliberately NOT upgraded alongside Synapse |
| Appservice | registration `id: aipa-bridge`, but the service behind it is **`wt-matrix-bridge`**, the Worldtree PERSONAL instance on corviduo-dev `10.250.50.152:8010`. The `aipa` name is legacy. |
| Accounts | 15 — 13 appservice-owned, plus `@vhoang` (admin) and `@miranda` |
| Encryption | **zero encrypted rooms**, server-wide |
| Federation | `federation_domain_whitelist: []` — present but EMPTY, so federation is effectively OFF. The federation endpoint still answers, which is misleading. |

**Ops ownership:** worldtree-dev writes the bridge code; **infra-ops operates this
instance** and has full authority to change it (operator, 2026-09-01). Do not route
provisioning or config requests to worldtree-dev.

### Upgrade procedure (learned 2026-09-01)

**Synapse schema migrations are one-way.** v1.120 cannot start against a v1.159
database. Rollback is restore-from-dump, not revert-the-tag. Therefore:

```bash
D=/opt/docker/backups/synapse-preupgrade-$(date -u +%Y%m%dT%H%M%SZ)
docker exec synapse-db pg_dump -U synapse -d synapse --format=custom --compress=6 > $D/synapse.dump
docker exec -i synapse-db pg_restore --list < $D/synapse.dump | wc -l   # verify it reads back
```

Snapshot `compose.yaml`, `homeserver.yaml`, `aipa_appservice.yaml` and
`element-config.json` alongside it. Then bump tags and `up -d`; watch for
`Schema now up to date` in the logs.

A benign startup warning — `Failed to listen on 0.0.0.0 ... Address already in use`
— is normal Linux dual-stack behaviour after it binds `[::]:8008`. Not a fault.

### Appservice namespace — why `exclusive` is false

The `aipa-bridge` registration claims a user namespace of
`@[a-z][a-z0-9_-]*:matrix.phasefinal.com` — effectively every localpart on the
server. It was `exclusive: true` until 2026-09-01.

**`exclusive` governs who ELSE may act, not what the appservice may do.** On a
homeserver with registration disabled, one admin, and no competing actor, it bought
anti-squatting protection against a threat that cannot occur, while locking out
every other means of account creation — admin shared-secret registration returned
`M_EXCLUSIVE` for any conventional name, with no explanation of why.

Now `exclusive: false`, regex unchanged. The bridge is unaffected: it creates and
puppets exactly as before. The narrow **aliases** namespace (`#aipa-debug-*`) is
left exclusive — it is specific and costs nothing.

⚠ **Do not narrow the users regex to a prefix.** All 13 appservice accounts fall
inside it; narrowing would orphan them. ⚠ **Do not rename the `id`** — Synapse keys
account ownership on `aipa-bridge` in the `users` table. The FILE may be renamed.

### Shared-secret registration — the HMAC has no trailing null

`POST /_synapse/admin/v1/register`. The MAC is
`HMAC-SHA1(secret, nonce \0 user \0 password \0 "notadmin")` — the null **separates**,
it does not **terminate**. A trailing `\x00` after `notadmin` yields `HMAC incorrect`.
Run it inside the container against `localhost:8008`; port 8008 is not published to
the host.

---

## Conventions

### User IDs

`@<agent>:matrix.phasefinal.com` — one account per agent or persona.

### Room aliases — ratified by the operator 2026-09-01

```
#<agent>-<purpose>:matrix.phasefinal.com
```

Proposed by ledger-dev, ratified by the operator. Mirrors the user-ID convention
rather than introducing a second, independent axis.

- **Ownership is legible from the alias alone** — `#miranda-summons` is
  unambiguously Miranda's without opening the room.
- **Namespaced on the AGENT, not a product**, so it survives a consuming project's
  rename.
- **`#aipa-debug-*` is avoided by construction** — no agent is named `aipa-debug`,
  and that range is the bridge's exclusive alias namespace.
- **Load-bearing rationale, not tidiness:** purpose-as-tier is what makes a summons
  triageable **so the room IDENTITY carries the tier**. (Stated carefully: it is
  *not* that the push payload carries the room name — see the pusher note below.)

**Known cost, accepted deliberately:** a genuinely shared room with no single owner
has no natural slot. Preferred over pre-building a namespace for a room that may
never exist; name that case deliberately if it arrives.

**Pre-existing rooms are not renamed.** "The High Seat"
(`!NiVVoMsyoHCBRPrrrn`) predates the convention, has no canonical alias, and works.
Renaming a live proven channel to satisfy a scheme it did not exist under is the
churn the convention exists to prevent. New rooms follow it.

---

## Push notifications — what actually reaches a phone

Established 2026-09-01 while proving Ledger's summons channel.

The registered pusher (`@vhoang`, Element X iOS) uses
**`"format": "event_id_only"`** via `https://matrix.org/_matrix/push/v1/notify`.
That payload carries `event_id`, `room_id` and counts — **no room name, no sender,
no content.**

It still produces a useful notification because `mutable-content: 1` means Element X
runs a **Notification Service Extension**: iOS wakes it with the near-empty payload,
and the extension then **fetches the event from the homeserver and renders the
notification on the device.**

Three consequences:

1. **The tier-in-room-identity scheme works, but via the client fetch** — not
   because the push carried the name. `m.room.name` must still be set at creation.
2. **`push: include_content: false` is irrelevant for clients with an NSE.** The
   extension reads the event directly. It *does* bite clients without one.
3. **Server-invisible failure mode:** if the phone cannot reach
   `matrix.phasefinal.com` at wake time, the fetch fails and iOS shows the fallback
   — the bare word "Notification". **Synapse records `last_success` and sees a
   delivered push.** The degradation happens entirely on the device.

Because the payload carries nothing meaningful, **self-hosted sygnal was considered
and rejected** — it would remove matrix.org from a path that already carries no
content, room name or sender.

⚠ Sygnal is a **relay** to FCM/APNs, not a replacement for them. Self-hosting it
removes matrix.org and nothing else. "Notifications must not leave the fleet" is not
satisfiable with mobile push at any price.

---

## Sign in with QR code — requires Matrix Authentication Service

Asked 2026-09-01; **not enabled.** From Synapse 1.159.0's own config validator:

```python
if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) \
        and not auth_delegated:
    raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled")
```

QR sign-in is **MSC4108**, an OIDC flow that hard-requires **MAS**. Synapse refuses
to start with the flag set and no MAS.

**MSC4388** (`msc4388_mode: off|open|authenticated`) enables independently, but it is
only the **rendezvous channel** — the transport, not a login flow.

Deferred because MAS is a service, a database, and a migration of every account off
built-in auth — and the v1.139.0 upgrade note warns that `/register` from **old
application service implementations may break under MAS**, which is precisely the
bridge owning 13 of the 15 accounts. Revisit when MAS becomes mandatory rather than
optional, in its own change window, with the bridge regression-tested either side.

---

## Architecture

```
User (Element client)
       │  m.room.message events
       ▼
┌──────────────────────┐
│  Synapse             │  Matrix homeserver — event routing, auth, persistence
│  (homeserver)        │  Port 8008 (Client-Server API)
└──────────┬───────────┘
           │  Appservice push  PUT /_matrix/app/v1/transactions/{txnId}
           ▼
┌──────────────────────────────────────────┐
│  AIPA Matrix Bridge  (core/matrix_bridge.py)  │
│  Port 8009                                │
└──────────────────────────────────────────┘
```

> Full appservice bridge details: [VM 102 — Matrix Appservice Configuration](vm-102-matrix-appservice.md)

---

## Components

| Component       | Image                          | Port  | Purpose                           |
|-----------------|--------------------------------|-------|-----------------------------------|
| `synapse`       | `matrixdotorg/synapse:latest`  | 8008  | Matrix homeserver (Client-Server API) |
| `synapse-db`    | `postgres:16`                  | —     | Synapse database                  |
| `element-web`   | `vectorim/element-web:latest`  | 8080  | Web client for users              |

---

## Storage Paths

| Host Path                          | Purpose                                 |
|------------------------------------|-----------------------------------------|
| `/opt/docker/compose/synapse/`     | Compose file (managed by Dockge)        |
| `/opt/docker/conf/synapse/`        | Configuration files                     |
| `/opt/docker/data/synapse/`        | Synapse data (`homeserver.yaml`, signing keys, appservice reg) |
| `/opt/docker/conf/synapse/element-config.json` | Element Web client config    |

> Follows the VM 102 convention: compose in `/opt/docker/compose/<service>/`, config in `/opt/docker/conf/<service>/`.

### Volumes

| Volume / Path                  | Container Path                | Purpose                     |
|--------------------------------|-------------------------------|-----------------------------|
| `/opt/docker/data/synapse`     | `/data`                       | Synapse config, signing keys, media store |
| `synapse-db-data` (named vol)  | `/var/lib/postgresql/data`    | PostgreSQL persistent data   |

---

## Step 1 — Create Directory Structure

```bash
mkdir -p /opt/docker/data/synapse
mkdir -p /opt/docker/conf/synapse
mkdir -p /opt/docker/compose/synapse
```

---

## Step 2 — Generate Synapse Config

```bash
docker run --rm \
  -v /opt/docker/data/synapse:/data \
  -e SYNAPSE_SERVER_NAME=matrix.pfi.local \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate
```

This writes `/opt/docker/data/synapse/homeserver.yaml` and
`/opt/docker/data/synapse/matrix.pfi.local.signing.key`.

**Do not modify `server_name` after generation — it is permanent.**

---

## Step 3 — Edit homeserver.yaml

Open `/opt/docker/data/synapse/homeserver.yaml` and apply:

```yaml
# Use PostgreSQL instead of SQLite (required for production)
database:
  name: psycopg2
  args:
    user: synapse
    password: synapse_db_password   # match POSTGRES_PASSWORD in compose
    database: synapse
    host: synapse-db
    cp_min: 5
    cp_max: 10

# Disable open registration — accounts are created by admin only
enable_registration: false

# Disable federation (internal deployment only)
federation_domain_whitelist: []

# Allow the application service to be registered (add AFTER generating the AS file)
app_service_config_files:
  - /data/aipa_appservice.yaml
```

---

## Step 4 — Docker Compose

File: `/opt/docker/compose/synapse/docker-compose.yml`

```yaml
---
# =============================================================================
# Synapse Matrix Homeserver — AIPA internal deployment on VM 102
# =============================================================================
#
# Conventions:
#   - Config:  /opt/docker/conf/synapse/
#   - Data:    /opt/docker/data/synapse/  (bind mount) + synapse-db-data (named vol)
#   - Compose: /opt/docker/compose/synapse/
#   - Network: synapse-net (dedicated, not on traefik-net)
#
# Notes:
#   - Federation disabled (internal only)
#   - Port 8448 (federation) commented out
#   - Resource limits set for VM 102 (8 vCPU, 16 GB RAM)

services:
  synapse-db:
    image: postgres:16
    container_name: synapse-db
    restart: unless-stopped
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: synapse_db_password
      POSTGRES_DB: synapse
      POSTGRES_INITDB_ARGS: "--encoding=UTF-8 --lc-collate=C --lc-ctype=C"
    volumes:
      - synapse-db-data:/var/lib/postgresql/data
    networks:
      - synapse-net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse"]
      interval: 10s
      timeout: 5s
      retries: 5

  synapse:
    image: matrixdotorg/synapse:latest
    container_name: synapse
    restart: unless-stopped
    depends_on:
      synapse-db:
        condition: service_healthy
    ports:
      - "8008:8008"   # Client-Server API (HTTP)
      # - "8448:8448" # Server-Server API (federation) — disabled for internal use
    volumes:
      - /opt/docker/data/synapse:/data
    networks:
      - synapse-net
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: "2.0"
        reservations:
          memory: 256M
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:8008/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s

  element-web:
    image: vectorim/element-web:latest
    container_name: element-web
    restart: unless-stopped
    volumes:
      - /opt/docker/conf/synapse/element-config.json:/app/config.json:ro
    ports:
      - "8080:80"
    networks:
      - synapse-net

volumes:
  synapse-db-data:

networks:
  synapse-net:
    name: synapse-net
```

### Start

```bash
cd /opt/docker/compose/synapse
docker compose up -d
```

### Verify

```bash
docker compose ps
curl http://localhost:8008/health
```

---

## Step 5 — Element Web Configuration

File: `/opt/docker/conf/synapse/element-config.json`

```json
{
    "default_server_config": {
        "m.homeserver": {
            "base_url": "http://10.250.50.70:8008",
            "server_name": "matrix.pfi.local"
        }
    },
    "brand": "AIPA",
    "default_theme": "dark",
    "disable_guests": true,
    "disable_login_language_selector": true
}
```

---

## Step 6 — Create Admin User

```bash
docker exec -it synapse register_new_matrix_user \
  -u admin \
  -p 'yourpassword' \
  -a \
  http://localhost:8008
```

> The `-a` flag makes the user an admin. Omit for regular users.

**Agent users** (`@atlas`, `@linus`, `@hermione`) are **virtual** — managed by the
appservice. Do **not** register them as real accounts.

---

## Port Allocation

| Port | Service       | Purpose                    | Protocol |
|------|---------------|----------------------------|----------|
| 8008 | Synapse       | Client-Server API          | HTTP     |
| 8080 | Element Web   | Web client                 | HTTP     |
| 8009 | AIPA Bridge   | Appservice endpoint        | HTTP     |

> See [docker-stack.md](docker-stack.md) for the full VM 102 port allocation table.

---

## Network

This deployment uses a **dedicated `synapse-net` network** (not `traefik-net`), because:

- Synapse is accessed directly by IP (no public domain routing needed)
- The AIPA bridge connects to Synapse at `http://localhost:8008` from the host
- Element Web connects at the VM IP:8008 from the browser

If TLS/reverse proxy is added later, join `traefik-net` and add Traefik labels.

---

## Security Notes

- Synapse is exposed on port 8008 (HTTP). For any externally accessible deployment,
  put it behind a TLS-terminating reverse proxy (Traefik/nginx) and restrict 8009
  to internal access only.
- Registration is disabled (`enable_registration: false`) — accounts created by admin only.
- Federation is disabled (`federation_domain_whitelist: []`) — internal use only.
- The `as_token` and `hs_token` in the appservice registration are secrets equivalent to
  admin credentials. Store in `env.sh` (gitignored), never commit.

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| Bridge starts but Synapse doesn't push events | Appservice URL wrong in registration YAML | Verify `url:` is reachable from the Synapse container; check `docker inspect synapse` network |
| 401 errors from Synapse | `as_token` mismatch | Verify token in `aipa_appservice.yaml` matches `MATRIX_AS_TOKEN` env var |
| Element can't connect to homeserver | Wrong `base_url` in element-config.json | Must be the IP/hostname Element's browser can reach, not the Docker container name |
| Synapse won't start | Database connection failure | Verify `synapse-db` is healthy first; check password matches in `homeserver.yaml` and compose |

---

## Next Steps

After Synapse is running and healthy, configure the AIPA appservice bridge:
→ [VM 102 — Matrix Appservice Configuration](vm-102-matrix-appservice.md)

---

## Sources

- Source deployment guide: `projects/matrix/matrix-deployment.md` (2026-04-11)
- VM 102 Proxmox config: `configs/pfi-ana/proxmox/vm-102.conf`
- Docker Stack conventions: `docs/pfi-ana/docker-stack.md`
- Matrix Protocol Reference: `infrastructure/matrix-docker-deployment.md` (KB)