# ana-docker

General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0/16` that doesn't need a GPU — pair it with ana-ml2, which handles the GPU workloads.

## Network

- **LAN IP:** 10.250.50.70
- **FQDN:** `ana-docker.phasefinal.com`
- **SSH:** standard port 22
- **Docker CE from the bookworm repo since 2026-10-05** (29.8.2). Until then `/etc/apt/sources.list.d/docker.list` said
  `bullseye` on this bookworm host. Pre-change copy: `/root/docker.list.pre-bookworm-20261005`.
- ⚠ **`live-restore: true` + bind-mounted `/var/run/docker.sock` = stale sockets after any engine upgrade WITHOUT a
  reboot.** Containers keep running through the dockerd restart, but dockerd recreates the socket, so every container that
  bind-mounts the socket file still points at the old, deleted inode ("Cannot connect to the Docker daemon"). After an
  engine bump, restart them: crowdsec FIRST, then traefik, dozzle, beszel-agent, dockge, gitea-runner (find them with
  `docker inspect` Mounts Source=/var/run/docker.sock). Found 2026-10-05 when dozzle went unhealthy and traefik's
  docker provider stopped seeing changes.
- **Traefik entrypoints:** 80/443 terminate here; cert resolver `anaprod` (Let's Encrypt)
  - ⚠ **`readTimeout: 10m` on both entrypoints since 2026-10-05.** Traefik v3 defaults it to 60s, which cut every
    request body slower than 60 s. Measured: skaldsong's 2.5 GB registry blob PUTs died at exactly 60.0 s (504, then 499),
    and 16 of 16 deploys failed from 2026-05-28. The runner here reaches gitea through the FortiGate hairpin
    (ClientHost 10.250.50.1, ~40 MB/s). Static config is at `/opt/docker/conf/traefik-ana/config/traefik.yml` on the host;
    it is NOT in `stacks/` (pre-edit copy: `traefik.yml.pre-readtimeout-20261005`). A static change needs a
    `docker restart traefik` (~20 s of public HTTPS down).

## Hardware

- **CPU:** 8 vCPU (QEMU virtual — this is a VM)
- **RAM:** 15.6 GB
- **GPUs:** none
- **Storage:** 245 GB root (ext4) + NFS mounts from `10.250.50.50` (Debian 12 NFS/SMB file server; historically TrueNAS SCALE, since reprovisioned)
- **OS:** Debian 12 (bookworm), kernel 6.1.x
- **Docker:** 20.10.24

## Key paths

| Path | Purpose |
|------|---------|
| `/opt/docker/compose/<stack>/` | Compose files (also a git repo) |
| `/opt/docker/conf/<stack>/` | Config bind mounts |
| `/opt/docker/data/` | Service state (legacy — most stacks now use named volumes) |
| `/mnt/compose` | NFS — `10.250.50.50:/mnt/docker` |
| `/mnt/backup` | NFS — `10.250.50.50:/mnt/backup` (restic target) |
| `/mnt/tnvms` | NFS — `10.250.50.50:/mnt/pve-VMStorage` |

## Running stacks

| Stack | Port(s) | Notes |
|-------|---------|-------|
| traefik | 80 / 443 / 8380 | Reverse proxy + TLS (cert resolver `anaprod`) |
| crowdsec (+ blocklist-mirror) | 41412 | IPS; bouncer runs as a Traefik plugin. ⚠ The bouncer FAILS CLOSED: if LAPI does not answer, every public route returns 403. SQLite runs with `use_wal: true` since 2026-10-05. Before that, blocklist-mirror's stream reads held the DB for 26–58 s and caused three public 403 blips (0310/0342/0415); WAL took it to 0.04–2.4 s. Config: `/opt/docker/conf/crowdsec/config.yaml` on the host, not in `stacks/` |
| gitea | 3000 / 222 | Git hosting — `gitea.phasefinal.com` |
| vaultwarden | 9080 | Password vault — `vaultwarden.phasefinal.com` |
| synapse (+ synapse-db, element-web) | internal | Matrix homeserver — `matrix.phasefinal.com`, client at `chat.phasefinal.com` |
| seafile (+ mariadb, memcached) | 9180 | File sync — `seafile.phasefinal.com` |
| searxng | 9996 | Private search metaengine |
| openwebui | 3100 | Chat UI frontend |
| sillytavern | 8100 | Chat UI |
| mailrise | 8025 | SMTP-to-notification gateway |
| rustdesk (hbbs + hbbr) | host-net 21115-21119 | Remote desktop relay — `rustdesk.phasefinal.com`, server **1.1.16** (updated 2026-10-03 from 1.1.14; `:latest`, image 2026-07-20), public key `vVIChMRsCxCqxjg7h1rDrrxqh5zGpS6ghs+814xV6pI=` (from `id_ed25519.pub` in volume `rustdesk_rustdesk_data`). Client install steps: Booth `rustdesk-client-setup` (kept). Private key vaulted `ana-docker/rustdesk/id_ed25519` (2026-10-03). ⚠ The client's "Change ID" is refused by this OSS server by design (a Pro feature). Workaround from the client source (`hbb_common` `Config::load`), untested: stop RustDesk, delete `enc_id` from `RustDesk.toml`, set `id = '<new>'`, restart; the client re-encrypts it into `enc_id` |
| dockge | 5001 | Docker stack management UI |
| beszel | 8090 | Fleet metrics hub (+ local agent); agents on the other hosts report here |
| dozzle (hub as `dozzle-hub`) | 8088 | Fleet log viewer; agents on the other hosts report here |
| restic rest-server | 8000 | Anaheim-side restic endpoint (writes to the NFS mount at `/mnt/backup/restic/repo/ana/`, backed by the Debian file server at `10.250.50.50`); paired with `rest-server-nh3` on the Synology for the NH3 side |
| backrest | 9898 | Fleet-wide restic snapshot viewer / restore UI — points at both rest-servers |
| it-tools | 8780 | Dev utilities |
| hrafn | internal only | Browser-fetch service (real Chromium behind a REST API) for bot-gated sites; consumers reach `http://hrafn:8080` on `traefik-net`. No host port by design. See `stacks/hrafn/` |
| mattermost | — | Stopped; kept around for reference |

Portainer was retired from this host; stack management is now handled via Dockge + Beszel.

## Refresh state

```bash
scripts/refresh-server-info.sh ana-docker
```

Latest snapshot: `system-details.txt` (regenerate as needed).

## Placement rule

If a new stack needs a GPU it goes on **ana-ml2**, otherwise it lands here.