feat(hrafn): register the browser-fetch service and tighten its env perms
nevermore-claude handed hrafn to infra-ops for uptime ownership. Intake: verified health independently, brought it into the inventory, and fixed one security defect found during the check. - canonicalize stacks/hrafn/ (compose.yaml + .env.example + README) - list hrafn in the ana-docker running-stacks table - README records the load-bearing bits: shm_size 1gb is required for Chromium, playwright and the base image are version-coupled, SSRF denies private targets by default, and restart:unless-stopped does NOT act on healthcheck failure Security fix applied on the host, not in this tree: /opt/docker/compose/ hrafn/.env was 0644 with a live 57-char bearer token in it, readable by every local account (verified by reading it as `nobody`). Tightened to 0600; owner unchanged so the deploy path still works, confirmed via `docker compose config` as lkraven. Container untouched and still healthy. Known gaps recorded rather than silently accepted: no CI deploy, and the image builds from an rsync'd working tree living in the compose directory, so the running image has no commit provenance.
This commit is contained in:
@@ -50,6 +50,7 @@ General-purpose Docker host for the Anaheim colo. Runs everything at `10.250.0.0
|
|||||||
| restic rest-server | 8000 | Anaheim-side restic endpoint (writes to the NFS mount at `/mnt/backup/restic/repo/ana/`, backed by the Debian file server at `10.250.50.50`); paired with `rest-server-nh3` on the Synology for the NH3 side |
|
| restic rest-server | 8000 | Anaheim-side restic endpoint (writes to the NFS mount at `/mnt/backup/restic/repo/ana/`, backed by the Debian file server at `10.250.50.50`); paired with `rest-server-nh3` on the Synology for the NH3 side |
|
||||||
| backrest | 9898 | Fleet-wide restic snapshot viewer / restore UI — points at both rest-servers |
|
| backrest | 9898 | Fleet-wide restic snapshot viewer / restore UI — points at both rest-servers |
|
||||||
| it-tools | 8780 | Dev utilities |
|
| it-tools | 8780 | Dev utilities |
|
||||||
|
| hrafn | internal only | Browser-fetch service (real Chromium behind a REST API) for bot-gated sites; consumers reach `http://hrafn:8080` on `traefik-net`. No host port by design. See `stacks/hrafn/` |
|
||||||
| mattermost | — | Stopped; kept around for reference |
|
| mattermost | — | Stopped; kept around for reference |
|
||||||
|
|
||||||
Portainer was retired from this host; stack management is now handled via Dockge + Beszel.
|
Portainer was retired from this host; stack management is now handled via Dockge + Beszel.
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# hrafn — fleet browser-fetch service. Copy to `.env` and set HRAFN_TOKEN.
|
||||||
|
# This file is the canonical config surface (contract § Configuration).
|
||||||
|
|
||||||
|
# ── auth (required) ──────────────────────────────────────────────────
|
||||||
|
# Shared bearer token required on POST /v1/fetch. Provision via the
|
||||||
|
# vault (secrets-broker); rotation = rotate the vault entry + recycle.
|
||||||
|
HRAFN_TOKEN=CHANGE_ME
|
||||||
|
|
||||||
|
# ── network ──────────────────────────────────────────────────────────
|
||||||
|
HRAFN_BIND=0.0.0.0
|
||||||
|
HRAFN_PORT=8080
|
||||||
|
|
||||||
|
# ── fetch limits ─────────────────────────────────────────────────────
|
||||||
|
# Max post-content-decoding body bytes of the main navigation response;
|
||||||
|
# oversize is rejected (413), never truncated. Both raw and dom modes.
|
||||||
|
HRAFN_MAX_BODY_BYTES=10485760
|
||||||
|
# Whole-request budget default + hard cap (seconds). timeout_s INCLUDES
|
||||||
|
# politeness-queue wait. A request may not exceed the cap.
|
||||||
|
HRAFN_DEFAULT_TIMEOUT_S=30
|
||||||
|
HRAFN_MAX_TIMEOUT_S=120
|
||||||
|
# Default dom-mode wait state: load | domcontentloaded | networkidle.
|
||||||
|
HRAFN_DEFAULT_WAIT_UNTIL=domcontentloaded
|
||||||
|
|
||||||
|
# ── concurrency + browser lifecycle ──────────────────────────────────
|
||||||
|
# Global in-flight cap (acquired on leaving the politeness queue).
|
||||||
|
HRAFN_CONCURRENCY=4
|
||||||
|
# Recycle the browser after N requests or M seconds of age.
|
||||||
|
HRAFN_BROWSER_MAX_REQUESTS=200
|
||||||
|
HRAFN_BROWSER_MAX_AGE_S=1800
|
||||||
|
|
||||||
|
# ── politeness gate (per registrable domain) ─────────────────────────
|
||||||
|
# Minimum seconds between fetch STARTS per domain (0 = no throttle).
|
||||||
|
HRAFN_DOMAIN_MIN_INTERVAL_S=5
|
||||||
|
# Per-domain interval overrides: comma-separated domain=seconds pairs.
|
||||||
|
# HRAFN_DOMAIN_OVERRIDES=reddit.com=10, example.com=2
|
||||||
|
HRAFN_DOMAIN_OVERRIDES=
|
||||||
|
# Waiting-request queue depth per domain (excludes the in-flight fetch;
|
||||||
|
# 0 = no queue). Overflow → 429 queue_full.
|
||||||
|
HRAFN_QUEUE_DEPTH=4
|
||||||
|
# Upstream Retry-After (on 429/503) embargoes the domain; clamped to
|
||||||
|
# this many seconds so a hostile Retry-After can't lock a domain for a day.
|
||||||
|
HRAFN_MAX_EMBARGO_S=300
|
||||||
|
|
||||||
|
# ── SSRF ─────────────────────────────────────────────────────────────
|
||||||
|
# Private/reserved targets are DENIED by default. To permit specific
|
||||||
|
# internal targets, list IPs/CIDRs (comma-separated). Empty = deny all
|
||||||
|
# private ranges (the safe default). Write entries in IPv4 form for IPv4
|
||||||
|
# targets: resolved IPv4-mapped IPv6 addresses are canonicalised to IPv4
|
||||||
|
# before matching, so an IPv6-mapped CIDR (::ffff:10.0.0.0/120) will not
|
||||||
|
# match them.
|
||||||
|
# HRAFN_PRIVATE_ALLOWLIST=10.0.0.0/8, 192.168.1.5
|
||||||
|
HRAFN_PRIVATE_ALLOWLIST=
|
||||||
|
|
||||||
|
# ── live smoke (dev only) ────────────────────────────────────────────
|
||||||
|
# Set to 1 to enable the opt-in Reddit RSS smoke test. Off in CI.
|
||||||
|
# HRAFN_LIVE_SMOKE=1
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# hrafn
|
||||||
|
|
||||||
|
Browser-fetch service — a genuine Chromium kept behind a REST API so consumers
|
||||||
|
never have to embed a browser themselves. It exists because bot-gated sites
|
||||||
|
(Reddit first) 403 every non-browser client; hrafn is the one place in the fleet
|
||||||
|
that keeps a real browser warm.
|
||||||
|
|
||||||
|
**Deploys to:**
|
||||||
|
- **ana-docker** (10.250.50.70) — internal-only, no published host port.
|
||||||
|
Consumers reach it at `http://hrafn:8080` on `traefik-net`.
|
||||||
|
|
||||||
|
Built and handed to infra-ops for uptime ownership by nevermore-claude,
|
||||||
|
2026-08-22. Upstream source: `vh/hrafn` (private gitea repo).
|
||||||
|
|
||||||
|
## Reaching it
|
||||||
|
|
||||||
|
There is **no host port** — this is deliberate (contract § Deployment). Only
|
||||||
|
containers on `traefik-net` can talk to it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from another container on traefik-net
|
||||||
|
curl -H "Authorization: Bearer $HRAFN_TOKEN" \
|
||||||
|
-X POST http://hrafn:8080/v1/fetch -d '{"url":"https://example.com"}'
|
||||||
|
|
||||||
|
# from the host, for diagnostics only
|
||||||
|
ssh infra-ops@10.250.50.70 \
|
||||||
|
'sudo docker run --rm --network traefik-net curlimages/curl:latest \
|
||||||
|
-s -o /dev/null -w "%{http_code}\n" http://hrafn:8080/readyz'
|
||||||
|
```
|
||||||
|
|
||||||
|
`/readyz` and the other health endpoints are unauthenticated; `POST /v1/fetch`
|
||||||
|
requires the bearer token.
|
||||||
|
|
||||||
|
## Credentials
|
||||||
|
|
||||||
|
`HRAFN_TOKEN` lives in `/opt/docker/compose/hrafn/.env` on the host and is
|
||||||
|
vaulted at `ana-docker/hrafn/bearer-token` (secrets-broker). The `.env` is
|
||||||
|
**0600** — it was 0644 on handoff, which exposed a live bearer token to every
|
||||||
|
local account on the box; tightened 2026-08-22. Keep it 0600.
|
||||||
|
|
||||||
|
## Operational notes
|
||||||
|
|
||||||
|
- **`shm_size: 1gb` is load-bearing.** Chromium crashes on some pages with
|
||||||
|
Docker's 64 MB default. Don't trim it to save memory.
|
||||||
|
- **Playwright and the base image are version-coupled.** Base is
|
||||||
|
`mcr.microsoft.com/playwright/python:v1.62.0-noble` with playwright pinned to
|
||||||
|
`1.62.0`. Bump both together or the browser binary and the client library
|
||||||
|
disagree.
|
||||||
|
- **SSRF is denied by default** — private and reserved targets are refused
|
||||||
|
unless listed in `HRAFN_PRIVATE_ALLOWLIST`, which is empty. Leave it empty
|
||||||
|
unless there is a specific internal target and a reason.
|
||||||
|
- **`restart: unless-stopped` does not act on the healthcheck.** Docker restarts
|
||||||
|
on *exit*, not on *unhealthy*. A wedged-but-running Chromium will sit there
|
||||||
|
marked unhealthy indefinitely. Beszel on this host reports container status,
|
||||||
|
so that is the alerting path.
|
||||||
|
|
||||||
|
## Deploy
|
||||||
|
|
||||||
|
**No CI deploy is wired.** As of the handoff the image builds from a working
|
||||||
|
tree rsync'd into the compose directory, so `/opt/docker/compose/hrafn/`
|
||||||
|
contains the full source (`src/`, `tests/`, `pyproject.toml`, …) rather than
|
||||||
|
just a compose file. That is a deviation from the house convention and means
|
||||||
|
the running image has no commit provenance.
|
||||||
|
|
||||||
|
Current update path:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh infra-ops@10.250.50.70
|
||||||
|
cd /opt/docker/compose/hrafn
|
||||||
|
# rsync the updated working tree in, then:
|
||||||
|
sudo docker compose build && sudo docker compose up -d
|
||||||
|
sudo docker compose ps # expect (healthy)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Known gaps
|
||||||
|
|
||||||
|
| gap | impact | status |
|
||||||
|
|---|---|---|
|
||||||
|
| No CI deploy | manual rsync + build; no provenance for what is running | offered by nevermore-claude, not yet built |
|
||||||
|
| Source tree in the compose dir | deviates from `/opt/docker/compose/<stack>/compose.yaml` convention | needs a host-clone or registry-image deploy shape |
|
||||||
|
| No homepage entry | internal-only service, so arguably correct | intentional for now |
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# hrafn — internal fleet service. No published host port; consumers reach
|
||||||
|
# it at http://hrafn:8080 on the shared network (contract § Deployment).
|
||||||
|
services:
|
||||||
|
hrafn:
|
||||||
|
build: .
|
||||||
|
image: local/hrafn:${HRAFN_TAG:-v1}
|
||||||
|
container_name: hrafn
|
||||||
|
restart: unless-stopped
|
||||||
|
env_file: .env
|
||||||
|
networks:
|
||||||
|
- traefik-net
|
||||||
|
# Chromium wants a larger shared-memory segment than Docker's 64MB
|
||||||
|
# default or it crashes on some pages.
|
||||||
|
shm_size: "1gb"
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "python", "-c",
|
||||||
|
"import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/readyz').status==200 else 1)"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 3
|
||||||
|
start_period: 20s
|
||||||
|
|
||||||
|
networks:
|
||||||
|
traefik-net:
|
||||||
|
external: true
|
||||||
Reference in New Issue
Block a user