Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences).
hrafn
Browser-fetch service — a genuine Chromium kept behind a REST API so consumers never have to embed a browser themselves. It exists because bot-gated sites (Reddit first) 403 every non-browser client; hrafn is the one place in the fleet that keeps a real browser warm.
Deploys to:
- ana-docker (10.250.50.70) — internal-only, no published host port.
Consumers reach it at
http://hrafn:8080ontraefik-net.
Built and handed to infra-ops for uptime ownership by nevermore-claude,
2026-08-22. Upstream source: vh/hrafn (private gitea repo).
Reaching it
There is no host port — this is deliberate (contract § Deployment). Only
containers on traefik-net can talk to it:
# from another container on traefik-net
curl -H "Authorization: Bearer $HRAFN_TOKEN" \
-X POST http://hrafn:8080/v1/fetch -d '{"url":"https://example.com"}'
# from the host, for diagnostics only
ssh infra-ops@10.250.50.70 \
'sudo docker run --rm --network traefik-net curlimages/curl:latest \
-s -o /dev/null -w "%{http_code}\n" http://hrafn:8080/readyz'
/readyz and the other health endpoints are unauthenticated; POST /v1/fetch
requires the bearer token.
Credentials
HRAFN_TOKEN lives in /opt/docker/compose/hrafn/.env on the host and is
vaulted at ana-docker/hrafn/bearer-token (secrets-broker). The .env is
0600 — it was 0644 on handoff, which exposed a live bearer token to every
local account on the box; tightened 2026-08-22. Keep it 0600.
Operational notes
shm_size: 1gbis load-bearing. Chromium crashes on some pages with Docker's 64 MB default. Don't trim it to save memory.- Playwright and the base image are version-coupled. Base is
mcr.microsoft.com/playwright/python:v1.62.0-noblewith playwright pinned to1.62.0. Bump both together or the browser binary and the client library disagree. - SSRF is denied by default — private and reserved targets are refused
unless listed in
HRAFN_PRIVATE_ALLOWLIST, which is empty. Leave it empty unless there is a specific internal target and a reason. restart: unless-stoppeddoes not act on the healthcheck. Docker restarts on exit, not on unhealthy. A wedged-but-running Chromium will sit there marked unhealthy indefinitely. Beszel on this host reports container status, so that is the alerting path.
Deploy
No CI deploy is wired. As of the handoff the image builds from a working
tree rsync'd into the compose directory, so /opt/docker/compose/hrafn/
contains the full source (src/, tests/, pyproject.toml, …) rather than
just a compose file. That is a deviation from the house convention and means
the running image has no commit provenance.
Current update path:
ssh infra-ops@10.250.50.70
cd /opt/docker/compose/hrafn
# rsync the updated working tree in, then:
sudo docker compose build && sudo docker compose up -d
sudo docker compose ps # expect (healthy)
Known gaps
| gap | impact | status |
|---|---|---|
| No CI deploy | manual rsync + build; no provenance for what is running | offered by nevermore-claude, not yet built |
| Source tree in the compose dir | deviates from /opt/docker/compose/<stack>/compose.yaml convention |
needs a host-clone or registry-image deploy shape |
| No homepage entry | internal-only service, so arguably correct | intentional for now |