esh-pfi-infrastructure/stacks/hrafn/ci/README.md

# hrafn CI deploy — mirror of what runs in `vh/hrafn`

These two files are hrafn's deploy pipeline. The **canonical copy now lives in
`vh/hrafn`** — as of 2026-08-23 claude-bot holds `write` there, so infra-ops
maintains the pipeline it owns directly instead of routing patches through the
repo holder.

| file here | canonical location in `vh/hrafn` |
|---|---|
| `playbooks-deploy.yaml` | `playbooks/deploy.yaml` |
| `gitea-workflows-deploy.yaml` | `.gitea/workflows/deploy.yaml` |

This copy is a **mirror kept for review and history**, verified byte-identical
to the live files at commit `22e0eb9d75a6`. If you change one, change both —
or check with:

```bash
curl -s "https://gitea.phasefinal.com/api/v1/repos/vh/hrafn/contents/playbooks/deploy.yaml?ref=main" \
  -H "Authorization: token $(cat ~/.config/claude-bot/gitea-token-repo-create)" \
  | python3 -c 'import json,sys,base64,hashlib; print(hashlib.sha256(base64.b64decode(json.load(sys.stdin)["content"])).hexdigest()[:12])'
```

## What the change buys

The pre-CI shape was: rsync a working tree into `/opt/docker/compose/hrafn/`,
then `docker compose build && up`. Two problems, both fixed here.

1. **No provenance.** The image was always `local/hrafn:v1`, so nothing on
   the box could answer "what commit is running". The image is now tagged
   with the commit SHA, and `/opt/docker/compose/hrafn/.deployed` records
   the SHA and timestamp. Rollback becomes a retag.
2. **The whole repo lived in the compose directory** — `tests/`, `docs/`,
   `ROADMAP.md`, `persistent-memory.md`, `CLAUDE.md`. Only the build context
   ships now (`Dockerfile`, `compose.yaml`, `pyproject.toml`, `README.md`,
   `src/`).

## Two design calls worth knowing

- **Tarball, not per-file upload steps.** nevermore's playbook enumerates
  every source file as its own `upload:` step. That is explicit, but it
  fails *open*: add `src/hrafn/newthing.py`, forget the matching step, and
  the deploy silently ships without it. hrafn's build context travels as one
  archive so it cannot go partial.
- **`.env` is never deployed.** It is host-owned, `0600`, and holds the
  bearer token. The playbook *refuses to run* if it is missing or not `0600`
  — a guard added because the file arrived at `0644` on handoff.
- **The deploy converges, it does not accrete.** The first version unpacked
  the tarball in place, which overwrote tracked files but never removed
  anything — so leftovers from the pre-CI hand-rsync (`tests/`, `docs/`,
  `ROADMAP.md`, `persistent-memory.md`, `CLAUDE.md`, `LICENSE`) survived the
  first CI deploy and had to be cleaned off the host by hand. Now the
  tarball unpacks to a staging dir and `rsync --delete` converges the compose
  directory onto it, so a stray file cannot outlive the next deploy. The
  consequence: **the tar list in the workflow is authoritative** — anything
  omitted from it is deleted from the host, except `.env` and `.deployed`.

## The frozen-source defect (fixed 2026-08-22)

The converge fix above had a defect that made every deploy a no-op for
source content, while still reporting success. Worth reading before
touching this playbook.

**The bug:** the staging directory was `$compose_dir/.stage` — *inside* the
rsync target. `rsync -a --delete $compose_dir/.stage/ $compose_dir/` then
deleted `.stage` from the destination (it is not in the source listing)
**during** the transfer, destroying the source mid-copy. Reproduced exactly:

```
before:  app.py="OLD"   leftover.txt   .stage/app.py="NEW"
after:   app.py="OLD"   (leftover.txt GONE, .stage GONE)
```

Note which half worked. Deletion succeeded, so the directory *looked*
converged; the copy silently did not happen. The host source sat frozen at
the first manual rsync while `.deployed` and the image tag advanced with
every commit — and because the build's `COPY` inputs never changed, Docker
full-cache-hit and every SHA tag aliased one image. The provenance the SHA
tagging exists to provide was false the whole time.

**Why nothing caught it:** the verify steps asserted the marker, container
health, and a 200 from `/readyz`. All three pass on a frozen host. None of
them measured *content*. A deploy that reports success without asserting the
bytes changed is verifying an uptime, not a deploy.

**The fixes:**
- Stage outside the target (`/tmp/hrafn-deploy-stage`).
- CI computes a `context_sha256` over the shipped file list; the playbook
  recomputes it on the host after the converge and fails if they differ.
  End-to-end from the CI checkout to the host filesystem.
- Compare the running container's `src/**/*.py` against the host's, catching
  a SHA tag that names layers the image does not contain.
- `clean: true` on the checkout plus an explicit `HEAD == GITHUB_SHA`
  assertion, so a reused runner workspace fails the job instead of shipping
  a stale tree.

**No `--no-cache`.** A cache hit is *correct* when the build context is
genuinely unchanged, and rebuilding a Chromium base image every deploy to
paper over a bug is the wrong trade. The content assertions prove the
property directly instead of brute-forcing it.

**Gotcha in the check itself:** compare only `*.py`. `pip install .`
generates `src/hrafn.egg-info/*` inside the image (6 files the host lacks),
and `__pycache__` appears at runtime, so a naive `find src -type f` compare
fails on every healthy deploy. Verified against a known-good container
before shipping: 12 host files, 18 in the container, 0 content differences.

## Two failed runs getting there (9920, 9921)

Worth reading before adding anything to the workflow's prerequisite step.

The frozen-source fix originally shipped with a `HEAD == GITHUB_SHA`
assertion. It failed twice, both times for reasons the change itself
introduced:

- **run 9920** — `git: command not found` (exit 127). act_runner's job image
  ships no git binary, and `actions/checkout@v4` does not need one.
- **run 9921** — adding `git` made the checkout fail with `server certificate
  verification failed. CAfile: none`. Installing git flips checkout@v4 from
  its **node implementation** — which every working run of this pipeline had
  used — to the **git binary**, and the image has no CA bundle.

So a nice-to-have assertion changed the checkout's code path and broke a
working pipeline. It was removed rather than patched with `ca-certificates`:
it guarded a hypothesis that turned out to be **wrong** (the frozen source was
a self-referential rsync in the playbook, not a stale checkout), and
`clean: true` plus the host-side content hash already cover the real risk.

**Do not add `git` to the prerequisite step.** Run 9922 is green with it out.

## Validation

The playbook parses and interpolates clean under elway's own parser:

```bash
scripts/elway ana-docker --playbook stacks/hrafn/ci/playbooks-deploy.yaml \
  --var hrafn_sha=abc123def456 --dry-run
```

That dry-run is worth running after any edit — it caught a real bug during
authoring, where a comment containing a literal `{{ ... }}` identifier was
picked up by elway's variable substitution and failed the run.

No new Actions secrets are required: `DEPLOY_SSH_KEY` and `MGMT_REPO_TOKEN`
already exist at user scope on `vh` from the nevermore/task-board wiring.