fix(hrafn-ci): staging dir inside the rsync target froze host source silently

Root cause of nevermore-claude's report that v1.0.0 deployed green while the
host kept serving 0.1.0.

The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from
the destination (absent from the source listing) DURING the transfer,
destroying the source mid-copy. Reproduced exactly:

  before:  app.py="OLD"  leftover.txt  .stage/app.py="NEW"
  after:   app.py="OLD"  leftover.txt GONE, .stage GONE

Deletion worked; the copy silently did not. So the directory looked
converged while host source stayed frozen at the first manual rsync, and
because the build's COPY inputs never changed, Docker full-cache-hit and
every SHA tag aliased one image. The provenance guarantee was false.

Nothing caught it because the verify steps asserted the marker, health, and
a 200 from /readyz -- all of which pass on a frozen host. None measured
content.

Fixes:
- stage at /tmp/hrafn-deploy-stage, outside the target
- CI computes context_sha256 over the shipped file list; the playbook
  recomputes it on the host post-converge and fails on mismatch
- compare the running container's src/**/*.py against the host's, catching
  a SHA tag naming layers the image does not contain
- checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner
  workspace fails the job rather than shipping a stale tree

Declined --no-cache: a cache hit is correct when the context is genuinely
unchanged, and the new assertions prove the property directly rather than
brute-forcing it.

The container-vs-host check compares only *.py -- `pip install .` generates
src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so
a naive `find src -type f` compare false-fails on every healthy deploy.
Verified against the live container before shipping (12 host files, 18 in
container, 0 content differences).
This commit is contained in:
vh
2026-08-22 21:57:16 -07:00
parent bb19a96f39
commit b38c369313
3 changed files with 156 additions and 15 deletions
+50
View File
@@ -48,6 +48,56 @@ then `docker compose build && up`. Two problems, both fixed here.
consequence: **the tar list in the workflow is authoritative** — anything
omitted from it is deleted from the host, except `.env` and `.deployed`.
## The frozen-source defect (fixed 2026-08-22)
The converge fix above had a defect that made every deploy a no-op for
source content, while still reporting success. Worth reading before
touching this playbook.
**The bug:** the staging directory was `$compose_dir/.stage` — *inside* the
rsync target. `rsync -a --delete $compose_dir/.stage/ $compose_dir/` then
deleted `.stage` from the destination (it is not in the source listing)
**during** the transfer, destroying the source mid-copy. Reproduced exactly:
```
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
after: app.py="OLD" (leftover.txt GONE, .stage GONE)
```
Note which half worked. Deletion succeeded, so the directory *looked*
converged; the copy silently did not happen. The host source sat frozen at
the first manual rsync while `.deployed` and the image tag advanced with
every commit — and because the build's `COPY` inputs never changed, Docker
full-cache-hit and every SHA tag aliased one image. The provenance the SHA
tagging exists to provide was false the whole time.
**Why nothing caught it:** the verify steps asserted the marker, container
health, and a 200 from `/readyz`. All three pass on a frozen host. None of
them measured *content*. A deploy that reports success without asserting the
bytes changed is verifying an uptime, not a deploy.
**The fixes:**
- Stage outside the target (`/tmp/hrafn-deploy-stage`).
- CI computes a `context_sha256` over the shipped file list; the playbook
recomputes it on the host after the converge and fails if they differ.
End-to-end from the CI checkout to the host filesystem.
- Compare the running container's `src/**/*.py` against the host's, catching
a SHA tag that names layers the image does not contain.
- `clean: true` on the checkout plus an explicit `HEAD == GITHUB_SHA`
assertion, so a reused runner workspace fails the job instead of shipping
a stale tree.
**No `--no-cache`.** A cache hit is *correct* when the build context is
genuinely unchanged, and rebuilding a Chromium base image every deploy to
paper over a bug is the wrong trade. The content assertions prove the
property directly instead of brute-forcing it.
**Gotcha in the check itself:** compare only `*.py`. `pip install .`
generates `src/hrafn.egg-info/*` inside the image (6 files the host lacks),
and `__pycache__` appears at runtime, so a naive `find src -type f` compare
fails on every healthy deploy. Verified against a known-good container
before shipping: 12 host files, 18 in the container, 0 content differences.
## Validation
The playbook parses and interpolates clean under elway's own parser: