fix(hrafn-ci): staging dir inside the rsync target froze host source silently
Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences).
This commit is contained in:
@@ -48,6 +48,56 @@ then `docker compose build && up`. Two problems, both fixed here.
|
||||
consequence: **the tar list in the workflow is authoritative** — anything
|
||||
omitted from it is deleted from the host, except `.env` and `.deployed`.
|
||||
|
||||
## The frozen-source defect (fixed 2026-08-22)
|
||||
|
||||
The converge fix above had a defect that made every deploy a no-op for
|
||||
source content, while still reporting success. Worth reading before
|
||||
touching this playbook.
|
||||
|
||||
**The bug:** the staging directory was `$compose_dir/.stage` — *inside* the
|
||||
rsync target. `rsync -a --delete $compose_dir/.stage/ $compose_dir/` then
|
||||
deleted `.stage` from the destination (it is not in the source listing)
|
||||
**during** the transfer, destroying the source mid-copy. Reproduced exactly:
|
||||
|
||||
```
|
||||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||||
after: app.py="OLD" (leftover.txt GONE, .stage GONE)
|
||||
```
|
||||
|
||||
Note which half worked. Deletion succeeded, so the directory *looked*
|
||||
converged; the copy silently did not happen. The host source sat frozen at
|
||||
the first manual rsync while `.deployed` and the image tag advanced with
|
||||
every commit — and because the build's `COPY` inputs never changed, Docker
|
||||
full-cache-hit and every SHA tag aliased one image. The provenance the SHA
|
||||
tagging exists to provide was false the whole time.
|
||||
|
||||
**Why nothing caught it:** the verify steps asserted the marker, container
|
||||
health, and a 200 from `/readyz`. All three pass on a frozen host. None of
|
||||
them measured *content*. A deploy that reports success without asserting the
|
||||
bytes changed is verifying an uptime, not a deploy.
|
||||
|
||||
**The fixes:**
|
||||
- Stage outside the target (`/tmp/hrafn-deploy-stage`).
|
||||
- CI computes a `context_sha256` over the shipped file list; the playbook
|
||||
recomputes it on the host after the converge and fails if they differ.
|
||||
End-to-end from the CI checkout to the host filesystem.
|
||||
- Compare the running container's `src/**/*.py` against the host's, catching
|
||||
a SHA tag that names layers the image does not contain.
|
||||
- `clean: true` on the checkout plus an explicit `HEAD == GITHUB_SHA`
|
||||
assertion, so a reused runner workspace fails the job instead of shipping
|
||||
a stale tree.
|
||||
|
||||
**No `--no-cache`.** A cache hit is *correct* when the build context is
|
||||
genuinely unchanged, and rebuilding a Chromium base image every deploy to
|
||||
paper over a bug is the wrong trade. The content assertions prove the
|
||||
property directly instead of brute-forcing it.
|
||||
|
||||
**Gotcha in the check itself:** compare only `*.py`. `pip install .`
|
||||
generates `src/hrafn.egg-info/*` inside the image (6 files the host lacks),
|
||||
and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||||
fails on every healthy deploy. Verified against a known-good container
|
||||
before shipping: 12 host files, 18 in the container, 0 content differences.
|
||||
|
||||
## Validation
|
||||
|
||||
The playbook parses and interpolates clean under elway's own parser:
|
||||
|
||||
Reference in New Issue
Block a user