fix(hrafn-ci): staging dir inside the rsync target froze host source silently
Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences).
This commit is contained in:
@@ -48,6 +48,56 @@ then `docker compose build && up`. Two problems, both fixed here.
|
||||
consequence: **the tar list in the workflow is authoritative** — anything
|
||||
omitted from it is deleted from the host, except `.env` and `.deployed`.
|
||||
|
||||
## The frozen-source defect (fixed 2026-08-22)
|
||||
|
||||
The converge fix above had a defect that made every deploy a no-op for
|
||||
source content, while still reporting success. Worth reading before
|
||||
touching this playbook.
|
||||
|
||||
**The bug:** the staging directory was `$compose_dir/.stage` — *inside* the
|
||||
rsync target. `rsync -a --delete $compose_dir/.stage/ $compose_dir/` then
|
||||
deleted `.stage` from the destination (it is not in the source listing)
|
||||
**during** the transfer, destroying the source mid-copy. Reproduced exactly:
|
||||
|
||||
```
|
||||
before: app.py="OLD" leftover.txt .stage/app.py="NEW"
|
||||
after: app.py="OLD" (leftover.txt GONE, .stage GONE)
|
||||
```
|
||||
|
||||
Note which half worked. Deletion succeeded, so the directory *looked*
|
||||
converged; the copy silently did not happen. The host source sat frozen at
|
||||
the first manual rsync while `.deployed` and the image tag advanced with
|
||||
every commit — and because the build's `COPY` inputs never changed, Docker
|
||||
full-cache-hit and every SHA tag aliased one image. The provenance the SHA
|
||||
tagging exists to provide was false the whole time.
|
||||
|
||||
**Why nothing caught it:** the verify steps asserted the marker, container
|
||||
health, and a 200 from `/readyz`. All three pass on a frozen host. None of
|
||||
them measured *content*. A deploy that reports success without asserting the
|
||||
bytes changed is verifying an uptime, not a deploy.
|
||||
|
||||
**The fixes:**
|
||||
- Stage outside the target (`/tmp/hrafn-deploy-stage`).
|
||||
- CI computes a `context_sha256` over the shipped file list; the playbook
|
||||
recomputes it on the host after the converge and fails if they differ.
|
||||
End-to-end from the CI checkout to the host filesystem.
|
||||
- Compare the running container's `src/**/*.py` against the host's, catching
|
||||
a SHA tag that names layers the image does not contain.
|
||||
- `clean: true` on the checkout plus an explicit `HEAD == GITHUB_SHA`
|
||||
assertion, so a reused runner workspace fails the job instead of shipping
|
||||
a stale tree.
|
||||
|
||||
**No `--no-cache`.** A cache hit is *correct* when the build context is
|
||||
genuinely unchanged, and rebuilding a Chromium base image every deploy to
|
||||
paper over a bug is the wrong trade. The content assertions prove the
|
||||
property directly instead of brute-forcing it.
|
||||
|
||||
**Gotcha in the check itself:** compare only `*.py`. `pip install .`
|
||||
generates `src/hrafn.egg-info/*` inside the image (6 files the host lacks),
|
||||
and `__pycache__` appears at runtime, so a naive `find src -type f` compare
|
||||
fails on every healthy deploy. Verified against a known-good container
|
||||
before shipping: 12 host files, 18 in the container, 0 content differences.
|
||||
|
||||
## Validation
|
||||
|
||||
The playbook parses and interpolates clean under elway's own parser:
|
||||
|
||||
@@ -44,6 +44,22 @@ jobs:
|
||||
|
||||
- name: Checkout hrafn (triggering repo)
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
# Self-hosted runners reuse workspaces. Without a clean checkout a
|
||||
# stale tree can be tarred while GITHUB_SHA claims the new commit —
|
||||
# exactly the kind of drift the SHA tagging is supposed to prevent.
|
||||
clean: true
|
||||
|
||||
- name: Assert the checkout really is the triggering commit
|
||||
# Cheap, and it converts "the runner shipped stale files" from a
|
||||
# silent deploy into a failed job.
|
||||
run: |
|
||||
head=$(git rev-parse HEAD)
|
||||
echo "checkout HEAD : $head"
|
||||
echo "GITHUB_SHA : $GITHUB_SHA"
|
||||
test "$head" = "$GITHUB_SHA" || {
|
||||
echo "CHECKOUT DRIFT — refusing to deploy a tree that is not $GITHUB_SHA"
|
||||
exit 1; }
|
||||
|
||||
- name: Checkout management repo (for elway)
|
||||
uses: actions/checkout@v4
|
||||
@@ -83,10 +99,24 @@ jobs:
|
||||
echo "context contents:"
|
||||
tar tzf dist/hrafn-context.tgz
|
||||
|
||||
# Content hash over the same file list, in the same order the
|
||||
# playbook recomputes it on the host. This is what turns "the
|
||||
# deploy said OK" into "the bytes on the host are the bytes CI
|
||||
# built" — the assertion whose absence let the host source stay
|
||||
# frozen at 0.1.0 through every green deploy.
|
||||
CTX=$(find Dockerfile compose.yaml pyproject.toml README.md .env.example src \
|
||||
-type f | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1)
|
||||
echo "context_sha256=$CTX" | tee -a "$GITHUB_ENV"
|
||||
echo " version in this context:"
|
||||
grep -h '__version__' src/hrafn/__init__.py || true
|
||||
|
||||
- name: Deploy hrafn (in-repo elway playbook)
|
||||
# hrafn_sha becomes the image tag and is written to
|
||||
# /opt/docker/compose/hrafn/.deployed on the host.
|
||||
# /opt/docker/compose/hrafn/.deployed on the host. context_sha256
|
||||
# is asserted against the host tree AFTER the converge, so a deploy
|
||||
# that fails to land new source fails the job instead of passing.
|
||||
run: |
|
||||
_mgmt/scripts/elway ana-docker \
|
||||
--playbook playbooks/deploy.yaml \
|
||||
--var hrafn_sha=${GITHUB_SHA::12}
|
||||
--var hrafn_sha=${GITHUB_SHA::12} \
|
||||
--var context_sha256=${context_sha256}
|
||||
|
||||
@@ -33,9 +33,15 @@
|
||||
|
||||
vars:
|
||||
compose_dir: /opt/docker/compose/hrafn
|
||||
# MUST live outside compose_dir — see the unpack step for why staging
|
||||
# inside the target silently destroyed the source mid-rsync.
|
||||
stage_dir: /tmp/hrafn-deploy-stage
|
||||
# Overridden by CI with the triggering commit. `manual` marks a
|
||||
# hand-run deploy so an un-provenanced image is obvious on the host.
|
||||
hrafn_sha: manual
|
||||
# sha256 of the shipped context, computed in the workflow. Empty means
|
||||
# "skip the content assertion" so a hand-run without it still works.
|
||||
context_sha256: ""
|
||||
|
||||
steps:
|
||||
# ── preconditions ───────────────────────────────────────────────────
|
||||
@@ -63,26 +69,32 @@ steps:
|
||||
dest: "{{ compose_dir }}/.hrafn-context.tgz"
|
||||
mode: "0600"
|
||||
|
||||
- name: Unpack build context into a staging dir
|
||||
- name: Unpack build context into a staging dir OUTSIDE the target
|
||||
# {{ stage_dir }} must NOT live under {{ compose_dir }}. The first
|
||||
# version staged at {{ compose_dir }}/.stage, and rsync --delete then
|
||||
# deleted .stage from the destination (it is not in the source listing)
|
||||
# DURING the transfer — destroying the source mid-copy. Net effect:
|
||||
# strays were removed but NEW SOURCE NEVER LANDED. Reproduced exactly:
|
||||
# before: app.py="OLD", leftover.txt, .stage/app.py="NEW"
|
||||
# after: app.py="OLD", leftover.txt GONE, .stage GONE
|
||||
# The half that worked (deletion) is the half the old verify steps
|
||||
# could see, so the deploy reported success for weeks while the host
|
||||
# source stayed frozen at the first manual rsync.
|
||||
shell: |
|
||||
rm -rf {{ compose_dir }}/.stage && mkdir -p {{ compose_dir }}/.stage
|
||||
tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ compose_dir }}/.stage
|
||||
rm -rf {{ stage_dir }} && mkdir -p {{ stage_dir }}
|
||||
tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ stage_dir }}
|
||||
rm -f {{ compose_dir }}/.hrafn-context.tgz
|
||||
|
||||
- name: Converge the compose dir onto the build context
|
||||
# The first version of this playbook unpacked in place, which
|
||||
# ACCRETED: it overwrote tracked files but never removed anything,
|
||||
# so leftovers from the pre-CI hand-rsync (tests/, docs/, ROADMAP.md,
|
||||
# persistent-memory.md, CLAUDE.md, LICENSE) survived the first CI
|
||||
# deploy and had to be cleaned off the host by hand. rsync --delete
|
||||
# makes the directory CONVERGE on the build context, so a stray file
|
||||
# — from an old deploy or a debugging session — cannot outlive the
|
||||
# next deploy. Host-owned state is protected by name.
|
||||
# Unpacking in place ACCRETED — it overwrote tracked files but never
|
||||
# removed anything, so leftovers outlived every deploy. rsync --delete
|
||||
# makes the directory CONVERGE on the build context instead. Host-owned
|
||||
# state is protected by name.
|
||||
shell: |
|
||||
rsync -a --delete \
|
||||
--exclude '.env' --exclude '.deployed' \
|
||||
{{ compose_dir }}/.stage/ {{ compose_dir }}/
|
||||
rm -rf {{ compose_dir }}/.stage
|
||||
{{ stage_dir }}/ {{ compose_dir }}/
|
||||
rm -rf {{ stage_dir }}
|
||||
|
||||
# ── build + start ───────────────────────────────────────────────────
|
||||
|
||||
@@ -127,3 +139,52 @@ verify:
|
||||
- name: Deployed SHA matches what we just shipped
|
||||
shell: grep -q "^sha={{ hrafn_sha }}$" {{ compose_dir }}/.deployed
|
||||
changed_when: "false"
|
||||
|
||||
# ── the checks that would have caught the frozen-source defect ──────
|
||||
# The original verify set asserted the marker, container health, and a
|
||||
# 200 from /readyz. Every one of those passed for weeks while the host
|
||||
# source sat frozen at the first manual rsync, because none of them
|
||||
# measured CONTENT. A deploy that reports success without asserting the
|
||||
# bytes changed is not verifying a deploy, it is verifying an uptime.
|
||||
|
||||
- name: Host tree content matches the context CI shipped
|
||||
# context_sha256 is computed in the workflow over the same file list
|
||||
# the tarball carries, so this is an end-to-end assertion from the CI
|
||||
# checkout to the host filesystem. Empty value skips the check, so a
|
||||
# hand-run without it still works.
|
||||
when: "test -n '{{ context_sha256 }}'"
|
||||
shell: |
|
||||
cd {{ compose_dir }}
|
||||
actual=$(find Dockerfile compose.yaml pyproject.toml README.md .env.example src \
|
||||
-type f | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1)
|
||||
test "$actual" = "{{ context_sha256 }}" || {
|
||||
echo "CONTENT MISMATCH — the converge did not land what CI shipped"
|
||||
echo " expected {{ context_sha256 }}"
|
||||
echo " on host $actual"
|
||||
exit 1; }
|
||||
echo "host tree matches shipped context ($actual)"
|
||||
changed_when: "false"
|
||||
|
||||
- name: Running image was built from the source now on the host
|
||||
# Catches a SHA-tagged image whose layers predate the source — the build
|
||||
# cache full-hits when COPY inputs never change, so a tag can name a
|
||||
# commit the image does not contain. Comparing the container's /app/src
|
||||
# against the host src proves the image really was built from this tree.
|
||||
#
|
||||
# ONLY *.py. A naive `find src -type f` compare fails on every healthy
|
||||
# deploy: `pip install .` generates src/hrafn.egg-info/* inside the image
|
||||
# (6 files) that the host tree does not have, and __pycache__ appears at
|
||||
# runtime. Verified against a known-good container before shipping this —
|
||||
# 12 host files, 18 in the container, 0 content differences. A check that
|
||||
# cries wolf on every green deploy is worse than no check.
|
||||
shell: |
|
||||
list() { find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1; }
|
||||
h=$(cd {{ compose_dir }} && list)
|
||||
c=$(docker exec hrafn sh -c "cd /app && find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum" | cut -d' ' -f1)
|
||||
test "$h" = "$c" || {
|
||||
echo "IMAGE IS STALE — the running container's source differs from the host tree"
|
||||
echo " host $h"
|
||||
echo " container $c"
|
||||
exit 1; }
|
||||
echo "image source matches host source ($h)"
|
||||
changed_when: "false"
|
||||
|
||||
Reference in New Issue
Block a user