From b38c36931332d7c3f1acfd2a4de5926a97cf638c Mon Sep 17 00:00:00 2001 From: Vuong Hoang Date: Sat, 22 Aug 2026 21:57:16 -0700 Subject: [PATCH] fix(hrafn-ci): staging dir inside the rsync target froze host source silently Root cause of nevermore-claude's report that v1.0.0 deployed green while the host kept serving 0.1.0. The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So `rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from the destination (absent from the source listing) DURING the transfer, destroying the source mid-copy. Reproduced exactly: before: app.py="OLD" leftover.txt .stage/app.py="NEW" after: app.py="OLD" leftover.txt GONE, .stage GONE Deletion worked; the copy silently did not. So the directory looked converged while host source stayed frozen at the first manual rsync, and because the build's COPY inputs never changed, Docker full-cache-hit and every SHA tag aliased one image. The provenance guarantee was false. Nothing caught it because the verify steps asserted the marker, health, and a 200 from /readyz -- all of which pass on a frozen host. None measured content. Fixes: - stage at /tmp/hrafn-deploy-stage, outside the target - CI computes context_sha256 over the shipped file list; the playbook recomputes it on the host post-converge and fails on mismatch - compare the running container's src/**/*.py against the host's, catching a SHA tag naming layers the image does not contain - checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner workspace fails the job rather than shipping a stale tree Declined --no-cache: a cache hit is correct when the context is genuinely unchanged, and the new assertions prove the property directly rather than brute-forcing it. The container-vs-host check compares only *.py -- `pip install .` generates src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so a naive `find src -type f` compare false-fails on every healthy deploy. Verified against the live container before shipping (12 host files, 18 in container, 0 content differences). --- stacks/hrafn/ci/README.md | 50 ++++++++++++ stacks/hrafn/ci/gitea-workflows-deploy.yaml | 34 +++++++- stacks/hrafn/ci/playbooks-deploy.yaml | 87 ++++++++++++++++++--- 3 files changed, 156 insertions(+), 15 deletions(-) diff --git a/stacks/hrafn/ci/README.md b/stacks/hrafn/ci/README.md index 6cca569..d9ceeee 100644 --- a/stacks/hrafn/ci/README.md +++ b/stacks/hrafn/ci/README.md @@ -48,6 +48,56 @@ then `docker compose build && up`. Two problems, both fixed here. consequence: **the tar list in the workflow is authoritative** — anything omitted from it is deleted from the host, except `.env` and `.deployed`. +## The frozen-source defect (fixed 2026-08-22) + +The converge fix above had a defect that made every deploy a no-op for +source content, while still reporting success. Worth reading before +touching this playbook. + +**The bug:** the staging directory was `$compose_dir/.stage` — *inside* the +rsync target. `rsync -a --delete $compose_dir/.stage/ $compose_dir/` then +deleted `.stage` from the destination (it is not in the source listing) +**during** the transfer, destroying the source mid-copy. Reproduced exactly: + +``` +before: app.py="OLD" leftover.txt .stage/app.py="NEW" +after: app.py="OLD" (leftover.txt GONE, .stage GONE) +``` + +Note which half worked. Deletion succeeded, so the directory *looked* +converged; the copy silently did not happen. The host source sat frozen at +the first manual rsync while `.deployed` and the image tag advanced with +every commit — and because the build's `COPY` inputs never changed, Docker +full-cache-hit and every SHA tag aliased one image. The provenance the SHA +tagging exists to provide was false the whole time. + +**Why nothing caught it:** the verify steps asserted the marker, container +health, and a 200 from `/readyz`. All three pass on a frozen host. None of +them measured *content*. A deploy that reports success without asserting the +bytes changed is verifying an uptime, not a deploy. + +**The fixes:** +- Stage outside the target (`/tmp/hrafn-deploy-stage`). +- CI computes a `context_sha256` over the shipped file list; the playbook + recomputes it on the host after the converge and fails if they differ. + End-to-end from the CI checkout to the host filesystem. +- Compare the running container's `src/**/*.py` against the host's, catching + a SHA tag that names layers the image does not contain. +- `clean: true` on the checkout plus an explicit `HEAD == GITHUB_SHA` + assertion, so a reused runner workspace fails the job instead of shipping + a stale tree. + +**No `--no-cache`.** A cache hit is *correct* when the build context is +genuinely unchanged, and rebuilding a Chromium base image every deploy to +paper over a bug is the wrong trade. The content assertions prove the +property directly instead of brute-forcing it. + +**Gotcha in the check itself:** compare only `*.py`. `pip install .` +generates `src/hrafn.egg-info/*` inside the image (6 files the host lacks), +and `__pycache__` appears at runtime, so a naive `find src -type f` compare +fails on every healthy deploy. Verified against a known-good container +before shipping: 12 host files, 18 in the container, 0 content differences. + ## Validation The playbook parses and interpolates clean under elway's own parser: diff --git a/stacks/hrafn/ci/gitea-workflows-deploy.yaml b/stacks/hrafn/ci/gitea-workflows-deploy.yaml index e7e87cc..03bbdf5 100644 --- a/stacks/hrafn/ci/gitea-workflows-deploy.yaml +++ b/stacks/hrafn/ci/gitea-workflows-deploy.yaml @@ -44,6 +44,22 @@ jobs: - name: Checkout hrafn (triggering repo) uses: actions/checkout@v4 + with: + # Self-hosted runners reuse workspaces. Without a clean checkout a + # stale tree can be tarred while GITHUB_SHA claims the new commit — + # exactly the kind of drift the SHA tagging is supposed to prevent. + clean: true + + - name: Assert the checkout really is the triggering commit + # Cheap, and it converts "the runner shipped stale files" from a + # silent deploy into a failed job. + run: | + head=$(git rev-parse HEAD) + echo "checkout HEAD : $head" + echo "GITHUB_SHA : $GITHUB_SHA" + test "$head" = "$GITHUB_SHA" || { + echo "CHECKOUT DRIFT — refusing to deploy a tree that is not $GITHUB_SHA" + exit 1; } - name: Checkout management repo (for elway) uses: actions/checkout@v4 @@ -83,10 +99,24 @@ jobs: echo "context contents:" tar tzf dist/hrafn-context.tgz + # Content hash over the same file list, in the same order the + # playbook recomputes it on the host. This is what turns "the + # deploy said OK" into "the bytes on the host are the bytes CI + # built" — the assertion whose absence let the host source stay + # frozen at 0.1.0 through every green deploy. + CTX=$(find Dockerfile compose.yaml pyproject.toml README.md .env.example src \ + -type f | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1) + echo "context_sha256=$CTX" | tee -a "$GITHUB_ENV" + echo " version in this context:" + grep -h '__version__' src/hrafn/__init__.py || true + - name: Deploy hrafn (in-repo elway playbook) # hrafn_sha becomes the image tag and is written to - # /opt/docker/compose/hrafn/.deployed on the host. + # /opt/docker/compose/hrafn/.deployed on the host. context_sha256 + # is asserted against the host tree AFTER the converge, so a deploy + # that fails to land new source fails the job instead of passing. run: | _mgmt/scripts/elway ana-docker \ --playbook playbooks/deploy.yaml \ - --var hrafn_sha=${GITHUB_SHA::12} + --var hrafn_sha=${GITHUB_SHA::12} \ + --var context_sha256=${context_sha256} diff --git a/stacks/hrafn/ci/playbooks-deploy.yaml b/stacks/hrafn/ci/playbooks-deploy.yaml index ef9f276..34c4763 100644 --- a/stacks/hrafn/ci/playbooks-deploy.yaml +++ b/stacks/hrafn/ci/playbooks-deploy.yaml @@ -33,9 +33,15 @@ vars: compose_dir: /opt/docker/compose/hrafn + # MUST live outside compose_dir — see the unpack step for why staging + # inside the target silently destroyed the source mid-rsync. + stage_dir: /tmp/hrafn-deploy-stage # Overridden by CI with the triggering commit. `manual` marks a # hand-run deploy so an un-provenanced image is obvious on the host. hrafn_sha: manual + # sha256 of the shipped context, computed in the workflow. Empty means + # "skip the content assertion" so a hand-run without it still works. + context_sha256: "" steps: # ── preconditions ─────────────────────────────────────────────────── @@ -63,26 +69,32 @@ steps: dest: "{{ compose_dir }}/.hrafn-context.tgz" mode: "0600" - - name: Unpack build context into a staging dir + - name: Unpack build context into a staging dir OUTSIDE the target + # {{ stage_dir }} must NOT live under {{ compose_dir }}. The first + # version staged at {{ compose_dir }}/.stage, and rsync --delete then + # deleted .stage from the destination (it is not in the source listing) + # DURING the transfer — destroying the source mid-copy. Net effect: + # strays were removed but NEW SOURCE NEVER LANDED. Reproduced exactly: + # before: app.py="OLD", leftover.txt, .stage/app.py="NEW" + # after: app.py="OLD", leftover.txt GONE, .stage GONE + # The half that worked (deletion) is the half the old verify steps + # could see, so the deploy reported success for weeks while the host + # source stayed frozen at the first manual rsync. shell: | - rm -rf {{ compose_dir }}/.stage && mkdir -p {{ compose_dir }}/.stage - tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ compose_dir }}/.stage + rm -rf {{ stage_dir }} && mkdir -p {{ stage_dir }} + tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ stage_dir }} rm -f {{ compose_dir }}/.hrafn-context.tgz - name: Converge the compose dir onto the build context - # The first version of this playbook unpacked in place, which - # ACCRETED: it overwrote tracked files but never removed anything, - # so leftovers from the pre-CI hand-rsync (tests/, docs/, ROADMAP.md, - # persistent-memory.md, CLAUDE.md, LICENSE) survived the first CI - # deploy and had to be cleaned off the host by hand. rsync --delete - # makes the directory CONVERGE on the build context, so a stray file - # — from an old deploy or a debugging session — cannot outlive the - # next deploy. Host-owned state is protected by name. + # Unpacking in place ACCRETED — it overwrote tracked files but never + # removed anything, so leftovers outlived every deploy. rsync --delete + # makes the directory CONVERGE on the build context instead. Host-owned + # state is protected by name. shell: | rsync -a --delete \ --exclude '.env' --exclude '.deployed' \ - {{ compose_dir }}/.stage/ {{ compose_dir }}/ - rm -rf {{ compose_dir }}/.stage + {{ stage_dir }}/ {{ compose_dir }}/ + rm -rf {{ stage_dir }} # ── build + start ─────────────────────────────────────────────────── @@ -127,3 +139,52 @@ verify: - name: Deployed SHA matches what we just shipped shell: grep -q "^sha={{ hrafn_sha }}$" {{ compose_dir }}/.deployed changed_when: "false" + + # ── the checks that would have caught the frozen-source defect ────── + # The original verify set asserted the marker, container health, and a + # 200 from /readyz. Every one of those passed for weeks while the host + # source sat frozen at the first manual rsync, because none of them + # measured CONTENT. A deploy that reports success without asserting the + # bytes changed is not verifying a deploy, it is verifying an uptime. + + - name: Host tree content matches the context CI shipped + # context_sha256 is computed in the workflow over the same file list + # the tarball carries, so this is an end-to-end assertion from the CI + # checkout to the host filesystem. Empty value skips the check, so a + # hand-run without it still works. + when: "test -n '{{ context_sha256 }}'" + shell: | + cd {{ compose_dir }} + actual=$(find Dockerfile compose.yaml pyproject.toml README.md .env.example src \ + -type f | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1) + test "$actual" = "{{ context_sha256 }}" || { + echo "CONTENT MISMATCH — the converge did not land what CI shipped" + echo " expected {{ context_sha256 }}" + echo " on host $actual" + exit 1; } + echo "host tree matches shipped context ($actual)" + changed_when: "false" + + - name: Running image was built from the source now on the host + # Catches a SHA-tagged image whose layers predate the source — the build + # cache full-hits when COPY inputs never change, so a tag can name a + # commit the image does not contain. Comparing the container's /app/src + # against the host src proves the image really was built from this tree. + # + # ONLY *.py. A naive `find src -type f` compare fails on every healthy + # deploy: `pip install .` generates src/hrafn.egg-info/* inside the image + # (6 files) that the host tree does not have, and __pycache__ appears at + # runtime. Verified against a known-good container before shipping this — + # 12 host files, 18 in the container, 0 content differences. A check that + # cries wolf on every green deploy is worse than no check. + shell: | + list() { find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1; } + h=$(cd {{ compose_dir }} && list) + c=$(docker exec hrafn sh -c "cd /app && find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum" | cut -d' ' -f1) + test "$h" = "$c" || { + echo "IMAGE IS STALE — the running container's source differs from the host tree" + echo " host $h" + echo " container $c" + exit 1; } + echo "image source matches host source ($h)" + changed_when: "false"