fix(hrafn-ci): staging dir inside the rsync target froze host source silently

Root cause of nevermore-claude's report that v1.0.0 deployed green while the
host kept serving 0.1.0.

The staging dir was $compose_dir/.stage -- INSIDE the rsync target. So
`rsync -a --delete $compose_dir/.stage/ $compose_dir/` deleted .stage from
the destination (absent from the source listing) DURING the transfer,
destroying the source mid-copy. Reproduced exactly:

  before:  app.py="OLD"  leftover.txt  .stage/app.py="NEW"
  after:   app.py="OLD"  leftover.txt GONE, .stage GONE

Deletion worked; the copy silently did not. So the directory looked
converged while host source stayed frozen at the first manual rsync, and
because the build's COPY inputs never changed, Docker full-cache-hit and
every SHA tag aliased one image. The provenance guarantee was false.

Nothing caught it because the verify steps asserted the marker, health, and
a 200 from /readyz -- all of which pass on a frozen host. None measured
content.

Fixes:
- stage at /tmp/hrafn-deploy-stage, outside the target
- CI computes context_sha256 over the shipped file list; the playbook
  recomputes it on the host post-converge and fails on mismatch
- compare the running container's src/**/*.py against the host's, catching
  a SHA tag naming layers the image does not contain
- checkout clean:true + assert HEAD == GITHUB_SHA so a reused runner
  workspace fails the job rather than shipping a stale tree

Declined --no-cache: a cache hit is correct when the context is genuinely
unchanged, and the new assertions prove the property directly rather than
brute-forcing it.

The container-vs-host check compares only *.py -- `pip install .` generates
src/hrafn.egg-info/* inside the image and __pycache__ appears at runtime, so
a naive `find src -type f` compare false-fails on every healthy deploy.
Verified against the live container before shipping (12 host files, 18 in
container, 0 content differences).
This commit is contained in:
vh
2026-08-22 21:57:16 -07:00
parent bb19a96f39
commit b38c369313
3 changed files with 156 additions and 15 deletions
+74 -13
View File
@@ -33,9 +33,15 @@
vars:
compose_dir: /opt/docker/compose/hrafn
# MUST live outside compose_dir — see the unpack step for why staging
# inside the target silently destroyed the source mid-rsync.
stage_dir: /tmp/hrafn-deploy-stage
# Overridden by CI with the triggering commit. `manual` marks a
# hand-run deploy so an un-provenanced image is obvious on the host.
hrafn_sha: manual
# sha256 of the shipped context, computed in the workflow. Empty means
# "skip the content assertion" so a hand-run without it still works.
context_sha256: ""
steps:
# ── preconditions ───────────────────────────────────────────────────
@@ -63,26 +69,32 @@ steps:
dest: "{{ compose_dir }}/.hrafn-context.tgz"
mode: "0600"
- name: Unpack build context into a staging dir
- name: Unpack build context into a staging dir OUTSIDE the target
# {{ stage_dir }} must NOT live under {{ compose_dir }}. The first
# version staged at {{ compose_dir }}/.stage, and rsync --delete then
# deleted .stage from the destination (it is not in the source listing)
# DURING the transfer — destroying the source mid-copy. Net effect:
# strays were removed but NEW SOURCE NEVER LANDED. Reproduced exactly:
# before: app.py="OLD", leftover.txt, .stage/app.py="NEW"
# after: app.py="OLD", leftover.txt GONE, .stage GONE
# The half that worked (deletion) is the half the old verify steps
# could see, so the deploy reported success for weeks while the host
# source stayed frozen at the first manual rsync.
shell: |
rm -rf {{ compose_dir }}/.stage && mkdir -p {{ compose_dir }}/.stage
tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ compose_dir }}/.stage
rm -rf {{ stage_dir }} && mkdir -p {{ stage_dir }}
tar xzf {{ compose_dir }}/.hrafn-context.tgz -C {{ stage_dir }}
rm -f {{ compose_dir }}/.hrafn-context.tgz
- name: Converge the compose dir onto the build context
# The first version of this playbook unpacked in place, which
# ACCRETED: it overwrote tracked files but never removed anything,
# so leftovers from the pre-CI hand-rsync (tests/, docs/, ROADMAP.md,
# persistent-memory.md, CLAUDE.md, LICENSE) survived the first CI
# deploy and had to be cleaned off the host by hand. rsync --delete
# makes the directory CONVERGE on the build context, so a stray file
# — from an old deploy or a debugging session — cannot outlive the
# next deploy. Host-owned state is protected by name.
# Unpacking in place ACCRETED — it overwrote tracked files but never
# removed anything, so leftovers outlived every deploy. rsync --delete
# makes the directory CONVERGE on the build context instead. Host-owned
# state is protected by name.
shell: |
rsync -a --delete \
--exclude '.env' --exclude '.deployed' \
{{ compose_dir }}/.stage/ {{ compose_dir }}/
rm -rf {{ compose_dir }}/.stage
{{ stage_dir }}/ {{ compose_dir }}/
rm -rf {{ stage_dir }}
# ── build + start ───────────────────────────────────────────────────
@@ -127,3 +139,52 @@ verify:
- name: Deployed SHA matches what we just shipped
shell: grep -q "^sha={{ hrafn_sha }}$" {{ compose_dir }}/.deployed
changed_when: "false"
# ── the checks that would have caught the frozen-source defect ──────
# The original verify set asserted the marker, container health, and a
# 200 from /readyz. Every one of those passed for weeks while the host
# source sat frozen at the first manual rsync, because none of them
# measured CONTENT. A deploy that reports success without asserting the
# bytes changed is not verifying a deploy, it is verifying an uptime.
- name: Host tree content matches the context CI shipped
# context_sha256 is computed in the workflow over the same file list
# the tarball carries, so this is an end-to-end assertion from the CI
# checkout to the host filesystem. Empty value skips the check, so a
# hand-run without it still works.
when: "test -n '{{ context_sha256 }}'"
shell: |
cd {{ compose_dir }}
actual=$(find Dockerfile compose.yaml pyproject.toml README.md .env.example src \
-type f | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1)
test "$actual" = "{{ context_sha256 }}" || {
echo "CONTENT MISMATCH — the converge did not land what CI shipped"
echo " expected {{ context_sha256 }}"
echo " on host $actual"
exit 1; }
echo "host tree matches shipped context ($actual)"
changed_when: "false"
- name: Running image was built from the source now on the host
# Catches a SHA-tagged image whose layers predate the source — the build
# cache full-hits when COPY inputs never change, so a tag can name a
# commit the image does not contain. Comparing the container's /app/src
# against the host src proves the image really was built from this tree.
#
# ONLY *.py. A naive `find src -type f` compare fails on every healthy
# deploy: `pip install .` generates src/hrafn.egg-info/* inside the image
# (6 files) that the host tree does not have, and __pycache__ appears at
# runtime. Verified against a known-good container before shipping this —
# 12 host files, 18 in the container, 0 content differences. A check that
# cries wolf on every green deploy is worse than no check.
shell: |
list() { find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum | cut -d' ' -f1; }
h=$(cd {{ compose_dir }} && list)
c=$(docker exec hrafn sh -c "cd /app && find src -type f -name '*.py' | LC_ALL=C sort | xargs sha256sum | sha256sum" | cut -d' ' -f1)
test "$h" = "$c" || {
echo "IMAGE IS STALE — the running container's source differs from the host tree"
echo " host $h"
echo " container $c"
exit 1; }
echo "image source matches host source ($h)"
changed_when: "false"