Files
esh-pfi-infrastructure/playbooks/repin-worldtree-image.yaml
T
vh bb19a96f39 fix(playbooks): compare image IDs, not the created-from tag; make the repin idempotent
Two defects the worldtree-pinned case exposed in the guard written an hour
ago.

1. The guard compared the container's .Config.Image STRING against the tag
   being pinned. That string is only the tag the container was CREATED from,
   which can differ from what it actually runs: worldtree-pinned was created
   from `:latest` back when that tag pointed at 446e5807, and `:latest` has
   since moved to b19afd71d7cc. So the guard refused an instance whose
   pinning was correct and necessary. Now it resolves the target tag to an
   image ID and compares that against the running image ID -- asserting the
   thing actually cared about, that this tag names the bytes now running.
   It also fails closed when no such local tag exists.

2. The sed step reported CHANGED unconditionally, so a re-run on an
   already-pinned instance claimed work it had not done. Gated behind a
   `when:` that skips when the line is already correct; a second run on demo
   now reports "2 ok, 0 changed, 2 skipped / overall: OK".

Applied to worldtree-pinned under worldtree-dev authorization. That instance
needed a `docker tag` first -- its image was DANGLING (no repo tags, kept
alive only by the running container), so the fleet's frozen reference was one
`docker rm` from garbage collection. Tagged as :446e5807bf43, then pinned.

All three instances now render a SHA with no floating tag anywhere:
  worldtree          -> :ae88a057c0ed
  worldtree-personal -> :f63529168c13
  worldtree-pinned   -> :446e5807bf43
Nothing restarted -- pinned still Up 3 months, its start time unchanged.
2026-08-22 21:52:20 -07:00

116 lines
5.5 KiB
YAML

# Re-pin a Worldtree instance's WORLDTREE_IMAGE to the SHA it is actually
# running, closing the stale-`:latest` recreate hazard.
#
# THE HAZARD (found 2026-08-22, Worldtree #410): both corviduo-dev instances
# had `WORLDTREE_IMAGE=…/worldtree:latest` in their .env while running
# SHA-tagged images from that day. The local `:latest` tag pointed at
# b19afd71d7cc, built 2026-06-14 — 69 days stale. So ANY `docker compose up`
# on either instance, by anyone, for any reason, silently DOWNGRADED that
# service by 69 days. This is the same footgun that caused the 2026-06-15
# outage; the pin is what disarms it.
#
# This is a stopgap. The durable fix is worldtree-dev's deploy workflow
# stamping the deployed SHA into .env at each deploy (queued repo-side).
# Until that lands, re-run this after any deploy that moves the image.
#
# Usage — one run per instance:
# scripts/elway corviduo-dev --playbook playbooks/repin-worldtree-image.yaml \
# --var instance_dir=/opt/worldtree \
# --var api_container=worldtree-worldtree-api-1 \
# --var expect_sha=ae88a057c0ed
#
# The edit is INERT until the next recreate — it changes what the NEXT
# `compose up` resolves to, not the running container. That is the intent:
# make the next recreate safe rather than dangerous.
#
# ⚠ THE `worldtree-pinned` INSTANCE NEEDS AN EXTRA STEP FIRST. It runs image
# sha256:446e5807… which has NO repo tags at all — it is dangling, kept alive
# only by the running container referencing it. So there is no tag to pin to,
# and this playbook's guard will (correctly) refuse. Tag it before re-pinning:
#
# docker tag sha256:446e5807bf43639be7d285864a7716816880e0f5c0892427e72800f4aa8ffc56 \
# gitea.phasefinal.com/vh/worldtree:446e5807bf43
#
# That is also worth doing on its own merits: an untagged image referenced
# only by a container is one `docker rm` away from being garbage-collected,
# and this one is the frozen reference the whole instance exists to provide.
vars:
instance_dir: /opt/worldtree
api_container: worldtree-worldtree-api-1
expect_sha: ""
registry: gitea.phasefinal.com/vh/worldtree
steps:
- name: Refuse to run without an explicit target SHA
shell: test -n "{{ expect_sha }}"
changed_when: "false"
- name: Confirm the tag we are about to pin resolves to the running image
# Guards against a deploy landing between reading the SHA and writing it —
# pinning a SHA that is NOT running would arm the exact hazard we are
# disarming, just with a different image.
#
# Compares IMAGE IDs, not the container's .Config.Image string. The string
# is only the tag the container was CREATED from, which can differ from
# what it actually runs: worldtree-pinned was created from `:latest` back
# when that tag pointed at 446e5807, and `:latest` has since moved. A
# string compare rejects that instance even though pinning it is correct;
# an ID compare asserts the thing we actually care about — that this tag
# names the bytes currently running.
shell: |
running=$(docker inspect {{ api_container }} --format '{{.Image}}')
tagged=$(docker image inspect {{ registry }}:{{ expect_sha }} --format '{{.Id}}' 2>/dev/null) || {
echo "REFUSING: no local image tagged {{ registry }}:{{ expect_sha }} — tag it first"
exit 1; }
test "$running" = "$tagged" || {
echo "REFUSING: {{ api_container }} runs $running but {{ registry }}:{{ expect_sha }} is $tagged"
exit 1; }
echo "confirmed: {{ registry }}:{{ expect_sha }} == running image $running"
changed_when: "false"
- name: Back up .env
sudo: true
shell: cp -n {{ instance_dir }}/.env {{ instance_dir }}/.env.bak-pre-repin-{{ expect_sha }}
creates: "{{ instance_dir }}/.env.bak-pre-repin-{{ expect_sha }}"
- name: Re-pin WORLDTREE_IMAGE to the running SHA
sudo: true
# Skipped when already correct, so a re-run reports OK rather than a
# phantom CHANGED — a playbook that always claims to have changed
# something trains you to stop reading the summary.
when: "! sudo grep -q '^WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}$' {{ instance_dir }}/.env"
# `|` delimiter because the image reference contains slashes.
shell: |
grep -q '^WORLDTREE_IMAGE=' {{ instance_dir }}/.env || {
echo "REFUSING: no WORLDTREE_IMAGE line to replace"; exit 1; }
sed -i 's|^WORLDTREE_IMAGE=.*|WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}|' {{ instance_dir }}/.env
chown deploy:deploy {{ instance_dir }}/.env
chmod 600 {{ instance_dir }}/.env
changed_when: "true"
verify:
- name: .env now names the SHA, not a floating tag
sudo: true
shell: grep -q '^WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}$' {{ instance_dir }}/.env
changed_when: "false"
- name: compose resolves every service to the pinned SHA (no ':latest' anywhere)
# The assertion that matters. Reading the .env proves the line changed;
# only rendering the compose file proves what a recreate would actually
# pull.
sudo: true
shell: |
cd {{ instance_dir }}
if docker compose config 2>/dev/null | grep -E '^\s+image:' | grep -q ':latest'; then
echo "STILL RESOLVING TO :latest"
docker compose config 2>/dev/null | grep -E '^\s+image:'
exit 1
fi
docker compose config 2>/dev/null | grep -E '^\s+image:' | sort -u
changed_when: "false"
- name: Running containers untouched (this edit must not restart anything)
shell: docker inspect {{ api_container }} --format '{{.State.Status}} since {{.State.StartedAt}}'
changed_when: "false"