bb19a96f39
Two defects the worldtree-pinned case exposed in the guard written an hour ago. 1. The guard compared the container's .Config.Image STRING against the tag being pinned. That string is only the tag the container was CREATED from, which can differ from what it actually runs: worldtree-pinned was created from `:latest` back when that tag pointed at 446e5807, and `:latest` has since moved to b19afd71d7cc. So the guard refused an instance whose pinning was correct and necessary. Now it resolves the target tag to an image ID and compares that against the running image ID -- asserting the thing actually cared about, that this tag names the bytes now running. It also fails closed when no such local tag exists. 2. The sed step reported CHANGED unconditionally, so a re-run on an already-pinned instance claimed work it had not done. Gated behind a `when:` that skips when the line is already correct; a second run on demo now reports "2 ok, 0 changed, 2 skipped / overall: OK". Applied to worldtree-pinned under worldtree-dev authorization. That instance needed a `docker tag` first -- its image was DANGLING (no repo tags, kept alive only by the running container), so the fleet's frozen reference was one `docker rm` from garbage collection. Tagged as :446e5807bf43, then pinned. All three instances now render a SHA with no floating tag anywhere: worldtree -> :ae88a057c0ed worldtree-personal -> :f63529168c13 worldtree-pinned -> :446e5807bf43 Nothing restarted -- pinned still Up 3 months, its start time unchanged.
116 lines
5.5 KiB
YAML
116 lines
5.5 KiB
YAML
# Re-pin a Worldtree instance's WORLDTREE_IMAGE to the SHA it is actually
|
|
# running, closing the stale-`:latest` recreate hazard.
|
|
#
|
|
# THE HAZARD (found 2026-08-22, Worldtree #410): both corviduo-dev instances
|
|
# had `WORLDTREE_IMAGE=…/worldtree:latest` in their .env while running
|
|
# SHA-tagged images from that day. The local `:latest` tag pointed at
|
|
# b19afd71d7cc, built 2026-06-14 — 69 days stale. So ANY `docker compose up`
|
|
# on either instance, by anyone, for any reason, silently DOWNGRADED that
|
|
# service by 69 days. This is the same footgun that caused the 2026-06-15
|
|
# outage; the pin is what disarms it.
|
|
#
|
|
# This is a stopgap. The durable fix is worldtree-dev's deploy workflow
|
|
# stamping the deployed SHA into .env at each deploy (queued repo-side).
|
|
# Until that lands, re-run this after any deploy that moves the image.
|
|
#
|
|
# Usage — one run per instance:
|
|
# scripts/elway corviduo-dev --playbook playbooks/repin-worldtree-image.yaml \
|
|
# --var instance_dir=/opt/worldtree \
|
|
# --var api_container=worldtree-worldtree-api-1 \
|
|
# --var expect_sha=ae88a057c0ed
|
|
#
|
|
# The edit is INERT until the next recreate — it changes what the NEXT
|
|
# `compose up` resolves to, not the running container. That is the intent:
|
|
# make the next recreate safe rather than dangerous.
|
|
#
|
|
# ⚠ THE `worldtree-pinned` INSTANCE NEEDS AN EXTRA STEP FIRST. It runs image
|
|
# sha256:446e5807… which has NO repo tags at all — it is dangling, kept alive
|
|
# only by the running container referencing it. So there is no tag to pin to,
|
|
# and this playbook's guard will (correctly) refuse. Tag it before re-pinning:
|
|
#
|
|
# docker tag sha256:446e5807bf43639be7d285864a7716816880e0f5c0892427e72800f4aa8ffc56 \
|
|
# gitea.phasefinal.com/vh/worldtree:446e5807bf43
|
|
#
|
|
# That is also worth doing on its own merits: an untagged image referenced
|
|
# only by a container is one `docker rm` away from being garbage-collected,
|
|
# and this one is the frozen reference the whole instance exists to provide.
|
|
|
|
vars:
|
|
instance_dir: /opt/worldtree
|
|
api_container: worldtree-worldtree-api-1
|
|
expect_sha: ""
|
|
registry: gitea.phasefinal.com/vh/worldtree
|
|
|
|
steps:
|
|
- name: Refuse to run without an explicit target SHA
|
|
shell: test -n "{{ expect_sha }}"
|
|
changed_when: "false"
|
|
|
|
- name: Confirm the tag we are about to pin resolves to the running image
|
|
# Guards against a deploy landing between reading the SHA and writing it —
|
|
# pinning a SHA that is NOT running would arm the exact hazard we are
|
|
# disarming, just with a different image.
|
|
#
|
|
# Compares IMAGE IDs, not the container's .Config.Image string. The string
|
|
# is only the tag the container was CREATED from, which can differ from
|
|
# what it actually runs: worldtree-pinned was created from `:latest` back
|
|
# when that tag pointed at 446e5807, and `:latest` has since moved. A
|
|
# string compare rejects that instance even though pinning it is correct;
|
|
# an ID compare asserts the thing we actually care about — that this tag
|
|
# names the bytes currently running.
|
|
shell: |
|
|
running=$(docker inspect {{ api_container }} --format '{{.Image}}')
|
|
tagged=$(docker image inspect {{ registry }}:{{ expect_sha }} --format '{{.Id}}' 2>/dev/null) || {
|
|
echo "REFUSING: no local image tagged {{ registry }}:{{ expect_sha }} — tag it first"
|
|
exit 1; }
|
|
test "$running" = "$tagged" || {
|
|
echo "REFUSING: {{ api_container }} runs $running but {{ registry }}:{{ expect_sha }} is $tagged"
|
|
exit 1; }
|
|
echo "confirmed: {{ registry }}:{{ expect_sha }} == running image $running"
|
|
changed_when: "false"
|
|
|
|
- name: Back up .env
|
|
sudo: true
|
|
shell: cp -n {{ instance_dir }}/.env {{ instance_dir }}/.env.bak-pre-repin-{{ expect_sha }}
|
|
creates: "{{ instance_dir }}/.env.bak-pre-repin-{{ expect_sha }}"
|
|
|
|
- name: Re-pin WORLDTREE_IMAGE to the running SHA
|
|
sudo: true
|
|
# Skipped when already correct, so a re-run reports OK rather than a
|
|
# phantom CHANGED — a playbook that always claims to have changed
|
|
# something trains you to stop reading the summary.
|
|
when: "! sudo grep -q '^WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}$' {{ instance_dir }}/.env"
|
|
# `|` delimiter because the image reference contains slashes.
|
|
shell: |
|
|
grep -q '^WORLDTREE_IMAGE=' {{ instance_dir }}/.env || {
|
|
echo "REFUSING: no WORLDTREE_IMAGE line to replace"; exit 1; }
|
|
sed -i 's|^WORLDTREE_IMAGE=.*|WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}|' {{ instance_dir }}/.env
|
|
chown deploy:deploy {{ instance_dir }}/.env
|
|
chmod 600 {{ instance_dir }}/.env
|
|
changed_when: "true"
|
|
|
|
verify:
|
|
- name: .env now names the SHA, not a floating tag
|
|
sudo: true
|
|
shell: grep -q '^WORLDTREE_IMAGE={{ registry }}:{{ expect_sha }}$' {{ instance_dir }}/.env
|
|
changed_when: "false"
|
|
|
|
- name: compose resolves every service to the pinned SHA (no ':latest' anywhere)
|
|
# The assertion that matters. Reading the .env proves the line changed;
|
|
# only rendering the compose file proves what a recreate would actually
|
|
# pull.
|
|
sudo: true
|
|
shell: |
|
|
cd {{ instance_dir }}
|
|
if docker compose config 2>/dev/null | grep -E '^\s+image:' | grep -q ':latest'; then
|
|
echo "STILL RESOLVING TO :latest"
|
|
docker compose config 2>/dev/null | grep -E '^\s+image:'
|
|
exit 1
|
|
fi
|
|
docker compose config 2>/dev/null | grep -E '^\s+image:' | sort -u
|
|
changed_when: "false"
|
|
|
|
- name: Running containers untouched (this edit must not restart anything)
|
|
shell: docker inspect {{ api_container }} --format '{{.State.Status}} since {{.State.StartedAt}}'
|
|
changed_when: "false"
|