vh/esh-pfi-infrastructure
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

CD-driven version-tag mirror + per-release cleanup (supersedes #8) #9

New Issue
Open
opened 2026-05-23 18:03:48 -07:00 by vh · 0 comments
vh commented 2026-05-23 18:03:48 -07:00
Owner
Copy Link
Copy Source

Motivation

Two coupled problems this addresses:

  1. Disk pressure from CI image churn: each commit to vh/skaldsong, vh/Worldtree, etc. pushes a new SHA-pinned image; they accumulate indefinitely. Snapshot 2026-05-23: ana-docker at 89% disk, 96 GB of reclaimable gitea registry blobs. Manually pruned (~113 GB recovered) but the pattern recurs.
  2. Loss of release-grade durability: operator wants every versioned commit (semver-tagged) preserved indefinitely, but today every push gets ONLY a :<40-char-sha> tag — there are zero :v0.27.0-style tags in the registry. The registry cannot distinguish "release worth preserving" from "ephemeral SHA churn." Issue #8 proposed gitea-side cleanup rules with a keep-pattern for ^v[0-9]+\.[0-9]+\.[0-9]+$, but the inputs that pattern matches don't exist yet.

Why supersede #8

Issue #8 attacked symptom 1 (disk pressure) with per-owner cleanup rules and assumed version tags would be available as the protected set. With versioned tags absent from the registry, those rules can't do their intended job — they'd protect latest and nothing else, deleting everything between releases. We need to fix the upstream tagging problem AND introduce offsite durability for releases. The CD-driven model below does both atomically, replacing the gitea-side rules entirely.

Proposed shape: CD does the work on every release

When CI builds a commit that has a semver git tag, the same pipeline:

  1. Multi-tag the build: push :<sha> + :v<version> + :latest in one buildx invocation.
  2. Mirror the versioned tag offsite: crane copy gitea.phasefinal.com/vh/<repo>:v<x> $MIRROR/<repo>:v<x>.
  3. Prune the gitea registry of non-protected tags: keep :v*, :latest, and the current :<sha>; delete the rest via gitea's DELETE /api/v1/packages/{owner}/container/{name}/{version} API.

Non-versioned pushes (regular main commits) still publish :<sha> for CD's pull-and-recreate, but skip steps 2-3. Accumulation between releases is bounded by release cadence.

Workflow sketch (each consumer repo's .gitea/workflows/deploy.yaml):

- name: Push image with version + latest + sha tags
  if: startsWith(github.ref, 'refs/tags/v')
  run: |
    VER=${GITHUB_REF#refs/tags/}
    docker buildx build \
      --tag $REG/$REPO:${{ github.sha }} \
      --tag $REG/$REPO:$VER \
      --tag $REG/$REPO:latest \
      --push .

- name: Mirror version tag to offsite registry
  if: startsWith(github.ref, 'refs/tags/v')
  run: |
    VER=${GITHUB_REF#refs/tags/}
    crane copy $REG/$REPO:$VER $MIRROR_REG/$REPO:$VER

- name: Prune old SHA-only tags from gitea registry
  if: startsWith(github.ref, 'refs/tags/v')
  run: |
    VER=${GITHUB_REF#refs/tags/}
    KEEP="${VER}|latest|${{ github.sha }}|^v[0-9]+\\.[0-9]+\\.[0-9]+$"
    curl -sf -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \
      "$REG_API/packages/$OWNER?type=container&q=$REPO" \
      | jq -r '.[] | .version' \
      | grep -vE "$KEEP" \
      | xargs -I{} curl -X DELETE -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \
          "$REG_API/packages/$OWNER/container/$REPO/{}"

Sketch — real version needs pagination, retries, mirror-down handling.

DRY: reusable workflow

To avoid copying the 30-line block into N consumer repos, factor as a reusable workflow at vh/ci-shared/.gitea/workflows/release-mirror.yaml that each consumer repo uses: from its own CI. One source of truth; one place to fix bugs / add features.

Trade-offs vs scheduled cron approach

CD-driven (this) Scheduled mirror cron
Atomicity Per-release atomic Eventual consistency
Where logic lives Per consumer repo (factored via shared workflow) infra-ops repo, one place
Failure mode if mirror down CD step soft-fails (configurable) Next cron tick retries
Tracks "what is a release" Definitionally — the tag IS the trigger Has to discover via tag-name regex
Replaces #8's gitea cleanup rules Yes — the prune step IS the cleanup No, both needed

For the operator's fleet (4-6 Bifrost-consumer repos, all theirs, all on Gitea Actions), CD-driven is the cleaner shape.

Open questions

  1. Offsite mirror target. Lean: Distribution registry (the OCI reference) on nh3-docker, NFS-backed to nh3-nas. Lightweight, OCI-compliant, crane/docker push works natively. Alternatives: second Gitea instance (heavyweight), ghcr.io (free for public, $ for private), Cloudflare R2, OR crane pull to OCI tarballs on nh3-nas (no registry — restorability needs crane push from tarball later).

  2. Mirror-side cleanup: do versioned tags on the mirror live forever or also age out? "Preserve every versioned commit" implies forever. ~200 MB/image × ~weekly releases × N repos = ~10 GB/year/repo. Trivial for SAN-class storage.

  3. Mirror auth: per-repo tokens (better blast radius) vs shared MIRROR_TOKEN secret (single rotation point). Likely shared for v1, per-repo if a compromise scare materializes.

  4. Mirror push failures and CD coupling: should mirror+prune block the deploy step, or run in a needs: deploy post-deploy job so a flaky deploy doesn't leave a half-state? Lean: post-deploy job (atomicity per release-as-a-whole, not per-step).

  5. Per-repo gitea token scope for the prune step: needs write:package at minimum (the existing tea token in this repo only has write:misc,write:notification,write:organization,write:issue,write:repository,write:user — no package scope). Mint a CI-only token with just read:package,write:package scope.

Sequencing

  1. Pick the offsite target (operator decision; lean: Distribution registry on nh3-docker → nh3-nas backing).
  2. Stand up the mirror registry as a new stack in stacks/registry-mirror/. Infra-ops work.
  3. Author the reusable workflow at vh/ci-shared/.gitea/workflows/release-mirror.yaml. Could be infra-ops or skaldsong-dev — neutral.
  4. Per-repo CI updates: each consumer repo (skaldsong, Worldtree, mead-hall, althing-chamber, ratatoskr) adds the if: startsWith(github.ref, 'refs/tags/v') block invoking the shared workflow. One small PR per repo, owned by that repo's dev.
  5. Mint a CI-scoped gitea token with package read+write, distribute as GITEA_TOKEN secret to each consumer repo's Gitea Actions config.
  6. Close #8 as superseded once this lands.

Acceptance

  • A versioned commit on any consumer repo results in: :v<x> + :latest + :<sha> in gitea; :v<x> in the mirror; all prior non-protected tags for that repo removed from gitea.
  • Mirror survives the gitea host being completely rebuilt — versioned releases are pull-able from the mirror.
  • Disk usage on ana-docker for gitea registry blobs trends flat over time rather than growing.

Related

  • Supersedes #8.
  • Driven by 2026-05-23 disk snapshot (89% on ana-docker) and the subsequent discovery that no versioned tags exist in the registry today.
## Motivation Two coupled problems this addresses: 1. **Disk pressure from CI image churn**: each commit to `vh/skaldsong`, `vh/Worldtree`, etc. pushes a new SHA-pinned image; they accumulate indefinitely. Snapshot 2026-05-23: ana-docker at 89% disk, 96 GB of reclaimable gitea registry blobs. Manually pruned (~113 GB recovered) but the pattern recurs. 2. **Loss of release-grade durability**: operator wants every versioned commit (semver-tagged) preserved indefinitely, but today every push gets ONLY a `:<40-char-sha>` tag — there are zero `:v0.27.0`-style tags in the registry. The registry cannot distinguish "release worth preserving" from "ephemeral SHA churn." Issue #8 proposed gitea-side cleanup rules with a keep-pattern for `^v[0-9]+\.[0-9]+\.[0-9]+$`, but the inputs that pattern matches don't exist yet. ## Why supersede #8 Issue #8 attacked symptom 1 (disk pressure) with per-owner cleanup rules and assumed version tags would be available as the protected set. With versioned tags absent from the registry, those rules can't do their intended job — they'd protect `latest` and nothing else, deleting everything between releases. We need to fix the upstream tagging problem AND introduce offsite durability for releases. The CD-driven model below does both atomically, replacing the gitea-side rules entirely. ## Proposed shape: CD does the work on every release When CI builds a commit that has a semver git tag, the same pipeline: 1. **Multi-tag the build**: push `:<sha>` + `:v<version>` + `:latest` in one buildx invocation. 2. **Mirror the versioned tag offsite**: `crane copy gitea.phasefinal.com/vh/<repo>:v<x> $MIRROR/<repo>:v<x>`. 3. **Prune the gitea registry of non-protected tags**: keep `:v*`, `:latest`, and the current `:<sha>`; delete the rest via gitea's `DELETE /api/v1/packages/{owner}/container/{name}/{version}` API. Non-versioned pushes (regular `main` commits) still publish `:<sha>` for CD's pull-and-recreate, but skip steps 2-3. Accumulation between releases is bounded by release cadence. Workflow sketch (each consumer repo's `.gitea/workflows/deploy.yaml`): ```yaml - name: Push image with version + latest + sha tags if: startsWith(github.ref, 'refs/tags/v') run: | VER=${GITHUB_REF#refs/tags/} docker buildx build \ --tag $REG/$REPO:${{ github.sha }} \ --tag $REG/$REPO:$VER \ --tag $REG/$REPO:latest \ --push . - name: Mirror version tag to offsite registry if: startsWith(github.ref, 'refs/tags/v') run: | VER=${GITHUB_REF#refs/tags/} crane copy $REG/$REPO:$VER $MIRROR_REG/$REPO:$VER - name: Prune old SHA-only tags from gitea registry if: startsWith(github.ref, 'refs/tags/v') run: | VER=${GITHUB_REF#refs/tags/} KEEP="${VER}|latest|${{ github.sha }}|^v[0-9]+\\.[0-9]+\\.[0-9]+$" curl -sf -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \ "$REG_API/packages/$OWNER?type=container&q=$REPO" \ | jq -r '.[] | .version' \ | grep -vE "$KEEP" \ | xargs -I{} curl -X DELETE -H "Authorization: token ${{ secrets.GITEA_TOKEN }}" \ "$REG_API/packages/$OWNER/container/$REPO/{}" ``` Sketch — real version needs pagination, retries, mirror-down handling. ## DRY: reusable workflow To avoid copying the 30-line block into N consumer repos, factor as a reusable workflow at `vh/ci-shared/.gitea/workflows/release-mirror.yaml` that each consumer repo `uses:` from its own CI. One source of truth; one place to fix bugs / add features. ## Trade-offs vs scheduled cron approach | | CD-driven (this) | Scheduled mirror cron | |---|---|---| | Atomicity | Per-release atomic | Eventual consistency | | Where logic lives | Per consumer repo (factored via shared workflow) | infra-ops repo, one place | | Failure mode if mirror down | CD step soft-fails (configurable) | Next cron tick retries | | Tracks "what is a release" | Definitionally — the tag IS the trigger | Has to discover via tag-name regex | | Replaces #8's gitea cleanup rules | Yes — the prune step IS the cleanup | No, both needed | For the operator's fleet (4-6 Bifrost-consumer repos, all theirs, all on Gitea Actions), CD-driven is the cleaner shape. ## Open questions 1. **Offsite mirror target.** Lean: Distribution registry (the OCI reference) on nh3-docker, NFS-backed to nh3-nas. Lightweight, OCI-compliant, `crane`/`docker` push works natively. Alternatives: second Gitea instance (heavyweight), `ghcr.io` (free for public, $ for private), Cloudflare R2, OR `crane pull` to OCI tarballs on nh3-nas (no registry — restorability needs `crane push` from tarball later). 2. **Mirror-side cleanup**: do versioned tags on the mirror live forever or also age out? "Preserve every versioned commit" implies forever. ~200 MB/image × ~weekly releases × N repos = ~10 GB/year/repo. Trivial for SAN-class storage. 3. **Mirror auth**: per-repo tokens (better blast radius) vs shared `MIRROR_TOKEN` secret (single rotation point). Likely shared for v1, per-repo if a compromise scare materializes. 4. **Mirror push failures and CD coupling**: should mirror+prune block the deploy step, or run in a `needs: deploy` post-deploy job so a flaky deploy doesn't leave a half-state? Lean: post-deploy job (atomicity per release-as-a-whole, not per-step). 5. **Per-repo gitea token scope** for the prune step: needs `write:package` at minimum (the existing tea token in this repo only has `write:misc,write:notification,write:organization,write:issue,write:repository,write:user` — no package scope). Mint a CI-only token with just `read:package,write:package` scope. ## Sequencing 1. **Pick the offsite target** (operator decision; lean: Distribution registry on nh3-docker → nh3-nas backing). 2. **Stand up the mirror registry** as a new stack in `stacks/registry-mirror/`. Infra-ops work. 3. **Author the reusable workflow** at `vh/ci-shared/.gitea/workflows/release-mirror.yaml`. Could be infra-ops or skaldsong-dev — neutral. 4. **Per-repo CI updates**: each consumer repo (skaldsong, Worldtree, mead-hall, althing-chamber, ratatoskr) adds the `if: startsWith(github.ref, 'refs/tags/v')` block invoking the shared workflow. One small PR per repo, owned by that repo's dev. 5. **Mint a CI-scoped gitea token** with package read+write, distribute as `GITEA_TOKEN` secret to each consumer repo's Gitea Actions config. 6. **Close #8** as superseded once this lands. ## Acceptance - A versioned commit on any consumer repo results in: `:v<x>` + `:latest` + `:<sha>` in gitea; `:v<x>` in the mirror; all prior non-protected tags for that repo removed from gitea. - Mirror survives the gitea host being completely rebuilt — versioned releases are pull-able from the mirror. - Disk usage on ana-docker for gitea registry blobs trends flat over time rather than growing. ## Related - Supersedes #8. - Driven by 2026-05-23 disk snapshot (89% on ana-docker) and the subsequent discovery that no versioned tags exist in the registry today.
vh added the enhancement label 2026-05-23 18:03:48 -07:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working
 documentation
README / runbook updates
 enhancement
Improvement to an existing stack or script
 investigation
Needs reproduction / diagnosis before it's actionable
 question
Open design question — needs a decision before work
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
vh
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vh/esh-pfi-infrastructure#9
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?